From knepley at gmail.com  Mon Oct  1 05:49:41 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Oct 2018 06:49:41 -0400
Subject: [petsc-users] Fwd: Implementing a homotopy solver
In-Reply-To: <CABm8tB408JPpDavGwXb8s3krAwtxxihV-41mwkHZrrfukTP3Cg@mail.gmail.com>
References: <CABm8tB45FanRe-68v7T4pZTYNwj-ttYqeH=6MkeznVqacB5AZw@mail.gmail.com>
	<CABm8tB7qqa5TPdn6AHBkkn-v8RAAcAsEaQOULrEN4HAbYfY90A@mail.gmail.com>
	<6377AF62-D548-4276-A2A6-158EE4513593@anl.gov>
	<CABm8tB6_gEvUTM8oXBSMBAvtPnGmnyf07nAZj0gBmffDPe=V2A@mail.gmail.com>
	<00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk>
	<CAMYG4Gkh-2rUaDpj0BUesjwyRsuKJxiK9ottXpVsuw4+LWzHzg@mail.gmail.com>
	<CABm8tB6+8yk9UVXd+R5DNFgD+NsUAfj2r5Kdny9d_wbd27V7kQ@mail.gmail.com>
	<CAMYG4G=XYWNsjY0Y8nYa0FJQj5w4fuKttX+SNVUcMQsvDB_0DQ@mail.gmail.com>
	<CABm8tB59bFuvXp=Zoc1mB_zYJ=gk4hNF9qzxhBs=jcah=BY59w@mail.gmail.com>
	<CAMYG4GmcCeAU80bm4tV7h6aisanF1W3fm0UXapLkA2D92B7jaw@mail.gmail.com>
	<CABm8tB7rweyeVkBh96dhaY46vdY+pp0oy0bJCwV++_3uOB+Rsw@mail.gmail.com>
	<CABm8tB4W3NZXMu_ux2sbGnZCgx5F=Piq+kBuKVWbfp3DLLCc+A@mail.gmail.com>
	<29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov>
	<CABm8tB71F8_YKBJ=D_h0hKVr7GE6Ww=yuaB4MOXpFNSS+1WzRg@mail.gmail.com>
	<CABm8tB4qnjP=WrcMDnQgO7UEePG_yhfQ41PZxrLY-L+suBrYpg@mail.gmail.com>
	<CAMYG4GkX7jGuvsH4H2XdH2_wRK=Qi4LPqLq+pJv1EW6Oe2W-WQ@mail.gmail.com>
	<CABm8tB63yjedZ3O7Ocuv+=vXQnBRfoaUMb6G5fTCAa68F7h-2A@mail.gmail.com>
	<CAMYG4GnVW9khT0O5WtiE0p8iMDGoBcOJsvuc3BPnTfB5RiqQmg@mail.gmail.com>
	<CABm8tB408JPpDavGwXb8s3krAwtxxihV-41mwkHZrrfukTP3Cg@mail.gmail.com>
Message-ID: <CAMYG4Gk3k9=fVPesu1xOV79Z-jEgrsZDDXf65U-00uEVengzaQ@mail.gmail.com>

On Sun, Sep 30, 2018 at 6:06 PM zakaryah <zakaryah at gmail.com> wrote:

> OK, thanks.
>
> I'm using a composite DM, DM_packer.  To make a separate KSP, I did the
> following in the FormJacobian() routine, after assembling the Jacobian
> matrix A and the RHS for the tangent vector, b:
>
>    - KSPCreate(PETSC_COMM_WORLD,&ksp)
>    - KSPSetDM(ksp,DM_packer)
>    - KSPSetDMActive(ksp,PETSC_FALSE) - because I want to set the operators
>    - KSPSetOperators(ksp,A,P)
>    - VecSet(n,0) - set initial guess to zero
>    - KSPSolve(ksp,b,n) - this solve works correctly
>    - VecNormalize(n,NULL)
>    -
>    MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp)
>    - MatSetNullSpace(A,nullsp)
>
> Then, with -snes_type newtonls -pc_type none -ksp_monitor
> -ksp_monitor_true_residual -ksp_view, the output for the KSPSolve described
> above, i.e. for the tangent vector, looks correct:
>
>   0 KSP preconditioned resid norm 1.000000000000e+03 true resid norm
> 1.000000000000e+03 ||r(i)||/||b|| 1.000000000000e+00
> ...
>
> 185 KSP preconditioned resid norm 9.900713131874e-03 true resid norm
> 9.900713131904e-03 ||r(i)||/||b|| 9.900713131904e-06
>
> Linear solve converged due to CONVERGED_RTOL iterations 185
>
> KSP Object: 1 MPI processes
>
>   type: gmres
>
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>
>     happy breakdown tolerance 1e-30
>
>   maximum iterations=10000, initial guess is zero
>
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>
>   left preconditioning
>
>   using PRECONDITIONED norm type for convergence test
>
> PC Object: 1 MPI processes
>
>   type: none
>
>   linear system matrix = precond matrix:
>
>   Mat Object: 1 MPI processes
>
>     type: seqaij
>
>     rows=78247, cols=78247
>
>     total: nonzeros=6063481, allocated nonzeros=6063481
>
>     total number of mallocs used during MatSetValues calls =0
>
>       using I-node routines: found 26083 nodes, limit used is 5
>
>
> But the next KSP, i.e. the one in the SNES, doesn't converge in the true
> residual:
>
>
>   0 KSP preconditioned resid norm 2.045599092896e-04 true resid norm
> 2.803828296212e-04 ||r(i)||/||b|| 1.000000000000e+00
>
> 191 KSP preconditioned resid norm 2.009941278534e-09 true resid norm
> 1.636010142734e-04 ||r(i)||/||b|| 5.834915586465e-01
>
>
> KSP Object: 1 MPI processes
>
>   type: gmres
>
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>
>     happy breakdown tolerance 1e-30
>
>   maximum iterations=10000, initial guess is zero
>
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>
>   left preconditioning
>
>   using PRECONDITIONED norm type for convergence test
>
> PC Object: 1 MPI processes
>
>   type: none
>
>   linear system matrix = precond matrix:
>
>   Mat Object: 1 MPI processes
>
>     type: seqaij
>
>     rows=78247, cols=78247
>
>     total: nonzeros=6063481, allocated nonzeros=6063481
>
>     total number of mallocs used during MatSetValues calls =0
>
>       has attached null space
>
>       using I-node routines: found 26083 nodes, limit used is 5
>
>
> My guess about what's going on is that the tangent vector n isn't really
> in the nullspace of A.
>

I do not understand the rest, but this is the problem. Maybe it would make
more sense if you wrote things
in linear algebraic notation.

  Thanks,

     Matt


> Rather, it's in the nullspace of the m x (m+1) submatrix of A.  So, An=c
> e_{m+1}, where c is an arbitrary constant and e_{m+1} is the m+1 th basis
> vector.  The nonlinear function also has an added row, F_{m+1}, which is
> set to zero in the FormFunction() routine.  I don't care about the value of
> F_{m+1}, but I suppose that if it's included in the true residual, and NOT
> in the "preconditioned" residual, even with pc_type none, then it will be
> tricky to diagnose the performance.  Should I be using my own routine to
> evaluate the residual, so that F_{m+1} is not included?
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/72fe2e22/attachment-0001.html>

From phil.tooley at sheffield.ac.uk  Mon Oct  1 08:59:13 2018
From: phil.tooley at sheffield.ac.uk (Phil Tooley)
Date: Mon, 1 Oct 2018 14:59:13 +0100
Subject: [petsc-users] DMDA with dimension of size 1
Message-ID: <754c2a19-a0cc-19dd-7449-aaddc91d507e@sheffield.ac.uk>

Hi all,

Is it valid to have a DMDA with one of the dimensions of size 1.? I was
hoping to avoid having to write explicit logic to handle the case that I
am working on a 2D rather than a 3D image for my current application.??
Unfortunately when I try to construct such a DMDA I get an error:

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Overflow in integer operation:
[0]PETSC ERROR: Mesh of 10 by 10 by 1 (dof) is too large for 32 bit indices

Is there a workaround other than "write everything twice"?

Thanks

Phil

-- 
Phil Tooley
Research Software Engineering
University of Sheffield


From patrick.sanan at gmail.com  Mon Oct  1 09:55:03 2018
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Mon, 1 Oct 2018 16:55:03 +0200
Subject: [petsc-users] DMDA with dimension of size 1
In-Reply-To: <754c2a19-a0cc-19dd-7449-aaddc91d507e@sheffield.ac.uk>
References: <754c2a19-a0cc-19dd-7449-aaddc91d507e@sheffield.ac.uk>
Message-ID: <CA+z91Tfbpff3A0+aM1_kxThnG_inSKwy4HMOc1sQqZbQB_a_dQ@mail.gmail.com>

Meshes of size 1 should work.

Looks like there is a bug in the code to produce this error message (patch
for maint attached). It's not outputting the "P" (size in the 3rd
dimension) component.

This is just speculation without a full error message or code to reproduce,
but perhaps the size in the third dimension is an uninitialized value which
is triggering this warning with a 10 x 10 x (huge garbage) x 1 (dof) mesh.

Am Mo., 1. Okt. 2018 um 15:59 Uhr schrieb Phil Tooley <
phil.tooley at sheffield.ac.uk>:

> Hi all,
>
> Is it valid to have a DMDA with one of the dimensions of size 1.  I was
> hoping to avoid having to write explicit logic to handle the case that I
> am working on a 2D rather than a 3D image for my current application.
> Unfortunately when I try to construct such a DMDA I get an error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Overflow in integer operation:
> [0]PETSC ERROR: Mesh of 10 by 10 by 1 (dof) is too large for 32 bit indices
>
> Is there a workaround other than "write everything twice"?
>
> Thanks
>
> Phil
>
> --
> Phil Tooley
> Research Software Engineering
> University of Sheffield
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/6fcfdf89/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-DMSetUp_DA_3D-fix-warning-message-for-int32-mesh-siz.patch
Type: application/octet-stream
Size: 1391 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/6fcfdf89/attachment.obj>

From patrick.sanan at gmail.com  Mon Oct  1 09:57:16 2018
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Mon, 1 Oct 2018 16:57:16 +0200
Subject: [petsc-users] DMDA with dimension of size 1
In-Reply-To: <CA+z91Tfbpff3A0+aM1_kxThnG_inSKwy4HMOc1sQqZbQB_a_dQ@mail.gmail.com>
References: <754c2a19-a0cc-19dd-7449-aaddc91d507e@sheffield.ac.uk>
	<CA+z91Tfbpff3A0+aM1_kxThnG_inSKwy4HMOc1sQqZbQB_a_dQ@mail.gmail.com>
Message-ID: <CA+z91Te=XZBFBnV0gMHUCN1yGb_PWfVZEMUpqUuNCmO3iakcqw@mail.gmail.com>

Whoops, sent that patch too fast (forgot to update SETERRQ3 to SETERRQ4).
Updated patch for maint attached.

Am Mo., 1. Okt. 2018 um 16:55 Uhr schrieb Patrick Sanan <
patrick.sanan at gmail.com>:

> Meshes of size 1 should work.
>
> Looks like there is a bug in the code to produce this error message (patch
> for maint attached). It's not outputting the "P" (size in the 3rd
> dimension) component.
>
> This is just speculation without a full error message or code to
> reproduce, but perhaps the size in the third dimension is an uninitialized
> value which is triggering this warning with a 10 x 10 x (huge garbage) x 1
> (dof) mesh.
>
> Am Mo., 1. Okt. 2018 um 15:59 Uhr schrieb Phil Tooley <
> phil.tooley at sheffield.ac.uk>:
>
>> Hi all,
>>
>> Is it valid to have a DMDA with one of the dimensions of size 1.  I was
>> hoping to avoid having to write explicit logic to handle the case that I
>> am working on a 2D rather than a 3D image for my current application.
>> Unfortunately when I try to construct such a DMDA I get an error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Overflow in integer operation:
>> [0]PETSC ERROR: Mesh of 10 by 10 by 1 (dof) is too large for 32 bit
>> indices
>>
>> Is there a workaround other than "write everything twice"?
>>
>> Thanks
>>
>> Phil
>>
>> --
>> Phil Tooley
>> Research Software Engineering
>> University of Sheffield
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/7d04df3e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-DMSetUp_DA_3D-fix-warning-message-for-int32-mesh-siz.patch
Type: application/octet-stream
Size: 1391 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/7d04df3e/attachment.obj>

From fdkong.jd at gmail.com  Mon Oct  1 10:01:15 2018
From: fdkong.jd at gmail.com (Fande Kong)
Date: Mon, 1 Oct 2018 09:01:15 -0600
Subject: [petsc-users] MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable
In-Reply-To: <878t3llzrm.fsf@jedbrown.org>
References: <CAN5Wd-+=vHcuh-Wymkamz6UMxNikPDs-AZ7fj-niqZ2S_F-hTg@mail.gmail.com>
	<878t3llzrm.fsf@jedbrown.org>
Message-ID: <CAN5Wd-+hFA1xeLdr32KpWeZuToWk3QmPUnUQXRnrMYRerwE9=A@mail.gmail.com>

Thanks, Jed. I figured out. I just simply made P more sparse, then the
product would not take too much memory.

Fande,

On Fri, Sep 28, 2018 at 9:59 PM Jed Brown <jed at jedbrown.org> wrote:

> It depends entirely on your matrices.  For example, if A is an arrowhead
> matrix (graph of a star -- one hub and many leaves) then A^2 is dense.
> If you have particular stencils for A and P, then we could tell you the
> fill ratio.
>
> Fande Kong <fdkong.jd at gmail.com> writes:
>
> > Hi All,
> >
> > I was wondering how much memory is required to get PtAP done? Do you have
> > any simple formula to this? So that I can have an  estimate.
> >
> >
> > Fande,
> >
> >
> > [132]PETSC ERROR: --------------------- Error Message
> > --------------------------------------------------------------
> > [132]PETSC ERROR: Out of memory. This could be due to allocating
> > [132]PETSC ERROR: too large an object or bleeding by not properly
> > [132]PETSC ERROR: destroying unneeded objects.
> > [132]PETSC ERROR: Memory allocated 0 Memory used by process 3249920
> > [132]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
> > [132]PETSC ERROR: Memory requested 89148704
> > [132]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html
> > for trouble shooting.
> > [132]PETSC ERROR: Petsc Release Version 3.9.4, unknown
> > [132]PETSC ERROR: ../../rattlesnake-opt on a arch-theta-avx512-64-opt
> named
> > nid03830 by fdkong Fri Sep 28 22:43:45 2018
> > [132]PETSC ERROR: Configure options --LIBS=-lstdc++
> > --known-64-bit-blas-indices=0 --known-bits-per-byte=8
> > --known-has-attribute-aligned=1 --known-level1-dcache-assoc=8
> > --known-level1-dcache-linesize=64 --known-level1-dcache-size=32768
> > --known-memcmp-ok=1 --known-mklspblas-supports-zero-based=0
> > --known-mpi-c-double-complex=1 --known-mpi-int64_t=1
> > --known-mpi-long-double=1 --known-mpi-shared-libraries=0
> > --known-sdot-returns-double=0 --known-sizeof-MPI_Comm=4
> > --known-sizeof-MPI_Fint=4 --known-sizeof-char=1 --known-sizeof-double=8
> > --known-sizeof-float=4 --known-sizeof-int=4 --known-sizeof-long-long=8
> > --known-sizeof-long=8 --known-sizeof-short=2 --known-sizeof-size_t=8
> > --known-sizeof-void-p=8 --known-snrm2-returns-double=0 --with-batch=1
> > --with-blaslapack-lib="-mkl
> > -L/opt/intel/compilers_and_libraries_2018.0.128/linux/mkl/lib/intel64"
> > --with-cc=cc --with-clib-autodetect=0 --with-cxx=CC
> > --with-cxxlib-autodetect=0 --with-debugging=0 --with-fc=ftn
> > --with-fortranlib-autodetect=0 --with-hdf5=0 --with-memalign=64
> > --with-mpiexec=aprun --with-shared-libraries=0 --download-metis=1
> > --download-parmetis=1 --download-superlu_dist=1 --download-hypre=1
> > --download-ptscotch=1 COPTFLAGS="-O3 -xMIC-AVX512" CXXOPTFLAGS="-O3
> > -xMIC-AVX512" FOPTFLAGS="-O3 -xMIC-AVX512"
> > PETSC_ARCH=arch-theta-avx512-64-opt --with-64-bit-indices=1
> > [132]PETSC ERROR: #1 PetscSegBufferCreate() line 64 in
> > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c
> > [132]PETSC ERROR: #2 PetscSegBufferCreate() line 64 in
> > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c
> > [132]PETSC ERROR: #3 PetscSegBufferExtractInPlace() line 227 in
> > /gpfs/mira-home/fdkong/petsc/src/sys/utils/segbuffer.c
> > [132]PETSC ERROR: #4 MatStashScatterBegin_BTS() line 854 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c
> > [132]PETSC ERROR: #5 MatStashScatterBegin_Private() line 461 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/utils/matstash.c
> > [132]PETSC ERROR: #6 MatAssemblyBegin_MPIAIJ() line 683 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiaij.c
> > [132]PETSC ERROR: #7 MatAssemblyBegin() line 5158 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c
> > [132]PETSC ERROR: #8 MatPtAPNumeric_MPIAIJ_MPIAIJ_scalable() line 262 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c
> > [132]PETSC ERROR: #9 MatPtAP_MPIAIJ_MPIAIJ() line 172 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/impls/aij/mpi/mpiptap.c
> > [132]PETSC ERROR: #10 MatPtAP() line 9182 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c
> > [132]PETSC ERROR: #11 MatGalerkin() line 10615 in
> > /gpfs/mira-home/fdkong/petsc/src/mat/interface/matrix.c
> > [132]PETSC ERROR: #12 PCSetUp_MG() line 730 in
> > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/mg/mg.c
> > [132]PETSC ERROR: #13 PCSetUp_HMG() line 336 in
> > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/impls/hmg/hmg.c
> > [132]PETSC ERROR: #14 PCSetUp() line 923 in
> > /gpfs/mira-home/fdkong/petsc/src/ksp/pc/interface/precon.c
> > [132]PETSC ERROR: #15 KSPSetUp() line 381 in
> > /gpfs/mira-home/fdkong/petsc/src/ksp/ksp/interface/itfunc.c
> > [136]PETSC ERROR: --------------------- Error Message
> > ----------------------------------------------------------
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/93d73a38/attachment-0001.html>

From phil.tooley at sheffield.ac.uk  Mon Oct  1 10:04:03 2018
From: phil.tooley at sheffield.ac.uk (Phil Tooley)
Date: Mon, 1 Oct 2018 16:04:03 +0100
Subject: [petsc-users] DMDA with dimension of size 1
In-Reply-To: <CA+z91Te=XZBFBnV0gMHUCN1yGb_PWfVZEMUpqUuNCmO3iakcqw@mail.gmail.com>
References: <754c2a19-a0cc-19dd-7449-aaddc91d507e@sheffield.ac.uk>
	<CA+z91Tfbpff3A0+aM1_kxThnG_inSKwy4HMOc1sQqZbQB_a_dQ@mail.gmail.com>
	<CA+z91Te=XZBFBnV0gMHUCN1yGb_PWfVZEMUpqUuNCmO3iakcqw@mail.gmail.com>
Message-ID: <f0399c2d-d129-91a8-a02a-95786139ba3d@sheffield.ac.uk>

Hi Patrick,

You are spot on, I was copying in a vector and not taking care to pad
with 1s to length 3.

Thanks

Phil

On 01/10/2018 15:57, Patrick Sanan wrote:
> Whoops, sent that patch too fast (forgot to update SETERRQ3 to
> SETERRQ4). Updated patch for maint attached.
>
> Am Mo., 1. Okt. 2018 um 16:55?Uhr schrieb Patrick Sanan
> <patrick.sanan at gmail.com <mailto:patrick.sanan at gmail.com>>:
>
>     Meshes of size 1 should work.
>
>     Looks like there is a bug in the code to produce this error
>     message (patch for maint attached). It's not outputting the "P"
>     (size in the 3rd dimension) component.
>
>     This is just speculation without a full error message or code to
>     reproduce, but perhaps the size in the third dimension is an
>     uninitialized value which is triggering this warning with a 10 x
>     10 x (huge garbage) x 1 (dof) mesh.
>
>     Am Mo., 1. Okt. 2018 um 15:59?Uhr schrieb Phil Tooley
>     <phil.tooley at sheffield.ac.uk <mailto:phil.tooley at sheffield.ac.uk>>:
>
>         Hi all,
>
>         Is it valid to have a DMDA with one of the dimensions of size
>         1.? I was
>         hoping to avoid having to write explicit logic to handle the
>         case that I
>         am working on a 2D rather than a 3D image for my current
>         application.??
>         Unfortunately when I try to construct such a DMDA I get an error:
>
>         [0]PETSC ERROR: --------------------- Error Message
>         --------------------------------------------------------------
>         [0]PETSC ERROR: Overflow in integer operation:
>         [0]PETSC ERROR: Mesh of 10 by 10 by 1 (dof) is too large for
>         32 bit indices
>
>         Is there a workaround other than "write everything twice"?
>
>         Thanks
>
>         Phil
>
>         -- 
>         Phil Tooley
>         Research Software Engineering
>         University of Sheffield
>
-- 
Phil Tooley
Research Software Engineering
University of Sheffield

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/ab50d462/attachment.html>

From ysjosh.lo at gmail.com  Mon Oct  1 13:43:58 2018
From: ysjosh.lo at gmail.com (Josh L)
Date: Mon, 1 Oct 2018 13:43:58 -0500
Subject: [petsc-users] Fortran undefined reference
Message-ID: <CA+2ebbAHyC5fPemhk00aakHCfbZgU5udB2f6R5BixW=-W05Rsw@mail.gmail.com>

Hi,

I am using PETSc makefile to compile my code. few DMPlex function ,
e.g. PetscPartitionerSetType, DMPlexGetCellNumbering etc, fails the
linking(undefined reference to xxxxx).  I am using PETSc/3.9.
Anyone knows what is causing it ?


Thanks,
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/cd770b70/attachment.html>

From knepley at gmail.com  Mon Oct  1 15:36:29 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Oct 2018 16:36:29 -0400
Subject: [petsc-users] Fortran undefined reference
In-Reply-To: <CA+2ebbAHyC5fPemhk00aakHCfbZgU5udB2f6R5BixW=-W05Rsw@mail.gmail.com>
References: <CA+2ebbAHyC5fPemhk00aakHCfbZgU5udB2f6R5BixW=-W05Rsw@mail.gmail.com>
Message-ID: <CAMYG4Gn2=C4eZ980AgXip-AUz6aOp+U=8Lm9VAQqGCCguvH06Q@mail.gmail.com>

On Mon, Oct 1, 2018 at 2:44 PM Josh L <ysjosh.lo at gmail.com> wrote:

> Hi,
>
> I am using PETSc makefile to compile my code. few DMPlex function ,
> e.g. PetscPartitionerSetType, DMPlexGetCellNumbering etc, fails the
> linking(undefined reference to xxxxx).  I am using PETSc/3.9.
> Anyone knows what is causing it ?
>

Yes. We have to write Fortran bindings by hand, so not all functions has
them yet. Could you make
a list of all the ones you want fixed, and I will do it?

  Thanks,

     Matt


> Thanks,
> Josh
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/19cbf290/attachment.html>

From weizhuo2 at illinois.edu  Mon Oct  1 17:31:22 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 1 Oct 2018 17:31:22 -0500
Subject: [petsc-users] Increasing norm with finer mesh
Message-ID: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>

Hi!

I'm recently trying out the example code provided with the KSP solver
(ex12.c). I noticed that the mean norm of the grid increases as I use finer
meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
since most of the time error should decreases when using finer grid. Am I
doing this wrong?

Thanks!
-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/5dd38fb8/attachment.html>

From knepley at gmail.com  Mon Oct  1 19:44:49 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Oct 2018 20:44:49 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
Message-ID: <CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>

On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> Hi!
>
> I'm recently trying out the example code provided with the KSP solver
> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
> since most of the time error should decreases when using finer grid. Am I
> doing this wrong?
>

The norm is misleading in that it is the l_2 norm, meaning just the sqrt of
the sum of the squares of
the vector entries. It should be scaled by the volume element to
approximate a scale-independent
norm (like the L_2 norm).

  Thanks,

     Matt


> Thanks!
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/fbb5ff45/attachment.html>

From weizhuo2 at illinois.edu  Mon Oct  1 19:51:00 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 1 Oct 2018 19:51:00 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
Message-ID: <CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>

I also tried to divide the norm by m*n , which is the number of grids, the
trend of norm still increases.

Thanks!

Weizhuo

Matthew Knepley <knepley at gmail.com> ?2018?10?1??? ??7:45???

> On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> Hi!
>>
>> I'm recently trying out the example code provided with the KSP solver
>> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
>> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
>> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
>> since most of the time error should decreases when using finer grid. Am I
>> doing this wrong?
>>
>
> The norm is misleading in that it is the l_2 norm, meaning just the sqrt
> of the sum of the squares of
> the vector entries. It should be scaled by the volume element to
> approximate a scale-independent
> norm (like the L_2 norm).
>
>   Thanks,
>
>      Matt
>
>
>> Thanks!
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/be373895/attachment.html>

From knepley at gmail.com  Mon Oct  1 19:56:46 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 1 Oct 2018 20:56:46 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
Message-ID: <CAMYG4GmccGHdM9zivtCB6tG2wMNagDRZL5BVCNe27QQCsDQQLA@mail.gmail.com>

On Mon, Oct 1, 2018 at 8:51 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> I also tried to divide the norm by m*n , which is the number of grids, the
> trend of norm still increases.
>

We need to be precise. First, look at the initial residual, because that is
what you control with the initial
guess. You are saying that the initial residual does not asymptote? I would
be reluctant to believe that.

  Thanks,

     Matt


> Thanks!
>
> Weizhuo
>
> Matthew Knepley <knepley at gmail.com> ?2018?10?1??? ??7:45???
>
>> On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Hi!
>>>
>>> I'm recently trying out the example code provided with the KSP solver
>>> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
>>> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
>>> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
>>> since most of the time error should decreases when using finer grid. Am I
>>> doing this wrong?
>>>
>>
>> The norm is misleading in that it is the l_2 norm, meaning just the sqrt
>> of the sum of the squares of
>> the vector entries. It should be scaled by the volume element to
>> approximate a scale-independent
>> norm (like the L_2 norm).
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks!
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181001/63fdba10/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Oct  1 23:03:49 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 2 Oct 2018 04:03:49 +0000
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
Message-ID: <BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>


   Please send your version of the example that computes the mean norm of the grid; I suspect we are talking apples and oranges

   Barry



> On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
> 
> I also tried to divide the norm by m*n , which is the number of grids, the trend of norm still increases.
> 
> Thanks!
> 
> Weizhuo
> 
> Matthew Knepley <knepley at gmail.com> ?2018?10?1??? ??7:45???
> On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
> Hi!
> 
> I'm recently trying out the example code provided with the KSP solver (ex12.c). I noticed that the mean norm of the grid increases as I use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive, since most of the time error should decreases when using finer grid. Am I doing this wrong?
> 
> The norm is misleading in that it is the l_2 norm, meaning just the sqrt of the sum of the squares of
> the vector entries. It should be scaled by the volume element to approximate a scale-independent
> norm (like the L_2 norm).
> 
>   Thanks,
> 
>      Matt
>  
> Thanks! 
> -- 
> Wang Weizhuo
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> 
> 
> -- 
> Wang Weizhuo


From mfadams at lbl.gov  Tue Oct  2 08:29:00 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 2 Oct 2018 09:29:00 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
Message-ID: <CADOhEh5QyZz+_sWhHsDYYEGFHLQcJsY63RH72S19OJvvh3Z70g@mail.gmail.com>

As Matt said you should see the initial 2-norm residual asymptote to a
constant with scaling, but it will rise.

I prefer the max norm for this reason. You can use -ksp_monitor_max, but
this does compute an extra residual, apparently, but I don't understand why
it should ...

If this does not asymptotic to constant and your problem is linear, you
have a bug.

On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> Hi!
>
> I'm recently trying out the example code provided with the KSP solver
> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
> since most of the time error should decreases when using finer grid. Am I
> doing this wrong?
>
> Thanks!
> --
> Wang Weizhuo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/7bf58b81/attachment.html>

From david97sommer at gmail.com  Tue Oct  2 09:58:21 2018
From: david97sommer at gmail.com (David Sommer)
Date: Tue, 2 Oct 2018 16:58:21 +0200
Subject: [petsc-users] Parallel LU with MPIDENSE
Message-ID: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>

Hello,

i was trying to solve a linear system with a MPIDENSE matrix. I did
PC pc;
KSPGetPC(solver, &pc);
PCSetType(pc, PCLU);
KSPSetOperators(solver, a_petsc, a_petsc);
for some KSP solver and some Mat a_petsc. When i run the code sequentially
(mpirun -n 1) it works, but once i start it in parallel it breaks with the
error message







*[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: See
http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
<http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html> for
possible LU and Cholesky
solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must
./configure with
--download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
<http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble
shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3,
unknown
[0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named manjaroDesktop
by david Tue Oct  2 16:47:10
2018
[0]PETSC ERROR: Configure options --download-superlu_dist
--download-parmetis --download-metis
--download-ptscotch
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in
/home/david/lib/petsc/src/mat/interface/matrix.c     *

I recompiled petsc with the recommended options (and also trying some other
options) but the error persists. on the linear solve table there is an X at
the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

Thanks

David Sommer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/436b7bd9/attachment.html>

From yangyiwei.yang at mfm.tu-darmstadt.de  Tue Oct  2 10:10:15 2018
From: yangyiwei.yang at mfm.tu-darmstadt.de (Y. Yang)
Date: Tue, 2 Oct 2018 17:10:15 +0200
Subject: [petsc-users] UCX ERROR KNEM inline copy failed
Message-ID: <4f35edc3-b2c3-5b93-7b95-cb60bc51fee8@mfm.tu-darmstadt.de>

Dear PETSc team

Recently I'm using MOOSE (http://www.mooseframework.org/) which is built 
with PETSc and, Unfortunately, I encountered some problems with 
following PETSc options:

petsc_options_iname = '-pc_type -ksp_gmres_restart -sub_ksp_type 
-sub_pc_type -pc_asm_overlap -pc_factor_mat_solver_package'

petsc_options_value = 'asm? ? ? ? ? 1201 ?preonly? ? ? ? ? ? ?ilu? ? ? ? 
 ? ? ? ? ? ?4 ? ?superlu_dist'


the error message is:

Time Step 1, time = 1
 ??????????????? dt = 1

 ??? |residual|_2 of individual variables:
 ??????????????????????? c:?? 779.034
 ??????????????????????? w:?? 0
 ??????????????????????? T:?? 6.57948e+07
 ??????????????????????? gr0: 211.617
 ??????????????????????? gr1: 206.973
 ??????????????????????? gr2: 209.382
 ??????????????????????? gr3: 191.089
 ??????????????????????? gr4: 185.242
 ??????????????????????? gr5: 157.361
 ??????????????????????? gr6: 128.473
 ??????????????????????? gr7: 87.6029

 ?0 Nonlinear |R| = 6.579482e+07
[1538482623.976180] [hpb0085:22501:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482605.111342] [hpb0085:22502:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482606.761138] [hpb0085:22502:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482607.107478] [hpb0085:22502:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482605.882817] [hpb0085:22503:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482607.133543] [hpb0085:22503:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482621.905475] [hpb0085:22510:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482626.531234] [hpb0085:22510:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482627.613343] [hpb0085:22515:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482627.830489] [hpb0085:22515:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482629.852351] [hpb0085:22515:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482630.194620] [hpb0085:22515:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482630.280636] [hpb0085:22515:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482600.219314] [hpb0085:22516:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482658.960350] [hpb0085:22516:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482622.949471] [hpb0085:22517:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482612.502017] [hpb0085:22500:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482613.231970] [hpb0085:22500:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482621.417530] [hpb0085:22520:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482622.020998] [hpb0085:22520:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482606.221292] [hpb0085:22521:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482606.676987] [hpb0085:22521:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482606.896865] [hpb0085:22521:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482639.611427] [hpb0085:22522:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482631.435277] [hpb0085:22523:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482658.278343] [hpb0085:22512:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482658.396945] [hpb0085:22512:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482659.917476] [hpb0085:22512:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
[1538482660.162064] [hpb0085:22512:0]??????? knem_ep.c:84?? UCX ERROR 
KNEM inline copy failed, err = -1 Invalid argument
2 total processes killed (some possibly by mpirun during cleanup)


Here's the status of the simulation

Parallelism:
 ? Num Processors:? ? ? ? ? 100
 ? Num Threads:? ? ? ? ? ? ?1

Mesh:
 ? Parallel Type:? ? ? ? ? ?distributed
 ? Mesh Dimension:? ? ? ? ? 3
 ? Spatial Dimension:? ? ? ?3
 ? Nodes:
 ? ? Total:? ? ? ? ? ? ? ? ?2065551
 ? ? Local:? ? ? ? ? ? ? ? ?22774
 ? Elems:
 ? ? Total:? ? ? ? ? ? ? ? ?2000000
 ? ? Local:? ? ? ? ? ? ? ? ?20006
 ? Num Subdomains:? ? ? ? ? 1
 ? Num Partitions:? ? ? ? ? 100
 ? Partitioner:? ? ? ? ? ? ?parmetis

Nonlinear System:
 ? Num DOFs:? ? ? ? ? ? ? ? 18589959
 ? Num Local DOFs:? ? ? ? ? 204966
 ? Variables:? ? ? ? ? ? ? ?{ "c" "w" "T" "gr0" "gr1" "gr2" "gr3" "gr4" 
"gr5" }
 ? Finite Element Types:? ? "LAGRANGE"
 ? Approximation Orders:? ? "FIRST"

Auxiliary System:
 ? Num DOFs:? ? ? ? ? ? ? ? 10065551
 ? Num Local DOFs:? ? ? ? ? 102798
 ? Variables:? ? ? ? ? ? ? ?"bnds" { "var_indices" "unique_grains" } { 
"M" "dM/dT" }
 ? Finite Element Types:? ? "LAGRANGE" "MONOMIAL" "MONOMIAL"
 ? Approximation Orders:? ? "FIRST" "CONSTANT" "CONSTANT"

Relationship Managers:
 ? Geometric? ? ? ? ? ? ? ? : GrainTrackerHaloRM (2 layers)

Execution Information:
 ? Executioner:? ? ? ? ? ? ?Transient
 ? TimeStepper:? ? ? ? ? ? ?IterationAdaptiveDT
 ? Solver Mode:? ? ? ? ? ? ?Preconditioned JFNK


I tried modifying the parameters and other preconditioning option, the 
problem is much the same. So I don't know where I did wrong or there is 
actually suitable PETSc option to deal with such problem with large 
mesh.?I would like to hear your response.

Sincerely,
Yang

-- 
______________________________________________________
  
Yangyiwei Yang
Wissenschaftliche Hilfskraft

TU Darmstadt
Fachbereich 11 - Material- und Geowissenschaften
Fachgebiet Mechanik funktionaler Materialien

L1 | 08 402
Otto Berndt Stra?e 3
D-64287 Darmstadt

Tel: +49 (0)6151-16-22923
Email: yangyiwei.yang at mfm.tu-darmstadt.de
Homepage: http://www.mawi.tu-darmstadt.de/mfm
ORCID: 0000-0001-5505-7117

______________________________________________________


From knepley at gmail.com  Tue Oct  2 10:21:01 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Oct 2018 11:21:01 -0400
Subject: [petsc-users] UCX ERROR KNEM inline copy failed
In-Reply-To: <4f35edc3-b2c3-5b93-7b95-cb60bc51fee8@mfm.tu-darmstadt.de>
References: <4f35edc3-b2c3-5b93-7b95-cb60bc51fee8@mfm.tu-darmstadt.de>
Message-ID: <CAMYG4GnxydDvZ5As1DXV-EimV56uEunZ-tpyKcyUzWfTM_J2Ug@mail.gmail.com>

On Tue, Oct 2, 2018 at 11:16 AM Y. Yang <yangyiwei.yang at mfm.tu-darmstadt.de>
wrote:

> Dear PETSc team
>
> Recently I'm using MOOSE (http://www.mooseframework.org/) which is built
> with PETSc and, Unfortunately, I encountered some problems with
> following PETSc options:
>

I do not know what problem you are reporting.I don't know what package
knem_ep.c is
part of, but its not PETSc.

  Thanks,

     Matt


> petsc_options_iname = '-pc_type -ksp_gmres_restart -sub_ksp_type
> -sub_pc_type -pc_asm_overlap -pc_factor_mat_solver_package'
>
> petsc_options_value = 'asm          1201  preonly             ilu
>             4    superlu_dist'
>
>
> the error message is:
>
> Time Step 1, time = 1
>                  dt = 1
>
>      |residual|_2 of individual variables:
>                          c:   779.034
>                          w:   0
>                          T:   6.57948e+07
>                          gr0: 211.617
>                          gr1: 206.973
>                          gr2: 209.382
>                          gr3: 191.089
>                          gr4: 185.242
>                          gr5: 157.361
>                          gr6: 128.473
>                          gr7: 87.6029
>
>   0 Nonlinear |R| =  [32m6.579482e+07 [39m
> [1538482623.976180] [hpb0085:22501:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482605.111342] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482606.761138] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482607.107478] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482605.882817] [hpb0085:22503:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482607.133543] [hpb0085:22503:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482621.905475] [hpb0085:22510:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482626.531234] [hpb0085:22510:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482627.613343] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482627.830489] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482629.852351] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482630.194620] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482630.280636] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482600.219314] [hpb0085:22516:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482658.960350] [hpb0085:22516:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482622.949471] [hpb0085:22517:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482612.502017] [hpb0085:22500:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482613.231970] [hpb0085:22500:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482621.417530] [hpb0085:22520:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482622.020998] [hpb0085:22520:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482606.221292] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482606.676987] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482606.896865] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482639.611427] [hpb0085:22522:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482631.435277] [hpb0085:22523:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482658.278343] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482658.396945] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482659.917476] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> [1538482660.162064] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
> KNEM inline copy failed, err = -1 Invalid argument
> 2 total processes killed (some possibly by mpirun during cleanup)
>
>
> Here's the status of the simulation
>
> Parallelism:
>    Num Processors:          100
>    Num Threads:             1
>
> Mesh:
>    Parallel Type:           distributed
>    Mesh Dimension:          3
>    Spatial Dimension:       3
>    Nodes:
>      Total:                 2065551
>      Local:                 22774
>    Elems:
>      Total:                 2000000
>      Local:                 20006
>    Num Subdomains:          1
>    Num Partitions:          100
>    Partitioner:             parmetis
>
> Nonlinear System:
>    Num DOFs:                18589959
>    Num Local DOFs:          204966
>    Variables:               { "c" "w" "T" "gr0" "gr1" "gr2" "gr3" "gr4"
> "gr5" }
>    Finite Element Types:    "LAGRANGE"
>    Approximation Orders:    "FIRST"
>
> Auxiliary System:
>    Num DOFs:                10065551
>    Num Local DOFs:          102798
>    Variables:               "bnds" { "var_indices" "unique_grains" } {
> "M" "dM/dT" }
>    Finite Element Types:    "LAGRANGE" "MONOMIAL" "MONOMIAL"
>    Approximation Orders:    "FIRST" "CONSTANT" "CONSTANT"
>
> Relationship Managers:
>    Geometric                : GrainTrackerHaloRM (2 layers)
>
> Execution Information:
>    Executioner:             Transient
>    TimeStepper:             IterationAdaptiveDT
>    Solver Mode:             Preconditioned JFNK
>
>
> I tried modifying the parameters and other preconditioning option, the
> problem is much the same. So I don't know where I did wrong or there is
> actually suitable PETSc option to deal with such problem with large
> mesh. I would like to hear your response.
>
> Sincerely,
> Yang
>
> --
> ______________________________________________________
>
> Yangyiwei Yang
> Wissenschaftliche Hilfskraft
>
> TU Darmstadt
> Fachbereich 11 - Material- und Geowissenschaften
> Fachgebiet Mechanik funktionaler Materialien
>
> L1 | 08 402
> Otto Berndt Stra?e 3
> D-64287 Darmstadt
>
> Tel: +49 (0)6151-16-22923
> Email: yangyiwei.yang at mfm.tu-darmstadt.de
> Homepage: http://www.mawi.tu-darmstadt.de/mfm
> ORCID: 0000-0001-5505-7117
>
> ______________________________________________________
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/c94e55f0/attachment-0001.html>

From hzhang at mcs.anl.gov  Tue Oct  2 10:26:25 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 2 Oct 2018 15:26:25 +0000
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
Message-ID: <CAGCphBstiZw6fggZFCoTUZ4UCUVuJugOnp3R0zMzy9nrLtHC7Q@mail.gmail.com>

David :
Petsc does not support parallel LU. For dense matrix, you can use Elemental. See example
petsc/src/ksp/ksp/examples/tests/ex40.c
Hong
Hello,

i was trying to solve a linear system with a MPIDENSE matrix. I did
PC pc;
KSPGetPC(solver, &pc);
PCSetType(pc, PCLU);
KSPSetOperators(solver, a_petsc, a_petsc);
for some KSP solver and some Mat a_petsc. When i run the code sequentially (mpirun -n 1) it works, but once i start it in parallel it breaks with the error message
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
[0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named manjaroDesktop by david Tue Oct  2 16:47:10 2018
[0]PETSC ERROR: Configure options --download-superlu_dist --download-parmetis --download-metis --download-ptscotch
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/david/lib/petsc/src/mat/interface/matrix.c
I recompiled petsc with the recommended options (and also trying some other options) but the error persists. on the linear solve table there is an X at the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

Thanks

David Sommer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/6a6753ee/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct  2 10:26:25 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 2 Oct 2018 15:26:25 +0000
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
Message-ID: <CAGCphBstiZw6fggZFCoTUZ4UCUVuJugOnp3R0zMzy9nrLtHC7Q@mail.gmail.com>

David :
Petsc does not support parallel LU. For dense matrix, you can use Elemental. See example
petsc/src/ksp/ksp/examples/tests/ex40.c
Hong
Hello,

i was trying to solve a linear system with a MPIDENSE matrix. I did
PC pc;
KSPGetPC(solver, &pc);
PCSetType(pc, PCLU);
KSPSetOperators(solver, a_petsc, a_petsc);
for some KSP solver and some Mat a_petsc. When i run the code sequentially (mpirun -n 1) it works, but once i start it in parallel it breaks with the error message
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
[0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named manjaroDesktop by david Tue Oct  2 16:47:10 2018
[0]PETSC ERROR: Configure options --download-superlu_dist --download-parmetis --download-metis --download-ptscotch
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/david/lib/petsc/src/mat/interface/matrix.c
I recompiled petsc with the recommended options (and also trying some other options) but the error persists. on the linear solve table there is an X at the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

Thanks

David Sommer
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/6a6753ee/attachment-0001.html>

From knepley at gmail.com  Tue Oct  2 10:31:11 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Oct 2018 11:31:11 -0400
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
Message-ID: <CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>

On Tue, Oct 2, 2018 at 10:59 AM David Sommer <david97sommer at gmail.com>
wrote:

> Hello,
>
> i was trying to solve a linear system with a MPIDENSE matrix. I did
> PC pc;
> KSPGetPC(solver, &pc);
> PCSetType(pc, PCLU);
> KSPSetOperators(solver, a_petsc, a_petsc);
> for some KSP solver and some Mat a_petsc. When i run the code sequentially
> (mpirun -n 1) it works, but once i start it in parallel it breaks with the
> error message
>
>
>
>
>
>
>
> *[0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html
> <http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html> for
> possible LU and Cholesky
> solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
> ./configure with
> --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble
> shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.3,
> unknown
> [0]PETSC ERROR: Unknown Name on a arch-linux2-c-debug named manjaroDesktop
> by david Tue Oct  2 16:47:10
> 2018
> [0]PETSC ERROR: Configure options --download-superlu_dist
> --download-parmetis --download-metis
> --download-ptscotch
> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in
> /home/david/lib/petsc/src/mat/interface/matrix.c     *
>
> I recompiled petsc with the recommended options (and also trying some
> other options) but the error persists. on the linear solve table there is
> an X at the required solver package column for dense LU. What does it mean?
> On
> https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
> it says "PLAPACK". Did you deprecate it? What is the replacement?
>

I do not see PLAPACK. You want Elemental.

  Thanks,

    Matt



> Thanks
>
> David Sommer
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/1f4d2a5d/attachment-0001.html>

From fdkong.jd at gmail.com  Tue Oct  2 10:52:58 2018
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 2 Oct 2018 09:52:58 -0600
Subject: [petsc-users] UCX ERROR KNEM inline copy failed
In-Reply-To: <CAMYG4GnxydDvZ5As1DXV-EimV56uEunZ-tpyKcyUzWfTM_J2Ug@mail.gmail.com>
References: <4f35edc3-b2c3-5b93-7b95-cb60bc51fee8@mfm.tu-darmstadt.de>
	<CAMYG4GnxydDvZ5As1DXV-EimV56uEunZ-tpyKcyUzWfTM_J2Ug@mail.gmail.com>
Message-ID: <CAN5Wd-LgamC7z7O964XJ_kjut1mPDTWHNuGc6v3e4c5BNnCPQg@mail.gmail.com>

The error messages may have nothing to do with PETSc and MOOSE.

It might be from a package for MPI communication
https://github.com/openucx/ucx.  I have no experiences on such things. It
may be helpful to contact your HPC administer.

Thanks,

Fande,

On Tue, Oct 2, 2018 at 9:24 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 2, 2018 at 11:16 AM Y. Yang <
> yangyiwei.yang at mfm.tu-darmstadt.de> wrote:
>
>> Dear PETSc team
>>
>> Recently I'm using MOOSE (http://www.mooseframework.org/) which is built
>> with PETSc and, Unfortunately, I encountered some problems with
>> following PETSc options:
>>
>
> I do not know what problem you are reporting.I don't know what package
> knem_ep.c is
> part of, but its not PETSc.
>
>   Thanks,
>
>      Matt
>
>
>> petsc_options_iname = '-pc_type -ksp_gmres_restart -sub_ksp_type
>> -sub_pc_type -pc_asm_overlap -pc_factor_mat_solver_package'
>>
>> petsc_options_value = 'asm          1201  preonly             ilu
>>             4    superlu_dist'
>>
>>
>> the error message is:
>>
>> Time Step 1, time = 1
>>                  dt = 1
>>
>>      |residual|_2 of individual variables:
>>                          c:   779.034
>>                          w:   0
>>                          T:   6.57948e+07
>>                          gr0: 211.617
>>                          gr1: 206.973
>>                          gr2: 209.382
>>                          gr3: 191.089
>>                          gr4: 185.242
>>                          gr5: 157.361
>>                          gr6: 128.473
>>                          gr7: 87.6029
>>
>>   0 Nonlinear |R| =  [32m6.579482e+07 [39m
>> [1538482623.976180] [hpb0085:22501:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482605.111342] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482606.761138] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482607.107478] [hpb0085:22502:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482605.882817] [hpb0085:22503:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482607.133543] [hpb0085:22503:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482621.905475] [hpb0085:22510:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482626.531234] [hpb0085:22510:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482627.613343] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482627.830489] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482629.852351] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482630.194620] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482630.280636] [hpb0085:22515:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482600.219314] [hpb0085:22516:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482658.960350] [hpb0085:22516:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482622.949471] [hpb0085:22517:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482612.502017] [hpb0085:22500:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482613.231970] [hpb0085:22500:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482621.417530] [hpb0085:22520:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482622.020998] [hpb0085:22520:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482606.221292] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482606.676987] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482606.896865] [hpb0085:22521:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482639.611427] [hpb0085:22522:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482631.435277] [hpb0085:22523:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482658.278343] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482658.396945] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482659.917476] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> [1538482660.162064] [hpb0085:22512:0]        knem_ep.c:84   UCX ERROR
>> KNEM inline copy failed, err = -1 Invalid argument
>> 2 total processes killed (some possibly by mpirun during cleanup)
>>
>>
>> Here's the status of the simulation
>>
>> Parallelism:
>>    Num Processors:          100
>>    Num Threads:             1
>>
>> Mesh:
>>    Parallel Type:           distributed
>>    Mesh Dimension:          3
>>    Spatial Dimension:       3
>>    Nodes:
>>      Total:                 2065551
>>      Local:                 22774
>>    Elems:
>>      Total:                 2000000
>>      Local:                 20006
>>    Num Subdomains:          1
>>    Num Partitions:          100
>>    Partitioner:             parmetis
>>
>> Nonlinear System:
>>    Num DOFs:                18589959
>>    Num Local DOFs:          204966
>>    Variables:               { "c" "w" "T" "gr0" "gr1" "gr2" "gr3" "gr4"
>> "gr5" }
>>    Finite Element Types:    "LAGRANGE"
>>    Approximation Orders:    "FIRST"
>>
>> Auxiliary System:
>>    Num DOFs:                10065551
>>    Num Local DOFs:          102798
>>    Variables:               "bnds" { "var_indices" "unique_grains" } {
>> "M" "dM/dT" }
>>    Finite Element Types:    "LAGRANGE" "MONOMIAL" "MONOMIAL"
>>    Approximation Orders:    "FIRST" "CONSTANT" "CONSTANT"
>>
>> Relationship Managers:
>>    Geometric                : GrainTrackerHaloRM (2 layers)
>>
>> Execution Information:
>>    Executioner:             Transient
>>    TimeStepper:             IterationAdaptiveDT
>>    Solver Mode:             Preconditioned JFNK
>>
>>
>> I tried modifying the parameters and other preconditioning option, the
>> problem is much the same. So I don't know where I did wrong or there is
>> actually suitable PETSc option to deal with such problem with large
>> mesh. I would like to hear your response.
>>
>> Sincerely,
>> Yang
>>
>> --
>> ______________________________________________________
>>
>> Yangyiwei Yang
>> Wissenschaftliche Hilfskraft
>>
>> TU Darmstadt
>> Fachbereich 11 - Material- und Geowissenschaften
>> Fachgebiet Mechanik funktionaler Materialien
>>
>> L1 | 08 402
>> Otto Berndt Stra?e 3
>> D-64287 Darmstadt
>>
>> Tel: +49 (0)6151-16-22923
>> Email: yangyiwei.yang at mfm.tu-darmstadt.de
>> Homepage: http://www.mawi.tu-darmstadt.de/mfm
>> ORCID: 0000-0001-5505-7117
>>
>> ______________________________________________________
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/322022de/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct  2 11:12:21 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 2 Oct 2018 16:12:21 +0000
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
	<CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
Message-ID: <CAGCphBs4=xpm0X3Jk_zfQy4XBYHjKcQdUCR6GgUV+Rxdjo8Sjw@mail.gmail.com>

Matt:

I recompiled petsc with the recommended options (and also trying some other options) but the error persists. on the linear solve table there is an X at the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

I do not see PLAPACK. You want Elemental.
petsc-3.2 uses  PLAPACK. We replaced  PLAPACK with Elemental long time ago.
Hong

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/fe812830/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct  2 11:12:21 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 2 Oct 2018 16:12:21 +0000
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
	<CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
Message-ID: <CAGCphBs4=xpm0X3Jk_zfQy4XBYHjKcQdUCR6GgUV+Rxdjo8Sjw@mail.gmail.com>

Matt:

I recompiled petsc with the recommended options (and also trying some other options) but the error persists. on the linear solve table there is an X at the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

I do not see PLAPACK. You want Elemental.
petsc-3.2 uses  PLAPACK. We replaced  PLAPACK with Elemental long time ago.
Hong

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/fe812830/attachment-0003.html>

From hzhang at mcs.anl.gov  Tue Oct  2 11:12:21 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Tue, 2 Oct 2018 16:12:21 +0000
Subject: [petsc-users] Parallel LU with MPIDENSE
In-Reply-To: <CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
References: <CAHNyna0BoxJ885qsYTatzT2qRVL5e0TuHzG8tXb-pbGkuZsX0g@mail.gmail.com>
	<CAMYG4G=iBEieS1xFh=g5azAV6XSFGH38u=znfwLZinzMtVPrug@mail.gmail.com>
Message-ID: <CAGCphBs4=xpm0X3Jk_zfQy4XBYHjKcQdUCR6GgUV+Rxdjo8Sjw@mail.gmail.com>

Matt:

I recompiled petsc with the recommended options (and also trying some other options) but the error persists. on the linear solve table there is an X at the required solver package column for dense LU. What does it mean? On
https://www.mcs.anl.gov/petsc/petsc-3.2/docs/linearsolvertable.html
it says "PLAPACK". Did you deprecate it? What is the replacement?

I do not see PLAPACK. You want Elemental.
petsc-3.2 uses  PLAPACK. We replaced  PLAPACK with Elemental long time ago.
Hong

--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/fe812830/attachment-0004.html>

From weizhuo2 at illinois.edu  Tue Oct  2 13:24:37 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Tue, 2 Oct 2018 13:24:37 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
Message-ID: <CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>

The example code and makefile are attached below. The whole thing started
as I tried to build a Helmholtz solver, and the mean error (calculated by:
sum( | numerical_sol - analytical_sol | / analytical_sol ) ) increases as I
use finer and finer grids. Then I looked at the example 12 (Laplacian
solver) which is similar to what I did to see if I have missed something.
The example is using 2_norm. I have made some minor modifications (3
places) on the code, you can search 'Modified' in the code to see them.

If this helps: I configured the PETSc to use real and double precision.
Changed the name of the example code from ex12.c to ex12c.c

Thanks for all your reply!

Weizhuo


Smith, Barry F. <bsmith at mcs.anl.gov>


>    Please send your version of the example that computes the mean norm of
> the grid; I suspect we are talking apples and oranges
>
>    Barry
>
>
>
> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
> >
> > I also tried to divide the norm by m*n , which is the number of grids,
> the trend of norm still increases.
> >
> > Thanks!
> >
> > Weizhuo
> >
> > Matthew Knepley <knepley at gmail.com>
> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
> wrote:
> > Hi!
> >
> > I'm recently trying out the example code provided with the KSP solver
> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
> since most of the time error should decreases when using finer grid. Am I
> doing this wrong?
> >
> > The norm is misleading in that it is the l_2 norm, meaning just the sqrt
> of the sum of the squares of
> > the vector entries. It should be scaled by the volume element to
> approximate a scale-independent
> > norm (like the L_2 norm).
> >
> >   Thanks,
> >
> >      Matt
> >
> > Thanks!
> > --
> > Wang Weizhuo
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> >
> >
> > --
> > Wang Weizhuo
>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/049c9268/attachment.html>
-------------- next part --------------

static char help[] = "Solves a linear system in parallel with KSP.\n\
Input parameters include:\n\
  -m <mesh_x>       : number of mesh points in x-direction\n\
  -n <mesh_n>       : number of mesh points in y-direction\n\n";

/*T
   Concepts: KSP^solving a system of linear equations
   Concepts: KSP^Laplacian, 2d
   Concepts: PC^registering preconditioners
   Processors: n
T*/

/*
   Demonstrates registering a new preconditioner (PC) type.

   To register a PC type whose code is linked into the executable,
   use PCRegister(). To register a PC type in a dynamic library use PCRegister()

   Also provide the prototype for your PCCreate_XXX() function. In
   this example we use the PETSc implementation of the Jacobi method,
   PCCreate_Jacobi() just as an example.

   See the file src/ksp/pc/impls/jacobi/jacobi.c for details on how to
   write a new PC component.

   See the manual page PCRegister() for details on how to register a method.
*/

/*
  Include "petscksp.h" so that we can use KSP solvers.  Note that this file
  automatically includes:
     petscsys.h       - base PETSc routines   petscvec.h - vectors
     petscmat.h - matrices
     petscis.h     - index sets            petscksp.h - Krylov subspace methods
     petscviewer.h - viewers               petscpc.h  - preconditioners
*/
#include <petscksp.h>

PETSC_EXTERN PetscErrorCode PCCreate_Jacobi(PC);

int main(int argc,char **args)
{
  Vec            x,b,u;  /* approx solution, RHS, exact solution */
  Mat            A;        /* linear system matrix */
  KSP            ksp;     /* linear solver context */
  PetscReal      norm;     /* norm of solution error */
  PetscInt       i,j,Ii,J,Istart,Iend,m = 8,n = 7,its;
  PetscErrorCode ierr;
  PetscScalar    v,one = 1.0;
  PC             pc;      /* preconditioner context */

  ierr = PetscInitialize(&argc,&args,(char*)0,help);if (ierr) return ierr;
  ierr = PetscOptionsGetInt(NULL,NULL,"-m",&m,NULL);CHKERRQ(ierr);
  ierr = PetscOptionsGetInt(NULL,NULL,"-n",&n,NULL);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
         Compute the matrix and right-hand-side vector that define
         the linear system, Ax = b.
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */
  /*
     Create parallel matrix, specifying only its global dimensions.
     When using MatCreate(), the matrix format can be specified at
     runtime. Also, the parallel partitioning of the matrix can be
     determined by PETSc at runtime.
  */
  ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr);
  ierr = MatSetSizes(A,PETSC_DECIDE,PETSC_DECIDE,m*n,m*n);CHKERRQ(ierr);
  ierr = MatSetFromOptions(A);CHKERRQ(ierr);
  ierr = MatSetUp(A);CHKERRQ(ierr);

  /*
     Currently, all PETSc parallel matrix formats are partitioned by
     contiguous chunks of rows across the processors.  Determine which
     rows of the matrix are locally owned.
  */
  ierr = MatGetOwnershipRange(A,&Istart,&Iend);CHKERRQ(ierr);

  /*
     Set matrix elements for the 2-D, five-point stencil in parallel.
      - Each processor needs to insert only elements that it owns
        locally (but any non-local elements will be sent to the
        appropriate processor during matrix assembly).
      - Always specify global rows and columns of matrix entries.
   */
  for (Ii=Istart; Ii<Iend; Ii++) {
    v = -1.0; i = Ii/n; j = Ii - i*n;
    if (i>0)   {J = Ii - n; ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);}
    if (i<m-1) {J = Ii + n; ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);}
    if (j>0)   {J = Ii - 1; ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);}
    if (j<n-1) {J = Ii + 1; ierr = MatSetValues(A,1,&Ii,1,&J,&v,INSERT_VALUES);CHKERRQ(ierr);}
    v = 4.0; ierr = MatSetValues(A,1,&Ii,1,&Ii,&v,INSERT_VALUES);CHKERRQ(ierr);
  }

  /*
     Assemble matrix, using the 2-step process:
       MatAssemblyBegin(), MatAssemblyEnd()
     Computations can be done while messages are in transition
     by placing code between these two statements.
  */
  ierr = MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);
  ierr = MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY);CHKERRQ(ierr);

  /*
     Create parallel vectors.
      - When using VecCreate(), VecSetSizes() and VecSetFromOptions(),
      we specify only the vector's global
        dimension; the parallel partitioning is determined at runtime.
      - When solving a linear system, the vectors and matrices MUST
        be partitioned accordingly.  PETSc automatically generates
        appropriately partitioned matrices and vectors when MatCreate()
        and VecCreate() are used with the same communicator.
      - Note: We form 1 vector from scratch and then duplicate as needed.
  */
  ierr = VecCreate(PETSC_COMM_WORLD,&u);CHKERRQ(ierr);
  ierr = VecSetSizes(u,PETSC_DECIDE,m*n);CHKERRQ(ierr);
  ierr = VecSetFromOptions(u);CHKERRQ(ierr);
  ierr = VecDuplicate(u,&b);CHKERRQ(ierr);
  ierr = VecDuplicate(b,&x);CHKERRQ(ierr);

  /*
     Set exact solution; then compute right-hand-side vector.
     Use an exact solution of a vector with all elements of 1.0;
  */
  ierr = VecSet(u,one);CHKERRQ(ierr);
  ierr = MatMult(A,u,b);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                Create the linear solver and set various options
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  /*
     Create linear solver context
  */
  ierr = KSPCreate(PETSC_COMM_WORLD,&ksp);CHKERRQ(ierr);

  /*
     Set operators. Here the matrix that defines the linear system
     also serves as the preconditioning matrix.
  */
  ierr = KSPSetOperators(ksp,A,A);CHKERRQ(ierr);

  // Modified: Make sure it uses GMRES
  ierr = KSPSetType(ksp,KSPGMRES);
  // Modified: Make sure it uses desired tolerance
  ierr = KSPSetOptionsPrefix(ksp,"ksp_atol 1e-12");
  ierr = KSPSetOptionsPrefix(ksp,"ksp_rtol 1e-9");

  /*
       First register a new PC type with the command PCRegister()
  */
  ierr = PCRegister("ourjacobi",PCCreate_Jacobi);CHKERRQ(ierr);

  /*
     Set the PC type to be the new method
  */
  ierr = KSPGetPC(ksp,&pc);CHKERRQ(ierr);
  ierr = PCSetType(pc,"ourjacobi");CHKERRQ(ierr);

  /*
    Set runtime options, e.g.,
        -ksp_type <type> -pc_type <type> -ksp_monitor -ksp_rtol <rtol>
    These options will override those specified above as long as
    KSPSetFromOptions() is called _after_ any other customization
    routines.
  */
  ierr = KSPSetFromOptions(ksp);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                      Solve the linear system
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  ierr = KSPSolve(ksp,b,x);CHKERRQ(ierr);

  /* - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                      Check solution and clean up
     - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - */

  /*
     Check the error
  */
  ierr = VecAXPY(x,-1.0,u);CHKERRQ(ierr);
  ierr = VecNorm(x,NORM_2,&norm);CHKERRQ(ierr);
  ierr = KSPGetIterationNumber(ksp,&its);CHKERRQ(ierr);

  /*
     Print convergence information.  PetscPrintf() produces a single
     print statement from all processes that share a communicator.
  */
  
  // Modified: replaced  (double)norm   with   (double)norm/(m*n)
  ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations %D\n",(double)norm/(m*n),its);CHKERRQ(ierr);

  /*
     Free work space.  All PETSc objects should be destroyed when they
     are no longer needed.
  */
  ierr = KSPDestroy(&ksp);CHKERRQ(ierr);
  ierr = VecDestroy(&u);CHKERRQ(ierr);  ierr = VecDestroy(&x);CHKERRQ(ierr);
  ierr = VecDestroy(&b);CHKERRQ(ierr);  ierr = MatDestroy(&A);CHKERRQ(ierr);

  /*
     Always call PetscFinalize() before exiting a program.  This routine
       - finalizes the PETSc libraries as well as MPI
       - provides summary and diagnostic information if certain runtime
         options are chosen (e.g., -log_view).
  */
  ierr = PetscFinalize();
  return ierr;
}


/*TEST

   test:
      args: -ksp_gmres_cgs_refinement_type refine_always

TEST*/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 959 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/049c9268/attachment.obj>

From mfadams at lbl.gov  Tue Oct  2 14:27:12 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 2 Oct 2018 15:27:12 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
Message-ID: <CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>

On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> The example code and makefile are attached below. The whole thing started
> as I tried to build a Helmholtz solver, and the mean error (calculated by:
> sum( | numerical_sol - analytical_sol | / analytical_sol ) )
>

This is a one norm. If you use max (instead of sum) then you don't need to
scale. You do have to be careful about dividing by (near) zero.


> increases as I use finer and finer grids.
>

What was the rate of increase?


> Then I looked at the example 12 (Laplacian solver) which is similar to
> what I did to see if I have missed something. The example is using 2_norm.
> I have made some minor modifications (3 places) on the code, you can search
> 'Modified' in the code to see them.
>
> If this helps: I configured the PETSc to use real and double precision.
> Changed the name of the example code from ex12.c to ex12c.c
>
> Thanks for all your reply!
>
> Weizhuo
>
>
> Smith, Barry F. <bsmith at mcs.anl.gov>
>
>
>>    Please send your version of the example that computes the mean norm of
>> the grid; I suspect we are talking apples and oranges
>>
>>    Barry
>>
>>
>>
>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>> >
>> > I also tried to divide the norm by m*n , which is the number of grids,
>> the trend of norm still increases.
>> >
>> > Thanks!
>> >
>> > Weizhuo
>> >
>> > Matthew Knepley <knepley at gmail.com>
>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>> > Hi!
>> >
>> > I'm recently trying out the example code provided with the KSP solver
>> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
>> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
>> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
>> since most of the time error should decreases when using finer grid. Am I
>> doing this wrong?
>> >
>> > The norm is misleading in that it is the l_2 norm, meaning just the
>> sqrt of the sum of the squares of
>> > the vector entries. It should be scaled by the volume element to
>> approximate a scale-independent
>> > norm (like the L_2 norm).
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > Thanks!
>> > --
>> > Wang Weizhuo
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>> >
>> >
>> > --
>> > Wang Weizhuo
>>
>>
>
> --
> Wang Weizhuo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/8a82549f/attachment.html>

From weizhuo2 at illinois.edu  Tue Oct  2 16:04:27 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Tue, 2 Oct 2018 16:04:27 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
Message-ID: <CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>

Yes I was using one norm in my Helmholtz code, the example code used 2
norm. But now I am using 2 norm in both code.

  /*
     Check the error
  */
  ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
  ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
  ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
  ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
%D\n",(double)norm/(m*n),its); CHKERRQ(ierr);

 I made a plot to show the increase:

[image: Norm comparison.png]

Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??2:27???

>
>
> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> The example code and makefile are attached below. The whole thing started
>> as I tried to build a Helmholtz solver, and the mean error (calculated by:
>> sum( | numerical_sol - analytical_sol | / analytical_sol ) )
>>
>
> This is a one norm. If you use max (instead of sum) then you don't need to
> scale. You do have to be careful about dividing by (near) zero.
>
>
>> increases as I use finer and finer grids.
>>
>
> What was the rate of increase?
>
>
>> Then I looked at the example 12 (Laplacian solver) which is similar to
>> what I did to see if I have missed something. The example is using 2_norm.
>> I have made some minor modifications (3 places) on the code, you can search
>> 'Modified' in the code to see them.
>>
>> If this helps: I configured the PETSc to use real and double precision.
>> Changed the name of the example code from ex12.c to ex12c.c
>>
>> Thanks for all your reply!
>>
>> Weizhuo
>>
>>
>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>
>>
>>>    Please send your version of the example that computes the mean norm
>>> of the grid; I suspect we are talking apples and oranges
>>>
>>>    Barry
>>>
>>>
>>>
>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>> >
>>> > I also tried to divide the norm by m*n , which is the number of grids,
>>> the trend of norm still increases.
>>> >
>>> > Thanks!
>>> >
>>> > Weizhuo
>>> >
>>> > Matthew Knepley <knepley at gmail.com>
>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>> > Hi!
>>> >
>>> > I'm recently trying out the example code provided with the KSP solver
>>> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
>>> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
>>> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
>>> since most of the time error should decreases when using finer grid. Am I
>>> doing this wrong?
>>> >
>>> > The norm is misleading in that it is the l_2 norm, meaning just the
>>> sqrt of the sum of the squares of
>>> > the vector entries. It should be scaled by the volume element to
>>> approximate a scale-independent
>>> > norm (like the L_2 norm).
>>> >
>>> >   Thanks,
>>> >
>>> >      Matt
>>> >
>>> > Thanks!
>>> > --
>>> > Wang Weizhuo
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>> >
>>> >
>>> > --
>>> > Wang Weizhuo
>>>
>>>
>>
>> --
>> Wang Weizhuo
>>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/70505382/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/70505382/attachment-0001.png>

From mfadams at lbl.gov  Tue Oct  2 16:10:49 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 2 Oct 2018 17:10:49 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
Message-ID: <CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>

On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> Yes I was using one norm in my Helmholtz code, the example code used 2
> norm. But now I am using 2 norm in both code.
>
>   /*
>      Check the error
>   */
>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>
>  I made a plot to show the increase:
>


FYI, this is asymptoting to a constant.  What solver tolerance are
you using?


>
> [image: Norm comparison.png]
>
> Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??2:27???
>
>>
>>
>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> The example code and makefile are attached below. The whole thing
>>> started as I tried to build a Helmholtz solver, and the mean error
>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>> )
>>>
>>
>> This is a one norm. If you use max (instead of sum) then you don't need
>> to scale. You do have to be careful about dividing by (near) zero.
>>
>>
>>> increases as I use finer and finer grids.
>>>
>>
>> What was the rate of increase?
>>
>>
>>> Then I looked at the example 12 (Laplacian solver) which is similar to
>>> what I did to see if I have missed something. The example is using 2_norm.
>>> I have made some minor modifications (3 places) on the code, you can search
>>> 'Modified' in the code to see them.
>>>
>>> If this helps: I configured the PETSc to use real and double precision.
>>> Changed the name of the example code from ex12.c to ex12c.c
>>>
>>> Thanks for all your reply!
>>>
>>> Weizhuo
>>>
>>>
>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>
>>>
>>>>    Please send your version of the example that computes the mean norm
>>>> of the grid; I suspect we are talking apples and oranges
>>>>
>>>>    Barry
>>>>
>>>>
>>>>
>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>> >
>>>> > I also tried to divide the norm by m*n , which is the number of
>>>> grids, the trend of norm still increases.
>>>> >
>>>> > Thanks!
>>>> >
>>>> > Weizhuo
>>>> >
>>>> > Matthew Knepley <knepley at gmail.com>
>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>> > Hi!
>>>> >
>>>> > I'm recently trying out the example code provided with the KSP solver
>>>> (ex12.c). I noticed that the mean norm of the grid increases as I use finer
>>>> meshes. For example, the mean norm is 5.72e-8 at m=10 n=10. However at
>>>> m=100, n=100, mean norm increases to 9.55e-6. This seems counter intuitive,
>>>> since most of the time error should decreases when using finer grid. Am I
>>>> doing this wrong?
>>>> >
>>>> > The norm is misleading in that it is the l_2 norm, meaning just the
>>>> sqrt of the sum of the squares of
>>>> > the vector entries. It should be scaled by the volume element to
>>>> approximate a scale-independent
>>>> > norm (like the L_2 norm).
>>>> >
>>>> >   Thanks,
>>>> >
>>>> >      Matt
>>>> >
>>>> > Thanks!
>>>> > --
>>>> > Wang Weizhuo
>>>> >
>>>> >
>>>> > --
>>>> > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > -- Norbert Wiener
>>>> >
>>>> > https://www.cse.buffalo.edu/~knepley/
>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>> >
>>>> >
>>>> > --
>>>> > Wang Weizhuo
>>>>
>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>
> --
> Wang Weizhuo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2fb85482/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2fb85482/attachment-0001.png>

From weizhuo2 at illinois.edu  Tue Oct  2 16:26:42 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Tue, 2 Oct 2018 16:26:42 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
Message-ID: <CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>

I didn't specify a tolerance, it was using the default tolerance. Doesn't
the asymptoting norm implies finer grid won't help to get finer solution?

Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??4:11???

>
>
> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> Yes I was using one norm in my Helmholtz code, the example code used 2
>> norm. But now I am using 2 norm in both code.
>>
>>   /*
>>      Check the error
>>   */
>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>
>>  I made a plot to show the increase:
>>
>
>
> FYI, this is asymptoting to a constant.  What solver tolerance are
> you using?
>
>
>>
>> [image: Norm comparison.png]
>>
>> Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??2:27???
>>
>>>
>>>
>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> The example code and makefile are attached below. The whole thing
>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>> )
>>>>
>>>
>>> This is a one norm. If you use max (instead of sum) then you don't need
>>> to scale. You do have to be careful about dividing by (near) zero.
>>>
>>>
>>>> increases as I use finer and finer grids.
>>>>
>>>
>>> What was the rate of increase?
>>>
>>>
>>>> Then I looked at the example 12 (Laplacian solver) which is similar to
>>>> what I did to see if I have missed something. The example is using 2_norm.
>>>> I have made some minor modifications (3 places) on the code, you can search
>>>> 'Modified' in the code to see them.
>>>>
>>>> If this helps: I configured the PETSc to use real and double precision.
>>>> Changed the name of the example code from ex12.c to ex12c.c
>>>>
>>>> Thanks for all your reply!
>>>>
>>>> Weizhuo
>>>>
>>>>
>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>
>>>>
>>>>>    Please send your version of the example that computes the mean norm
>>>>> of the grid; I suspect we are talking apples and oranges
>>>>>
>>>>>    Barry
>>>>>
>>>>>
>>>>>
>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>> >
>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>> grids, the trend of norm still increases.
>>>>> >
>>>>> > Thanks!
>>>>> >
>>>>> > Weizhuo
>>>>> >
>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>> > Hi!
>>>>> >
>>>>> > I'm recently trying out the example code provided with the KSP
>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>> intuitive, since most of the time error should decreases when using finer
>>>>> grid. Am I doing this wrong?
>>>>> >
>>>>> > The norm is misleading in that it is the l_2 norm, meaning just the
>>>>> sqrt of the sum of the squares of
>>>>> > the vector entries. It should be scaled by the volume element to
>>>>> approximate a scale-independent
>>>>> > norm (like the L_2 norm).
>>>>> >
>>>>> >   Thanks,
>>>>> >
>>>>> >      Matt
>>>>> >
>>>>> > Thanks!
>>>>> > --
>>>>> > Wang Weizhuo
>>>>> >
>>>>> >
>>>>> > --
>>>>> > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > -- Norbert Wiener
>>>>> >
>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>> >
>>>>> >
>>>>> > --
>>>>> > Wang Weizhuo
>>>>>
>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>
>> --
>> Wang Weizhuo
>>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2d77356a/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2d77356a/attachment-0001.png>

From nsime at carnegiescience.edu  Tue Oct  2 17:47:11 2018
From: nsime at carnegiescience.edu (Nate Sime)
Date: Tue, 2 Oct 2018 18:47:11 -0400
Subject: [petsc-users] MatNest with pc_fieldsplit_type multiplicative yields
 PetscLayoutSetUp error
Message-ID: <a74e3462-e861-f46b-baad-bc2ab68df4e4@carnegiescience.edu>

Initially, I apologise for not being able to provide a minimal working 
example.

I have a system of 4 PDEs defined in 2 spatial dimensions which I have 
discretised by the FEM.

This includes a Taylor-Hood Stokes system, a modified Helmholtz type 
problem, and a nonlinear advection equation.

I assemble each of the blocks of this 4x4 system into Mats and then put 
those into a MatNest.

Additionally I assemble a 4x4 block preconditioner in the same manner, 
using the standard pressure mass matrix for the continuity equation block.

I am using the following simple options:

-ksp_type fgmres
-pc_type fieldsplit
-pc_fieldsplit_type multiplicative

-fieldsplit_0_ksp_type preonly
-fieldsplit_0_pc_type hypre

-fieldsplit_1_ksp_type preonly
-fieldsplit_1_pc_type hypre

-fieldsplit_2_ksp_type preonly
-fieldsplit_2_pc_type hypre

-fieldsplit_3_ksp_type preonly
-fieldsplit_3_pc_type hypre


Solving the system works and scales reasonably well up until the problem 
extends beyond a size of approximately 50e6 DoF with 192 processes 
across 4 nodes.

Beyond that, I run into the following error with PetscLayoutFindOwner 
which I have attached the error message at the end of the email.

Tracing through the PETSc source, I find that the error arises dealing 
with -pc_fieldsplit_type multiplicative.

Changing to -pc_fieldsplit_type additive, I am able to continue my 
scaling experiments beyond 50e6 with no problem so far.

I was hoping someone may have some basic insight into what I may have 
done naively and incorrectly to cause this issue.

Additionally, please let me know what I have inevitably neglected to 
provide for elucidation.

Nate.


[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: PetscLayoutSetUp() must be called first
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.0, unknown
[0]PETSC ERROR: Unknown Name on a? named 
c456-101.stampede2.tacc.utexas.edu by Tue Oct? 2 17:11:27 2018
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: #1 PetscLayoutFindOwner() line 246 in 
petsc/petsc/include/petscis.h
[0]PETSC ERROR: #2 PetscSFSetGraphLayout() line 511 in 
petsc/petsc/src/vec/is/utils/pmap.c
[0]PETSC ERROR: #3 MatNestCreateAggregateL2G_Private() line 1188 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #4 MatSetUp_NestIS_Private() line 1336 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #5 MatNestSetSubMats_Nest() line 1073 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #6 MatNestSetSubMats() line 1130 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #7 MatCreateNest() line 1400 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #8 MatNestGetRow() line 360 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #9 MatNestFindSubMat() line 398 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #10 MatCreateSubMatrix_Nest() line 425 in 
petsc/petsc/src/mat/impls/nest/matnest.c
[0]PETSC ERROR: #11 MatCreateSubMatrix() line 8023 in 
petsc/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #12 PCSetUp_FieldSplit() line 649 in 
petsc/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
[0]PETSC ERROR: #13 PCSetUp() line 932 in 
petsc/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #14 KSPSetUp() line 391 in 
petsc/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #15 KSPSolve() line 723 in 
petsc/petsc/src/ksp/ksp/interface/itfunc.c



From knepley at gmail.com  Tue Oct  2 18:52:23 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Oct 2018 19:52:23 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
Message-ID: <CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>

On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> I didn't specify a tolerance, it was using the default tolerance. Doesn't
> the asymptoting norm implies finer grid won't help to get finer solution?
>

There are two things going on in your test, discretization error controlled
by the grid, and algebraic error controlled by the solver. This makes it
difficult to isolate what is happening. However, it seems clear that your
plot is looking at algebraic error. You can confirm this by using

  -pc_type lu

for the solve. Then all you have is discretization error.

  Thanks,

     Matt


> Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??4:11???
>
>>
>>
>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Yes I was using one norm in my Helmholtz code, the example code used 2
>>> norm. But now I am using 2 norm in both code.
>>>
>>>   /*
>>>      Check the error
>>>   */
>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>
>>>  I made a plot to show the increase:
>>>
>>
>>
>> FYI, this is asymptoting to a constant.  What solver tolerance are
>> you using?
>>
>>
>>>
>>> [image: Norm comparison.png]
>>>
>>> Mark Adams <mfadams at lbl.gov> ?2018?10?2??? ??2:27???
>>>
>>>>
>>>>
>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> The example code and makefile are attached below. The whole thing
>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>> )
>>>>>
>>>>
>>>> This is a one norm. If you use max (instead of sum) then you don't need
>>>> to scale. You do have to be careful about dividing by (near) zero.
>>>>
>>>>
>>>>> increases as I use finer and finer grids.
>>>>>
>>>>
>>>> What was the rate of increase?
>>>>
>>>>
>>>>> Then I looked at the example 12 (Laplacian solver) which is similar to
>>>>> what I did to see if I have missed something. The example is using 2_norm.
>>>>> I have made some minor modifications (3 places) on the code, you can search
>>>>> 'Modified' in the code to see them.
>>>>>
>>>>> If this helps: I configured the PETSc to use real and double
>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>
>>>>> Thanks for all your reply!
>>>>>
>>>>> Weizhuo
>>>>>
>>>>>
>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>
>>>>>
>>>>>>    Please send your version of the example that computes the mean
>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>>
>>>>>>
>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>> >
>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>> grids, the trend of norm still increases.
>>>>>> >
>>>>>> > Thanks!
>>>>>> >
>>>>>> > Weizhuo
>>>>>> >
>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>> > Hi!
>>>>>> >
>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>> grid. Am I doing this wrong?
>>>>>> >
>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just the
>>>>>> sqrt of the sum of the squares of
>>>>>> > the vector entries. It should be scaled by the volume element to
>>>>>> approximate a scale-independent
>>>>>> > norm (like the L_2 norm).
>>>>>> >
>>>>>> >   Thanks,
>>>>>> >
>>>>>> >      Matt
>>>>>> >
>>>>>> > Thanks!
>>>>>> > --
>>>>>> > Wang Weizhuo
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > -- Norbert Wiener
>>>>>> >
>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Wang Weizhuo
>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Wang Weizhuo
>>>>>
>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2abc126c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/2abc126c/attachment-0001.png>

From knepley at gmail.com  Tue Oct  2 19:07:55 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 2 Oct 2018 20:07:55 -0400
Subject: [petsc-users] MatNest with pc_fieldsplit_type multiplicative
 yields PetscLayoutSetUp error
In-Reply-To: <a74e3462-e861-f46b-baad-bc2ab68df4e4@carnegiescience.edu>
References: <a74e3462-e861-f46b-baad-bc2ab68df4e4@carnegiescience.edu>
Message-ID: <CAMYG4GnXpvPjbM2Yk=zr=00Q8CnEnE9fNtmg7QGiJGu8tHPBeA@mail.gmail.com>

On Tue, Oct 2, 2018 at 6:47 PM Nate Sime <nsime at carnegiescience.edu> wrote:

> Initially, I apologise for not being able to provide a minimal working
> example.
>
> I have a system of 4 PDEs defined in 2 spatial dimensions which I have
> discretised by the FEM.
>
> This includes a Taylor-Hood Stokes system, a modified Helmholtz type
> problem, and a nonlinear advection equation.
>

This sounds like magma dynamics, but that should have linear advection.


> I assemble each of the blocks of this 4x4 system into Mats and then put
> those into a MatNest.
>

We can track down this error. However, I want to first say that I don't
think
MatNest should ever be in your user code. MatNest is only an optimization
of the storage format, like SBAIJ. It does not contribute anything to
functionality.
If you want the ability to insert into a submatrix, you can use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html

or


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateLocalRef.html#MatCreateLocalRef

Then you can start off using just AIJ for testing. I doubt the MAtNest
optimization would
make much of a difference here, but testing it after everything works is
much easier.

  Thanks,

     Matt


> Additionally I assemble a 4x4 block preconditioner in the same manner,
> using the standard pressure mass matrix for the continuity equation block.
>
> I am using the following simple options:
>
> -ksp_type fgmres
> -pc_type fieldsplit
> -pc_fieldsplit_type multiplicative
>
> -fieldsplit_0_ksp_type preonly
> -fieldsplit_0_pc_type hypre
>
> -fieldsplit_1_ksp_type preonly
> -fieldsplit_1_pc_type hypre
>
> -fieldsplit_2_ksp_type preonly
> -fieldsplit_2_pc_type hypre
>
> -fieldsplit_3_ksp_type preonly
> -fieldsplit_3_pc_type hypre
>
>
> Solving the system works and scales reasonably well up until the problem
> extends beyond a size of approximately 50e6 DoF with 192 processes
> across 4 nodes.
>
> Beyond that, I run into the following error with PetscLayoutFindOwner
> which I have attached the error message at the end of the email.
>
> Tracing through the PETSc source, I find that the error arises dealing
> with -pc_fieldsplit_type multiplicative.
>
> Changing to -pc_fieldsplit_type additive, I am able to continue my
> scaling experiments beyond 50e6 with no problem so far.
>
> I was hoping someone may have some basic insight into what I may have
> done naively and incorrectly to cause this issue.
>
> Additionally, please let me know what I have inevitably neglected to
> provide for elucidation.
>
> Nate.
>
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: PetscLayoutSetUp() must be called first
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.0, unknown
> [0]PETSC ERROR: Unknown Name on a  named
> c456-101.stampede2.tacc.utexas.edu by Tue Oct  2 17:11:27 2018
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: #1 PetscLayoutFindOwner() line 246 in
> petsc/petsc/include/petscis.h
> [0]PETSC ERROR: #2 PetscSFSetGraphLayout() line 511 in
> petsc/petsc/src/vec/is/utils/pmap.c
> [0]PETSC ERROR: #3 MatNestCreateAggregateL2G_Private() line 1188 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #4 MatSetUp_NestIS_Private() line 1336 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #5 MatNestSetSubMats_Nest() line 1073 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #6 MatNestSetSubMats() line 1130 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #7 MatCreateNest() line 1400 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #8 MatNestGetRow() line 360 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #9 MatNestFindSubMat() line 398 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #10 MatCreateSubMatrix_Nest() line 425 in
> petsc/petsc/src/mat/impls/nest/matnest.c
> [0]PETSC ERROR: #11 MatCreateSubMatrix() line 8023 in
> petsc/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #12 PCSetUp_FieldSplit() line 649 in
> petsc/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #13 PCSetUp() line 932 in
> petsc/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #14 KSPSetUp() line 391 in
> petsc/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #15 KSPSolve() line 723 in
> petsc/petsc/src/ksp/ksp/interface/itfunc.c
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181002/9bfa48f8/attachment.html>

From andreaceresoli91 at gmail.com  Wed Oct  3 08:00:36 2018
From: andreaceresoli91 at gmail.com (Andrea Ceresoli)
Date: Wed, 3 Oct 2018 15:00:36 +0200
Subject: [petsc-users] DMNetwork
Message-ID: <CADXvvquBPJAk=90y30thHD_2SUp_sa43uY9av7rNO-MhzgirSA@mail.gmail.com>

Good morning,
I am contacting you for some clarification regarding DMNetwork routines.

>From the documentation I found the flow for the creation of an application
code is the following:
1. Create a Network and register components
2. Set network size and layout
3. Set components and number of variables for nodes/edges
4. Set up the network
5. Distribute the network

Following that flow, from point 1 to point 4 the Network is "owned" only by
the 0 rank processor while after point 5 the network is partitioned and
"owned" by the n processors involved in the computation (according to the
partitioner used).

I was trying to let all the n processors involved read a portion of the
network in such a way that the flow listed before will be a multiprocessor
process and not restricted only to the rank 0 processor (parallel input
data). In this way after point 4 I will have a distributed network and I
don't need to call DMNetworkDistribute(). Unfortunately the documentation I
found doesn't explain how to do that, so I would like to ask you if it is
feasible and if you have some examples or documentation that you can share
for this purpose.

Thank you in advance for your availability,

Sincerely,

Andrea Ceresoli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181003/bbed5b0a/attachment.html>

From hzhang at mcs.anl.gov  Wed Oct  3 10:57:54 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Wed, 3 Oct 2018 15:57:54 +0000
Subject: [petsc-users] DMNetwork
In-Reply-To: <CADXvvquBPJAk=90y30thHD_2SUp_sa43uY9av7rNO-MhzgirSA@mail.gmail.com>
References: <CADXvvquBPJAk=90y30thHD_2SUp_sa43uY9av7rNO-MhzgirSA@mail.gmail.com>
Message-ID: <CAGCphBujDU=0UvoVOwOFFQdWo0myJFjgWQD3TVrdzqKiPB+_zA@mail.gmail.com>

 Andrea:
From the documentation I found the flow for the creation of an application code is the following:
1. Create a Network and register components
2. Set network size and layout
3. Set components and number of variables for nodes/edges
4. Set up the network
5. Distribute the network

Following that flow, from point 1 to point 4 the Network is "owned" only by the 0 rank processor while after point 5 the network is partitioned and "owned" by the n processors involved in the computation (according to the partitioner used).

Correct. Currently, DMNetwork only supports sequential setup of network. I just started parallelizing it. I'll let you know when I'm done.

I was trying to let all the n processors involved read a portion of the network in such a way that the flow listed before will be a multiprocessor process and not restricted only to the rank 0 processor (parallel input data). In this way after point 4 I will have a distributed network and I don't need to call DMNetworkDistribute(). Unfortunately the documentation I found doesn't explain how to do that, so I would like to ask you if it is feasible and if you have some examples or documentation that you can share for this purpose.

It is not supported yet.  Internal works are needed. You may follow my development branch
https://bitbucket.org/petsc/petsc/branch/hzhang/pdmnetwork-setup

If you have idea/suggestions, let us know.
Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181003/95a34d3b/attachment-0002.html>

From hzhang at mcs.anl.gov  Wed Oct  3 10:57:54 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Wed, 3 Oct 2018 15:57:54 +0000
Subject: [petsc-users] DMNetwork
In-Reply-To: <CADXvvquBPJAk=90y30thHD_2SUp_sa43uY9av7rNO-MhzgirSA@mail.gmail.com>
References: <CADXvvquBPJAk=90y30thHD_2SUp_sa43uY9av7rNO-MhzgirSA@mail.gmail.com>
Message-ID: <CAGCphBujDU=0UvoVOwOFFQdWo0myJFjgWQD3TVrdzqKiPB+_zA@mail.gmail.com>

 Andrea:
From the documentation I found the flow for the creation of an application code is the following:
1. Create a Network and register components
2. Set network size and layout
3. Set components and number of variables for nodes/edges
4. Set up the network
5. Distribute the network

Following that flow, from point 1 to point 4 the Network is "owned" only by the 0 rank processor while after point 5 the network is partitioned and "owned" by the n processors involved in the computation (according to the partitioner used).

Correct. Currently, DMNetwork only supports sequential setup of network. I just started parallelizing it. I'll let you know when I'm done.

I was trying to let all the n processors involved read a portion of the network in such a way that the flow listed before will be a multiprocessor process and not restricted only to the rank 0 processor (parallel input data). In this way after point 4 I will have a distributed network and I don't need to call DMNetworkDistribute(). Unfortunately the documentation I found doesn't explain how to do that, so I would like to ask you if it is feasible and if you have some examples or documentation that you can share for this purpose.

It is not supported yet.  Internal works are needed. You may follow my development branch
https://bitbucket.org/petsc/petsc/branch/hzhang/pdmnetwork-setup

If you have idea/suggestions, let us know.
Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181003/95a34d3b/attachment-0003.html>

From amfoggia at gmail.com  Thu Oct  4 03:42:46 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Thu, 4 Oct 2018 10:42:46 +0200
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
Message-ID: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>

Hello all,

I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of a
matrix with the following characteristics:

* type: real, Hermitian, sparse
* linear size: 2333606220
* distributed in 2048 processes (64 nodes, 32 procs per node)

My code first preallocates the necessary memory with
*MatMPIAIJSetPreallocation*, then fills it with the values and finally it
calls the following functions to create the solver and diagonalize the
matrix:

EPSCreate(PETSC_COMM_WORLD, &solver);
EPSSetOperators(solver,matrix,NULL);
EPSSetProblemType(solver, EPS_HEP);
EPSSetType(solver, EPSLANCZOS);
EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
EPSSetFromOptions(solver);
EPSSolve(solver);

I want to make an estimation for larger size problems of the memory used by
the program (at every step) because I would like to keep it under 16 GB per
node. I've used the "memory usage" functions provided by PETSc, but
something happens during the solver stage that I can't explain. This brings
up two questions.

1) In each step I put a call to four memory functions and between them I
print the value of mem:

mem = 0;
PetscMallocGetCurrentUsage(&mem);
PetscMallocGetMaximumUsage(&mem);
PetscMemoryGetCurrentUsage(&mem);
PetscMemoryGetMaximumUsage(&mem);

I've read some other question in the mailing list regarding the same issue
but I can't fully understand this. What is the difference between all of
them? What information are they actually giving me? (I know this is only a
"per process" output). I copy the output of two steps of the program as an
example:

==================== step N ====================
MallocGetCurrent: 314513664.0 B
MallocGetMaximum: 332723328.0 B
MemoryGetCurrent: 539996160.0 B
MemoryGetMaximum: 0.0 B
==================== step N+1 ====================
MallocGetCurrent: 395902912.0 B
MallocGetMaximum: 415178624.0 B
MemoryGetCurrent: 623783936.0 B
MemoryGetMaximum: 623775744.0 B

2) I was using this information to make the calculation of the memory
required per node to run my problem. Also, I'm able to login to the
computing node while running and I can check the memory consumption (with
*top*). The memory used that I see with top is more or less the same as the
one reported by PETSc functions at the beginning. But during the
inialization of the solver and during the solving, *top* reports a
consumption two times bigger than the one the functions report. Is it
possible to know from where this extra memory consumption comes from? What
things does SLEPc allocate that need that much memory? I've been trying to
do the math but I think there are things I'm missing. I thought that part
of it comes from the "BV" that the option -eps_view reports:

BV Object: 2048 MPI processes
  type: svec
  17 columns of global length 2333606220
  vector orthogonalization method: modified Gram-Schmidt
  orthogonalization refinement: if needed (eta: 0.7071)
  block orthogonalization method: GS
  doing matmult as a single matrix-matrix product

But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less of
the "extra" memory.

Ale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/5e6881aa/attachment.html>

From knepley at gmail.com  Thu Oct  4 05:49:25 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Oct 2018 06:49:25 -0400
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
Message-ID: <CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>

On Thu, Oct 4, 2018 at 4:43 AM Ale Foggia <amfoggia at gmail.com> wrote:

> Hello all,
>
> I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of a
> matrix with the following characteristics:
>
> * type: real, Hermitian, sparse
> * linear size: 2333606220
> * distributed in 2048 processes (64 nodes, 32 procs per node)
>
> My code first preallocates the necessary memory with
> *MatMPIAIJSetPreallocation*, then fills it with the values and finally it
> calls the following functions to create the solver and diagonalize the
> matrix:
>
> EPSCreate(PETSC_COMM_WORLD, &solver);
> EPSSetOperators(solver,matrix,NULL);
> EPSSetProblemType(solver, EPS_HEP);
> EPSSetType(solver, EPSLANCZOS);
> EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
> EPSSetFromOptions(solver);
> EPSSolve(solver);
>
> I want to make an estimation for larger size problems of the memory used
> by the program (at every step) because I would like to keep it under 16 GB
> per node. I've used the "memory usage" functions provided by PETSc, but
> something happens during the solver stage that I can't explain. This brings
> up two questions.
>
> 1) In each step I put a call to four memory functions and between them I
> print the value of mem:
>

Did you call PetscMemorySetGetMaximumUsage() first?

We are computing https://en.wikipedia.org/wiki/Resident_set_size however we
can. Usually with getrusage().
>From this (https://www.binarytides.com/linux-command-check-memory-usage/),
it looks like top also reports
paged out memory.

   Matt


> mem = 0;
> PetscMallocGetCurrentUsage(&mem);
> PetscMallocGetMaximumUsage(&mem);
> PetscMemoryGetCurrentUsage(&mem);
> PetscMemoryGetMaximumUsage(&mem);
>
> I've read some other question in the mailing list regarding the same issue
> but I can't fully understand this. What is the difference between all of
> them? What information are they actually giving me? (I know this is only a
> "per process" output). I copy the output of two steps of the program as an
> example:
>
> ==================== step N ====================
> MallocGetCurrent: 314513664.0 B
> MallocGetMaximum: 332723328.0 B
> MemoryGetCurrent: 539996160.0 B
> MemoryGetMaximum: 0.0 B
> ==================== step N+1 ====================
> MallocGetCurrent: 395902912.0 B
> MallocGetMaximum: 415178624.0 B
> MemoryGetCurrent: 623783936.0 B
> MemoryGetMaximum: 623775744.0 B
>
> 2) I was using this information to make the calculation of the memory
> required per node to run my problem. Also, I'm able to login to the
> computing node while running and I can check the memory consumption (with
> *top*). The memory used that I see with top is more or less the same as the
> one reported by PETSc functions at the beginning. But during the
> inialization of the solver and during the solving, *top* reports a
> consumption two times bigger than the one the functions report. Is it
> possible to know from where this extra memory consumption comes from? What
> things does SLEPc allocate that need that much memory? I've been trying to
> do the math but I think there are things I'm missing. I thought that part
> of it comes from the "BV" that the option -eps_view reports:
>
> BV Object: 2048 MPI processes
>   type: svec
>   17 columns of global length 2333606220
>   vector orthogonalization method: modified Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
>
> But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less
> of the "extra" memory.
>
> Ale
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/34e3e565/attachment.html>

From jroman at dsic.upv.es  Thu Oct  4 06:12:18 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 4 Oct 2018 13:12:18 +0200
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
Message-ID: <E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>

Regarding the SLEPc part:
- What do you mean by "each step"? Are you calling EPSSolve() several times?
- Yes, the BV object is generally what takes most of the memory. It is allocated at the beginning of EPSSolve(). Depending on the solver/options, other memory may be allocated as well.
- You can also see the memory reported at the end of -log_view
- I would suggest using the default solver Krylov-Schur - it will do Lanczos with implicit restart, which will give faster convergence than the EPSLANCZOS solver.

Jose


> El 4 oct 2018, a las 12:49, Matthew Knepley <knepley at gmail.com> escribi?:
> 
> On Thu, Oct 4, 2018 at 4:43 AM Ale Foggia <amfoggia at gmail.com> wrote:
> Hello all,
> 
> I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of a matrix with the following characteristics:
> 
> * type: real, Hermitian, sparse
> * linear size: 2333606220 
> * distributed in 2048 processes (64 nodes, 32 procs per node)
> 
> My code first preallocates the necessary memory with *MatMPIAIJSetPreallocation*, then fills it with the values and finally it calls the following functions to create the solver and diagonalize the matrix:
> 
> EPSCreate(PETSC_COMM_WORLD, &solver);
> EPSSetOperators(solver,matrix,NULL);
> EPSSetProblemType(solver, EPS_HEP);
> EPSSetType(solver, EPSLANCZOS);
> EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
> EPSSetFromOptions(solver);
> EPSSolve(solver);
> 
> I want to make an estimation for larger size problems of the memory used by the program (at every step) because I would like to keep it under 16 GB per node. I've used the "memory usage" functions provided by PETSc, but something happens during the solver stage that I can't explain. This brings up two questions.
> 
> 1) In each step I put a call to four memory functions and between them I print the value of mem:
> 
> Did you call PetscMemorySetGetMaximumUsage() first?
> 
> We are computing https://en.wikipedia.org/wiki/Resident_set_size however we can. Usually with getrusage().
> From this (https://www.binarytides.com/linux-command-check-memory-usage/), it looks like top also reports
> paged out memory.
> 
>    Matt
>  
> mem = 0;
> PetscMallocGetCurrentUsage(&mem);
> PetscMallocGetMaximumUsage(&mem);
> PetscMemoryGetCurrentUsage(&mem);
> PetscMemoryGetMaximumUsage(&mem);
> 
> I've read some other question in the mailing list regarding the same issue but I can't fully understand this. What is the difference between all of them? What information are they actually giving me? (I know this is only a "per process" output). I copy the output of two steps of the program as an example:
> 
> ==================== step N ====================
> MallocGetCurrent: 314513664.0 B
> MallocGetMaximum: 332723328.0 B
> MemoryGetCurrent: 539996160.0 B
> MemoryGetMaximum: 0.0 B
> ==================== step N+1 ====================
> MallocGetCurrent: 395902912.0 B
> MallocGetMaximum: 415178624.0 B
> MemoryGetCurrent: 623783936.0 B
> MemoryGetMaximum: 623775744.0 B
> 
> 2) I was using this information to make the calculation of the memory required per node to run my problem. Also, I'm able to login to the computing node while running and I can check the memory consumption (with *top*). The memory used that I see with top is more or less the same as the one reported by PETSc functions at the beginning. But during the inialization of the solver and during the solving, *top* reports a consumption two times bigger than the one the functions report. Is it possible to know from where this extra memory consumption comes from? What things does SLEPc allocate that need that much memory? I've been trying to do the math but I think there are things I'm missing. I thought that part of it comes from the "BV" that the option -eps_view reports:
> 
> BV Object: 2048 MPI processes
>   type: svec
>   17 columns of global length 2333606220
>   vector orthogonalization method: modified Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
> 
> But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less of the "extra" memory.
> 
> Ale
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From phil.tooley at sheffield.ac.uk  Thu Oct  4 07:56:09 2018
From: phil.tooley at sheffield.ac.uk (Phil Tooley)
Date: Thu, 4 Oct 2018 13:56:09 +0100
Subject: [petsc-users] Better understanding VecScatter
Message-ID: <95861ed3-7c9f-820c-8a54-9e84923be54c@sheffield.ac.uk>

Hi all,

I am trying to understand how to create a custom scatter from petsc
ordering in a local vector to natural ordering in a global vector.

I have a 3D DMDA and local vector containing field data and am
calculating the x y and z gradients into their own local vectors.? I
then need to scatter these gradients to different parts of a single
vector so they are arranged [x_1 .. x_n y_1 .. y_n z_1 .. z_n] with
natural ordering.? (This is for use in a custom numerical scheme that I
have been asked to parallelise)

I can of course do this using multiple scatters but I need to perform
this operation regularly and would like to express it as a single
scatter operation if possible.

I know I need to get the relevant AO from my DMDA and use it to
construct appropriate ISs to build the scatter context but I am unsure
how exactly to go about this.

Can someone point me in the right direction please?

Many Thanks

Phil

-- 
Phil Tooley
Research Software Engineering
University of Sheffield


From mfadams at lbl.gov  Thu Oct  4 08:59:11 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 4 Oct 2018 09:59:11 -0400
Subject: [petsc-users] Better understanding VecScatter
In-Reply-To: <95861ed3-7c9f-820c-8a54-9e84923be54c@sheffield.ac.uk>
References: <95861ed3-7c9f-820c-8a54-9e84923be54c@sheffield.ac.uk>
Message-ID: <CADOhEh5ZJTRh-2pFs0f-=9ZrbnNGpC77gf7We2o1trL+cFv0iA@mail.gmail.com>

So if you have three processors you would want the first processor to have
all x gradients, etc.?

You could use ISCreateGeneral:
https://www.mcs.anl.gov/petsc/petsc-current/src/dm/examples/tutorials/ex6.c.html

You need to create an integer array with the global indices that you want
to bring to your new vector locally.

On Thu, Oct 4, 2018 at 8:56 AM Phil Tooley <phil.tooley at sheffield.ac.uk>
wrote:

> Hi all,
>
> I am trying to understand how to create a custom scatter from petsc
> ordering in a local vector to natural ordering in a global vector.
>
> I have a 3D DMDA and local vector containing field data and am
> calculating the x y and z gradients into their own local vectors.  I
> then need to scatter these gradients to different parts of a single
> vector so they are arranged [x_1 .. x_n y_1 .. y_n z_1 .. z_n] with
> natural ordering.  (This is for use in a custom numerical scheme that I
> have been asked to parallelise)
>
> I can of course do this using multiple scatters but I need to perform
> this operation regularly and would like to express it as a single
> scatter operation if possible.
>
> I know I need to get the relevant AO from my DMDA and use it to
> construct appropriate ISs to build the scatter context but I am unsure
> how exactly to go about this.
>
> Can someone point me in the right direction please?
>
> Many Thanks
>
> Phil
>
> --
> Phil Tooley
> Research Software Engineering
> University of Sheffield
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/769a42ca/attachment.html>

From knepley at gmail.com  Thu Oct  4 09:34:48 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Oct 2018 10:34:48 -0400
Subject: [petsc-users] Better understanding VecScatter
In-Reply-To: <95861ed3-7c9f-820c-8a54-9e84923be54c@sheffield.ac.uk>
References: <95861ed3-7c9f-820c-8a54-9e84923be54c@sheffield.ac.uk>
Message-ID: <CAMYG4GmFD5R-n0YukrWtYWVsgreci9rrUQavrYAyrJrrsE6dtA@mail.gmail.com>

On Thu, Oct 4, 2018 at 8:56 AM Phil Tooley <phil.tooley at sheffield.ac.uk>
wrote:

> Hi all,
>
> I am trying to understand how to create a custom scatter from petsc
> ordering in a local vector to natural ordering in a global vector.
>
> I have a 3D DMDA and local vector containing field data and am
> calculating the x y and z gradients into their own local vectors.  I
> then need to scatter these gradients to different parts of a single
> vector so they are arranged [x_1 .. x_n y_1 .. y_n z_1 .. z_n] with
> natural ordering.  (This is for use in a custom numerical scheme that I
> have been asked to parallelise)
>

I may not be understanding correctly, but it sounds like this can be done
simply
using two DMs. The first DM is your original one holding the field values.
The
second one would set dof = dim in your new DM but everything else the same.
You can use


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMDA/DMDACreateCompatibleDMDA.html

Then stick the gradients in a global vector for the new DM. Then do
GlobalToNatural
for that DM.

Is this what you want?

  Thanks,

     Matt


> I can of course do this using multiple scatters but I need to perform
> this operation regularly and would like to express it as a single
> scatter operation if possible.
>
> I know I need to get the relevant AO from my DMDA and use it to
> construct appropriate ISs to build the scatter context but I am unsure
> how exactly to go about this.
>
> Can someone point me in the right direction please?
>
> Many Thanks
>
> Phil
>
> --
> Phil Tooley
> Research Software Engineering
> University of Sheffield
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/0e1e1b1d/attachment.html>

From amfoggia at gmail.com  Thu Oct  4 12:54:05 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Thu, 4 Oct 2018 19:54:05 +0200
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
Message-ID: <CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>

Thank you both for your answers :)

Matt:
-Yes, sorry I forgot to tell you that, but I've also called
PetscMemorySetGetMaximumUsage() right after initializing SLEPc. Also I've
seen a strange behaviour: if I ran the same code in my computer and in the
cluster *without* the command line option -malloc_dump, in the cluster the
output of PetscMallocGetCurrentUsage and PetscMallocGetMaximumUsage is
always zero, but that doesn't happen in my computer.

- This is the output of the code for the solving part (after EPSCreate and
after EPSSolve), and I've compared it with the output of *top* during those
moments of peak memory consumption. *top* provides in one of the columns
the resident set size (RES) and the numbers are around 1 GB per process,
while, considering the numbers reported by the PETSc functions, the one
that is more similar to that is given by MemoryGetCurrentUsage and is only
800 MB in the solving stage. Maybe, we can consider that those numbers are
the same plus/minus something? Is it safe to say that MemoryGetCurrentUsage
is measuring the "ru_maxss" member of "rusage" (or something similar)? If
that's the case, what do the other functions report?

==================== SOLVER INIT ====================
MallocGetCurrent (init): 396096192.0 B
MallocGetMaximum (init): 415178624.0 B
MemoryGetCurrent (init): 624050176.0 B
MemoryGetMaximum (init): 623775744.0 B
==================== SOLVER ====================
MallocGetCurrent (solver): 560320256.0 B
MallocGetMaximum (solver): 560333440.0 B
MemoryGetCurrent (solver): 820961280.0 B
MemoryGetMaximum (solver): 623775744.0 B

Jose:
- By each step I mean each of the step of the the program in order to
diagonalize the matrix. For me, those are: creation of basis, preallocation
of matrix, setting values of matrix, initializing solver,
solving/diagonalizing and cleaning. I'm only diagonalizing once.

- Regarding the information provided by -log_view, it's confusing for me:
for example, it reports the creation of Vecs scattered across the various
stages that I've set up (with PetscLogStageRegister and
PetscLogStagePush/Pop), but almost all the deletions are presented in the
"Main Stage". What does that "Main Stage" consider? Why are more deletions
in there that creations? It's nor completely for me clear how things are
presented there.

- Thanks for the suggestion about the solver. Does "faster convergence" for
Krylov-Schur mean less memory and less computation, or just less
computation?

Ale


El jue., 4 oct. 2018 a las 13:12, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

> Regarding the SLEPc part:
> - What do you mean by "each step"? Are you calling EPSSolve() several
> times?
> - Yes, the BV object is generally what takes most of the memory. It is
> allocated at the beginning of EPSSolve(). Depending on the solver/options,
> other memory may be allocated as well.
> - You can also see the memory reported at the end of -log_view
> - I would suggest using the default solver Krylov-Schur - it will do
> Lanczos with implicit restart, which will give faster convergence than the
> EPSLANCZOS solver.
>
> Jose
>
>
> > El 4 oct 2018, a las 12:49, Matthew Knepley <knepley at gmail.com>
> escribi?:
> >
> > On Thu, Oct 4, 2018 at 4:43 AM Ale Foggia <amfoggia at gmail.com> wrote:
> > Hello all,
> >
> > I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of
> a matrix with the following characteristics:
> >
> > * type: real, Hermitian, sparse
> > * linear size: 2333606220
> > * distributed in 2048 processes (64 nodes, 32 procs per node)
> >
> > My code first preallocates the necessary memory with
> *MatMPIAIJSetPreallocation*, then fills it with the values and finally it
> calls the following functions to create the solver and diagonalize the
> matrix:
> >
> > EPSCreate(PETSC_COMM_WORLD, &solver);
> > EPSSetOperators(solver,matrix,NULL);
> > EPSSetProblemType(solver, EPS_HEP);
> > EPSSetType(solver, EPSLANCZOS);
> > EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
> > EPSSetFromOptions(solver);
> > EPSSolve(solver);
> >
> > I want to make an estimation for larger size problems of the memory used
> by the program (at every step) because I would like to keep it under 16 GB
> per node. I've used the "memory usage" functions provided by PETSc, but
> something happens during the solver stage that I can't explain. This brings
> up two questions.
> >
> > 1) In each step I put a call to four memory functions and between them I
> print the value of mem:
> >
> > Did you call PetscMemorySetGetMaximumUsage() first?
> >
> > We are computing https://en.wikipedia.org/wiki/Resident_set_size
> however we can. Usually with getrusage().
> > From this (https://www.binarytides.com/linux-command-check-memory-usage/),
> it looks like top also reports
> > paged out memory.
> >
> >    Matt
> >
> > mem = 0;
> > PetscMallocGetCurrentUsage(&mem);
> > PetscMallocGetMaximumUsage(&mem);
> > PetscMemoryGetCurrentUsage(&mem);
> > PetscMemoryGetMaximumUsage(&mem);
> >
> > I've read some other question in the mailing list regarding the same
> issue but I can't fully understand this. What is the difference between all
> of them? What information are they actually giving me? (I know this is only
> a "per process" output). I copy the output of two steps of the program as
> an example:
> >
> > ==================== step N ====================
> > MallocGetCurrent: 314513664.0 B
> > MallocGetMaximum: 332723328.0 B
> > MemoryGetCurrent: 539996160.0 B
> > MemoryGetMaximum: 0.0 B
> > ==================== step N+1 ====================
> > MallocGetCurrent: 395902912.0 B
> > MallocGetMaximum: 415178624.0 B
> > MemoryGetCurrent: 623783936.0 B
> > MemoryGetMaximum: 623775744.0 B
> >
> > 2) I was using this information to make the calculation of the memory
> required per node to run my problem. Also, I'm able to login to the
> computing node while running and I can check the memory consumption (with
> *top*). The memory used that I see with top is more or less the same as the
> one reported by PETSc functions at the beginning. But during the
> inialization of the solver and during the solving, *top* reports a
> consumption two times bigger than the one the functions report. Is it
> possible to know from where this extra memory consumption comes from? What
> things does SLEPc allocate that need that much memory? I've been trying to
> do the math but I think there are things I'm missing. I thought that part
> of it comes from the "BV" that the option -eps_view reports:
> >
> > BV Object: 2048 MPI processes
> >   type: svec
> >   17 columns of global length 2333606220
> >   vector orthogonalization method: modified Gram-Schmidt
> >   orthogonalization refinement: if needed (eta: 0.7071)
> >   block orthogonalization method: GS
> >   doing matmult as a single matrix-matrix product
> >
> > But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less
> of the "extra" memory.
> >
> > Ale
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/0801effc/attachment.html>

From knepley at gmail.com  Thu Oct  4 13:57:52 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 4 Oct 2018 14:57:52 -0400
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
	<CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
Message-ID: <CAMYG4Gm=iOC3rD=zfhtFaB6S+5oiuqP60-9nwdHAHS7b3S7CfA@mail.gmail.com>

On Thu, Oct 4, 2018 at 1:54 PM Ale Foggia <amfoggia at gmail.com> wrote:

> Thank you both for your answers :)
>
> Matt:
> -Yes, sorry I forgot to tell you that, but I've also called
> PetscMemorySetGetMaximumUsage() right after initializing SLEPc. Also I've
> seen a strange behaviour: if I ran the same code in my computer and in the
> cluster *without* the command line option -malloc_dump, in the cluster the
> output of PetscMallocGetCurrentUsage and PetscMallocGetMaximumUsage is
> always zero, but that doesn't happen in my computer.
>
> - This is the output of the code for the solving part (after EPSCreate and
> after EPSSolve), and I've compared it with the output of *top* during those
> moments of peak memory consumption. *top* provides in one of the columns
> the resident set size (RES) and the numbers are around 1 GB per process,
> while, considering the numbers reported by the PETSc functions, the one
> that is more similar to that is given by MemoryGetCurrentUsage and is only
> 800 MB in the solving stage. Maybe, we can consider that those numbers are
> the same plus/minus something? Is it safe to say that MemoryGetCurrentUsage
> is measuring the "ru_maxss" member of "rusage" (or something similar)? If
> that's the case, what do the other functions report?
>

This is a perennial problem, since RSS is no guarantee of stuff that is
actually being used, but only was allocated at some point. The best tool I
have seen for this is Massif. I really recommend it:

  http://valgrind.org/docs/manual/ms-manual.html

  Thanks,

     Matt


> ==================== SOLVER INIT ====================
> MallocGetCurrent (init): 396096192.0 B
> MallocGetMaximum (init): 415178624.0 B
> MemoryGetCurrent (init): 624050176.0 B
> MemoryGetMaximum (init): 623775744.0 B
> ==================== SOLVER ====================
> MallocGetCurrent (solver): 560320256.0 B
> MallocGetMaximum (solver): 560333440.0 B
> MemoryGetCurrent (solver): 820961280.0 B
> MemoryGetMaximum (solver): 623775744.0 B
>
> Jose:
> - By each step I mean each of the step of the the program in order to
> diagonalize the matrix. For me, those are: creation of basis, preallocation
> of matrix, setting values of matrix, initializing solver,
> solving/diagonalizing and cleaning. I'm only diagonalizing once.
>
> - Regarding the information provided by -log_view, it's confusing for me:
> for example, it reports the creation of Vecs scattered across the various
> stages that I've set up (with PetscLogStageRegister and
> PetscLogStagePush/Pop), but almost all the deletions are presented in the
> "Main Stage". What does that "Main Stage" consider? Why are more deletions
> in there that creations? It's nor completely for me clear how things are
> presented there.
>
> - Thanks for the suggestion about the solver. Does "faster convergence"
> for Krylov-Schur mean less memory and less computation, or just less
> computation?
>
> Ale
>
>
> El jue., 4 oct. 2018 a las 13:12, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
>
>> Regarding the SLEPc part:
>> - What do you mean by "each step"? Are you calling EPSSolve() several
>> times?
>> - Yes, the BV object is generally what takes most of the memory. It is
>> allocated at the beginning of EPSSolve(). Depending on the solver/options,
>> other memory may be allocated as well.
>> - You can also see the memory reported at the end of -log_view
>> - I would suggest using the default solver Krylov-Schur - it will do
>> Lanczos with implicit restart, which will give faster convergence than the
>> EPSLANCZOS solver.
>>
>> Jose
>>
>>
>> > El 4 oct 2018, a las 12:49, Matthew Knepley <knepley at gmail.com>
>> escribi?:
>> >
>> > On Thu, Oct 4, 2018 at 4:43 AM Ale Foggia <amfoggia at gmail.com> wrote:
>> > Hello all,
>> >
>> > I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of
>> a matrix with the following characteristics:
>> >
>> > * type: real, Hermitian, sparse
>> > * linear size: 2333606220
>> > * distributed in 2048 processes (64 nodes, 32 procs per node)
>> >
>> > My code first preallocates the necessary memory with
>> *MatMPIAIJSetPreallocation*, then fills it with the values and finally it
>> calls the following functions to create the solver and diagonalize the
>> matrix:
>> >
>> > EPSCreate(PETSC_COMM_WORLD, &solver);
>> > EPSSetOperators(solver,matrix,NULL);
>> > EPSSetProblemType(solver, EPS_HEP);
>> > EPSSetType(solver, EPSLANCZOS);
>> > EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
>> > EPSSetFromOptions(solver);
>> > EPSSolve(solver);
>> >
>> > I want to make an estimation for larger size problems of the memory
>> used by the program (at every step) because I would like to keep it under
>> 16 GB per node. I've used the "memory usage" functions provided by PETSc,
>> but something happens during the solver stage that I can't explain. This
>> brings up two questions.
>> >
>> > 1) In each step I put a call to four memory functions and between them
>> I print the value of mem:
>> >
>> > Did you call PetscMemorySetGetMaximumUsage() first?
>> >
>> > We are computing https://en.wikipedia.org/wiki/Resident_set_size
>> however we can. Usually with getrusage().
>> > From this (
>> https://www.binarytides.com/linux-command-check-memory-usage/), it looks
>> like top also reports
>> > paged out memory.
>> >
>> >    Matt
>> >
>> > mem = 0;
>> > PetscMallocGetCurrentUsage(&mem);
>> > PetscMallocGetMaximumUsage(&mem);
>> > PetscMemoryGetCurrentUsage(&mem);
>> > PetscMemoryGetMaximumUsage(&mem);
>> >
>> > I've read some other question in the mailing list regarding the same
>> issue but I can't fully understand this. What is the difference between all
>> of them? What information are they actually giving me? (I know this is only
>> a "per process" output). I copy the output of two steps of the program as
>> an example:
>> >
>> > ==================== step N ====================
>> > MallocGetCurrent: 314513664.0 B
>> > MallocGetMaximum: 332723328.0 B
>> > MemoryGetCurrent: 539996160.0 B
>> > MemoryGetMaximum: 0.0 B
>> > ==================== step N+1 ====================
>> > MallocGetCurrent: 395902912.0 B
>> > MallocGetMaximum: 415178624.0 B
>> > MemoryGetCurrent: 623783936.0 B
>> > MemoryGetMaximum: 623775744.0 B
>> >
>> > 2) I was using this information to make the calculation of the memory
>> required per node to run my problem. Also, I'm able to login to the
>> computing node while running and I can check the memory consumption (with
>> *top*). The memory used that I see with top is more or less the same as the
>> one reported by PETSc functions at the beginning. But during the
>> inialization of the solver and during the solving, *top* reports a
>> consumption two times bigger than the one the functions report. Is it
>> possible to know from where this extra memory consumption comes from? What
>> things does SLEPc allocate that need that much memory? I've been trying to
>> do the math but I think there are things I'm missing. I thought that part
>> of it comes from the "BV" that the option -eps_view reports:
>> >
>> > BV Object: 2048 MPI processes
>> >   type: svec
>> >   17 columns of global length 2333606220
>> >   vector orthogonalization method: modified Gram-Schmidt
>> >   orthogonalization refinement: if needed (eta: 0.7071)
>> >   block orthogonalization method: GS
>> >   doing matmult as a single matrix-matrix product
>> >
>> > But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less
>> of the "extra" memory.
>> >
>> > Ale
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/0da0c18c/attachment-0001.html>

From jed at jedbrown.org  Thu Oct  4 15:59:32 2018
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 04 Oct 2018 14:59:32 -0600
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CAMYG4Gm=iOC3rD=zfhtFaB6S+5oiuqP60-9nwdHAHS7b3S7CfA@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
	<CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
	<CAMYG4Gm=iOC3rD=zfhtFaB6S+5oiuqP60-9nwdHAHS7b3S7CfA@mail.gmail.com>
Message-ID: <87bm894edn.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, Oct 4, 2018 at 1:54 PM Ale Foggia <amfoggia at gmail.com> wrote:
>
>> Thank you both for your answers :)
>>
>> Matt:
>> -Yes, sorry I forgot to tell you that, but I've also called
>> PetscMemorySetGetMaximumUsage() right after initializing SLEPc. Also I've
>> seen a strange behaviour: if I ran the same code in my computer and in the
>> cluster *without* the command line option -malloc_dump, in the cluster the
>> output of PetscMallocGetCurrentUsage and PetscMallocGetMaximumUsage is
>> always zero, but that doesn't happen in my computer.
>>
>> - This is the output of the code for the solving part (after EPSCreate and
>> after EPSSolve), and I've compared it with the output of *top* during those
>> moments of peak memory consumption. *top* provides in one of the columns
>> the resident set size (RES) and the numbers are around 1 GB per process,
>> while, considering the numbers reported by the PETSc functions, the one
>> that is more similar to that is given by MemoryGetCurrentUsage and is only
>> 800 MB in the solving stage. Maybe, we can consider that those numbers are
>> the same plus/minus something? Is it safe to say that MemoryGetCurrentUsage
>> is measuring the "ru_maxss" member of "rusage" (or something similar)? If
>> that's the case, what do the other functions report?
>>
>
> This is a perennial problem, since RSS is no guarantee of stuff that is
> actually being used, but only was allocated at some point. 

No, allocation alone does not make it resident on most operating
systems.  If you run top, you see a column VIRT (memory that was
allocated/mmap'd) and RES (actually resident in physical memory).

PetscMemoryGetCurrentUsage tries to get resident memory usage via
/proc/{PID}/statm or getrusage() ru_maxrss.

PetscMallocGetMaximumUsage just says how much memory has been allocated
using PETSc's tracing malloc (off by default with an optimized build,
but you can turn on by running with -malloc or related diagnostic
options).

> The best tool I have seen for this is Massif. I really recommend it:
>
>   http://valgrind.org/docs/manual/ms-manual.html
>
>   Thanks,
>
>      Matt
>
>
>> ==================== SOLVER INIT ====================
>> MallocGetCurrent (init): 396096192.0 B
>> MallocGetMaximum (init): 415178624.0 B
>> MemoryGetCurrent (init): 624050176.0 B
>> MemoryGetMaximum (init): 623775744.0 B
>> ==================== SOLVER ====================
>> MallocGetCurrent (solver): 560320256.0 B
>> MallocGetMaximum (solver): 560333440.0 B
>> MemoryGetCurrent (solver): 820961280.0 B
>> MemoryGetMaximum (solver): 623775744.0 B
>>
>> Jose:
>> - By each step I mean each of the step of the the program in order to
>> diagonalize the matrix. For me, those are: creation of basis, preallocation
>> of matrix, setting values of matrix, initializing solver,
>> solving/diagonalizing and cleaning. I'm only diagonalizing once.
>>
>> - Regarding the information provided by -log_view, it's confusing for me:
>> for example, it reports the creation of Vecs scattered across the various
>> stages that I've set up (with PetscLogStageRegister and
>> PetscLogStagePush/Pop), but almost all the deletions are presented in the
>> "Main Stage". What does that "Main Stage" consider? Why are more deletions
>> in there that creations? It's nor completely for me clear how things are
>> presented there.
>>
>> - Thanks for the suggestion about the solver. Does "faster convergence"
>> for Krylov-Schur mean less memory and less computation, or just less
>> computation?
>>
>> Ale
>>
>>
>> El jue., 4 oct. 2018 a las 13:12, Jose E. Roman (<jroman at dsic.upv.es>)
>> escribi?:
>>
>>> Regarding the SLEPc part:
>>> - What do you mean by "each step"? Are you calling EPSSolve() several
>>> times?
>>> - Yes, the BV object is generally what takes most of the memory. It is
>>> allocated at the beginning of EPSSolve(). Depending on the solver/options,
>>> other memory may be allocated as well.
>>> - You can also see the memory reported at the end of -log_view
>>> - I would suggest using the default solver Krylov-Schur - it will do
>>> Lanczos with implicit restart, which will give faster convergence than the
>>> EPSLANCZOS solver.
>>>
>>> Jose
>>>
>>>
>>> > El 4 oct 2018, a las 12:49, Matthew Knepley <knepley at gmail.com>
>>> escribi?:
>>> >
>>> > On Thu, Oct 4, 2018 at 4:43 AM Ale Foggia <amfoggia at gmail.com> wrote:
>>> > Hello all,
>>> >
>>> > I'm using SLEPc 3.9.2 (and PETSc 3.9.3) to get the EPS_SMALLEST_REAL of
>>> a matrix with the following characteristics:
>>> >
>>> > * type: real, Hermitian, sparse
>>> > * linear size: 2333606220
>>> > * distributed in 2048 processes (64 nodes, 32 procs per node)
>>> >
>>> > My code first preallocates the necessary memory with
>>> *MatMPIAIJSetPreallocation*, then fills it with the values and finally it
>>> calls the following functions to create the solver and diagonalize the
>>> matrix:
>>> >
>>> > EPSCreate(PETSC_COMM_WORLD, &solver);
>>> > EPSSetOperators(solver,matrix,NULL);
>>> > EPSSetProblemType(solver, EPS_HEP);
>>> > EPSSetType(solver, EPSLANCZOS);
>>> > EPSSetWhichEigenpairs(solver, EPS_SMALLEST_REAL);
>>> > EPSSetFromOptions(solver);
>>> > EPSSolve(solver);
>>> >
>>> > I want to make an estimation for larger size problems of the memory
>>> used by the program (at every step) because I would like to keep it under
>>> 16 GB per node. I've used the "memory usage" functions provided by PETSc,
>>> but something happens during the solver stage that I can't explain. This
>>> brings up two questions.
>>> >
>>> > 1) In each step I put a call to four memory functions and between them
>>> I print the value of mem:
>>> >
>>> > Did you call PetscMemorySetGetMaximumUsage() first?
>>> >
>>> > We are computing https://en.wikipedia.org/wiki/Resident_set_size
>>> however we can. Usually with getrusage().
>>> > From this (
>>> https://www.binarytides.com/linux-command-check-memory-usage/), it looks
>>> like top also reports
>>> > paged out memory.
>>> >
>>> >    Matt
>>> >
>>> > mem = 0;
>>> > PetscMallocGetCurrentUsage(&mem);
>>> > PetscMallocGetMaximumUsage(&mem);
>>> > PetscMemoryGetCurrentUsage(&mem);
>>> > PetscMemoryGetMaximumUsage(&mem);
>>> >
>>> > I've read some other question in the mailing list regarding the same
>>> issue but I can't fully understand this. What is the difference between all
>>> of them? What information are they actually giving me? (I know this is only
>>> a "per process" output). I copy the output of two steps of the program as
>>> an example:
>>> >
>>> > ==================== step N ====================
>>> > MallocGetCurrent: 314513664.0 B
>>> > MallocGetMaximum: 332723328.0 B
>>> > MemoryGetCurrent: 539996160.0 B
>>> > MemoryGetMaximum: 0.0 B
>>> > ==================== step N+1 ====================
>>> > MallocGetCurrent: 395902912.0 B
>>> > MallocGetMaximum: 415178624.0 B
>>> > MemoryGetCurrent: 623783936.0 B
>>> > MemoryGetMaximum: 623775744.0 B
>>> >
>>> > 2) I was using this information to make the calculation of the memory
>>> required per node to run my problem. Also, I'm able to login to the
>>> computing node while running and I can check the memory consumption (with
>>> *top*). The memory used that I see with top is more or less the same as the
>>> one reported by PETSc functions at the beginning. But during the
>>> inialization of the solver and during the solving, *top* reports a
>>> consumption two times bigger than the one the functions report. Is it
>>> possible to know from where this extra memory consumption comes from? What
>>> things does SLEPc allocate that need that much memory? I've been trying to
>>> do the math but I think there are things I'm missing. I thought that part
>>> of it comes from the "BV" that the option -eps_view reports:
>>> >
>>> > BV Object: 2048 MPI processes
>>> >   type: svec
>>> >   17 columns of global length 2333606220
>>> >   vector orthogonalization method: modified Gram-Schmidt
>>> >   orthogonalization refinement: if needed (eta: 0.7071)
>>> >   block orthogonalization method: GS
>>> >   doing matmult as a single matrix-matrix product
>>> >
>>> > But "17 * 2333606220 * 8 Bytes / #nodes" only explains on third or less
>>> of the "extra" memory.
>>> >
>>> > Ale
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> > https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>

From heeho.park at gmail.com  Thu Oct  4 17:44:21 2018
From: heeho.park at gmail.com (HeeHo Park)
Date: Thu, 4 Oct 2018 17:44:21 -0500
Subject: [petsc-users] subprocess (block) tolerance.
Message-ID: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>

Hi, I'm running PFLOTRAN and in PFLOTRAN, we have flow_ and flow_sub_
processes. I was wondering what the red underlined values meant (each block
tolerance?) and how to change them (would it affect convergence?). Blue
marked bold values are changed from the default values for linear solvers.

FLOW Linear Solver
                       solver: bcgs
               preconditioner: asm
                         *atol: 1.000000E-10*
*                         rtol: 1.000000E-10*
                         dtol: 1.000000E+04
            maximum iteration: 10000
KSP Object: (flow_) 8 MPI processes
  type: bcgs
  maximum iterations=10000, initial guess is zero
  tolerances: * relative=1e-10, absolute=1e-10*, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: (flow_) 8 MPI processes
  type: asm
    Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1
    Additive Schwarz: restriction/interpolation type - RESTRICT
    [0] number of local blocks = 1
    [1] number of local blocks = 1
    [2] number of local blocks = 1
    [3] number of local blocks = 1
    [4] number of local blocks = 1
    [5] number of local blocks = 1
    [6] number of local blocks = 1
    [7] number of local blocks = 1
    Local solve info for each block is in the following KSP and PC objects:
    - - - - - - - - - - - - - - - - - -
    [0] local block number 0, size = 1389
    KSP Object: (flow_sub_) 1 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
>>>      tolerances:  *relative=1e-05, absolute=1e-50*, divergence=10000.
      left preconditioning
      using DEFAULT norm type for convergence test
    PC Object: (flow_sub_) 1 MPI processes
      type: ilu
      PC has not been set up so information may be incomplete
        out-of-place factorization
        0 levels of fill
        tolerance for zero pivot 2.22045e-14
        using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
        matrix ordering: natural
      linear system matrix = precond matrix:
      Mat Object: (flow_) 1 MPI processes
        type: seqbaij
        rows=1389, cols=1389, bs=3
        total: nonzeros=20025, allocated nonzeros=20025
        total number of mallocs used during MatSetValues calls =0
            block size is 3
    - - - - - - - - - - - - - - - - - -

-- 
HeeHo Daniel Park
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/6d81196a/attachment.html>

From jed at jedbrown.org  Thu Oct  4 17:47:19 2018
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 04 Oct 2018 16:47:19 -0600
Subject: [petsc-users] subprocess (block) tolerance.
In-Reply-To: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
References: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
Message-ID: <87r2h52utk.fsf@jedbrown.org>

The subdomain KSP (flow_sub_) has type "preonly" so it always does
exactly one iteration.  If you were to use an iterative subdomain solver
(e.g., -flow_sub_ksp_type gmres) then those tolerances would be used.

HeeHo Park <heeho.park at gmail.com> writes:

> Hi, I'm running PFLOTRAN and in PFLOTRAN, we have flow_ and flow_sub_
> processes. I was wondering what the red underlined values meant (each block
> tolerance?) and how to change them (would it affect convergence?). Blue
> marked bold values are changed from the default values for linear solvers.
>
> FLOW Linear Solver
>                        solver: bcgs
>                preconditioner: asm
>                          *atol: 1.000000E-10*
> *                         rtol: 1.000000E-10*
>                          dtol: 1.000000E+04
>             maximum iteration: 10000
> KSP Object: (flow_) 8 MPI processes
>   type: bcgs
>   maximum iterations=10000, initial guess is zero
>   tolerances: * relative=1e-10, absolute=1e-10*, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: (flow_) 8 MPI processes
>   type: asm
>     Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1
>     Additive Schwarz: restriction/interpolation type - RESTRICT
>     [0] number of local blocks = 1
>     [1] number of local blocks = 1
>     [2] number of local blocks = 1
>     [3] number of local blocks = 1
>     [4] number of local blocks = 1
>     [5] number of local blocks = 1
>     [6] number of local blocks = 1
>     [7] number of local blocks = 1
>     Local solve info for each block is in the following KSP and PC objects:
>     - - - - - - - - - - - - - - - - - -
>     [0] local block number 0, size = 1389
>     KSP Object: (flow_sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>>>>      tolerances:  *relative=1e-05, absolute=1e-50*, divergence=10000.
>       left preconditioning
>       using DEFAULT norm type for convergence test
>     PC Object: (flow_sub_) 1 MPI processes
>       type: ilu
>       PC has not been set up so information may be incomplete
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>         matrix ordering: natural
>       linear system matrix = precond matrix:
>       Mat Object: (flow_) 1 MPI processes
>         type: seqbaij
>         rows=1389, cols=1389, bs=3
>         total: nonzeros=20025, allocated nonzeros=20025
>         total number of mallocs used during MatSetValues calls =0
>             block size is 3
>     - - - - - - - - - - - - - - - - - -
>
> -- 
> HeeHo Daniel Park

From bsmith at mcs.anl.gov  Thu Oct  4 17:53:04 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 4 Oct 2018 22:53:04 +0000
Subject: [petsc-users] subprocess (block) tolerance.
In-Reply-To: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
References: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
Message-ID: <6F04DAF7-562F-4777-A3E1-70EA4A632929@anl.gov>


  Since 

KSP Object: (flow_sub_) 1 MPI processes
        type: preonly

this means only a single iteration of the inner solver is used so   the numbers in red are not used. 

You could do something like -flow_ksp_type fgmres -flow_sub_ksp_type gmres -flow_sub_ksp_rtol 1.e-2 but it wouldn't help maters. Likely the current values are the best.

   Barry


> On Oct 4, 2018, at 5:44 PM, HeeHo Park <heeho.park at gmail.com> wrote:
> 
> Hi, I'm running PFLOTRAN and in PFLOTRAN, we have flow_ and flow_sub_ processes. I was wondering what the red underlined values meant (each block tolerance?) and how to change them (would it affect convergence?). Blue marked bold values are changed from the default values for linear solvers.
> 
> FLOW Linear Solver
>                        solver: bcgs
>                preconditioner: asm
>                          atol: 1.000000E-10
>                          rtol: 1.000000E-10
>                          dtol: 1.000000E+04
>             maximum iteration: 10000
> KSP Object: (flow_) 8 MPI processes
>   type: bcgs
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-10, absolute=1e-10, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: (flow_) 8 MPI processes
>   type: asm
>     Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1
>     Additive Schwarz: restriction/interpolation type - RESTRICT
>     [0] number of local blocks = 1
>     [1] number of local blocks = 1
>     [2] number of local blocks = 1
>     [3] number of local blocks = 1
>     [4] number of local blocks = 1
>     [5] number of local blocks = 1
>     [6] number of local blocks = 1
>     [7] number of local blocks = 1
>     Local solve info for each block is in the following KSP and PC objects:
>     - - - - - - - - - - - - - - - - - -
>     [0] local block number 0, size = 1389
>     KSP Object: (flow_sub_) 1 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
> >>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using DEFAULT norm type for convergence test
>     PC Object: (flow_sub_) 1 MPI processes
>       type: ilu
>       PC has not been set up so information may be incomplete
>         out-of-place factorization
>         0 levels of fill
>         tolerance for zero pivot 2.22045e-14
>         using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>         matrix ordering: natural
>       linear system matrix = precond matrix:
>       Mat Object: (flow_) 1 MPI processes
>         type: seqbaij
>         rows=1389, cols=1389, bs=3
>         total: nonzeros=20025, allocated nonzeros=20025
>         total number of mallocs used during MatSetValues calls =0
>             block size is 3
>     - - - - - - - - - - - - - - - - - -
> 
> -- 
> HeeHo Daniel Park


From heeho.park at gmail.com  Thu Oct  4 18:09:51 2018
From: heeho.park at gmail.com (HeeHo Park)
Date: Thu, 4 Oct 2018 18:09:51 -0500
Subject: [petsc-users] subprocess (block) tolerance.
In-Reply-To: <6F04DAF7-562F-4777-A3E1-70EA4A632929@anl.gov>
References: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
	<6F04DAF7-562F-4777-A3E1-70EA4A632929@anl.gov>
Message-ID: <CAO2fGGqtUFTfZmvfU9K3eJkLNM1y6N_pXkhLddB=n=5nDMa9LQ@mail.gmail.com>

Barry and Jed,

Thank you for your answers. I think I need to learn more about domain
decomposition as I am a bit confused.
Is it true that we are using BiCGstab here to solve the system of
equations, using Additive Schwartz as a domain decomposition
preconditioner, and that precondition matrix for each block is
preconditioned by ilu[0]?

Thanks,


On Thu, Oct 4, 2018 at 5:53 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:

>
>   Since
>
> KSP Object: (flow_sub_) 1 MPI processes
>         type: preonly
>
> this means only a single iteration of the inner solver is used so   the
> numbers in red are not used.
>
> You could do something like -flow_ksp_type fgmres -flow_sub_ksp_type gmres
> -flow_sub_ksp_rtol 1.e-2 but it wouldn't help maters. Likely the current
> values are the best.
>
>    Barry
>
>
> > On Oct 4, 2018, at 5:44 PM, HeeHo Park <heeho.park at gmail.com> wrote:
> >
> > Hi, I'm running PFLOTRAN and in PFLOTRAN, we have flow_ and flow_sub_
> processes. I was wondering what the red underlined values meant (each block
> tolerance?) and how to change them (would it affect convergence?). Blue
> marked bold values are changed from the default values for linear solvers.
> >
> > FLOW Linear Solver
> >                        solver: bcgs
> >                preconditioner: asm
> >                          atol: 1.000000E-10
> >                          rtol: 1.000000E-10
> >                          dtol: 1.000000E+04
> >             maximum iteration: 10000
> > KSP Object: (flow_) 8 MPI processes
> >   type: bcgs
> >   maximum iterations=10000, initial guess is zero
> >   tolerances:  relative=1e-10, absolute=1e-10, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: (flow_) 8 MPI processes
> >   type: asm
> >     Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1
> >     Additive Schwarz: restriction/interpolation type - RESTRICT
> >     [0] number of local blocks = 1
> >     [1] number of local blocks = 1
> >     [2] number of local blocks = 1
> >     [3] number of local blocks = 1
> >     [4] number of local blocks = 1
> >     [5] number of local blocks = 1
> >     [6] number of local blocks = 1
> >     [7] number of local blocks = 1
> >     Local solve info for each block is in the following KSP and PC
> objects:
> >     - - - - - - - - - - - - - - - - - -
> >     [0] local block number 0, size = 1389
> >     KSP Object: (flow_sub_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> > >>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using DEFAULT norm type for convergence test
> >     PC Object: (flow_sub_) 1 MPI processes
> >       type: ilu
> >       PC has not been set up so information may be incomplete
> >         out-of-place factorization
> >         0 levels of fill
> >         tolerance for zero pivot 2.22045e-14
> >         using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
> >         matrix ordering: natural
> >       linear system matrix = precond matrix:
> >       Mat Object: (flow_) 1 MPI processes
> >         type: seqbaij
> >         rows=1389, cols=1389, bs=3
> >         total: nonzeros=20025, allocated nonzeros=20025
> >         total number of mallocs used during MatSetValues calls =0
> >             block size is 3
> >     - - - - - - - - - - - - - - - - - -
> >
> > --
> > HeeHo Daniel Park
>
>

-- 
HeeHo Daniel Park
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181004/1b0884dd/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Oct  4 18:26:39 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 4 Oct 2018 23:26:39 +0000
Subject: [petsc-users] subprocess (block) tolerance.
In-Reply-To: <CAO2fGGqtUFTfZmvfU9K3eJkLNM1y6N_pXkhLddB=n=5nDMa9LQ@mail.gmail.com>
References: <CAO2fGGqaEO+Q4PiGJtss-QCGgenoGZo7qjYJSz4q1XO4a4JYzA@mail.gmail.com>
	<6F04DAF7-562F-4777-A3E1-70EA4A632929@anl.gov>
	<CAO2fGGqtUFTfZmvfU9K3eJkLNM1y6N_pXkhLddB=n=5nDMa9LQ@mail.gmail.com>
Message-ID: <84FAD7D5-649E-4EFE-ADA6-B58374CE2F0B@anl.gov>



> On Oct 4, 2018, at 6:09 PM, HeeHo Park <heeho.park at gmail.com> wrote:
> 
> Barry and Jed,
> 
> Thank you for your answers. I think I need to learn more about domain decomposition as I am a bit confused. 
> Is it true that we are using BiCGstab here to solve the system of equations, using Additive Schwartz as a domain decomposition preconditioner, and that precondition matrix for each block is preconditioned by ilu[0]? 

   You can say it that way. 
> 
> Thanks,
> 
> 
> On Thu, Oct 4, 2018 at 5:53 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>   Since 
> 
> KSP Object: (flow_sub_) 1 MPI processes
>         type: preonly
> 
> this means only a single iteration of the inner solver is used so   the numbers in red are not used. 
> 
> You could do something like -flow_ksp_type fgmres -flow_sub_ksp_type gmres -flow_sub_ksp_rtol 1.e-2 but it wouldn't help maters. Likely the current values are the best.
> 
>    Barry
> 
> 
> > On Oct 4, 2018, at 5:44 PM, HeeHo Park <heeho.park at gmail.com> wrote:
> > 
> > Hi, I'm running PFLOTRAN and in PFLOTRAN, we have flow_ and flow_sub_ processes. I was wondering what the red underlined values meant (each block tolerance?) and how to change them (would it affect convergence?). Blue marked bold values are changed from the default values for linear solvers.
> > 
> > FLOW Linear Solver
> >                        solver: bcgs
> >                preconditioner: asm
> >                          atol: 1.000000E-10
> >                          rtol: 1.000000E-10
> >                          dtol: 1.000000E+04
> >             maximum iteration: 10000
> > KSP Object: (flow_) 8 MPI processes
> >   type: bcgs
> >   maximum iterations=10000, initial guess is zero
> >   tolerances:  relative=1e-10, absolute=1e-10, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: (flow_) 8 MPI processes
> >   type: asm
> >     Additive Schwarz: total subdomain blocks = 8, amount of overlap = 1
> >     Additive Schwarz: restriction/interpolation type - RESTRICT
> >     [0] number of local blocks = 1
> >     [1] number of local blocks = 1
> >     [2] number of local blocks = 1
> >     [3] number of local blocks = 1
> >     [4] number of local blocks = 1
> >     [5] number of local blocks = 1
> >     [6] number of local blocks = 1
> >     [7] number of local blocks = 1
> >     Local solve info for each block is in the following KSP and PC objects:
> >     - - - - - - - - - - - - - - - - - -
> >     [0] local block number 0, size = 1389
> >     KSP Object: (flow_sub_) 1 MPI processes
> >       type: preonly
> >       maximum iterations=10000, initial guess is zero
> > >>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >       left preconditioning
> >       using DEFAULT norm type for convergence test
> >     PC Object: (flow_sub_) 1 MPI processes
> >       type: ilu
> >       PC has not been set up so information may be incomplete
> >         out-of-place factorization
> >         0 levels of fill
> >         tolerance for zero pivot 2.22045e-14
> >         using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
> >         matrix ordering: natural
> >       linear system matrix = precond matrix:
> >       Mat Object: (flow_) 1 MPI processes
> >         type: seqbaij
> >         rows=1389, cols=1389, bs=3
> >         total: nonzeros=20025, allocated nonzeros=20025
> >         total number of mallocs used during MatSetValues calls =0
> >             block size is 3
> >     - - - - - - - - - - - - - - - - - -
> > 
> > -- 
> > HeeHo Daniel Park
> 
> 
> 
> -- 
> HeeHo Daniel Park


From jroman at dsic.upv.es  Fri Oct  5 02:38:28 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 5 Oct 2018 09:38:28 +0200
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
	<CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
Message-ID: <C19248E9-CD9D-4A28-9203-D7078F1FFCF0@dsic.upv.es>



> El 4 oct 2018, a las 19:54, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> Jose: 
> - By each step I mean each of the step of the the program in order to diagonalize the matrix. For me, those are: creation of basis, preallocation of matrix, setting values of matrix, initializing solver, solving/diagonalizing and cleaning. I'm only diagonalizing once. 
> 
> - Regarding the information provided by -log_view, it's confusing for me: for example, it reports the creation of Vecs scattered across the various stages that I've set up (with PetscLogStageRegister and PetscLogStagePush/Pop), but almost all the deletions are presented in the "Main Stage". What does that "Main Stage" consider? Why are more deletions in there that creations? It's nor completely for me clear how things are presented there.

I guess deletions should match creations. Seems to be related to using stages. Maybe someone from PETSc can give an explanation, but looking at a PETSc example that uses stages (e.g. dm/impls/plex/examples/tests/ex1.c) it seems that some destructions are counted in the main stage while the creation is counted in another stage - I guess it depends on the points where the stages are defined. The sum of creations matches the sum of destroys.

> 
> - Thanks for the suggestion about the solver. Does "faster convergence" for Krylov-Schur mean less memory and less computation, or just less computation? 
> 

It should be about the same memory with less iterations.

Jose


From zakaryah at gmail.com  Fri Oct  5 15:14:35 2018
From: zakaryah at gmail.com (zakaryah)
Date: Fri, 5 Oct 2018 16:14:35 -0400
Subject: [petsc-users] Fwd: Implementing a homotopy solver
In-Reply-To: <CAMYG4Gk3k9=fVPesu1xOV79Z-jEgrsZDDXf65U-00uEVengzaQ@mail.gmail.com>
References: <CABm8tB45FanRe-68v7T4pZTYNwj-ttYqeH=6MkeznVqacB5AZw@mail.gmail.com>
	<CABm8tB7qqa5TPdn6AHBkkn-v8RAAcAsEaQOULrEN4HAbYfY90A@mail.gmail.com>
	<6377AF62-D548-4276-A2A6-158EE4513593@anl.gov>
	<CABm8tB6_gEvUTM8oXBSMBAvtPnGmnyf07nAZj0gBmffDPe=V2A@mail.gmail.com>
	<00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk>
	<CAMYG4Gkh-2rUaDpj0BUesjwyRsuKJxiK9ottXpVsuw4+LWzHzg@mail.gmail.com>
	<CABm8tB6+8yk9UVXd+R5DNFgD+NsUAfj2r5Kdny9d_wbd27V7kQ@mail.gmail.com>
	<CAMYG4G=XYWNsjY0Y8nYa0FJQj5w4fuKttX+SNVUcMQsvDB_0DQ@mail.gmail.com>
	<CABm8tB59bFuvXp=Zoc1mB_zYJ=gk4hNF9qzxhBs=jcah=BY59w@mail.gmail.com>
	<CAMYG4GmcCeAU80bm4tV7h6aisanF1W3fm0UXapLkA2D92B7jaw@mail.gmail.com>
	<CABm8tB7rweyeVkBh96dhaY46vdY+pp0oy0bJCwV++_3uOB+Rsw@mail.gmail.com>
	<CABm8tB4W3NZXMu_ux2sbGnZCgx5F=Piq+kBuKVWbfp3DLLCc+A@mail.gmail.com>
	<29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov>
	<CABm8tB71F8_YKBJ=D_h0hKVr7GE6Ww=yuaB4MOXpFNSS+1WzRg@mail.gmail.com>
	<CABm8tB4qnjP=WrcMDnQgO7UEePG_yhfQ41PZxrLY-L+suBrYpg@mail.gmail.com>
	<CAMYG4GkX7jGuvsH4H2XdH2_wRK=Qi4LPqLq+pJv1EW6Oe2W-WQ@mail.gmail.com>
	<CABm8tB63yjedZ3O7Ocuv+=vXQnBRfoaUMb6G5fTCAa68F7h-2A@mail.gmail.com>
	<CAMYG4GnVW9khT0O5WtiE0p8iMDGoBcOJsvuc3BPnTfB5RiqQmg@mail.gmail.com>
	<CABm8tB408JPpDavGwXb8s3krAwtxxihV-41mwkHZrrfukTP3Cg@mail.gmail.com>
	<CAMYG4Gk3k9=fVPesu1xOV79Z-jEgrsZDDXf65U-00uEVengzaQ@mail.gmail.com>
Message-ID: <CABm8tB51_GB00rzUh+_fRmxsZCaiqttqp=Gnqtxq61wk=WLhog@mail.gmail.com>

Thanks Matt - I was trying to write it in linear algebra notation, but the
nature of the problem might produce some confusion.  I was also originally
a bit confused by a fairly major typo in the paper I was following, but I'm
almost sure I have it clear now.  I'll try to be concise - the meat is in
the last two paragraphs.

My system of nonlinear equations, F(x)=0, has dimension m.  Traditional
methods for solving, like Newton's method, tend to reach points at which
the Jacobian becomes singular.  I'm trying the homotopy trick, which
involves introducing an additional scalar variable, s, and solving the
system H(x,s) = sF(x) + (1-s)(x-x_0), starting at s=0 and following the
zero curve to, hopefully, a solution at s=1.

The homotopy map H only has m components, so its Jacobian J is m x (m+1),
and has rank m.  I need to find a vector u in its null space: J u = 0.
Note that u is size m+1.  Rather than solve J u = 0, I make an augmented
matrix A, which is (m+1)x(m+1), and consists of J in its first m rows, then
a somewhat arbitrary vector in the last row.  Then the vector u is the
solution to A u = b, where b is zero everywhere except its last element.
This solve seems to be working fine.

The SNES should then solve for a particular solution y of the Newton
update, J y = - H(y).  A (unique) particular solution can also be found by
solving A y = [-H(y) 0]^T.  Finally - and this is the step I can't figure
out how to implement properly, I want to find the norm-minimizing solution
z for the Newton update:
z = y - u(y^T u)/(u^T u).

I doubt there is any way to do this elegantly within the existing routines,
because u is not in the nullspace of A.  Rather, it's in the nullspace of
J, which is the submatrix I care about.  I think what I need is a way to
just apply the orthogonal projection in the preceding paragraph, after the
SNES solves the linear system but before the Newton update is applied.  I
don't know the SNES internals well at all.  Is it possible to hook this in
somehow, or put it in an existing function?

Thanks again for all your help.


On Oct 1, 2018 6:49 AM, "Matthew Knepley" <knepley at gmail.com> wrote:

On Sun, Sep 30, 2018 at 6:06 PM zakaryah <zakaryah at gmail.com> wrote:

> OK, thanks.
>
> I'm using a composite DM, DM_packer.  To make a separate KSP, I did the
> following in the FormJacobian() routine, after assembling the Jacobian
> matrix A and the RHS for the tangent vector, b:
>
>    - KSPCreate(PETSC_COMM_WORLD,&ksp)
>    - KSPSetDM(ksp,DM_packer)
>    - KSPSetDMActive(ksp,PETSC_FALSE) - because I want to set the operators
>    - KSPSetOperators(ksp,A,P)
>    - VecSet(n,0) - set initial guess to zero
>    - KSPSolve(ksp,b,n) - this solve works correctly
>    - VecNormalize(n,NULL)
>    -
>    MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp)
>    - MatSetNullSpace(A,nullsp)
>
> Then, with -snes_type newtonls -pc_type none -ksp_monitor
> -ksp_monitor_true_residual -ksp_view, the output for the KSPSolve described
> above, i.e. for the tangent vector, looks correct:
>
>   0 KSP preconditioned resid norm 1.000000000000e+03 true resid norm
> 1.000000000000e+03 ||r(i)||/||b|| 1.000000000000e+00
> ...
>
> 185 KSP preconditioned resid norm 9.900713131874e-03 true resid norm
> 9.900713131904e-03 ||r(i)||/||b|| 9.900713131904e-06
>
> Linear solve converged due to CONVERGED_RTOL iterations 185
>
> KSP Object: 1 MPI processes
>
>   type: gmres
>
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>
>     happy breakdown tolerance 1e-30
>
>   maximum iterations=10000, initial guess is zero
>
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>
>   left preconditioning
>
>   using PRECONDITIONED norm type for convergence test
>
> PC Object: 1 MPI processes
>
>   type: none
>
>   linear system matrix = precond matrix:
>
>   Mat Object: 1 MPI processes
>
>     type: seqaij
>
>     rows=78247, cols=78247
>
>     total: nonzeros=6063481, allocated nonzeros=6063481
>
>     total number of mallocs used during MatSetValues calls =0
>
>       using I-node routines: found 26083 nodes, limit used is 5
>
>
> But the next KSP, i.e. the one in the SNES, doesn't converge in the true
> residual:
>
>
>   0 KSP preconditioned resid norm 2.045599092896e-04 true resid norm
> 2.803828296212e-04 ||r(i)||/||b|| 1.000000000000e+00
>
> 191 KSP preconditioned resid norm 2.009941278534e-09 true resid norm
> 1.636010142734e-04 ||r(i)||/||b|| 5.834915586465e-01
>
>
> KSP Object: 1 MPI processes
>
>   type: gmres
>
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>
>     happy breakdown tolerance 1e-30
>
>   maximum iterations=10000, initial guess is zero
>
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>
>   left preconditioning
>
>   using PRECONDITIONED norm type for convergence test
>
> PC Object: 1 MPI processes
>
>   type: none
>
>   linear system matrix = precond matrix:
>
>   Mat Object: 1 MPI processes
>
>     type: seqaij
>
>     rows=78247, cols=78247
>
>     total: nonzeros=6063481, allocated nonzeros=6063481
>
>     total number of mallocs used during MatSetValues calls =0
>
>       has attached null space
>
>       using I-node routines: found 26083 nodes, limit used is 5
>
>
> My guess about what's going on is that the tangent vector n isn't really
> in the nullspace of A.
>

I do not understand the rest, but this is the problem. Maybe it would make
more sense if you wrote things
in linear algebraic notation.

  Thanks,

     Matt


> Rather, it's in the nullspace of the m x (m+1) submatrix of A.  So, An=c
> e_{m+1}, where c is an arbitrary constant and e_{m+1} is the m+1 th basis
> vector.  The nonlinear function also has an added row, F_{m+1}, which is
> set to zero in the FormFunction() routine.  I don't care about the value of
> F_{m+1}, but I suppose that if it's included in the true residual, and NOT
> in the "preconditioned" residual, even with pc_type none, then it will be
> tricky to diagnose the performance.  Should I be using my own routine to
> evaluate the residual, so that F_{m+1} is not included?
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/59e2049d/attachment-0001.html>

From knepley at gmail.com  Fri Oct  5 15:42:00 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 5 Oct 2018 16:42:00 -0400
Subject: [petsc-users] Fwd: Implementing a homotopy solver
In-Reply-To: <CABm8tB51_GB00rzUh+_fRmxsZCaiqttqp=Gnqtxq61wk=WLhog@mail.gmail.com>
References: <CABm8tB45FanRe-68v7T4pZTYNwj-ttYqeH=6MkeznVqacB5AZw@mail.gmail.com>
	<CABm8tB7qqa5TPdn6AHBkkn-v8RAAcAsEaQOULrEN4HAbYfY90A@mail.gmail.com>
	<6377AF62-D548-4276-A2A6-158EE4513593@anl.gov>
	<CABm8tB6_gEvUTM8oXBSMBAvtPnGmnyf07nAZj0gBmffDPe=V2A@mail.gmail.com>
	<00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk>
	<CAMYG4Gkh-2rUaDpj0BUesjwyRsuKJxiK9ottXpVsuw4+LWzHzg@mail.gmail.com>
	<CABm8tB6+8yk9UVXd+R5DNFgD+NsUAfj2r5Kdny9d_wbd27V7kQ@mail.gmail.com>
	<CAMYG4G=XYWNsjY0Y8nYa0FJQj5w4fuKttX+SNVUcMQsvDB_0DQ@mail.gmail.com>
	<CABm8tB59bFuvXp=Zoc1mB_zYJ=gk4hNF9qzxhBs=jcah=BY59w@mail.gmail.com>
	<CAMYG4GmcCeAU80bm4tV7h6aisanF1W3fm0UXapLkA2D92B7jaw@mail.gmail.com>
	<CABm8tB7rweyeVkBh96dhaY46vdY+pp0oy0bJCwV++_3uOB+Rsw@mail.gmail.com>
	<CABm8tB4W3NZXMu_ux2sbGnZCgx5F=Piq+kBuKVWbfp3DLLCc+A@mail.gmail.com>
	<29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov>
	<CABm8tB71F8_YKBJ=D_h0hKVr7GE6Ww=yuaB4MOXpFNSS+1WzRg@mail.gmail.com>
	<CABm8tB4qnjP=WrcMDnQgO7UEePG_yhfQ41PZxrLY-L+suBrYpg@mail.gmail.com>
	<CAMYG4GkX7jGuvsH4H2XdH2_wRK=Qi4LPqLq+pJv1EW6Oe2W-WQ@mail.gmail.com>
	<CABm8tB63yjedZ3O7Ocuv+=vXQnBRfoaUMb6G5fTCAa68F7h-2A@mail.gmail.com>
	<CAMYG4GnVW9khT0O5WtiE0p8iMDGoBcOJsvuc3BPnTfB5RiqQmg@mail.gmail.com>
	<CABm8tB408JPpDavGwXb8s3krAwtxxihV-41mwkHZrrfukTP3Cg@mail.gmail.com>
	<CAMYG4Gk3k9=fVPesu1xOV79Z-jEgrsZDDXf65U-00uEVengzaQ@mail.gmail.com>
	<CABm8tB51_GB00rzUh+_fRmxsZCaiqttqp=Gnqtxq61wk=WLhog@mail.gmail.com>
Message-ID: <CAMYG4GnZXYx-XywErHQrrUv-tYPk-tDacsdvJGG4qiDcuCHqpQ@mail.gmail.com>

On Fri, Oct 5, 2018 at 4:15 PM zakaryah <zakaryah at gmail.com> wrote:

> Thanks Matt - I was trying to write it in linear algebra notation, but the
> nature of the problem might produce some confusion.  I was also
> originally a bit confused by a fairly major typo in the paper I was
> following, but I'm almost sure I have it clear now.  I'll try to be
> concise - the meat is in the last two paragraphs.
>
> My system of nonlinear equations, F(x)=0, has dimension m.  Traditional
> methods for solving, like Newton's method, tend to reach points at which
> the Jacobian becomes singular.  I'm trying the homotopy trick, which
> involves introducing an additional scalar variable, s, and solving the
> system H(x,s) = sF(x) + (1-s)(x-x_0), starting at s=0 and following the
> zero curve to, hopefully, a solution at s=1.
>
> The homotopy map H only has m components, so its Jacobian J is m x (m+1),
> and has rank m.  I need to find a vector u in its null space: J u = 0.
> Note that u is size m+1.  Rather than solve J u = 0, I make an augmented
> matrix A, which is (m+1)x(m+1), and consists of J in its first m rows, then
> a somewhat arbitrary vector in the last row.  Then the vector u is the
> solution to A u = b, where b is zero everywhere except its last element.
> This solve seems to be working fine.
>
> The SNES should then solve for a particular solution y of the Newton
> update, J y = - H(y).  A (unique) particular solution can also be found by
> solving A y = [-H(y) 0]^T.  Finally - and this is the step I can't figure
> out how to implement properly, I want to find the norm-minimizing solution
> z for the Newton update:
> z = y - u(y^T u)/(u^T u).
>
> I doubt there is any way to do this elegantly within the existing
> routines, because u is not in the nullspace of A.  Rather, it's in the
> nullspace of J, which is the submatrix I care about.  I think what I need
> is a way to just apply the orthogonal projection in the preceding
> paragraph, after the SNES solves the linear system but before the Newton
> update is applied.  I don't know the SNES internals well at all.  Is it
> possible to hook this in somehow, or put it in an existing function?
>

I do not claim to fully understand the above (it would be easier for me to
write the equations explicitly on their own lines). However, the issue with
nullspaces is soluble I think. So you say that

  J \in R^{m \cross (m+1)} is full rank

so that its nullspace has dimension 1 and is spanned by u,

  J u = 0.

Now you augment J to get a square matrix A by adding a row,

  /  J  \   u = /   J u  \  =  / 0 \
 \ v^T /        \ v^T u /     \  0 /

if v is orthogonal to u. Thus, just choose v from the m-1 dimensional space
orthogonal to u.

  Thanks,

     Matt


> Thanks again for all your help.
>
>
> On Oct 1, 2018 6:49 AM, "Matthew Knepley" <knepley at gmail.com> wrote:
>
> On Sun, Sep 30, 2018 at 6:06 PM zakaryah <zakaryah at gmail.com> wrote:
>
>> OK, thanks.
>>
>> I'm using a composite DM, DM_packer.  To make a separate KSP, I did the
>> following in the FormJacobian() routine, after assembling the Jacobian
>> matrix A and the RHS for the tangent vector, b:
>>
>>    - KSPCreate(PETSC_COMM_WORLD,&ksp)
>>    - KSPSetDM(ksp,DM_packer)
>>    - KSPSetDMActive(ksp,PETSC_FALSE) - because I want to set the
>>    operators
>>    - KSPSetOperators(ksp,A,P)
>>    - VecSet(n,0) - set initial guess to zero
>>    - KSPSolve(ksp,b,n) - this solve works correctly
>>    - VecNormalize(n,NULL)
>>    -
>>    MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp)
>>    - MatSetNullSpace(A,nullsp)
>>
>> Then, with -snes_type newtonls -pc_type none -ksp_monitor
>> -ksp_monitor_true_residual -ksp_view, the output for the KSPSolve described
>> above, i.e. for the tangent vector, looks correct:
>>
>>   0 KSP preconditioned resid norm 1.000000000000e+03 true resid norm
>> 1.000000000000e+03 ||r(i)||/||b|| 1.000000000000e+00
>> ...
>>
>> 185 KSP preconditioned resid norm 9.900713131874e-03 true resid norm
>> 9.900713131904e-03 ||r(i)||/||b|| 9.900713131904e-06
>>
>> Linear solve converged due to CONVERGED_RTOL iterations 185
>>
>> KSP Object: 1 MPI processes
>>
>>   type: gmres
>>
>>     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>
>>     happy breakdown tolerance 1e-30
>>
>>   maximum iterations=10000, initial guess is zero
>>
>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>
>>   left preconditioning
>>
>>   using PRECONDITIONED norm type for convergence test
>>
>> PC Object: 1 MPI processes
>>
>>   type: none
>>
>>   linear system matrix = precond matrix:
>>
>>   Mat Object: 1 MPI processes
>>
>>     type: seqaij
>>
>>     rows=78247, cols=78247
>>
>>     total: nonzeros=6063481, allocated nonzeros=6063481
>>
>>     total number of mallocs used during MatSetValues calls =0
>>
>>       using I-node routines: found 26083 nodes, limit used is 5
>>
>>
>> But the next KSP, i.e. the one in the SNES, doesn't converge in the true
>> residual:
>>
>>
>>   0 KSP preconditioned resid norm 2.045599092896e-04 true resid norm
>> 2.803828296212e-04 ||r(i)||/||b|| 1.000000000000e+00
>>
>> 191 KSP preconditioned resid norm 2.009941278534e-09 true resid norm
>> 1.636010142734e-04 ||r(i)||/||b|| 5.834915586465e-01
>>
>>
>> KSP Object: 1 MPI processes
>>
>>   type: gmres
>>
>>     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>
>>     happy breakdown tolerance 1e-30
>>
>>   maximum iterations=10000, initial guess is zero
>>
>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>
>>   left preconditioning
>>
>>   using PRECONDITIONED norm type for convergence test
>>
>> PC Object: 1 MPI processes
>>
>>   type: none
>>
>>   linear system matrix = precond matrix:
>>
>>   Mat Object: 1 MPI processes
>>
>>     type: seqaij
>>
>>     rows=78247, cols=78247
>>
>>     total: nonzeros=6063481, allocated nonzeros=6063481
>>
>>     total number of mallocs used during MatSetValues calls =0
>>
>>       has attached null space
>>
>>       using I-node routines: found 26083 nodes, limit used is 5
>>
>>
>> My guess about what's going on is that the tangent vector n isn't really
>> in the nullspace of A.
>>
>
> I do not understand the rest, but this is the problem. Maybe it would make
> more sense if you wrote things
> in linear algebraic notation.
>
>   Thanks,
>
>      Matt
>
>
>> Rather, it's in the nullspace of the m x (m+1) submatrix of A.  So, An=c
>> e_{m+1}, where c is an arbitrary constant and e_{m+1} is the m+1 th basis
>> vector.  The nonlinear function also has an added row, F_{m+1}, which is
>> set to zero in the FormFunction() routine.  I don't care about the value of
>> F_{m+1}, but I suppose that if it's included in the true residual, and NOT
>> in the "preconditioned" residual, even with pc_type none, then it will be
>> tricky to diagnose the performance.  Should I be using my own routine to
>> evaluate the residual, so that F_{m+1} is not included?
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/aa0807d4/attachment-0001.html>

From zakaryah at gmail.com  Fri Oct  5 15:54:45 2018
From: zakaryah at gmail.com (zakaryah)
Date: Fri, 5 Oct 2018 16:54:45 -0400
Subject: [petsc-users] Fwd: Implementing a homotopy solver
In-Reply-To: <CAMYG4GnZXYx-XywErHQrrUv-tYPk-tDacsdvJGG4qiDcuCHqpQ@mail.gmail.com>
References: <CABm8tB45FanRe-68v7T4pZTYNwj-ttYqeH=6MkeznVqacB5AZw@mail.gmail.com>
	<CABm8tB7qqa5TPdn6AHBkkn-v8RAAcAsEaQOULrEN4HAbYfY90A@mail.gmail.com>
	<6377AF62-D548-4276-A2A6-158EE4513593@anl.gov>
	<CABm8tB6_gEvUTM8oXBSMBAvtPnGmnyf07nAZj0gBmffDPe=V2A@mail.gmail.com>
	<00D16613-4B28-4801-96EA-7CE2EF8BF8F7@imperial.ac.uk>
	<CAMYG4Gkh-2rUaDpj0BUesjwyRsuKJxiK9ottXpVsuw4+LWzHzg@mail.gmail.com>
	<CABm8tB6+8yk9UVXd+R5DNFgD+NsUAfj2r5Kdny9d_wbd27V7kQ@mail.gmail.com>
	<CAMYG4G=XYWNsjY0Y8nYa0FJQj5w4fuKttX+SNVUcMQsvDB_0DQ@mail.gmail.com>
	<CABm8tB59bFuvXp=Zoc1mB_zYJ=gk4hNF9qzxhBs=jcah=BY59w@mail.gmail.com>
	<CAMYG4GmcCeAU80bm4tV7h6aisanF1W3fm0UXapLkA2D92B7jaw@mail.gmail.com>
	<CABm8tB7rweyeVkBh96dhaY46vdY+pp0oy0bJCwV++_3uOB+Rsw@mail.gmail.com>
	<CABm8tB4W3NZXMu_ux2sbGnZCgx5F=Piq+kBuKVWbfp3DLLCc+A@mail.gmail.com>
	<29B68A54-9D29-4881-AD80-98A5E510FD82@anl.gov>
	<CABm8tB71F8_YKBJ=D_h0hKVr7GE6Ww=yuaB4MOXpFNSS+1WzRg@mail.gmail.com>
	<CABm8tB4qnjP=WrcMDnQgO7UEePG_yhfQ41PZxrLY-L+suBrYpg@mail.gmail.com>
	<CAMYG4GkX7jGuvsH4H2XdH2_wRK=Qi4LPqLq+pJv1EW6Oe2W-WQ@mail.gmail.com>
	<CABm8tB63yjedZ3O7Ocuv+=vXQnBRfoaUMb6G5fTCAa68F7h-2A@mail.gmail.com>
	<CAMYG4GnVW9khT0O5WtiE0p8iMDGoBcOJsvuc3BPnTfB5RiqQmg@mail.gmail.com>
	<CABm8tB408JPpDavGwXb8s3krAwtxxihV-41mwkHZrrfukTP3Cg@mail.gmail.com>
	<CAMYG4Gk3k9=fVPesu1xOV79Z-jEgrsZDDXf65U-00uEVengzaQ@mail.gmail.com>
	<CABm8tB51_GB00rzUh+_fRmxsZCaiqttqp=Gnqtxq61wk=WLhog@mail.gmail.com>
	<CAMYG4GnZXYx-XywErHQrrUv-tYPk-tDacsdvJGG4qiDcuCHqpQ@mail.gmail.com>
Message-ID: <CABm8tB7WemcVa-7oBjTT1H2q_d150pVOf=Uxti9pRt-7nq8pOw@mail.gmail.com>

Sorry again about the formatting - I struggle with notation in email.

I think I see what you are saying - my augmented matrix should differ in
the last row between the one used to calculate the nullspace, u, and the
one the SNES uses as the Jacobian.  If I construct the SNES Jacobian A so
that u is truly in its nullspace, then I can just add u to the nullspace of
A and let PETSc take care of the projection?  I will try this - thanks for
the excellent suggestion.

On Fri, Oct 5, 2018 at 4:42 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 5, 2018 at 4:15 PM zakaryah <zakaryah at gmail.com> wrote:
>
>> Thanks Matt - I was trying to write it in linear algebra notation, but
>> the nature of the problem might produce some confusion.  I was also
>> originally a bit confused by a fairly major typo in the paper I was
>> following, but I'm almost sure I have it clear now.  I'll try to be
>> concise - the meat is in the last two paragraphs.
>>
>> My system of nonlinear equations, F(x)=0, has dimension m.  Traditional
>> methods for solving, like Newton's method, tend to reach points at which
>> the Jacobian becomes singular.  I'm trying the homotopy trick, which
>> involves introducing an additional scalar variable, s, and solving the
>> system H(x,s) = sF(x) + (1-s)(x-x_0), starting at s=0 and following the
>> zero curve to, hopefully, a solution at s=1.
>>
>> The homotopy map H only has m components, so its Jacobian J is m x
>> (m+1), and has rank m.  I need to find a vector u in its null space: J u
>> = 0.  Note that u is size m+1.  Rather than solve J u = 0, I make an
>> augmented matrix A, which is (m+1)x(m+1), and consists of J in its first m
>> rows, then a somewhat arbitrary vector in the last row.  Then the vector
>> u is the solution to A u = b, where b is zero everywhere except its last
>> element.  This solve seems to be working fine.
>>
>> The SNES should then solve for a particular solution y of the Newton
>> update, J y = - H(y).  A (unique) particular solution can also be found by
>> solving A y = [-H(y) 0]^T.  Finally - and this is the step I can't figure
>> out how to implement properly, I want to find the norm-minimizing solution
>> z for the Newton update:
>> z = y - u(y^T u)/(u^T u).
>>
>> I doubt there is any way to do this elegantly within the existing
>> routines, because u is not in the nullspace of A.  Rather, it's in the
>> nullspace of J, which is the submatrix I care about.  I think what I need
>> is a way to just apply the orthogonal projection in the preceding
>> paragraph, after the SNES solves the linear system but before the Newton
>> update is applied.  I don't know the SNES internals well at all.  Is it
>> possible to hook this in somehow, or put it in an existing function?
>>
>
> I do not claim to fully understand the above (it would be easier for me to
> write the equations explicitly on their own lines). However, the issue with
> nullspaces is soluble I think. So you say that
>
>   J \in R^{m \cross (m+1)} is full rank
>
> so that its nullspace has dimension 1 and is spanned by u,
>
>   J u = 0.
>
> Now you augment J to get a square matrix A by adding a row,
>
>   /  J  \   u = /   J u  \  =  / 0 \
>  \ v^T /        \ v^T u /     \  0 /
>
> if v is orthogonal to u. Thus, just choose v from the m-1 dimensional
> space orthogonal to u.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks again for all your help.
>>
>>
>> On Oct 1, 2018 6:49 AM, "Matthew Knepley" <knepley at gmail.com> wrote:
>>
>> On Sun, Sep 30, 2018 at 6:06 PM zakaryah <zakaryah at gmail.com> wrote:
>>
>>> OK, thanks.
>>>
>>> I'm using a composite DM, DM_packer.  To make a separate KSP, I did the
>>> following in the FormJacobian() routine, after assembling the Jacobian
>>> matrix A and the RHS for the tangent vector, b:
>>>
>>>    - KSPCreate(PETSC_COMM_WORLD,&ksp)
>>>    - KSPSetDM(ksp,DM_packer)
>>>    - KSPSetDMActive(ksp,PETSC_FALSE) - because I want to set the
>>>    operators
>>>    - KSPSetOperators(ksp,A,P)
>>>    - VecSet(n,0) - set initial guess to zero
>>>    - KSPSolve(ksp,b,n) - this solve works correctly
>>>    - VecNormalize(n,NULL)
>>>    -
>>>    MatNullSpaceCreate(PetscObjectComm((PetscObject)A),PETSC_FALSE,1,&n,&nullsp)
>>>    - MatSetNullSpace(A,nullsp)
>>>
>>> Then, with -snes_type newtonls -pc_type none -ksp_monitor
>>> -ksp_monitor_true_residual -ksp_view, the output for the KSPSolve described
>>> above, i.e. for the tangent vector, looks correct:
>>>
>>>   0 KSP preconditioned resid norm 1.000000000000e+03 true resid norm
>>> 1.000000000000e+03 ||r(i)||/||b|| 1.000000000000e+00
>>> ...
>>>
>>> 185 KSP preconditioned resid norm 9.900713131874e-03 true resid norm
>>> 9.900713131904e-03 ||r(i)||/||b|| 9.900713131904e-06
>>>
>>> Linear solve converged due to CONVERGED_RTOL iterations 185
>>>
>>> KSP Object: 1 MPI processes
>>>
>>>   type: gmres
>>>
>>>     restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>
>>>     happy breakdown tolerance 1e-30
>>>
>>>   maximum iterations=10000, initial guess is zero
>>>
>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>
>>>   left preconditioning
>>>
>>>   using PRECONDITIONED norm type for convergence test
>>>
>>> PC Object: 1 MPI processes
>>>
>>>   type: none
>>>
>>>   linear system matrix = precond matrix:
>>>
>>>   Mat Object: 1 MPI processes
>>>
>>>     type: seqaij
>>>
>>>     rows=78247, cols=78247
>>>
>>>     total: nonzeros=6063481, allocated nonzeros=6063481
>>>
>>>     total number of mallocs used during MatSetValues calls =0
>>>
>>>       using I-node routines: found 26083 nodes, limit used is 5
>>>
>>>
>>> But the next KSP, i.e. the one in the SNES, doesn't converge in the true
>>> residual:
>>>
>>>
>>>   0 KSP preconditioned resid norm 2.045599092896e-04 true resid norm
>>> 2.803828296212e-04 ||r(i)||/||b|| 1.000000000000e+00
>>>
>>> 191 KSP preconditioned resid norm 2.009941278534e-09 true resid norm
>>> 1.636010142734e-04 ||r(i)||/||b|| 5.834915586465e-01
>>>
>>>
>>> KSP Object: 1 MPI processes
>>>
>>>   type: gmres
>>>
>>>     restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>>
>>>     happy breakdown tolerance 1e-30
>>>
>>>   maximum iterations=10000, initial guess is zero
>>>
>>>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>
>>>   left preconditioning
>>>
>>>   using PRECONDITIONED norm type for convergence test
>>>
>>> PC Object: 1 MPI processes
>>>
>>>   type: none
>>>
>>>   linear system matrix = precond matrix:
>>>
>>>   Mat Object: 1 MPI processes
>>>
>>>     type: seqaij
>>>
>>>     rows=78247, cols=78247
>>>
>>>     total: nonzeros=6063481, allocated nonzeros=6063481
>>>
>>>     total number of mallocs used during MatSetValues calls =0
>>>
>>>       has attached null space
>>>
>>>       using I-node routines: found 26083 nodes, limit used is 5
>>>
>>>
>>> My guess about what's going on is that the tangent vector n isn't really
>>> in the nullspace of A.
>>>
>>
>> I do not understand the rest, but this is the problem. Maybe it would
>> make more sense if you wrote things
>> in linear algebraic notation.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Rather, it's in the nullspace of the m x (m+1) submatrix of A.  So, An=c
>>> e_{m+1}, where c is an arbitrary constant and e_{m+1} is the m+1 th basis
>>> vector.  The nonlinear function also has an added row, F_{m+1}, which is
>>> set to zero in the FormFunction() routine.  I don't care about the value of
>>> F_{m+1}, but I suppose that if it's included in the true residual, and NOT
>>> in the "preconditioned" residual, even with pc_type none, then it will be
>>> tricky to diagnose the performance.  Should I be using my own routine to
>>> evaluate the residual, so that F_{m+1} is not included?
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/d741a541/attachment-0001.html>

From mvalera-w at sdsu.edu  Fri Oct  5 17:49:02 2018
From: mvalera-w at sdsu.edu (Manuel Valera)
Date: Fri, 5 Oct 2018 15:49:02 -0700
Subject: [petsc-users] Interpolation in staggered grid
Message-ID: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>

Hello,

I'm trying to do a simple variable interpolation, from a cell center to a
face in a staggered grid, my model data management is done with DMDAs, with
two different DMs one for each cell position,

I already did this task on a fortran only version of the model using the 4
closest neighbors of the scalars (cell center) to be interpolated at the
velocity site (cell face), i did it using a loop over the domain, somehow
this easy task is not translating into the DMDA framework,

I'm not sure what I'm doing wrong or not aware of a easier way Petsc may
have for this task, if you could point out an easier strategy or an example
I would be grateful,

Thanks,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/85385c8f/attachment.html>

From michael.wick.1980 at gmail.com  Fri Oct  5 20:08:35 2018
From: michael.wick.1980 at gmail.com (Mike Wick)
Date: Fri, 5 Oct 2018 18:08:35 -0700
Subject: [petsc-users] Failure of MUMPS
Message-ID: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>

Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The
matrix is getting denser and my experience is that MUMPS just outperforms
iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I
just wander if there is any suggestion to improve the robustness of MUMPS?
Or in general, any suggestion for interative solver with very high-order
finite elements?

Thanks!

Mike
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/0b5d63cd/attachment.html>

From hzhang at mcs.anl.gov  Fri Oct  5 20:12:43 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Sat, 6 Oct 2018 01:12:43 +0000
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
Message-ID: <CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>

Mike:
Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The matrix is getting denser and my experience is that MUMPS just outperforms iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I just wander if there is any suggestion to improve the robustness of MUMPS? Or in general, any suggestion for interative solver with very high-order finite elements?

What error message do you get when MUMPS fails? Out of memory, zero pivoting, or something?
 Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181006/a802daf4/attachment.html>

From hzhang at mcs.anl.gov  Fri Oct  5 20:12:43 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Sat, 6 Oct 2018 01:12:43 +0000
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
Message-ID: <CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>

Mike:
Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The matrix is getting denser and my experience is that MUMPS just outperforms iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I just wander if there is any suggestion to improve the robustness of MUMPS? Or in general, any suggestion for interative solver with very high-order finite elements?

What error message do you get when MUMPS fails? Out of memory, zero pivoting, or something?
 Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181006/a802daf4/attachment-0001.html>

From michael.wick.1980 at gmail.com  Fri Oct  5 20:15:26 2018
From: michael.wick.1980 at gmail.com (Mike Wick)
Date: Fri, 5 Oct 2018 18:15:26 -0700
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
Message-ID: <CAA24MXvwX=HDT9Y8_gujMvTw5hEdGhLD5RscMGkLCeVjq-4VLw@mail.gmail.com>

Hi Hong:

There is no explicit error message actually. The solver converge in 0
iteration and returns an nan number. Looks like a zero pivoting problem.

Mike

On Fri, Oct 5, 2018 at 6:12 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Mike:
>
>> Hello PETSc team:
>>
>> I am trying to solve a PDE problem with high-order finite elements. The
>> matrix is getting denser and my experience is that MUMPS just outperforms
>> iterative solvers.
>>
>> For certain problems, MUMPS just fail in the middle for no clear reason.
>> I just wander if there is any suggestion to improve the robustness of
>> MUMPS? Or in general, any suggestion for interative solver with very
>> high-order finite elements?
>>
>
> What error message do you get when MUMPS fails? Out of memory, zero
> pivoting, or something?
>  Hong
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/99decb34/attachment.html>

From david.knezevic at akselos.com  Fri Oct  5 20:22:17 2018
From: david.knezevic at akselos.com (David Knezevic)
Date: Fri, 5 Oct 2018 21:22:17 -0400
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAA24MXvwX=HDT9Y8_gujMvTw5hEdGhLD5RscMGkLCeVjq-4VLw@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
	<CAA24MXvwX=HDT9Y8_gujMvTw5hEdGhLD5RscMGkLCeVjq-4VLw@mail.gmail.com>
Message-ID: <CAJCWK9APjBr_xz+g41Ayca7hSLHV-WZr91X5WTrXBcs58DJ4uA@mail.gmail.com>

On Fri, Oct 5, 2018 at 9:16 PM Mike Wick <michael.wick.1980 at gmail.com>
wrote:

> Hi Hong:
>
> There is no explicit error message actually. The solver converge in 0
> iteration and returns an nan number. Looks like a zero pivoting problem.
>
> Mike
>

I use MatMumpsGetInfo
<https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMumpsGetInfo.html>
to
get error info returned by MUMPS, e.g.

PetscInt info_1;
MatMumpsGetInfo(pc_mat, 1, &info_1);

Then you can check the value of info_1 to get error diagnostics. One error
that I run into sometimes is when info_1 is -9, in which case I increase
icntl_14 and try again.

Best,
David



>
> On Fri, Oct 5, 2018 at 6:12 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Mike:
>>
>>> Hello PETSc team:
>>>
>>> I am trying to solve a PDE problem with high-order finite elements. The
>>> matrix is getting denser and my experience is that MUMPS just outperforms
>>> iterative solvers.
>>>
>>> For certain problems, MUMPS just fail in the middle for no clear reason.
>>> I just wander if there is any suggestion to improve the robustness of
>>> MUMPS? Or in general, any suggestion for interative solver with very
>>> high-order finite elements?
>>>
>>
>> What error message do you get when MUMPS fails? Out of memory, zero
>> pivoting, or something?
>>  Hong
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181005/dc1d738c/attachment-0001.html>

From stefano.zampini at gmail.com  Sat Oct  6 02:51:18 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Sat, 6 Oct 2018 10:51:18 +0300
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAJCWK9APjBr_xz+g41Ayca7hSLHV-WZr91X5WTrXBcs58DJ4uA@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
	<CAA24MXvwX=HDT9Y8_gujMvTw5hEdGhLD5RscMGkLCeVjq-4VLw@mail.gmail.com>
	<CAJCWK9APjBr_xz+g41Ayca7hSLHV-WZr91X5WTrXBcs58DJ4uA@mail.gmail.com>
Message-ID: <CAGPUisg0EeyPk4CiA0_xoedZh2couFqf0TQd89_Mzp7n4VphPA@mail.gmail.com>

I also recently noticed that mumps sometimes segfaults (or return NaNs) in
parallel on matrices coming from higher order fem (relatively dense
blocks). The fault goes away if not using scalapack to solve for the root
mode of the nested dissection tree. The same happens with mumps from a
package manager instead of --download-mumps. Probably this is a regression
on their side .

Il Sab 6 Ott 2018, 04:22 David Knezevic <david.knezevic at akselos.com> ha
scritto:

>
> On Fri, Oct 5, 2018 at 9:16 PM Mike Wick <michael.wick.1980 at gmail.com>
> wrote:
>
>> Hi Hong:
>>
>> There is no explicit error message actually. The solver converge in 0
>> iteration and returns an nan number. Looks like a zero pivoting problem.
>>
>> Mike
>>
>
> I use MatMumpsGetInfo
> <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMumpsGetInfo.html> to
> get error info returned by MUMPS, e.g.
>
> PetscInt info_1;
> MatMumpsGetInfo(pc_mat, 1, &info_1);
>
> Then you can check the value of info_1 to get error diagnostics. One error
> that I run into sometimes is when info_1 is -9, in which case I increase
> icntl_14 and try again.
>
> Best,
> David
>
>
>
>>
>> On Fri, Oct 5, 2018 at 6:12 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Mike:
>>>
>>>> Hello PETSc team:
>>>>
>>>> I am trying to solve a PDE problem with high-order finite elements. The
>>>> matrix is getting denser and my experience is that MUMPS just outperforms
>>>> iterative solvers.
>>>>
>>>> For certain problems, MUMPS just fail in the middle for no clear
>>>> reason. I just wander if there is any suggestion to improve the robustness
>>>> of MUMPS? Or in general, any suggestion for interative solver with very
>>>> high-order finite elements?
>>>>
>>>
>>> What error message do you get when MUMPS fails? Out of memory, zero
>>> pivoting, or something?
>>>  Hong
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181006/7645c669/attachment.html>

From knepley at gmail.com  Sat Oct  6 06:42:39 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 6 Oct 2018 07:42:39 -0400
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
Message-ID: <CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>

On Fri, Oct 5, 2018 at 9:08 PM Mike Wick <michael.wick.1980 at gmail.com>
wrote:

> Hello PETSc team:
>
> I am trying to solve a PDE problem with high-order finite elements. The
> matrix is getting denser and my experience is that MUMPS just outperforms
> iterative solvers.
>

If the problem is elliptic, there is a lot of evidence that the P1
preconditioner is descent for the system. Some people
just project the system to P1, invert that with multigrid, and use that as
the PC for Krylov. It should be worth trying.
Moreover, as Jed will tell you, forming matrices for higher order is
counterproductive. You should apply those matrix-free.

  Thanks,

     Matt


> For certain problems, MUMPS just fail in the middle for no clear reason. I
> just wander if there is any suggestion to improve the robustness of MUMPS?
> Or in general, any suggestion for interative solver with very high-order
> finite elements?
>
> Thanks!
>
> Mike
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181006/0a2cf46d/attachment.html>

From knepley at gmail.com  Sat Oct  6 06:45:12 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 6 Oct 2018 07:45:12 -0400
Subject: [petsc-users] Interpolation in staggered grid
In-Reply-To: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>
References: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>
Message-ID: <CAMYG4G=VLfR_EfBFV9trX6VA2ETUOE2Fhc=6926rtDWiDEtRuQ@mail.gmail.com>

On Fri, Oct 5, 2018 at 6:49 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:

> Hello,
>
> I'm trying to do a simple variable interpolation, from a cell center to a
> face in a staggered grid, my model data management is done with DMDAs, with
> two different DMs one for each cell position,
>
> I already did this task on a fortran only version of the model using the 4
> closest neighbors of the scalars (cell center) to be interpolated at the
> velocity site (cell face), i did it using a loop over the domain, somehow
> this easy task is not translating into the DMDA framework,
>

Its not clear to me what problem you are having. We have done this before.
For example,


https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex30.c.html

I would note that the development version of PETSc now has DMStag which
supports staggered grid discretizations directly.

  Thanks,

    Matt


> I'm not sure what I'm doing wrong or not aware of a easier way Petsc may
> have for this task, if you could point out an easier strategy or an example
> I would be grateful,
>
> Thanks,
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181006/3ef5bf2a/attachment.html>

From a.croucher at auckland.ac.nz  Sun Oct  7 20:09:06 2018
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Mon, 8 Oct 2018 14:09:06 +1300
Subject: [petsc-users] pkg-config
Message-ID: <00e0d38f-acaa-cd4b-5e01-91a21eb09235@auckland.ac.nz>

hi

I am experimenting with using pkg-config to help build my code, and 
using the PETSc.pc file in $PETSC_DIR/$PETSC_ARCH/lib/pkgconfig.

At present it is not working, because the PETSc.pc file sets Libs: 
-L${libdir} -lpetsc

where libdir=${prefix}/lib, and (in my case at least) prefix is equal to 
$PETSC_DIR.

But the PETSc library is not in $PETSC_DIR/lib, it is in 
$PETSC_DIR/$PETSC_ARCH/lib.

Have I messed something up in my PETSc config so that the PETSc.pc file 
is not quite right?

- Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 4611


From jed at jedbrown.org  Sun Oct  7 20:14:00 2018
From: jed at jedbrown.org (Jed Brown)
Date: Sun, 07 Oct 2018 19:14:00 -0600
Subject: [petsc-users] pkg-config
In-Reply-To: <00e0d38f-acaa-cd4b-5e01-91a21eb09235@auckland.ac.nz>
References: <00e0d38f-acaa-cd4b-5e01-91a21eb09235@auckland.ac.nz>
Message-ID: <87efd1z1d3.fsf@jedbrown.org>

This was a bug, now fixed in 'maint' and will be in the next patch
release.  You can repair the PETSc.pc, update to 'maint', or apply this
2-line patch.

https://bitbucket.org/petsc/petsc/commits/e1e675de8c3f48a6c27b15412e00ded76072f735

Adrian Croucher <a.croucher at auckland.ac.nz> writes:

> hi
>
> I am experimenting with using pkg-config to help build my code, and 
> using the PETSc.pc file in $PETSC_DIR/$PETSC_ARCH/lib/pkgconfig.
>
> At present it is not working, because the PETSc.pc file sets Libs: 
> -L${libdir} -lpetsc
>
> where libdir=${prefix}/lib, and (in my case at least) prefix is equal to 
> $PETSC_DIR.
>
> But the PETSc library is not in $PETSC_DIR/lib, it is in 
> $PETSC_DIR/$PETSC_ARCH/lib.
>
> Have I messed something up in my PETSc config so that the PETSc.pc file 
> is not quite right?
>
> - Adrian
>
> -- 
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 4611

From a.croucher at auckland.ac.nz  Sun Oct  7 21:18:03 2018
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Mon, 8 Oct 2018 15:18:03 +1300
Subject: [petsc-users] pkg-config
In-Reply-To: <87efd1z1d3.fsf@jedbrown.org>
References: <00e0d38f-acaa-cd4b-5e01-91a21eb09235@auckland.ac.nz>
	<87efd1z1d3.fsf@jedbrown.org>
Message-ID: <076ae03e-0d99-509a-866c-b716b88c1579@auckland.ac.nz>

OK thanks Jed, that's fixed it.

- Adrian

On 8/10/18 2:14 PM, Jed Brown wrote:
> This was a bug, now fixed in 'maint' and will be in the next patch
> release.  You can repair the PETSc.pc, update to 'maint', or apply this
> 2-line patch.
>
> https://bitbucket.org/petsc/petsc/commits/e1e675de8c3f48a6c27b15412e00ded76072f735
>
> Adrian Croucher <a.croucher at auckland.ac.nz> writes:
>
>> hi
>>
>> I am experimenting with using pkg-config to help build my code, and
>> using the PETSc.pc file in $PETSC_DIR/$PETSC_ARCH/lib/pkgconfig.
>>
>> At present it is not working, because the PETSc.pc file sets Libs:
>> -L${libdir} -lpetsc
>>
>> where libdir=${prefix}/lib, and (in my case at least) prefix is equal to
>> $PETSC_DIR.
>>
>> But the PETSc library is not in $PETSC_DIR/lib, it is in
>> $PETSC_DIR/$PETSC_ARCH/lib.
>>
>> Have I messed something up in my PETSc config so that the PETSc.pc file
>> is not quite right?
>>
>> - Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 4611


From Ling.Zou at inl.gov  Mon Oct  8 10:30:22 2018
From: Ling.Zou at inl.gov (Ling Zou)
Date: Mon, 8 Oct 2018 15:30:22 +0000
Subject: [petsc-users] Diffusion with unstructured mesh
Message-ID: <DM5PR09MB145274FF52A20BF1D40350DF83E60@DM5PR09MB1452.namprd09.prod.outlook.com>

Hi All,


It looks like DMPLEX is the built-in unstructured mesh library.

Is there an existing example on, for example, a simple diffusion problem using cell-centered finite volume method with the DMPLEX method?


Thanks,


Ling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/33613902/attachment.html>

From ellen.price at cfa.harvard.edu  Mon Oct  8 16:49:13 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Mon, 8 Oct 2018 17:49:13 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
Message-ID: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>

Can anyone shed some light on SNES example 12?

https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex12.c.html

I am solving a Poisson problem, but with a source term defined on mesh
points, so I don't have an analytic form. The part I am confused about
is the role of the functions f0, f1, and g3 that go into the DS.

Suppose I have the Poisson problem nabla^2 u = f. Then the weak form is,
for test function phi,

\int_{\partial \Omega} \phi \nabla u - \int_\Omega \nabla u \cdot \nabla
\phi = \int_\Omega \phi f

Based on the documentation of PetscDSSetResidual(), I'm a little
confused about f0 and f1:

\int_\Omega \phi f_0(u, u_t, \nabla u, x, t) + \nabla\phi \cdot {\vec
f}_1(u, u_t, \nabla u, x, t)

So f0 is, for the Poisson problem, zero, and f1 is just -grad u? Do we
need to do anything with the surface term, or is that handled
internally? Because neither of these terms are a surface integral term,
both are volume integrals.

I can't figure out the purpose of g0, g1, g2, g3 in
PetscDSSetJacobian(), either. Is g supposed to be the source term? Is
psi a second basis function? What even *is* the Jacobian in a FEM
problem? I've only seen them formulated as linear problems in tutorials,
so I'm not even clear why we need a nonlinear solver or Jacobian...

Any help is appreciated. I like the idea of a general framework for FEM,
as I think it will reduce human error on my part, but I also want to
understand what I'm doing.

Ellen Price

From weizhuo2 at illinois.edu  Mon Oct  8 17:13:29 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 8 Oct 2018 17:13:29 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
Message-ID: <CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>

Sorry I was caught up with midterms for the last few days. I tried the lu
decomposition today and the 2-norm is pretty stable at ~ 10^-15, which is
expected for double precision. Since the discretization error is so small,
it would be reasonable to assume the rest is majority the algebraic error.
Then looking at the result without the -pc_type lu flag(second graph), the
error is asymptoting to a constant several magnitudes larger than the
tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
behavior? Shouldn't it decrease with finer grid?
[image: LU.png]
[image: Total.png]

On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> I didn't specify a tolerance, it was using the default tolerance. Doesn't
>> the asymptoting norm implies finer grid won't help to get finer solution?
>>
>
> There are two things going on in your test, discretization error
> controlled by the grid, and algebraic error controlled by the solver. This
> makes it difficult to isolate what is happening. However, it seems clear
> that your plot is looking at algebraic error. You can confirm this by using
>
>   -pc_type lu
>
> for the solve. Then all you have is discretization error.
>
>   Thanks,
>
>      Matt
>
>
>> Mark Adams <mfadams at lbl.gov> ?
>>
>>>
>>>
>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> Yes I was using one norm in my Helmholtz code, the example code used 2
>>>> norm. But now I am using 2 norm in both code.
>>>>
>>>>   /*
>>>>      Check the error
>>>>   */
>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>
>>>>  I made a plot to show the increase:
>>>>
>>>
>>>
>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>> you using?
>>>
>>>
>>>>
>>>> [image: Norm comparison.png]
>>>>
>>>> Mark Adams <mfadams at lbl.gov>?
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> The example code and makefile are attached below. The whole thing
>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>> )
>>>>>>
>>>>>
>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>
>>>>>
>>>>>> increases as I use finer and finer grids.
>>>>>>
>>>>>
>>>>> What was the rate of increase?
>>>>>
>>>>>
>>>>>> Then I looked at the example 12 (Laplacian solver) which is similar
>>>>>> to what I did to see if I have missed something. The example is using
>>>>>> 2_norm. I have made some minor modifications (3 places) on the code, you
>>>>>> can search 'Modified' in the code to see them.
>>>>>>
>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>
>>>>>> Thanks for all your reply!
>>>>>>
>>>>>> Weizhuo
>>>>>>
>>>>>>
>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>
>>>>>>
>>>>>>>    Please send your version of the example that computes the mean
>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>
>>>>>>>    Barry
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>> grids, the trend of norm still increases.
>>>>>>> >
>>>>>>> > Thanks!
>>>>>>> >
>>>>>>> > Weizhuo
>>>>>>> >
>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>> > Hi!
>>>>>>> >
>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>> grid. Am I doing this wrong?
>>>>>>> >
>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>> the sqrt of the sum of the squares of
>>>>>>> > the vector entries. It should be scaled by the volume element to
>>>>>>> approximate a scale-independent
>>>>>>> > norm (like the L_2 norm).
>>>>>>> >
>>>>>>> >   Thanks,
>>>>>>> >
>>>>>>> >      Matt
>>>>>>> >
>>>>>>> > Thanks!
>>>>>>> > --
>>>>>>> > Wang Weizhuo
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > -- Norbert Wiener
>>>>>>> >
>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>> >
>>>>>>> >
>>>>>>> > --
>>>>>>> > Wang Weizhuo
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wang Weizhuo
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/812fd18c/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/812fd18c/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/812fd18c/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/812fd18c/attachment-0005.png>

From knepley at gmail.com  Mon Oct  8 17:28:11 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 8 Oct 2018 18:28:11 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
Message-ID: <CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>

On Mon, Oct 8, 2018 at 5:50 PM Ellen M. Price <ellen.price at cfa.harvard.edu>
wrote:

> Can anyone shed some light on SNES example 12?
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex12.c.html
>
> I am solving a Poisson problem, but with a source term defined on mesh
> points, so I don't have an analytic form.


Can you explain more about this source term? It sounds like a bunch of
delta functions. That
would still work in this framework, but the convergence rate for a rhs with
these singularities
is reduced (this is a generic feature of FEM).


> The part I am confused about
> is the role of the functions f0, f1, and g3 that go into the DS.
>
> Suppose I have the Poisson problem nabla^2 u = f. Then the weak form is,
> for test function phi,
>
> \int_{\partial \Omega} \phi \nabla u - \int_\Omega \nabla u \cdot \nabla
> \phi = \int_\Omega \phi f
>

I think you need \int_{\partial \Omega} \phi \nabla u \cdot n instead.


> Based on the documentation of PetscDSSetResidual(), I'm a little
> confused about f0 and f1:
>
> \int_\Omega \phi f_0(u, u_t, \nabla u, x, t) + \nabla\phi \cdot {\vec
> f}_1(u, u_t, \nabla u, x, t)
>
> So f0 is, for the Poisson problem, zero, and f1 is just -grad u?


No. f1 is indeed -grad u, but f0 is -f.

Note that ex12 solves -nabla^2 u = f since that operator is positive
definite.


> Do we
> need to do anything with the surface term, or is that handled
> internally?


The surface term can be included using


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscDSSetBdResidual.html

and we use that in ex12 for inhomogeneous Neumann conditions.


> Because neither of these terms are a surface integral term,
> both are volume integrals.
>
> I can't figure out the purpose of g0, g1, g2, g3 in
> PetscDSSetJacobian(), either. Is g supposed to be the source term?


No, it is the coefficient of the different combinations of basis functions
and their gradients.
For instance, the g3 term (grad phi and grad psi), g3 plays the same role
as the elasticity
tensor C in mechanics.


> Is
> psi a second basis function?


Yes. If the method is Galerkin, it comes from the same space as phi.


> What even *is* the Jacobian in a FEM
> problem?


The Frechet derivative of the residual.


> I've only seen them formulated as linear problems in tutorials,
> so I'm not even clear why we need a nonlinear solver or Jacobian...
>

The Frechet derivative of a linear operator is itself. ex12 also
incorporates
nonlinear variants of Poisson.


> Any help is appreciated. I like the idea of a general framework for FEM,
> as I think it will reduce human error on my part, but I also want to
> understand what I'm doing.
>

If you have any more questions, feel free to keep mailing.

  Thanks,

     Matt


> Ellen Price
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/4cabeb0f/attachment.html>

From knepley at gmail.com  Mon Oct  8 17:46:59 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 8 Oct 2018 18:46:59 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
Message-ID: <CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>

On Mon, Oct 8, 2018 at 6:13 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> Sorry I was caught up with midterms for the last few days. I tried the lu
> decomposition today and the 2-norm is pretty stable at ~ 10^-15, which is
> expected for double precision. Since the discretization error is so small,
> it would be reasonable to assume the rest is majority the algebraic error.
>

What are you plotting? It looks like only the algebraic error or residual.
There is absolutely no way your discretization error is 1e-14.

  Thanks,

    Matt


> Then looking at the result without the -pc_type lu flag(second graph), the
> error is asymptoting to a constant several magnitudes larger than the
> tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
> behavior? Shouldn't it decrease with finer grid?
> [image: LU.png]
> [image: Total.png]
>
> On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> I didn't specify a tolerance, it was using the default tolerance.
>>> Doesn't the asymptoting norm implies finer grid won't help to get finer
>>> solution?
>>>
>>
>> There are two things going on in your test, discretization error
>> controlled by the grid, and algebraic error controlled by the solver. This
>> makes it difficult to isolate what is happening. However, it seems clear
>> that your plot is looking at algebraic error. You can confirm this by using
>>
>>   -pc_type lu
>>
>> for the solve. Then all you have is discretization error.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Mark Adams <mfadams at lbl.gov> ?
>>>
>>>>
>>>>
>>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> Yes I was using one norm in my Helmholtz code, the example code used 2
>>>>> norm. But now I am using 2 norm in both code.
>>>>>
>>>>>   /*
>>>>>      Check the error
>>>>>   */
>>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>>
>>>>>  I made a plot to show the increase:
>>>>>
>>>>
>>>>
>>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>>> you using?
>>>>
>>>>
>>>>>
>>>>> [image: Norm comparison.png]
>>>>>
>>>>> Mark Adams <mfadams at lbl.gov>?
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> The example code and makefile are attached below. The whole thing
>>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>>> )
>>>>>>>
>>>>>>
>>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>>
>>>>>>
>>>>>>> increases as I use finer and finer grids.
>>>>>>>
>>>>>>
>>>>>> What was the rate of increase?
>>>>>>
>>>>>>
>>>>>>> Then I looked at the example 12 (Laplacian solver) which is similar
>>>>>>> to what I did to see if I have missed something. The example is using
>>>>>>> 2_norm. I have made some minor modifications (3 places) on the code, you
>>>>>>> can search 'Modified' in the code to see them.
>>>>>>>
>>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>>
>>>>>>> Thanks for all your reply!
>>>>>>>
>>>>>>> Weizhuo
>>>>>>>
>>>>>>>
>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>>
>>>>>>>
>>>>>>>>    Please send your version of the example that computes the mean
>>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>>
>>>>>>>>    Barry
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>>> grids, the trend of norm still increases.
>>>>>>>> >
>>>>>>>> > Thanks!
>>>>>>>> >
>>>>>>>> > Weizhuo
>>>>>>>> >
>>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <
>>>>>>>> weizhuo2 at illinois.edu> wrote:
>>>>>>>> > Hi!
>>>>>>>> >
>>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>>> grid. Am I doing this wrong?
>>>>>>>> >
>>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>>> the sqrt of the sum of the squares of
>>>>>>>> > the vector entries. It should be scaled by the volume element to
>>>>>>>> approximate a scale-independent
>>>>>>>> > norm (like the L_2 norm).
>>>>>>>> >
>>>>>>>> >   Thanks,
>>>>>>>> >
>>>>>>>> >      Matt
>>>>>>>> >
>>>>>>>> > Thanks!
>>>>>>>> > --
>>>>>>>> > Wang Weizhuo
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>> experiments lead.
>>>>>>>> > -- Norbert Wiener
>>>>>>>> >
>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Wang Weizhuo
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Wang Weizhuo
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Wang Weizhuo
>>>>>
>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/3cf54368/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/3cf54368/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/3cf54368/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/3cf54368/attachment-0005.png>

From mfadams at lbl.gov  Mon Oct  8 17:58:05 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 8 Oct 2018 18:58:05 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
Message-ID: <CADOhEh53G4grNFqEApvuyHWOKFii6V7WLPhVqzp771LpiFY9Fw@mail.gmail.com>

And what is the x-axis? And what solver (preconditioner) are you using w/o
LU (2nd graph)?

On Mon, Oct 8, 2018 at 6:47 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 8, 2018 at 6:13 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> Sorry I was caught up with midterms for the last few days. I tried the lu
>> decomposition today and the 2-norm is pretty stable at ~ 10^-15, which is
>> expected for double precision. Since the discretization error is so small,
>> it would be reasonable to assume the rest is majority the algebraic error.
>>
>
> What are you plotting? It looks like only the algebraic error or residual.
> There is absolutely no way your discretization error is 1e-14.
>
>   Thanks,
>
>     Matt
>
>
>> Then looking at the result without the -pc_type lu flag(second graph),
>> the error is asymptoting to a constant several magnitudes larger than the
>> tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
>> behavior? Shouldn't it decrease with finer grid?
>> [image: LU.png]
>> [image: Total.png]
>>
>> On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> I didn't specify a tolerance, it was using the default tolerance.
>>>> Doesn't the asymptoting norm implies finer grid won't help to get finer
>>>> solution?
>>>>
>>>
>>> There are two things going on in your test, discretization error
>>> controlled by the grid, and algebraic error controlled by the solver. This
>>> makes it difficult to isolate what is happening. However, it seems clear
>>> that your plot is looking at algebraic error. You can confirm this by using
>>>
>>>   -pc_type lu
>>>
>>> for the solve. Then all you have is discretization error.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Mark Adams <mfadams at lbl.gov> ?
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> Yes I was using one norm in my Helmholtz code, the example code used
>>>>>> 2 norm. But now I am using 2 norm in both code.
>>>>>>
>>>>>>   /*
>>>>>>      Check the error
>>>>>>   */
>>>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>>>
>>>>>>  I made a plot to show the increase:
>>>>>>
>>>>>
>>>>>
>>>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>>>> you using?
>>>>>
>>>>>
>>>>>>
>>>>>> [image: Norm comparison.png]
>>>>>>
>>>>>> Mark Adams <mfadams at lbl.gov>?
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The example code and makefile are attached below. The whole thing
>>>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>>>> )
>>>>>>>>
>>>>>>>
>>>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>>>
>>>>>>>
>>>>>>>> increases as I use finer and finer grids.
>>>>>>>>
>>>>>>>
>>>>>>> What was the rate of increase?
>>>>>>>
>>>>>>>
>>>>>>>> Then I looked at the example 12 (Laplacian solver) which is similar
>>>>>>>> to what I did to see if I have missed something. The example is using
>>>>>>>> 2_norm. I have made some minor modifications (3 places) on the code, you
>>>>>>>> can search 'Modified' in the code to see them.
>>>>>>>>
>>>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>>>
>>>>>>>> Thanks for all your reply!
>>>>>>>>
>>>>>>>> Weizhuo
>>>>>>>>
>>>>>>>>
>>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>>>
>>>>>>>>
>>>>>>>>>    Please send your version of the example that computes the mean
>>>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>>>
>>>>>>>>>    Barry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>>>> grids, the trend of norm still increases.
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> >
>>>>>>>>> > Weizhuo
>>>>>>>>> >
>>>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <
>>>>>>>>> weizhuo2 at illinois.edu> wrote:
>>>>>>>>> > Hi!
>>>>>>>>> >
>>>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>>>> grid. Am I doing this wrong?
>>>>>>>>> >
>>>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>>>> the sqrt of the sum of the squares of
>>>>>>>>> > the vector entries. It should be scaled by the volume element to
>>>>>>>>> approximate a scale-independent
>>>>>>>>> > norm (like the L_2 norm).
>>>>>>>>> >
>>>>>>>>> >   Thanks,
>>>>>>>>> >
>>>>>>>>> >      Matt
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> > --
>>>>>>>>> > Wang Weizhuo
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> > -- Norbert Wiener
>>>>>>>>> >
>>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Wang Weizhuo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Wang Weizhuo
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wang Weizhuo
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/081ef19d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/081ef19d/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/081ef19d/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/081ef19d/attachment-0005.png>

From weizhuo2 at illinois.edu  Mon Oct  8 17:57:55 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 8 Oct 2018 17:57:55 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
Message-ID: <CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>

The first plot is the norm with the flag -pc_type lu with respect to number
of grids in one axis (n), and the second plot is the norm without the flag
-pc_type lu.

Specifically, the norm is calculated by

       norm = norm_2(numerical - analytic)/(n*n)

On Mon, Oct 8, 2018 at 5:47 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 8, 2018 at 6:13 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> Sorry I was caught up with midterms for the last few days. I tried the lu
>> decomposition today and the 2-norm is pretty stable at ~ 10^-15, which is
>> expected for double precision. Since the discretization error is so small,
>> it would be reasonable to assume the rest is majority the algebraic error.
>>
>
> What are you plotting? It looks like only the algebraic error or residual.
> There is absolutely no way your discretization error is 1e-14.
>
>   Thanks,
>
>     Matt
>
>
>> Then looking at the result without the -pc_type lu flag(second graph),
>> the error is asymptoting to a constant several magnitudes larger than the
>> tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
>> behavior? Shouldn't it decrease with finer grid?
>> [image: LU.png]
>> [image: Total.png]
>>
>> On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> I didn't specify a tolerance, it was using the default tolerance.
>>>> Doesn't the asymptoting norm implies finer grid won't help to get finer
>>>> solution?
>>>>
>>>
>>> There are two things going on in your test, discretization error
>>> controlled by the grid, and algebraic error controlled by the solver. This
>>> makes it difficult to isolate what is happening. However, it seems clear
>>> that your plot is looking at algebraic error. You can confirm this by using
>>>
>>>   -pc_type lu
>>>
>>> for the solve. Then all you have is discretization error.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Mark Adams <mfadams at lbl.gov> ?
>>>>
>>>>>
>>>>>
>>>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> Yes I was using one norm in my Helmholtz code, the example code used
>>>>>> 2 norm. But now I am using 2 norm in both code.
>>>>>>
>>>>>>   /*
>>>>>>      Check the error
>>>>>>   */
>>>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>>>
>>>>>>  I made a plot to show the increase:
>>>>>>
>>>>>
>>>>>
>>>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>>>> you using?
>>>>>
>>>>>
>>>>>>
>>>>>> [image: Norm comparison.png]
>>>>>>
>>>>>> Mark Adams <mfadams at lbl.gov>?
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The example code and makefile are attached below. The whole thing
>>>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>>>> )
>>>>>>>>
>>>>>>>
>>>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>>>
>>>>>>>
>>>>>>>> increases as I use finer and finer grids.
>>>>>>>>
>>>>>>>
>>>>>>> What was the rate of increase?
>>>>>>>
>>>>>>>
>>>>>>>> Then I looked at the example 12 (Laplacian solver) which is similar
>>>>>>>> to what I did to see if I have missed something. The example is using
>>>>>>>> 2_norm. I have made some minor modifications (3 places) on the code, you
>>>>>>>> can search 'Modified' in the code to see them.
>>>>>>>>
>>>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>>>
>>>>>>>> Thanks for all your reply!
>>>>>>>>
>>>>>>>> Weizhuo
>>>>>>>>
>>>>>>>>
>>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>>>
>>>>>>>>
>>>>>>>>>    Please send your version of the example that computes the mean
>>>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>>>
>>>>>>>>>    Barry
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>>>> grids, the trend of norm still increases.
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> >
>>>>>>>>> > Weizhuo
>>>>>>>>> >
>>>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <
>>>>>>>>> weizhuo2 at illinois.edu> wrote:
>>>>>>>>> > Hi!
>>>>>>>>> >
>>>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>>>> grid. Am I doing this wrong?
>>>>>>>>> >
>>>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>>>> the sqrt of the sum of the squares of
>>>>>>>>> > the vector entries. It should be scaled by the volume element to
>>>>>>>>> approximate a scale-independent
>>>>>>>>> > norm (like the L_2 norm).
>>>>>>>>> >
>>>>>>>>> >   Thanks,
>>>>>>>>> >
>>>>>>>>> >      Matt
>>>>>>>>> >
>>>>>>>>> > Thanks!
>>>>>>>>> > --
>>>>>>>>> > Wang Weizhuo
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > What most experimenters take for granted before they begin their
>>>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>>>> experiments lead.
>>>>>>>>> > -- Norbert Wiener
>>>>>>>>> >
>>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > --
>>>>>>>>> > Wang Weizhuo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Wang Weizhuo
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wang Weizhuo
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/889fbaa1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/889fbaa1/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/889fbaa1/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/889fbaa1/attachment-0005.png>

From weizhuo2 at illinois.edu  Mon Oct  8 18:10:57 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 8 Oct 2018 18:10:57 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CADOhEh53G4grNFqEApvuyHWOKFii6V7WLPhVqzp771LpiFY9Fw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CADOhEh53G4grNFqEApvuyHWOKFii6V7WLPhVqzp771LpiFY9Fw@mail.gmail.com>
Message-ID: <CABvwWnKUpXZd+vo6S6yyScxtmo_BM6XHLrrXwSdCX1f3VQ2edg@mail.gmail.com>

x-axis is n - the number of grids in one axis, for example if the grid is
100*100, then n=100. I believe the preconditioner is ksp, since it is set
from options of a default KSP solver. I can send you the code if you want
to take a glimpse at it.

On Mon, Oct 8, 2018 at 5:58 PM Mark Adams <mfadams at lbl.gov> wrote:

> And what is the x-axis? And what solver (preconditioner) are you using w/o
> LU (2nd graph)?
>
> On Mon, Oct 8, 2018 at 6:47 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Oct 8, 2018 at 6:13 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Sorry I was caught up with midterms for the last few days. I tried the
>>> lu decomposition today and the 2-norm is pretty stable at ~ 10^-15, which
>>> is expected for double precision. Since the discretization error is so
>>> small, it would be reasonable to assume the rest is majority the algebraic
>>> error.
>>>
>>
>> What are you plotting? It looks like only the algebraic error or
>> residual. There is absolutely no way your discretization error is 1e-14.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Then looking at the result without the -pc_type lu flag(second graph),
>>> the error is asymptoting to a constant several magnitudes larger than the
>>> tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
>>> behavior? Shouldn't it decrease with finer grid?
>>> [image: LU.png]
>>> [image: Total.png]
>>>
>>> On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> I didn't specify a tolerance, it was using the default tolerance.
>>>>> Doesn't the asymptoting norm implies finer grid won't help to get finer
>>>>> solution?
>>>>>
>>>>
>>>> There are two things going on in your test, discretization error
>>>> controlled by the grid, and algebraic error controlled by the solver. This
>>>> makes it difficult to isolate what is happening. However, it seems clear
>>>> that your plot is looking at algebraic error. You can confirm this by using
>>>>
>>>>   -pc_type lu
>>>>
>>>> for the solve. Then all you have is discretization error.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Mark Adams <mfadams at lbl.gov> ?
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes I was using one norm in my Helmholtz code, the example code used
>>>>>>> 2 norm. But now I am using 2 norm in both code.
>>>>>>>
>>>>>>>   /*
>>>>>>>      Check the error
>>>>>>>   */
>>>>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>>>>
>>>>>>>  I made a plot to show the increase:
>>>>>>>
>>>>>>
>>>>>>
>>>>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>>>>> you using?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> [image: Norm comparison.png]
>>>>>>>
>>>>>>> Mark Adams <mfadams at lbl.gov>?
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The example code and makefile are attached below. The whole thing
>>>>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>>>>
>>>>>>>>
>>>>>>>>> increases as I use finer and finer grids.
>>>>>>>>>
>>>>>>>>
>>>>>>>> What was the rate of increase?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Then I looked at the example 12 (Laplacian solver) which is
>>>>>>>>> similar to what I did to see if I have missed something. The example is
>>>>>>>>> using 2_norm. I have made some minor modifications (3 places) on the code,
>>>>>>>>> you can search 'Modified' in the code to see them.
>>>>>>>>>
>>>>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>>>>
>>>>>>>>> Thanks for all your reply!
>>>>>>>>>
>>>>>>>>> Weizhuo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>    Please send your version of the example that computes the mean
>>>>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>>>>
>>>>>>>>>>    Barry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >
>>>>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>>>>> grids, the trend of norm still increases.
>>>>>>>>>> >
>>>>>>>>>> > Thanks!
>>>>>>>>>> >
>>>>>>>>>> > Weizhuo
>>>>>>>>>> >
>>>>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <
>>>>>>>>>> weizhuo2 at illinois.edu> wrote:
>>>>>>>>>> > Hi!
>>>>>>>>>> >
>>>>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>>>>> grid. Am I doing this wrong?
>>>>>>>>>> >
>>>>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>>>>> the sqrt of the sum of the squares of
>>>>>>>>>> > the vector entries. It should be scaled by the volume element
>>>>>>>>>> to approximate a scale-independent
>>>>>>>>>> > norm (like the L_2 norm).
>>>>>>>>>> >
>>>>>>>>>> >   Thanks,
>>>>>>>>>> >
>>>>>>>>>> >      Matt
>>>>>>>>>> >
>>>>>>>>>> > Thanks!
>>>>>>>>>> > --
>>>>>>>>>> > Wang Weizhuo
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > Wang Weizhuo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Wang Weizhuo
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Wang Weizhuo
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Wang Weizhuo
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=iJKTzLwSz91kwSX1J06Rddg52LnTKFxnqz9HZLA_KCk&s=bDOGCOeG7amWuKES2hcckRWXl8dp01UDiunq2BCVd0A&e=>
>>>>
>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=iJKTzLwSz91kwSX1J06Rddg52LnTKFxnqz9HZLA_KCk&s=bDOGCOeG7amWuKES2hcckRWXl8dp01UDiunq2BCVd0A&e=>
>>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/55bc4882/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/55bc4882/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/55bc4882/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/55bc4882/attachment-0005.png>

From knepley at gmail.com  Mon Oct  8 19:25:35 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 8 Oct 2018 20:25:35 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
Message-ID: <CAMYG4GnU1Neqr1=oz-ese__41VJ72hShc8fB5bJG6Q3LvznEkA@mail.gmail.com>

On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> The first plot is the norm with the flag -pc_type lu with respect to
> number of grids in one axis (n), and the second plot is the norm without
> the flag -pc_type lu.
>
> Specifically, the norm is calculated by
>
>        norm = norm_2(numerical - analytic)/(n*n)
>

I do not believe you are calculating this correctly. There is no way your
discretisation error is that low.

   Matt


> On Mon, Oct 8, 2018 at 5:47 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Oct 8, 2018 at 6:13 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Sorry I was caught up with midterms for the last few days. I tried the
>>> lu decomposition today and the 2-norm is pretty stable at ~ 10^-15, which
>>> is expected for double precision. Since the discretization error is so
>>> small, it would be reasonable to assume the rest is majority the algebraic
>>> error.
>>>
>>
>> What are you plotting? It looks like only the algebraic error or
>> residual. There is absolutely no way your discretization error is 1e-14.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Then looking at the result without the -pc_type lu flag(second graph),
>>> the error is asymptoting to a constant several magnitudes larger than the
>>> tolerance set for the solver. (atol=1e-12, rtol=1e-9) Is this the expected
>>> behavior? Shouldn't it decrease with finer grid?
>>> [image: LU.png]
>>> [image: Total.png]
>>>
>>> On Tue, Oct 2, 2018 at 6:52 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Oct 2, 2018 at 5:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> I didn't specify a tolerance, it was using the default tolerance.
>>>>> Doesn't the asymptoting norm implies finer grid won't help to get finer
>>>>> solution?
>>>>>
>>>>
>>>> There are two things going on in your test, discretization error
>>>> controlled by the grid, and algebraic error controlled by the solver. This
>>>> makes it difficult to isolate what is happening. However, it seems clear
>>>> that your plot is looking at algebraic error. You can confirm this by using
>>>>
>>>>   -pc_type lu
>>>>
>>>> for the solve. Then all you have is discretization error.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Mark Adams <mfadams at lbl.gov> ?
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 2, 2018 at 5:04 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> Yes I was using one norm in my Helmholtz code, the example code used
>>>>>>> 2 norm. But now I am using 2 norm in both code.
>>>>>>>
>>>>>>>   /*
>>>>>>>      Check the error
>>>>>>>   */
>>>>>>>   ierr = VecAXPY(x,-1.0,u); CHKERRQ(ierr);
>>>>>>>   ierr = VecNorm(x,NORM_1,&norm); CHKERRQ(ierr);
>>>>>>>   ierr = KSPGetIterationNumber(ksp,&its); CHKERRQ(ierr);
>>>>>>>   ierr = PetscPrintf(PETSC_COMM_WORLD,"Norm of error %g iterations
>>>>>>> %D\n",(double)norm/(m*n),its); CHKERRQ(ierr);
>>>>>>>
>>>>>>>  I made a plot to show the increase:
>>>>>>>
>>>>>>
>>>>>>
>>>>>> FYI, this is asymptoting to a constant.  What solver tolerance are
>>>>>> you using?
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> [image: Norm comparison.png]
>>>>>>>
>>>>>>> Mark Adams <mfadams at lbl.gov>?
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Oct 2, 2018 at 2:24 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The example code and makefile are attached below. The whole thing
>>>>>>>>> started as I tried to build a Helmholtz solver, and the mean error
>>>>>>>>> (calculated by: sum( | numerical_sol - analytical_sol | / analytical_sol )
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>
>>>>>>>> This is a one norm. If you use max (instead of sum) then you don't
>>>>>>>> need to scale. You do have to be careful about dividing by (near) zero.
>>>>>>>>
>>>>>>>>
>>>>>>>>> increases as I use finer and finer grids.
>>>>>>>>>
>>>>>>>>
>>>>>>>> What was the rate of increase?
>>>>>>>>
>>>>>>>>
>>>>>>>>> Then I looked at the example 12 (Laplacian solver) which is
>>>>>>>>> similar to what I did to see if I have missed something. The example is
>>>>>>>>> using 2_norm. I have made some minor modifications (3 places) on the code,
>>>>>>>>> you can search 'Modified' in the code to see them.
>>>>>>>>>
>>>>>>>>> If this helps: I configured the PETSc to use real and double
>>>>>>>>> precision. Changed the name of the example code from ex12.c to ex12c.c
>>>>>>>>>
>>>>>>>>> Thanks for all your reply!
>>>>>>>>>
>>>>>>>>> Weizhuo
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>    Please send your version of the example that computes the mean
>>>>>>>>>> norm of the grid; I suspect we are talking apples and oranges
>>>>>>>>>>
>>>>>>>>>>    Barry
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> > On Oct 1, 2018, at 7:51 PM, Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>>>> wrote:
>>>>>>>>>> >
>>>>>>>>>> > I also tried to divide the norm by m*n , which is the number of
>>>>>>>>>> grids, the trend of norm still increases.
>>>>>>>>>> >
>>>>>>>>>> > Thanks!
>>>>>>>>>> >
>>>>>>>>>> > Weizhuo
>>>>>>>>>> >
>>>>>>>>>> > Matthew Knepley <knepley at gmail.com>
>>>>>>>>>> > On Mon, Oct 1, 2018 at 6:31 PM Weizhuo Wang <
>>>>>>>>>> weizhuo2 at illinois.edu> wrote:
>>>>>>>>>> > Hi!
>>>>>>>>>> >
>>>>>>>>>> > I'm recently trying out the example code provided with the KSP
>>>>>>>>>> solver (ex12.c). I noticed that the mean norm of the grid increases as I
>>>>>>>>>> use finer meshes. For example, the mean norm is 5.72e-8 at m=10 n=10.
>>>>>>>>>> However at m=100, n=100, mean norm increases to 9.55e-6. This seems counter
>>>>>>>>>> intuitive, since most of the time error should decreases when using finer
>>>>>>>>>> grid. Am I doing this wrong?
>>>>>>>>>> >
>>>>>>>>>> > The norm is misleading in that it is the l_2 norm, meaning just
>>>>>>>>>> the sqrt of the sum of the squares of
>>>>>>>>>> > the vector entries. It should be scaled by the volume element
>>>>>>>>>> to approximate a scale-independent
>>>>>>>>>> > norm (like the L_2 norm).
>>>>>>>>>> >
>>>>>>>>>> >   Thanks,
>>>>>>>>>> >
>>>>>>>>>> >      Matt
>>>>>>>>>> >
>>>>>>>>>> > Thanks!
>>>>>>>>>> > --
>>>>>>>>>> > Wang Weizhuo
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > What most experimenters take for granted before they begin
>>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>>> their experiments lead.
>>>>>>>>>> > -- Norbert Wiener
>>>>>>>>>> >
>>>>>>>>>> > https://www.cse.buffalo.edu/~knepley/
>>>>>>>>>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=KjmDEsZ6w8LEry7nlv3Bw7-pczqWbKGueFU59VoIWZg&s=tEv9-AHhL2CIlmmVos0gFa5PAY9oMG3aTQlnfi62ivA&e=>
>>>>>>>>>> >
>>>>>>>>>> >
>>>>>>>>>> > --
>>>>>>>>>> > Wang Weizhuo
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Wang Weizhuo
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Wang Weizhuo
>>>>>>>
>>>>>>
>>>>>
>>>>> --
>>>>> Wang Weizhuo
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/2d8df6d6/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Norm comparison.png
Type: image/png
Size: 87887 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/2d8df6d6/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: LU.png
Type: image/png
Size: 38656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/2d8df6d6/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Total.png
Type: image/png
Size: 39046 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/2d8df6d6/attachment-0005.png>

From ellen.price at cfa.harvard.edu  Mon Oct  8 19:41:15 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Mon, 8 Oct 2018 20:41:15 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
	<CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
Message-ID: <dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>

Hi Matt,

Thanks for the speedy reply. I do have a few more questions/elaborations
on my previous correspondence.

As for the source term (and the boundary condition, actually), both are
only *known* at mesh points, but are assumed continuous everywhere. How
does this change things? Basically, I want to assume a linear
interpolation (I'm using linear elements) of the source term over each
tetrahedral cell and then the integral should be some linear combination
of mesh point values, with coefficients given by the inner product of
basis functions? The boundary term is actually given by a very nasty
integral equation, which is probably the most expensive part of the
simulation. So I only want to evaluate it at mesh points if possible.

You are correct, I forgot the \cdot n in that equation. Oops!

I need to think more about the Jacobian and residual before I can really
ask an intelligent question about them. I suppose that in practice, I
need to figure out exactly which Jacobian terms I need to include, and
which can be NULL. Some guidance on that would be immensely helpful.

If it helps, the equation I'm solving is just the Newtonian
gravitational potential equation (on a nontrivial mesh and with a
nontrivial, non-analytic mass density function).

Ellen


On 10/08/2018 06:28 PM, Matthew Knepley wrote:
> On Mon, Oct 8, 2018 at 5:50 PM Ellen M. Price
> <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>> wrote:
> 
>     Can anyone shed some light on SNES example 12?
> 
>     https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex12.c.html
> 
>     I am solving a Poisson problem, but with a source term defined on mesh
>     points, so I don't have an analytic form.
> 
> 
> Can you explain more about this source term? It sounds like a bunch of
> delta functions. That
> would still work in this framework, but the convergence rate for a rhs
> with these singularities
> is reduced (this is a generic feature of FEM).
> ?
> 
>     The part I am confused about
>     is the role of the functions f0, f1, and g3 that go into the DS.
> 
>     Suppose I have the Poisson problem nabla^2 u = f. Then the weak form is,
>     for test function phi,
> 
>     \int_{\partial \Omega} \phi \nabla u - \int_\Omega \nabla u \cdot \nabla
>     \phi = \int_\Omega \phi f
> 
> 
> I think you need \int_{\partial \Omega} \phi \nabla u \cdot n instead.
> ?
> 
>     Based on the documentation of PetscDSSetResidual(), I'm a little
>     confused about f0 and f1:
> 
>     \int_\Omega \phi f_0(u, u_t, \nabla u, x, t) + \nabla\phi \cdot {\vec
>     f}_1(u, u_t, \nabla u, x, t)
> 
>     So f0 is, for the Poisson problem, zero, and f1 is just -grad u?
> 
> 
> No. f1 is indeed -grad u, but f0 is -f.
> 
> Note that ex12 solves -nabla^2 u = f since that operator is positive
> definite.
> ?
> 
>     Do we
>     need to do anything with the surface term, or is that handled
>     internally?
> 
> 
> The surface term can be included using
> 
> ??https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscDSSetBdResidual.html
> 
> and we use that in ex12 for inhomogeneous Neumann conditions.
> ?
> 
>     Because neither of these terms are a surface integral term,
>     both are volume integrals.
> 
>     I can't figure out the purpose of g0, g1, g2, g3 in
>     PetscDSSetJacobian(), either. Is g supposed to be the source term?
> 
> 
> No, it is the coefficient of the different combinations of basis
> functions and their gradients.
> For instance, the g3 term (grad phi and grad psi), g3 plays the same
> role as the elasticity
> tensor C in mechanics.
> ?
> 
>     Is
>     psi a second basis function?
> 
> 
> Yes. If the method is Galerkin, it comes from the same space as phi.
> ?
> 
>     What even *is* the Jacobian in a FEM
>     problem?
> 
> 
> The Frechet derivative of the residual.
> ?
> 
>     I've only seen them formulated as linear problems in tutorials,
>     so I'm not even clear why we need a nonlinear solver or Jacobian...
> 
> 
> The Frechet derivative of a linear operator is itself. ex12 also
> incorporates
> nonlinear variants of Poisson.
> ?
> 
>     Any help is appreciated. I like the idea of a general framework for FEM,
>     as I think it will reduce human error on my part, but I also want to
>     understand what I'm doing.
> 
> 
> If you have any more questions, feel free to keep mailing.
> 
> ? Thanks,
> 
> ? ? ?Matt
> ?
> 
>     Ellen Price
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

From mfadams at lbl.gov  Mon Oct  8 19:58:09 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 8 Oct 2018 20:58:09 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
Message-ID: <CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>

On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> The first plot is the norm with the flag -pc_type lu with respect to
> number of grids in one axis (n), and the second plot is the norm without
> the flag -pc_type lu.
>

So you are using the default PC w/o LU. The default is ILU. This will
reduce high frequency effectively but is not effective on the low frequency
error. Don't expect your algebraic error reduction to be at the same scale
as the residual reduction (what KSP measures).


>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/db7d40fb/attachment.html>

From weizhuo2 at illinois.edu  Mon Oct  8 20:28:44 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 8 Oct 2018 20:28:44 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
Message-ID: <CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>

The code is attached in case anyone wants to take a look, I will try the
high frequency scenario later.

On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:

>
>
> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> The first plot is the norm with the flag -pc_type lu with respect to
>> number of grids in one axis (n), and the second plot is the norm without
>> the flag -pc_type lu.
>>
>
> So you are using the default PC w/o LU. The default is ILU. This will
> reduce high frequency effectively but is not effective on the low frequency
> error. Don't expect your algebraic error reduction to be at the same scale
> as the residual reduction (what KSP measures).
>
>
>>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/cb838652/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: makefile
Type: application/octet-stream
Size: 959 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/cb838652/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Helmholtz_demoV2C.cpp
Type: application/octet-stream
Size: 8084 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/cb838652/attachment-0003.obj>

From knepley at gmail.com  Mon Oct  8 20:30:28 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 8 Oct 2018 21:30:28 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
	<CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
	<dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>
Message-ID: <CAMYG4GmNWxGMWABOx3kBwekh_Cwk5W2ikWgG9Ho=kkO3=jm6VQ@mail.gmail.com>

On Mon, Oct 8, 2018 at 8:41 PM Ellen M. Price <ellen.price at cfa.harvard.edu>
wrote:

> Hi Matt,
>
> Thanks for the speedy reply. I do have a few more questions/elaborations
> on my previous correspondence.
>
> As for the source term (and the boundary condition, actually), both are
> only *known* at mesh points, but are assumed continuous everywhere. How
> does this change things? Basically, I want to assume a linear
> interpolation (I'm using linear elements) of the source term over each
> tetrahedral cell and then the integral should be some linear combination
> of mesh point values, with coefficients given by the inner product of
> basis functions?


Its a little bit strange to give a forcing term this way. It means that you
do not know
your problem independent of your discretization. For example, ex12 can use
quadratic
elements, or cubic, or DG, so it does not assume any discretization when
specifying the
problem.

You can do what you want, namely just pass in a rhs vector. However, that
also means you
need to understand the mesh ordering, dof ordering, etc. Its not a great
way to proceed.

How do you create these "mesh point" values in the first place?


> The boundary term is actually given by a very nasty
> integral equation, which is probably the most expensive part of the
> simulation. So I only want to evaluate it at mesh points if possible.
>

In order to maintain accuracy, you should really evaluate it at quadrature
points (you can always
choose a low-order quadrature if you don't care). You can look at the
Neumann term I use


https://bitbucket.org/petsc/petsc/src/760cff7d6b13b58cab0db4b57ca0fc45c2034c72/src/snes/examples/tutorials/ex12.c#lines-159


> You are correct, I forgot the \cdot n in that equation. Oops!
>
> I need to think more about the Jacobian and residual before I can really
> ask an intelligent question about them. I suppose that in practice, I
> need to figure out exactly which Jacobian terms I need to include, and
> which can be NULL. Some guidance on that would be immensely helpful.
>

Put them all in. Always check the full physics and then use approximations
as preconditioners.


> If it helps, the equation I'm solving is just the Newtonian
> gravitational potential equation (on a nontrivial mesh and with a
> nontrivial, non-analytic mass density function).
>

Hmm, FEM is not great for such rough source terms. At least not without
some kind of nice adaptivity.

Also, I assume you are doing a boundary integral for boundary conditions.
This is quickly going to
dominate the computational cost. I would figure out my strategy for that
pretty early.

  Thanks,

     Matt


> Ellen
>
>
> On 10/08/2018 06:28 PM, Matthew Knepley wrote:
> > On Mon, Oct 8, 2018 at 5:50 PM Ellen M. Price
> > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>>
> wrote:
> >
> >     Can anyone shed some light on SNES example 12?
> >
> >
> https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex12.c.html
> >
> >     I am solving a Poisson problem, but with a source term defined on
> mesh
> >     points, so I don't have an analytic form.
> >
> >
> > Can you explain more about this source term? It sounds like a bunch of
> > delta functions. That
> > would still work in this framework, but the convergence rate for a rhs
> > with these singularities
> > is reduced (this is a generic feature of FEM).
> >
> >
> >     The part I am confused about
> >     is the role of the functions f0, f1, and g3 that go into the DS.
> >
> >     Suppose I have the Poisson problem nabla^2 u = f. Then the weak form
> is,
> >     for test function phi,
> >
> >     \int_{\partial \Omega} \phi \nabla u - \int_\Omega \nabla u \cdot
> \nabla
> >     \phi = \int_\Omega \phi f
> >
> >
> > I think you need \int_{\partial \Omega} \phi \nabla u \cdot n instead.
> >
> >
> >     Based on the documentation of PetscDSSetResidual(), I'm a little
> >     confused about f0 and f1:
> >
> >     \int_\Omega \phi f_0(u, u_t, \nabla u, x, t) + \nabla\phi \cdot {\vec
> >     f}_1(u, u_t, \nabla u, x, t)
> >
> >     So f0 is, for the Poisson problem, zero, and f1 is just -grad u?
> >
> >
> > No. f1 is indeed -grad u, but f0 is -f.
> >
> > Note that ex12 solves -nabla^2 u = f since that operator is positive
> > definite.
> >
> >
> >     Do we
> >     need to do anything with the surface term, or is that handled
> >     internally?
> >
> >
> > The surface term can be included using
> >
> >
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DT/PetscDSSetBdResidual.html
> >
> > and we use that in ex12 for inhomogeneous Neumann conditions.
> >
> >
> >     Because neither of these terms are a surface integral term,
> >     both are volume integrals.
> >
> >     I can't figure out the purpose of g0, g1, g2, g3 in
> >     PetscDSSetJacobian(), either. Is g supposed to be the source term?
> >
> >
> > No, it is the coefficient of the different combinations of basis
> > functions and their gradients.
> > For instance, the g3 term (grad phi and grad psi), g3 plays the same
> > role as the elasticity
> > tensor C in mechanics.
> >
> >
> >     Is
> >     psi a second basis function?
> >
> >
> > Yes. If the method is Galerkin, it comes from the same space as phi.
> >
> >
> >     What even *is* the Jacobian in a FEM
> >     problem?
> >
> >
> > The Frechet derivative of the residual.
> >
> >
> >     I've only seen them formulated as linear problems in tutorials,
> >     so I'm not even clear why we need a nonlinear solver or Jacobian...
> >
> >
> > The Frechet derivative of a linear operator is itself. ex12 also
> > incorporates
> > nonlinear variants of Poisson.
> >
> >
> >     Any help is appreciated. I like the idea of a general framework for
> FEM,
> >     as I think it will reduce human error on my part, but I also want to
> >     understand what I'm doing.
> >
> >
> > If you have any more questions, feel free to keep mailing.
> >
> >   Thanks,
> >
> >      Matt
> >
> >
> >     Ellen Price
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/%7Eknepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/5f45fd80/attachment.html>

From knepley at gmail.com  Mon Oct  8 20:33:33 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 8 Oct 2018 21:33:33 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
Message-ID: <CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>

On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> The code is attached in case anyone wants to take a look, I will try the
> high frequency scenario later.
>

That is not the error. It is superconvergence at the vertices. The real
solution is trigonometric, so your
linear interpolants or whatever you use is not going to get the right value
in between mesh points. You
need to do a real integral over the whole interval to get the L_2 error.

  Thanks,

     Matt


> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>
>>
>>
>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> The first plot is the norm with the flag -pc_type lu with respect to
>>> number of grids in one axis (n), and the second plot is the norm without
>>> the flag -pc_type lu.
>>>
>>
>> So you are using the default PC w/o LU. The default is ILU. This will
>> reduce high frequency effectively but is not effective on the low frequency
>> error. Don't expect your algebraic error reduction to be at the same scale
>> as the residual reduction (what KSP measures).
>>
>>
>>>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181008/a1c8af0c/attachment.html>

From yjwu16 at gmail.com  Mon Oct  8 22:32:37 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Tue, 9 Oct 2018 11:32:37 +0800
Subject: [petsc-users] Problems about Block Jocabi and Matrix-Free Method in
 SNES
Message-ID: <CAAMEhBimeGsge9x7DH4iCpb7SwgrfViV_2gfcF5tKCrf4tzJQg@mail.gmail.com>

Dear Petsc developer:
Hi,

I've been studying Petsc recently about Precontioner and Metrix-Free, and I
have some questions that puzzle me.

1. I want to test block Jacobi preconditioner, so I chose
/snes/example/tutorial/ex3.c as an example. According to the reference in
the example, the input parameters are:
mpiexec -n 8./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
-pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -post_setsubksp
-sub_ksp_rtol 1.e-16
I want to export each block of KSP and PC information :
  -snes_view -ksp_view
However, the procedure is wrong and the wrong information is as follows:

       [0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Arguments must have same communicators
[0]PETSC ERROR: Different communicators in the two objects: Argument # 1
and 2 flag 3
[0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[3]PETSC ERROR: Arguments must have same communicators
[3]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[6]PETSC ERROR: Arguments must have same communicators
[6]PETSC ERROR: Different communicators in the two objects: Argument # 1
and 2 flag 3
[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[7]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[7]PETSC ERROR: Arguments must have same communicators
[7]PETSC ERROR: Different communicators in the two objects: Argument # 1
and 2 flag 3
[7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[7]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Mon Oct  8 22:35:34 2018
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[0]PETSC ERROR: #1 KSPView() line 118 in
/home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
[0]PETSC ERROR: #2 PCView_BJacobi() line 232 in
/home/yjwu/petsc-3.10.1/src/ksp/pc/impls/bjacobi/bjacobi.c
[0]PETSC ERROR: #3 PCView() line 1651 in
/home/yjwu/petsc-3.10.1/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: Arguments must have same communicators
[1]PETSC ERROR: Different communicators in the two objects: Argument # 1
and 2 flag 3
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Mon Oct  8 22:35:34 2018
[1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[1]PETSC ERROR: #1 KSPView() line 118 in
/home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
Arguments must have same communicators
[2]PETSC ERROR: Different communicators in the two objects: Argument # 1
and 2 flag 3
[2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.

If I want to get the information of KSP and PC in each block, what should I
do?

2. There is a requirement in my program to use Matrix-Free method to
approximate Jacobian matrix by finite difference of residual function in
solving nonlinear equations. But I'll also provide an analytic( incomplete,
some terms are missing or approximate) for preconditioning. Because my
problem is about a large set of equations composed of several physical
fields, I want to use block Jacobian precondition for each subfield(block),
and ILU sub_pc for each subfield. After reading the Users'Guide, I found
that using - snex_mf_operator can do the above, so I added:
 - snes_mf_operator
after the example above. However, the procedure is wrong and does not seem
to support this.
 The wrong information is:

   [1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[2]PETSC ERROR: No support for this operation for this object type
[2]PETSC ERROR: Mat type mffd
[2]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: --------------------- Error
Message --------------------------------------------------------------
[4]PETSC ERROR: No support for this operation for this object type
[4]PETSC ERROR: Mat type mffd
[4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[4]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[6]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[6]PETSC ERROR: No support for this operation for this object type
[6]PETSC ERROR: Mat type mffd
[6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[6]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[6]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Mon Oct  8 23:01:57 2018
[6]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: No support for this operation for this object type
[0]PETSC ERROR: Mat type mffd
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
[0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
Mon Oct  8 23:01:57 2018
[0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
--with-fc=gfortran --download-mpich --download-fblaslapack
[0]PETSC ERROR: #1 MatSetValues() line 1330 in
/home/yjwu/petsc-3.10.1/src/mat/interface/matrix.c
[1]PETSC ERROR: No support for this operation for this object type
[1]PETSC ERROR: Mat type mffd
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.

If I want to implement matrix-free and block jacobi precondition in my
program, what do I need to pay attention or add ?

The second question is very important for my research and I hope to get
your answer.

Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/36de62f5/attachment-0001.html>

From mfadams at lbl.gov  Tue Oct  9 02:50:01 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 9 Oct 2018 03:50:01 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
Message-ID: <CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>

To reiterate what Matt is saying, you seem to have the exact solution on a
10x10 grid. That makes no sense unless the solution can be represented
exactly by your FE space (eg, u(x,y) = x + y).

On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> The code is attached in case anyone wants to take a look, I will try the
>> high frequency scenario later.
>>
>
> That is not the error. It is superconvergence at the vertices. The real
> solution is trigonometric, so your
> linear interpolants or whatever you use is not going to get the right
> value in between mesh points. You
> need to do a real integral over the whole interval to get the L_2 error.
>
>   Thanks,
>
>      Matt
>
>
>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>>
>>>
>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> The first plot is the norm with the flag -pc_type lu with respect to
>>>> number of grids in one axis (n), and the second plot is the norm without
>>>> the flag -pc_type lu.
>>>>
>>>
>>> So you are using the default PC w/o LU. The default is ILU. This will
>>> reduce high frequency effectively but is not effective on the low frequency
>>> error. Don't expect your algebraic error reduction to be at the same scale
>>> as the residual reduction (what KSP measures).
>>>
>>>
>>>>
>>
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/a6ba6473/attachment.html>

From knepley at gmail.com  Tue Oct  9 10:34:19 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 9 Oct 2018 11:34:19 -0400
Subject: [petsc-users] Problems about Block Jocabi and Matrix-Free
 Method in SNES
In-Reply-To: <CAAMEhBimeGsge9x7DH4iCpb7SwgrfViV_2gfcF5tKCrf4tzJQg@mail.gmail.com>
References: <CAAMEhBimeGsge9x7DH4iCpb7SwgrfViV_2gfcF5tKCrf4tzJQg@mail.gmail.com>
Message-ID: <CAMYG4GkkevtfKEZmjvegcUA6y2J4NeU2P0tKfrZ+BS1NgryoUg@mail.gmail.com>

On Mon, Oct 8, 2018 at 11:33 PM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Dear Petsc developer:
> Hi,
>
> I've been studying Petsc recently about Precontioner and Metrix-Free, and
> I have some questions that puzzle me.
>
> 1. I want to test block Jacobi preconditioner, so I chose
> /snes/example/tutorial/ex3.c as an example. According to the reference in
> the example, the input parameters are:
> mpiexec -n 8./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
> -pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -post_setsubksp
> -sub_ksp_rtol 1.e-16
>

You do not care about recursive blocks, so just use

  $MPIEXEC -n 8 ./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
-pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -snes_view
-ksp_view

and I get the attached output.


> I want to export each block of KSP and PC information :
>   -snes_view -ksp_view
> However, the procedure is wrong and the wrong information is as follows:
>
>        [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Arguments must have same communicators
> [0]PETSC ERROR: Different communicators in the two objects: Argument # 1
> and 2 flag 3
> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3]PETSC ERROR: Arguments must have same communicators
> [3]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [6]PETSC ERROR: Arguments must have same communicators
> [6]PETSC ERROR: Different communicators in the two objects: Argument # 1
> and 2 flag 3
> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [7]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [7]PETSC ERROR: Arguments must have same communicators
> [7]PETSC ERROR: Different communicators in the two objects: Argument # 1
> and 2 flag 3
> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [7]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
> shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Mon Oct  8 22:35:34 2018
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [0]PETSC ERROR: #1 KSPView() line 118 in
> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
> [0]PETSC ERROR: #2 PCView_BJacobi() line 232 in
> /home/yjwu/petsc-3.10.1/src/ksp/pc/impls/bjacobi/bjacobi.c
> [0]PETSC ERROR: #3 PCView() line 1651 in
> /home/yjwu/petsc-3.10.1/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: Arguments must have same communicators
> [1]PETSC ERROR: Different communicators in the two objects: Argument # 1
> and 2 flag 3
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Mon Oct  8 22:35:34 2018
> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [1]PETSC ERROR: #1 KSPView() line 118 in
> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
> Arguments must have same communicators
> [2]PETSC ERROR: Different communicators in the two objects: Argument # 1
> and 2 flag 3
> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> If I want to get the information of KSP and PC in each block, what should
> I do?
>
> 2. There is a requirement in my program to use Matrix-Free method to
> approximate Jacobian matrix by finite difference of residual function in
> solving nonlinear equations. But I'll also provide an analytic( incomplete,
> some terms are missing or approximate) for preconditioning. Because my
> problem is about a large set of equations composed of several physical
> fields, I want to use block Jacobian precondition for each subfield(block),
> and ILU sub_pc for each subfield. After reading the Users'Guide, I found
> that using - snex_mf_operator can do the above, so I added:
>  - snes_mf_operator
>

You need to be careful what matrix you are adding values to in your
FormJacobian() routine. The primal matrix J is of type MFFD (finite
difference)
and thus cannot accept values. You put your approximate values in the
preconditioner M.

  Thanks,

    Matt


> after the example above. However, the procedure is wrong and does not seem
> to support this.
>  The wrong information is:
>
>    [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [2]PETSC ERROR: No support for this operation for this object type
> [2]PETSC ERROR: Mat type mffd
> [2]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: ---------------------
> Error Message --------------------------------------------------------------
> [4]PETSC ERROR: No support for this operation for this object type
> [4]PETSC ERROR: Mat type mffd
> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [4]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [6]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [6]PETSC ERROR: No support for this operation for this object type
> [6]PETSC ERROR: Mat type mffd
> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [6]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [6]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Mon Oct  8 23:01:57 2018
> [6]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: No support for this operation for this object type
> [0]PETSC ERROR: Mat type mffd
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by yjwu
> Mon Oct  8 23:01:57 2018
> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
> --with-fc=gfortran --download-mpich --download-fblaslapack
> [0]PETSC ERROR: #1 MatSetValues() line 1330 in
> /home/yjwu/petsc-3.10.1/src/mat/interface/matrix.c
> [1]PETSC ERROR: No support for this operation for this object type
> [1]PETSC ERROR: Mat type mffd
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
>
> If I want to implement matrix-free and block jacobi precondition in my
> program, what do I need to pay attention or add ?
>
> The second question is very important for my research and I hope to get
> your answer.
>
> Thanks,
> Yingjie
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/563830a9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: out
Type: application/octet-stream
Size: 12478 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/563830a9/attachment-0001.obj>

From jczhang at mcs.anl.gov  Tue Oct  9 15:39:07 2018
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 9 Oct 2018 20:39:07 +0000
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
Message-ID: <CA+MQGp8fOhD2BWnE0C9WULeu64Kjh1TvbRkEWZMKQdWg-2B25A@mail.gmail.com>

I met a case where MUMPS returned an out-of-memory code but PETSc continued to run.  When PETSc calls MUMPS, it checks if (A->erroriffailure). I added -mat_error_if_failure, but it did not work since it was overwritten by MatSetErrorIfFailure(pc->pmat,pc->erroriffailure)
Does it suggest we should add a new option -pc_factor_error_if_failure and check it in PCSetFromOptions_Factor()?

--Junchao Zhang

On Fri, Oct 5, 2018 at 8:12 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Mike:
Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The matrix is getting denser and my experience is that MUMPS just outperforms iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I just wander if there is any suggestion to improve the robustness of MUMPS? Or in general, any suggestion for interative solver with very high-order finite elements?

What error message do you get when MUMPS fails? Out of memory, zero pivoting, or something?
 Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/08586427/attachment.html>

From jczhang at mcs.anl.gov  Tue Oct  9 15:39:07 2018
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 9 Oct 2018 20:39:07 +0000
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
Message-ID: <CA+MQGp8fOhD2BWnE0C9WULeu64Kjh1TvbRkEWZMKQdWg-2B25A@mail.gmail.com>

I met a case where MUMPS returned an out-of-memory code but PETSc continued to run.  When PETSc calls MUMPS, it checks if (A->erroriffailure). I added -mat_error_if_failure, but it did not work since it was overwritten by MatSetErrorIfFailure(pc->pmat,pc->erroriffailure)
Does it suggest we should add a new option -pc_factor_error_if_failure and check it in PCSetFromOptions_Factor()?

--Junchao Zhang

On Fri, Oct 5, 2018 at 8:12 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Mike:
Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The matrix is getting denser and my experience is that MUMPS just outperforms iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I just wander if there is any suggestion to improve the robustness of MUMPS? Or in general, any suggestion for interative solver with very high-order finite elements?

What error message do you get when MUMPS fails? Out of memory, zero pivoting, or something?
 Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/08586427/attachment-0001.html>

From jczhang at mcs.anl.gov  Tue Oct  9 16:05:41 2018
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Tue, 9 Oct 2018 21:05:41 +0000
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CA+MQGp8fOhD2BWnE0C9WULeu64Kjh1TvbRkEWZMKQdWg-2B25A@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAGCphBsom2WR35ujMGo7SLH6HOg+dju8pqn_wEb4nkyM4=ckEg@mail.gmail.com>
	<CA+MQGp8fOhD2BWnE0C9WULeu64Kjh1TvbRkEWZMKQdWg-2B25A@mail.gmail.com>
Message-ID: <CA+MQGp8cczFzCjwMWfEbs-9KVb0Ui9FV40ctOgabXbRN1Vky6g@mail.gmail.com>

OK, I found -ksp_error_if_not_converged will trigger PETSc to fail in this case.

--Junchao Zhang


On Tue, Oct 9, 2018 at 3:38 PM Junchao Zhang <jczhang at mcs.anl.gov<mailto:jczhang at mcs.anl.gov>> wrote:
I met a case where MUMPS returned an out-of-memory code but PETSc continued to run.  When PETSc calls MUMPS, it checks if (A->erroriffailure). I added -mat_error_if_failure, but it did not work since it was overwritten by MatSetErrorIfFailure(pc->pmat,pc->erroriffailure)
Does it suggest we should add a new option -pc_factor_error_if_failure and check it in PCSetFromOptions_Factor()?

--Junchao Zhang

On Fri, Oct 5, 2018 at 8:12 PM Zhang, Hong <hzhang at mcs.anl.gov<mailto:hzhang at mcs.anl.gov>> wrote:
Mike:
Hello PETSc team:

I am trying to solve a PDE problem with high-order finite elements. The matrix is getting denser and my experience is that MUMPS just outperforms iterative solvers.

For certain problems, MUMPS just fail in the middle for no clear reason. I just wander if there is any suggestion to improve the robustness of MUMPS? Or in general, any suggestion for interative solver with very high-order finite elements?

What error message do you get when MUMPS fails? Out of memory, zero pivoting, or something?
 Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181009/2db66d9f/attachment.html>

From amfoggia at gmail.com  Wed Oct 10 02:38:23 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Wed, 10 Oct 2018 09:38:23 +0200
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <C19248E9-CD9D-4A28-9203-D7078F1FFCF0@dsic.upv.es>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
	<CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
	<C19248E9-CD9D-4A28-9203-D7078F1FFCF0@dsic.upv.es>
Message-ID: <CANCZWyv9pCbYS7qWCVnO=+Lwzwthxt3i9mGzoyaaTbYkhNeH7A@mail.gmail.com>

Jed, Jose and Matthew,
I've finally managed to make massif (it gives pretty detailed information,
I like it) work in the correct way in the cluster and I'm able to track
down the memory consumption, and what's more important (for me), I think
now I'm able to make a more accurate prediction of the memory I need for a
particular size of the problem. Thank you very much for all your answers
and suggestions.

El vie., 5 oct. 2018 a las 9:38, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

>
>
> > El 4 oct 2018, a las 19:54, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> > Jose:
> > - By each step I mean each of the step of the the program in order to
> diagonalize the matrix. For me, those are: creation of basis, preallocation
> of matrix, setting values of matrix, initializing solver,
> solving/diagonalizing and cleaning. I'm only diagonalizing once.
> >
> > - Regarding the information provided by -log_view, it's confusing for
> me: for example, it reports the creation of Vecs scattered across the
> various stages that I've set up (with PetscLogStageRegister and
> PetscLogStagePush/Pop), but almost all the deletions are presented in the
> "Main Stage". What does that "Main Stage" consider? Why are more deletions
> in there that creations? It's nor completely for me clear how things are
> presented there.
>
> I guess deletions should match creations. Seems to be related to using
> stages. Maybe someone from PETSc can give an explanation, but looking at a
> PETSc example that uses stages (e.g. dm/impls/plex/examples/tests/ex1.c) it
> seems that some destructions are counted in the main stage while the
> creation is counted in another stage - I guess it depends on the points
> where the stages are defined. The sum of creations matches the sum of
> destroys.
>
> >
> > - Thanks for the suggestion about the solver. Does "faster convergence"
> for Krylov-Schur mean less memory and less computation, or just less
> computation?
> >
>
> It should be about the same memory with less iterations.
>
> Jose
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/9130b561/attachment-0001.html>

From knepley at gmail.com  Wed Oct 10 06:51:23 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Oct 2018 07:51:23 -0400
Subject: [petsc-users] PETSc/SLEPc: Memory consumption,
 particularly during solver initialization/solve
In-Reply-To: <CANCZWyv9pCbYS7qWCVnO=+Lwzwthxt3i9mGzoyaaTbYkhNeH7A@mail.gmail.com>
References: <CANCZWyu0ehAjgtiSxscj0efz==2oMDX5Aq8if+u5bvc8ZqPYDQ@mail.gmail.com>
	<CAMYG4Gn3aoN1vZRP4ENbURQYLqYM1TXARONkVxNi+tkgvEd1Dg@mail.gmail.com>
	<E2C7DFD7-2B76-40FC-86C4-11A60D2AF730@dsic.upv.es>
	<CANCZWyt0dqrurx00LzTi7CgKGousCe9PAp5YhHyxHo0FpGgQ4A@mail.gmail.com>
	<C19248E9-CD9D-4A28-9203-D7078F1FFCF0@dsic.upv.es>
	<CANCZWyv9pCbYS7qWCVnO=+Lwzwthxt3i9mGzoyaaTbYkhNeH7A@mail.gmail.com>
Message-ID: <CAMYG4Gmid6OF0s9jJH9OrqRJcBC-8HYW5yUEQ+wiPAr6bOPzqQ@mail.gmail.com>

On Wed, Oct 10, 2018 at 3:38 AM Ale Foggia <amfoggia at gmail.com> wrote:

> Jed, Jose and Matthew,
> I've finally managed to make massif (it gives pretty detailed information,
> I like it) work in the correct way in the cluster and I'm able to track
> down the memory consumption, and what's more important (for me), I think
> now I'm able to make a more accurate prediction of the memory I need for a
> particular size of the problem. Thank you very much for all your answers
> and suggestions.
>

Great! Could you tell me in one line what was taking up memory? It always
good to hear about real applications.

  Thanks,

    Matt


> El vie., 5 oct. 2018 a las 9:38, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
>
>>
>>
>> > El 4 oct 2018, a las 19:54, Ale Foggia <amfoggia at gmail.com> escribi?:
>> >
>> > Jose:
>> > - By each step I mean each of the step of the the program in order to
>> diagonalize the matrix. For me, those are: creation of basis, preallocation
>> of matrix, setting values of matrix, initializing solver,
>> solving/diagonalizing and cleaning. I'm only diagonalizing once.
>> >
>> > - Regarding the information provided by -log_view, it's confusing for
>> me: for example, it reports the creation of Vecs scattered across the
>> various stages that I've set up (with PetscLogStageRegister and
>> PetscLogStagePush/Pop), but almost all the deletions are presented in the
>> "Main Stage". What does that "Main Stage" consider? Why are more deletions
>> in there that creations? It's nor completely for me clear how things are
>> presented there.
>>
>> I guess deletions should match creations. Seems to be related to using
>> stages. Maybe someone from PETSc can give an explanation, but looking at a
>> PETSc example that uses stages (e.g. dm/impls/plex/examples/tests/ex1.c) it
>> seems that some destructions are counted in the main stage while the
>> creation is counted in another stage - I guess it depends on the points
>> where the stages are defined. The sum of creations matches the sum of
>> destroys.
>>
>> >
>> > - Thanks for the suggestion about the solver. Does "faster convergence"
>> for Krylov-Schur mean less memory and less computation, or just less
>> computation?
>> >
>>
>> It should be about the same memory with less iterations.
>>
>> Jose
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/7c612d4f/attachment.html>

From yjwu16 at gmail.com  Wed Oct 10 07:50:20 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Wed, 10 Oct 2018 20:50:20 +0800
Subject: [petsc-users] Problems about Block Jocabi and Matrix-Free
 Method in SNES
In-Reply-To: <CAMYG4GkkevtfKEZmjvegcUA6y2J4NeU2P0tKfrZ+BS1NgryoUg@mail.gmail.com>
References: <CAAMEhBimeGsge9x7DH4iCpb7SwgrfViV_2gfcF5tKCrf4tzJQg@mail.gmail.com>
	<CAMYG4GkkevtfKEZmjvegcUA6y2J4NeU2P0tKfrZ+BS1NgryoUg@mail.gmail.com>
Message-ID: <CAAMEhBhZEU_ynG-_i0Dq7tGPy2P4+xdMnbpDtprPaV2RoKO+fg@mail.gmail.com>

Thank you very much, Matt.
 I followed your advice and learned some MATMFFD related functions. Since I
really wanted to use the Matrix-Free method, I learned the few examples of
this method. I would like to enhance my understanding of the Matrix-Free
method in PETSc according to my questions about these examples and hope to
get your answer.
1. In the example src/snes/example/tutorials/ex22.c

......
107:   packer->ops->creatematrix = DMCreateMatrix_MF;

110:   SNESCreate(PETSC_COMM_WORLD,&snes);
111:   SNESSetDM(snes,packer);
112:   SNESSetFunction(snes,NULL,ComputeFunction,NULL);
113:   SNESSetJacobian(snes,NULL, NULL,ComputeJacobian_MF,NULL);

115:   SNESSetFromOptions(snes);
......
285: PetscErrorCode DMCreateMatrix_MF(DM packer,Mat *A)
286: {
288:   Vec            t;
289:   PetscInt       m;

292:   DMGetGlobalVector(packer,&t);
293:   VecGetLocalSize(t,&m);
294:   DMRestoreGlobalVector(packer,&t);
295:
 MatCreateMFFD(PETSC_COMM_WORLD,m,m,PETSC_DETERMINE,PETSC_DETERMINE,A);
296:   MatSetUp(*A);
297:   return(0);
298: }

300: PetscErrorCode ComputeJacobian_MF(SNES snes,Vec x,Mat A,Mat B,void
*ctx)
301: {

305:   MatMFFDSetFunction(A,(PetscErrorCode
(*)(void*,Vec,Vec))SNESComputeFunction,snes);
306:   MatMFFDSetBase(A,x,NULL);
307:   return(0);
308: }

Why are the solution vector, Jacobian matrix, and precondition matrix set
to NULL in SNESSetJacobian? Is it because they are automatically generated
in SNESSetDM?
There is no precondition matrix B in ComputeJacobian_MF (Jacobian evaluate
program). If I want to add my own precondition matrix, then the Assembly
matrix B in this function, is that right?
What is the function of MatMFFDSetBase? I read the documentation but didn't
understand it. What would happen if I removed it?

2. In the example /src/snes/examples/tutorials/ex14.c

114:   SNESSetFunction(snes,r,FormFunction,(void*)&user);

131:   PetscOptionsGetBool(NULL,NULL,"-snes_mf",&matrix_free,NULL);
132:   PetscOptionsGetBool(NULL,NULL,"-fdcoloring",&coloring,NULL);
133:   PetscOptionsGetBool(NULL,NULL,"-fdcoloring_ds",&coloring_ds,NULL);
134:
 PetscOptionsGetBool(NULL,NULL,"-fdcoloring_local",&local_coloring,NULL);
135:   if (!matrix_free) {
136:     DMSetMatType(user.da,MATAIJ);
137:     DMCreateMatrix(user.da,&J);
138:     if (coloring) {
139:       ISColoring iscoloring;
140:       if (!local_coloring) {
141:         DMCreateColoring(user.da,IS_COLORING_GLOBAL,&iscoloring);
142:         MatFDColoringCreate(J,iscoloring,&matfdcoloring);
143:         MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode
(*)(void))FormFunction,&user);
144:       } else {
145:         DMCreateColoring(user.da,IS_COLORING_LOCAL,&iscoloring);
146:         MatFDColoringCreate(J,iscoloring,&matfdcoloring);
147:         MatFDColoringUseDM(J,matfdcoloring);
148:         MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode
(*)(void))FormFunctionLocal,&user);
149:       }
150:       if (coloring_ds) {
151:         MatFDColoringSetType(matfdcoloring,MATMFFD_DS);
152:       }
153:       MatFDColoringSetFromOptions(matfdcoloring);
154:       MatFDColoringSetUp(J,iscoloring,matfdcoloring);
155:
 SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring);
156:       ISColoringDestroy(&iscoloring);
157:     } else {
158:       SNESSetJacobian(snes,J,J,FormJacobian,&user);
159:     }
160:   }

If the command "- snes_mf" is used, the function SNESSetJacobian is not
called, and the Jacobian matrix type is initialized as MATMFFD. The
function in SNESSetFunction is used to approximate the matrix vector
product. Do I understand this correctly?
The description of my question may be complicated, but as a beginner, I
really want to learn something about matrix-free method from the examples,
and I hope to get your reply.

Thanks,
Yingjie

Matthew Knepley <knepley at gmail.com> ?2018?10?9??? ??11:34???

> On Mon, Oct 8, 2018 at 11:33 PM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Dear Petsc developer:
>> Hi,
>>
>> I've been studying Petsc recently about Precontioner and Metrix-Free, and
>> I have some questions that puzzle me.
>>
>> 1. I want to test block Jacobi preconditioner, so I chose
>> /snes/example/tutorial/ex3.c as an example. According to the reference in
>> the example, the input parameters are:
>> mpiexec -n 8./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
>> -pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -post_setsubksp
>> -sub_ksp_rtol 1.e-16
>>
>
> You do not care about recursive blocks, so just use
>
>   $MPIEXEC -n 8 ./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
> -pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -snes_view
> -ksp_view
>
> and I get the attached output.
>
>
>> I want to export each block of KSP and PC information :
>>   -snes_view -ksp_view
>> However, the procedure is wrong and the wrong information is as follows:
>>
>>        [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Arguments must have same communicators
>> [0]PETSC ERROR: Different communicators in the two objects: Argument # 1
>> and 2 flag 3
>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [3]PETSC ERROR: Arguments must have same communicators
>> [3]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [6]PETSC ERROR: Arguments must have same communicators
>> [6]PETSC ERROR: Different communicators in the two objects: Argument # 1
>> and 2 flag 3
>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [7]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [7]PETSC ERROR: Arguments must have same communicators
>> [7]PETSC ERROR: Different communicators in the two objects: Argument # 1
>> and 2 flag 3
>> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [7]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>> shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>> yjwu Mon Oct  8 22:35:34 2018
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-mpich --download-fblaslapack
>> [0]PETSC ERROR: #1 KSPView() line 118 in
>> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
>> [0]PETSC ERROR: #2 PCView_BJacobi() line 232 in
>> /home/yjwu/petsc-3.10.1/src/ksp/pc/impls/bjacobi/bjacobi.c
>> [0]PETSC ERROR: #3 PCView() line 1651 in
>> /home/yjwu/petsc-3.10.1/src/ksp/pc/interface/precon.c
>> [1]PETSC ERROR: Arguments must have same communicators
>> [1]PETSC ERROR: Different communicators in the two objects: Argument # 1
>> and 2 flag 3
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> [1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>> yjwu Mon Oct  8 22:35:34 2018
>> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-mpich --download-fblaslapack
>> [1]PETSC ERROR: #1 KSPView() line 118 in
>> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
>> Arguments must have same communicators
>> [2]PETSC ERROR: Different communicators in the two objects: Argument # 1
>> and 2 flag 3
>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>>
>> If I want to get the information of KSP and PC in each block, what should
>> I do?
>>
>> 2. There is a requirement in my program to use Matrix-Free method to
>> approximate Jacobian matrix by finite difference of residual function in
>> solving nonlinear equations. But I'll also provide an analytic( incomplete,
>> some terms are missing or approximate) for preconditioning. Because my
>> problem is about a large set of equations composed of several physical
>> fields, I want to use block Jacobian precondition for each subfield(block),
>> and ILU sub_pc for each subfield. After reading the Users'Guide, I found
>> that using - snex_mf_operator can do the above, so I added:
>>  - snes_mf_operator
>>
>
> You need to be careful what matrix you are adding values to in your
> FormJacobian() routine. The primal matrix J is of type MFFD (finite
> difference)
> and thus cannot accept values. You put your approximate values in the
> preconditioner M.
>
>   Thanks,
>
>     Matt
>
>
>> after the example above. However, the procedure is wrong and does not
>> seem to support this.
>>  The wrong information is:
>>
>>    [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [2]PETSC ERROR: No support for this operation for this object type
>> [2]PETSC ERROR: Mat type mffd
>> [2]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: ---------------------
>> Error Message --------------------------------------------------------------
>> [4]PETSC ERROR: No support for this operation for this object type
>> [4]PETSC ERROR: Mat type mffd
>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [4]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> [6]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [6]PETSC ERROR: No support for this operation for this object type
>> [6]PETSC ERROR: Mat type mffd
>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [6]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> [6]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>> yjwu Mon Oct  8 23:01:57 2018
>> [6]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-mpich --download-fblaslapack
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: No support for this operation for this object type
>> [0]PETSC ERROR: Mat type mffd
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>> yjwu Mon Oct  8 23:01:57 2018
>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>> --with-fc=gfortran --download-mpich --download-fblaslapack
>> [0]PETSC ERROR: #1 MatSetValues() line 1330 in
>> /home/yjwu/petsc-3.10.1/src/mat/interface/matrix.c
>> [1]PETSC ERROR: No support for this operation for this object type
>> [1]PETSC ERROR: Mat type mffd
>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>>
>> If I want to implement matrix-free and block jacobi precondition in my
>> program, what do I need to pay attention or add ?
>>
>> The second question is very important for my research and I hope to get
>> your answer.
>>
>> Thanks,
>> Yingjie
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/163a36ab/attachment-0001.html>

From knepley at gmail.com  Wed Oct 10 08:09:29 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Oct 2018 09:09:29 -0400
Subject: [petsc-users] Problems about Block Jocabi and Matrix-Free
 Method in SNES
In-Reply-To: <CAAMEhBhZEU_ynG-_i0Dq7tGPy2P4+xdMnbpDtprPaV2RoKO+fg@mail.gmail.com>
References: <CAAMEhBimeGsge9x7DH4iCpb7SwgrfViV_2gfcF5tKCrf4tzJQg@mail.gmail.com>
	<CAMYG4GkkevtfKEZmjvegcUA6y2J4NeU2P0tKfrZ+BS1NgryoUg@mail.gmail.com>
	<CAAMEhBhZEU_ynG-_i0Dq7tGPy2P4+xdMnbpDtprPaV2RoKO+fg@mail.gmail.com>
Message-ID: <CAMYG4G=trmf9__mFV0gffnziEPnxRXWpWq8qwDkne0WX2DwwPQ@mail.gmail.com>

On Wed, Oct 10, 2018 at 8:50 AM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Thank you very much, Matt.
>  I followed your advice and learned some MATMFFD related functions. Since
> I really wanted to use the Matrix-Free method, I learned the few examples
> of this method. I would like to enhance my understanding of the Matrix-Free
> method in PETSc according to my questions about these examples and hope to
> get your answer.
> 1. In the example src/snes/example/tutorials/ex22.c
>
> ......
> 107:   packer->ops->creatematrix = DMCreateMatrix_MF;
>
> 110:   SNESCreate(PETSC_COMM_WORLD,&snes);
> 111:   SNESSetDM(snes,packer);
> 112:   SNESSetFunction(snes,NULL,ComputeFunction,NULL);
> 113:   SNESSetJacobian(snes,NULL, NULL,ComputeJacobian_MF,NULL);
>
> 115:   SNESSetFromOptions(snes);
> ......
> 285: PetscErrorCode DMCreateMatrix_MF(DM packer,Mat *A)
> 286: {
> 288:   Vec            t;
> 289:   PetscInt       m;
>
> 292:   DMGetGlobalVector(packer,&t);
> 293:   VecGetLocalSize(t,&m);
> 294:   DMRestoreGlobalVector(packer,&t);
> 295:
>  MatCreateMFFD(PETSC_COMM_WORLD,m,m,PETSC_DETERMINE,PETSC_DETERMINE,A);
> 296:   MatSetUp(*A);
> 297:   return(0);
> 298: }
>
> 300: PetscErrorCode ComputeJacobian_MF(SNES snes,Vec x,Mat A,Mat B,void
> *ctx)
> 301: {
>
> 305:   MatMFFDSetFunction(A,(PetscErrorCode
> (*)(void*,Vec,Vec))SNESComputeFunction,snes);
> 306:   MatMFFDSetBase(A,x,NULL);
> 307:   return(0);
> 308: }
>
> Why are the solution vector, Jacobian matrix, and precondition matrix set
> to NULL in SNESSetJacobian? Is it because they are automatically generated
> in SNESSetDM?
>

Yes. This example show how to do it by hand. If you give NULL for your
FormJacobian function, we do it automatically. If you give _mf_operator,
then we do it, but pass through the matrices to your FormJacobian()
function.


> There is no precondition matrix B in ComputeJacobian_MF (Jacobian evaluate
> program). If I want to add my own precondition matrix, then the Assembly
> matrix B in this function, is that right?
> What is the function of MatMFFDSetBase? I read the documentation but
> didn't understand it. What would happen if I removed it?
>

SetBase() provides the initial vector for differencing. This is necessary.


> 2. In the example /src/snes/examples/tutorials/ex14.c
>
> 114:   SNESSetFunction(snes,r,FormFunction,(void*)&user);
>
> 131:   PetscOptionsGetBool(NULL,NULL,"-snes_mf",&matrix_free,NULL);
> 132:   PetscOptionsGetBool(NULL,NULL,"-fdcoloring",&coloring,NULL);
> 133:   PetscOptionsGetBool(NULL,NULL,"-fdcoloring_ds",&coloring_ds,NULL);
> 134:
>  PetscOptionsGetBool(NULL,NULL,"-fdcoloring_local",&local_coloring,NULL);
> 135:   if (!matrix_free) {
> 136:     DMSetMatType(user.da,MATAIJ);
> 137:     DMCreateMatrix(user.da,&J);
> 138:     if (coloring) {
> 139:       ISColoring iscoloring;
> 140:       if (!local_coloring) {
> 141:         DMCreateColoring(user.da,IS_COLORING_GLOBAL,&iscoloring);
> 142:         MatFDColoringCreate(J,iscoloring,&matfdcoloring);
> 143:         MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode
> (*)(void))FormFunction,&user);
> 144:       } else {
> 145:         DMCreateColoring(user.da,IS_COLORING_LOCAL,&iscoloring);
> 146:         MatFDColoringCreate(J,iscoloring,&matfdcoloring);
> 147:         MatFDColoringUseDM(J,matfdcoloring);
> 148:         MatFDColoringSetFunction(matfdcoloring,(PetscErrorCode
> (*)(void))FormFunctionLocal,&user);
> 149:       }
> 150:       if (coloring_ds) {
> 151:         MatFDColoringSetType(matfdcoloring,MATMFFD_DS);
> 152:       }
> 153:       MatFDColoringSetFromOptions(matfdcoloring);
> 154:       MatFDColoringSetUp(J,iscoloring,matfdcoloring);
> 155:
>  SNESSetJacobian(snes,J,J,SNESComputeJacobianDefaultColor,matfdcoloring);
> 156:       ISColoringDestroy(&iscoloring);
> 157:     } else {
> 158:       SNESSetJacobian(snes,J,J,FormJacobian,&user);
> 159:     }
> 160:   }
>
> If the command "- snes_mf" is used, the function SNESSetJacobian is not
> called, and the Jacobian matrix type is initialized as MATMFFD. The
> function in SNESSetFunction is used to approximate the matrix vector
> product. Do I understand this correctly?
>

Yes .

  Thanks,

     Matt


> The description of my question may be complicated, but as a beginner, I
> really want to learn something about matrix-free method from the examples,
> and I hope to get your reply.
>
> Thanks,
> Yingjie
>
> Matthew Knepley <knepley at gmail.com> ?2018?10?9??? ??11:34???
>
>> On Mon, Oct 8, 2018 at 11:33 PM Yingjie Wu <yjwu16 at gmail.com> wrote:
>>
>>> Dear Petsc developer:
>>> Hi,
>>>
>>> I've been studying Petsc recently about Precontioner and Metrix-Free,
>>> and I have some questions that puzzle me.
>>>
>>> 1. I want to test block Jacobi preconditioner, so I chose
>>> /snes/example/tutorial/ex3.c as an example. According to the reference in
>>> the example, the input parameters are:
>>> mpiexec -n 8./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
>>> -pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -post_setsubksp
>>> -sub_ksp_rtol 1.e-16
>>>
>>
>> You do not care about recursive blocks, so just use
>>
>>   $MPIEXEC -n 8 ./ex3 -nox -n 10000 -ksp_type fgmres -pc_type bjacobi
>> -pc_bjacobi_blocks 4 -sub_ksp_type gmres -sub_ksp_max_it 3 -snes_view
>> -ksp_view
>>
>> and I get the attached output.
>>
>>
>>> I want to export each block of KSP and PC information :
>>>   -snes_view -ksp_view
>>> However, the procedure is wrong and the wrong information is as
>>> follows:
>>>
>>>        [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Arguments must have same communicators
>>> [0]PETSC ERROR: Different communicators in the two objects: Argument # 1
>>> and 2 flag 3
>>> [0]PETSC ERROR: [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [2]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [2]PETSC ERROR: [3]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [3]PETSC ERROR: Arguments must have same communicators
>>> [3]PETSC ERROR: [6]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [6]PETSC ERROR: Arguments must have same communicators
>>> [6]PETSC ERROR: Different communicators in the two objects: Argument # 1
>>> and 2 flag 3
>>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [7]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [7]PETSC ERROR: Arguments must have same communicators
>>> [7]PETSC ERROR: Different communicators in the two objects: Argument # 1
>>> and 2 flag 3
>>> [7]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [7]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble
>>> shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>>> yjwu Mon Oct  8 22:35:34 2018
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --download-mpich --download-fblaslapack
>>> [0]PETSC ERROR: #1 KSPView() line 118 in
>>> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
>>> [0]PETSC ERROR: #2 PCView_BJacobi() line 232 in
>>> /home/yjwu/petsc-3.10.1/src/ksp/pc/impls/bjacobi/bjacobi.c
>>> [0]PETSC ERROR: #3 PCView() line 1651 in
>>> /home/yjwu/petsc-3.10.1/src/ksp/pc/interface/precon.c
>>> [1]PETSC ERROR: Arguments must have same communicators
>>> [1]PETSC ERROR: Different communicators in the two objects: Argument # 1
>>> and 2 flag 3
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [1]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> [1]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>>> yjwu Mon Oct  8 22:35:34 2018
>>> [1]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --download-mpich --download-fblaslapack
>>> [1]PETSC ERROR: #1 KSPView() line 118 in
>>> /home/yjwu/petsc-3.10.1/src/ksp/ksp/interface/itcreate.c
>>> Arguments must have same communicators
>>> [2]PETSC ERROR: Different communicators in the two objects: Argument # 1
>>> and 2 flag 3
>>> [2]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>>
>>> If I want to get the information of KSP and PC in each block, what
>>> should I do?
>>>
>>> 2. There is a requirement in my program to use Matrix-Free method to
>>> approximate Jacobian matrix by finite difference of residual function in
>>> solving nonlinear equations. But I'll also provide an analytic( incomplete,
>>> some terms are missing or approximate) for preconditioning. Because my
>>> problem is about a large set of equations composed of several physical
>>> fields, I want to use block Jacobian precondition for each subfield(block),
>>> and ILU sub_pc for each subfield. After reading the Users'Guide, I found
>>> that using - snex_mf_operator can do the above, so I added:
>>>  - snes_mf_operator
>>>
>>
>> You need to be careful what matrix you are adding values to in your
>> FormJacobian() routine. The primal matrix J is of type MFFD (finite
>> difference)
>> and thus cannot accept values. You put your approximate values in the
>> preconditioner M.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> after the example above. However, the procedure is wrong and does not
>>> seem to support this.
>>>  The wrong information is:
>>>
>>>    [1]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [2]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [2]PETSC ERROR: No support for this operation for this object type
>>> [2]PETSC ERROR: Mat type mffd
>>> [2]PETSC ERROR: [3]PETSC ERROR: [4]PETSC ERROR: ---------------------
>>> Error Message --------------------------------------------------------------
>>> [4]PETSC ERROR: No support for this operation for this object type
>>> [4]PETSC ERROR: Mat type mffd
>>> [4]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [4]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> [6]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [6]PETSC ERROR: No support for this operation for this object type
>>> [6]PETSC ERROR: Mat type mffd
>>> [6]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [6]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> [6]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>>> yjwu Mon Oct  8 23:01:57 2018
>>> [6]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --download-mpich --download-fblaslapack
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: No support for this operation for this object type
>>> [0]PETSC ERROR: Mat type mffd
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.1, Sep, 26, 2018
>>> [0]PETSC ERROR: ./ex3 on a arch-linux2-c-debug named yjwu-XPS-8910 by
>>> yjwu Mon Oct  8 23:01:57 2018
>>> [0]PETSC ERROR: Configure options --with-cc=gcc --with-cxx=g++
>>> --with-fc=gfortran --download-mpich --download-fblaslapack
>>> [0]PETSC ERROR: #1 MatSetValues() line 1330 in
>>> /home/yjwu/petsc-3.10.1/src/mat/interface/matrix.c
>>> [1]PETSC ERROR: No support for this operation for this object type
>>> [1]PETSC ERROR: Mat type mffd
>>> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>>
>>> If I want to implement matrix-free and block jacobi precondition in my
>>> program, what do I need to pay attention or add ?
>>>
>>> The second question is very important for my research and I hope to get
>>> your answer.
>>>
>>> Thanks,
>>> Yingjie
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/d7e0fa60/attachment-0001.html>

From Ling.Zou at inl.gov  Wed Oct 10 11:35:37 2018
From: Ling.Zou at inl.gov (Ling Zou)
Date: Wed, 10 Oct 2018 16:35:37 +0000
Subject: [petsc-users] =?windows-1252?q?error=3A_=91DMSetSection=92_was_n?=
 =?windows-1252?q?ot_declared_in_this_scope?=
Message-ID: <DM5PR09MB14526A5DA6A570C52BACCF5783E00@DM5PR09MB1452.namprd09.prod.outlook.com>

Hi All,


I am starting playing with DMPlex, but get this error:


error: ?DMSetSection? was not declared in this scope


However, other DMPlex basic functions work just fine, e.g.,

DMPlexCreateFluentFromFile

DMPlexCheckSymmetry

DMPlexCheckSkeleton

DMPlexCheckFaces

DMPlexGetChart

DMPlexGetHeightStratum


Is this function somewhat special?


PS: I am using cmake, and followed Jed's FindPETSc.cmake at https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake


Ling

[https://avatars1.githubusercontent.com/u/3303?s=400&v=4]<https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake>

jedbrown/cmake-modules - GitHub<https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake>
github.com
CMake modules for some scientific libraries. Contribute to jedbrown/cmake-modules development by creating an account on GitHub.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/6fca0a42/attachment.html>

From Ling.Zou at inl.gov  Wed Oct 10 11:41:51 2018
From: Ling.Zou at inl.gov (Ling Zou)
Date: Wed, 10 Oct 2018 16:41:51 +0000
Subject: [petsc-users] 
 =?windows-1252?q?error=3A_=91DMSetSection=92_was_n?=
 =?windows-1252?q?ot_declared_in_this_scope?=
In-Reply-To: <DM5PR09MB14526A5DA6A570C52BACCF5783E00@DM5PR09MB1452.namprd09.prod.outlook.com>
References: <DM5PR09MB14526A5DA6A570C52BACCF5783E00@DM5PR09MB1452.namprd09.prod.outlook.com>
Message-ID: <DM5PR09MB14526B553E8287C71B23A12083E00@DM5PR09MB1452.namprd09.prod.outlook.com>

BTW, if I move the source file to:

petsc-3.10.2/src/dm/impls/plex/examples/tutorials


Using the makefile there, it compiles just fine.


Ling

________________________________
From: petsc-users <petsc-users-bounces at mcs.anl.gov> on behalf of Ling Zou <Ling.Zou at inl.gov>
Sent: Wednesday, October 10, 2018 10:35:37 AM
To: PETSc users list
Subject: [petsc-users] error: ?DMSetSection? was not declared in this scope


Hi All,


I am starting playing with DMPlex, but get this error:


error: ?DMSetSection? was not declared in this scope


However, other DMPlex basic functions work just fine, e.g.,

DMPlexCreateFluentFromFile

DMPlexCheckSymmetry

DMPlexCheckSkeleton

DMPlexCheckFaces

DMPlexGetChart

DMPlexGetHeightStratum


Is this function somewhat special?


PS: I am using cmake, and followed Jed's FindPETSc.cmake at https://github.com/jedbrown/cmake-modules/blob/master/FindPETSc.cmake<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jedbrown_cmake-2Dmodules_blob_master_FindPETSc.cmake&d=DwMF-g&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=wpRJMLiF6hRbwE2a_dsAXp55Rw-IkQ-HAOQwTBee_cU&s=sOlqf87CtrvIcZleo9WzSyUDpHYbuat53hrtySkIzIo&e=>


Ling

[https://avatars1.githubusercontent.com/u/3303?s=400&v=4]<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jedbrown_cmake-2Dmodules_blob_master_FindPETSc.cmake&d=DwMF-g&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=wpRJMLiF6hRbwE2a_dsAXp55Rw-IkQ-HAOQwTBee_cU&s=sOlqf87CtrvIcZleo9WzSyUDpHYbuat53hrtySkIzIo&e=>

jedbrown/cmake-modules - GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_jedbrown_cmake-2Dmodules_blob_master_FindPETSc.cmake&d=DwMF-g&c=54IZrppPQZKX9mLzcGdPfFD1hxrcB__aEkJFOKJFd00&r=kuHHom1yjd94zUrBWecnYg&m=wpRJMLiF6hRbwE2a_dsAXp55Rw-IkQ-HAOQwTBee_cU&s=sOlqf87CtrvIcZleo9WzSyUDpHYbuat53hrtySkIzIo&e=>
github.com
CMake modules for some scientific libraries. Contribute to jedbrown/cmake-modules development by creating an account on GitHub.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/d3bc4a1a/attachment.html>

From mcygorek at uottawa.ca  Wed Oct 10 12:54:04 2018
From: mcygorek at uottawa.ca (Moritz Cygorek)
Date: Wed, 10 Oct 2018 17:54:04 +0000
Subject: [petsc-users] STFILTER in slepc
Message-ID: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>

Hi,


in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER.


Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.


Does anybody have an example of how to use STFILTER or any documentation about it?


Thanks in advance,

Moritz
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/038b5549/attachment.html>

From jroman at dsic.upv.es  Wed Oct 10 13:30:43 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 10 Oct 2018 20:30:43 +0200
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
Message-ID: <278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>


> El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> 
> Hi,
> 
> in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER. 
> 
> Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> 
> Does anybody have an example of how to use STFILTER or any documentation about it?
> 
> Thanks in advance,
> Moritz

There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here: http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html

In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.

Jose


From mcygorek at uottawa.ca  Wed Oct 10 14:10:34 2018
From: mcygorek at uottawa.ca (Moritz Cygorek)
Date: Wed, 10 Oct 2018 19:10:34 +0000
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>,
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
Message-ID: <867180c290734d1da494a13d49fcbc7e@uottawa.ca>

Thank you for the fast reply.


I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:


-st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000


and I get the following error message:


[0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
....

[0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
[0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
[0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
[0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
[0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
[0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c




Do you have a clue what I've missed?



Moritz


________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: Wednesday, October 10, 2018 2:30 PM
To: Moritz Cygorek
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] STFILTER in slepc


> El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
>
> Hi,
>
> in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER.
>
> Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
>
> Does anybody have an example of how to use STFILTER or any documentation about it?
>
> Thanks in advance,
> Moritz

There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here: http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html

In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.

Jose

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/32523f3c/attachment.html>

From jroman at dsic.upv.es  Wed Oct 10 14:48:36 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 10 Oct 2018 21:48:36 +0200
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <867180c290734d1da494a13d49fcbc7e@uottawa.ca>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>
Message-ID: <61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>

This type of method requires a very high degree polynomial; suggest using degree=100 at least (this is the default value), but larger values may be necessary. Also, for this particular filter the "range" must be approximately equal to the numerical range; if you have no clue where your first and last eigenvalues are, you may use EPSSolve() calls with EPS_LARGEST_REAL and EPS_SMALLEST_REAL.

Jose

> El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> 
> Thank you for the fast reply. 
> 
> I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:
> 
> -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000
> 
> and I get the following error message:
> 
> [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
> ....
> [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> [0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> [0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> [0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
> 
> 
> 
> Do you have a clue what I've missed?
> 
> 
> Moritz
> 
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: Wednesday, October 10, 2018 2:30 PM
> To: Moritz Cygorek
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] STFILTER in slepc
>  
> 
> > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > 
> > Hi,
> > 
> > in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER. 
> > 
> > Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> > 
> > Does anybody have an example of how to use STFILTER or any documentation about it?
> > 
> > Thanks in advance,
> > Moritz
> 
> There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here:http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
> 
> In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.
> 
> Jose


From mcygorek at uottawa.ca  Wed Oct 10 19:41:09 2018
From: mcygorek at uottawa.ca (Moritz Cygorek)
Date: Thu, 11 Oct 2018 00:41:09 +0000
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>,
	<61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
Message-ID: <0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>

Thank you very much. Apparently, I've misunderstood what the filter actually does. I thought about the much simpler process, where you diagonalize


-(A- tau*I)^2 +offset*I


where tau is my target an offset is large enough so that the global maximum is reached for eigenvalues around tau. Then you look for the largest eigenvalue of the modified problem and either calculate the Ritz value of the original matrix or calculate back from the eigenvalues of the modified problem.


Now, it looks to me like -st_type filter activates something like the package FILTLAN.


I guess I can define a MatShell to do the thing I intended in the first place.

But, I guess, this is a common thing, so I am wondering whether it is already implemented somewhere and I just didn't find it in the documentation.  Can you say something about this?


Regards,

Moritz



________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: Wednesday, October 10, 2018 3:48 PM
To: Moritz Cygorek
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] STFILTER in slepc

This type of method requires a very high degree polynomial; suggest using degree=100 at least (this is the default value), but larger values may be necessary. Also, for this particular filter the "range" must be approximately equal to the numerical range; if you have no clue where your first and last eigenvalues are, you may use EPSSolve() calls with EPS_LARGEST_REAL and EPS_SMALLEST_REAL.

Jose

> El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
>
> Thank you for the fast reply.
>
> I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:
>
> -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000
>
> and I get the following error message:
>
> [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
> ....
> [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> [0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> [0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> [0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
>
>
>
> Do you have a clue what I've missed?
>
>
> Moritz
>
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: Wednesday, October 10, 2018 2:30 PM
> To: Moritz Cygorek
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] STFILTER in slepc
>
>
> > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> >
> > Hi,
> >
> > in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER.
> >
> > Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> >
> > Does anybody have an example of how to use STFILTER or any documentation about it?
> >
> > Thanks in advance,
> > Moritz
>
> There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here:http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
>
> In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.
>
> Jose

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/e6b41620/attachment-0001.html>

From knepley at gmail.com  Wed Oct 10 19:43:44 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 10 Oct 2018 20:43:44 -0400
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>
	<61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
	<0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>
Message-ID: <CAMYG4G=9s03652Dzx92FzC4KiWS_iy5RTwWkMsK0uYZ0ahS5=w@mail.gmail.com>

On Wed, Oct 10, 2018 at 8:41 PM Moritz Cygorek <mcygorek at uottawa.ca> wrote:

> Thank you very much. Apparently, I've misunderstood what the filter
> actually does. I thought about the much simpler process, where you
> diagonalize
>
>
> -(A- tau*I)^2 +offset*I
>
>
> where tau is my target an offset is large enough so that the global
> maximum is reached for eigenvalues around tau.
>

Is this different from -eps_target_magnitude?

  Thanks,

    Matt


> Then you look for the largest eigenvalue of the modified problem and
> either calculate the Ritz value of the original matrix or calculate back
> from the eigenvalues of the modified problem.
>
>
> Now, it looks to me like -st_type filter activates something like the
> package FILTLAN.
>
>
> I guess I can define a MatShell to do the thing I intended in the first
> place.
>
> But, I guess, this is a common thing, so I am wondering whether it is
> already implemented somewhere and I just didn't find it in the
> documentation.  Can you say something about this?
>
>
> Regards,
>
> Moritz
>
>
>
>
> ------------------------------
> *From:* Jose E. Roman <jroman at dsic.upv.es>
> *Sent:* Wednesday, October 10, 2018 3:48 PM
> *To:* Moritz Cygorek
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] STFILTER in slepc
>
> This type of method requires a very high degree polynomial; suggest using
> degree=100 at least (this is the default value), but larger values may be
> necessary. Also, for this particular filter the "range" must be
> approximately equal to the numerical range; if you have no clue where your
> first and last eigenvalues are, you may use EPSSolve() calls with
> EPS_LARGEST_REAL and EPS_SMALLEST_REAL.
>
> Jose
>
> > El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca>
> escribi?:
> >
> > Thank you for the fast reply.
> >
> > I've tried running my program (using the defaul Krylov-Schur method for
> sparse MPI matrices) with the additional options:
> >
> > -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7
> -st_filter_range -2000,2000
> >
> > and I get the following error message:
> >
> > [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust
> your filter parameters (e.g. increasing the polynomial degree)
> > ....
> > [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in
> /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in
> /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in
> /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> > [0]PETSC ERROR: #4 STSetUp() line 271 in
> /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> > [0]PETSC ERROR: #5 EPSSetUp() line 263 in
> /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> > [0]PETSC ERROR: #6 EPSSolve() line 135 in
> /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
> >
> >
> >
> > Do you have a clue what I've missed?
> >
> >
> > Moritz
> >
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: Wednesday, October 10, 2018 2:30 PM
> > To: Moritz Cygorek
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] STFILTER in slepc
> >
> >
> > > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca>
> escribi?:
> > >
> > > Hi,
> > >
> > > in the list of changes to SLEPc version 3.8, it is stated that there
> is a preliminary implementation of polynomial filtering using STFILTER.
> > >
> > > Because I am struggling to obtain interior eigenvalues and harmonic
> extraction seems not to be stable enough in my case, I wanted to give it a
> try, but I could not find any documentation yet.
> > >
> > > Does anybody have an example of how to use STFILTER or any
> documentation about it?
> > >
> > > Thanks in advance,
> > > Moritz
> >
> > There are no examples. You just set the type to STFILTER and set some
> parameters such as the interval of interest or the polynomial degree. See
> functions starting with STFilter here:
> http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
> >
> > In some problems it works well, but don't expect too much. It is still
> in our to-do list to make it more usable. It will be good to have your
> feedback. If you want, send results to slepc-maint, maybe we can help
> tuning the parameters.
> >
> > Jose
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181010/67942d02/attachment.html>

From dave.mayhem23 at gmail.com  Thu Oct 11 03:50:44 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Thu, 11 Oct 2018 09:50:44 +0100
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
Message-ID: <CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>

On Sat, 6 Oct 2018 at 12:42, Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 5, 2018 at 9:08 PM Mike Wick <michael.wick.1980 at gmail.com>
> wrote:
>
>> Hello PETSc team:
>>
>> I am trying to solve a PDE problem with high-order finite elements. The
>> matrix is getting denser and my experience is that MUMPS just outperforms
>> iterative solvers.
>>
>
> If the problem is elliptic, there is a lot of evidence that the P1
> preconditioner is descent for the system. Some people
> just project the system to P1, invert that with multigrid, and use that as
> the PC for Krylov. It should be worth trying.
>

Matt means project to P1 directly from your high order function space in
one step. It is definitely worth trying.
For those interested, this approach is first described and discussed (to my
knowledge) in this paper:

Persson, Per-Olof, and Jaime Peraire. "An efficient low memory implicit DG
algorithm for time dependent problems." *44th AIAA Aerospace Sciences
Meeting and Exhibit*. 2006.


> Moreover, as Jed will tell you, forming matrices for higher order is
> counterproductive. You should apply those matrix-free.
>

I definitely agree with that.

Cheers,
  Dave



>
>   Thanks,
>
>      Matt
>
>
>> For certain problems, MUMPS just fail in the middle for no clear reason.
>> I just wander if there is any suggestion to improve the robustness of
>> MUMPS? Or in general, any suggestion for interative solver with very
>> high-order finite elements?
>>
>> Thanks!
>>
>> Mike
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/7bc2cdba/attachment.html>

From jroman at dsic.upv.es  Thu Oct 11 04:55:30 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 11 Oct 2018 11:55:30 +0200
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <CAMYG4G=9s03652Dzx92FzC4KiWS_iy5RTwWkMsK0uYZ0ahS5=w@mail.gmail.com>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>
	<61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
	<0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>
	<CAMYG4G=9s03652Dzx92FzC4KiWS_iy5RTwWkMsK0uYZ0ahS5=w@mail.gmail.com>
Message-ID: <61835618-52E0-42BC-BDAF-52FDF1662905@dsic.upv.es>

The filter technique must be used with an interval in EPS, e.g.
  -eps_interval 2.,2.7
(This interval will be passed to STFILTER, so no need to specify -st_filter_interval).
Therefore, it does not require -eps_target_magnitude or similar.

Regarding the simpler (A-tau*I)^2, we call it "spectrum folding" (see section 3.4.6 of SLEPc's manual) and it is implemented in ex24.c
http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex24.c.html

My experience is that spectrum folding will not give good convergence except for easy problems. For more difficult problems with clustered eigenvalues you need a polynomial filter. The technique of polynomial filter trades iterations of the Krylov subspace for matrix-vector products required by a high-degree polynomial.

If you want, send us the matrix to slepc-maint and we will have a try.

Jose
 

> El 11 oct 2018, a las 2:43, Matthew Knepley <knepley at gmail.com> escribi?:
> 
> On Wed, Oct 10, 2018 at 8:41 PM Moritz Cygorek <mcygorek at uottawa.ca> wrote:
> Thank you very much. Apparently, I've misunderstood what the filter actually does. I thought about the much simpler process, where you diagonalize
> 
> 
> 
> -(A- tau*I)^2 +offset*I
> 
> 
> 
> where tau is my target an offset is large enough so that the global maximum is reached for eigenvalues around tau.
> 
> 
> Is this different from -eps_target_magnitude?
> 
>   Thanks,
> 
>     Matt
>  
> Then you look for the largest eigenvalue of the modified problem and either calculate the Ritz value of the original matrix or calculate back from the eigenvalues of the modified problem.
> 
> 
> 
> Now, it looks to me like -st_type filter activates something like the package FILTLAN.
> 
> 
> 
> I guess I can define a MatShell to do the thing I intended in the first place.
> 
> But, I guess, this is a common thing, so I am wondering whether it is already implemented somewhere and I just didn't find it in the documentation.  Can you say something about this?
> 
> 
> 
> Regards,
> 
> Moritz
> 
> 
> 
> 
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: Wednesday, October 10, 2018 3:48 PM
> To: Moritz Cygorek
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] STFILTER in slepc
>  
> This type of method requires a very high degree polynomial; suggest using degree=100 at least (this is the default value), but larger values may be necessary. Also, for this particular filter the "range" must be approximately equal to the numerical range; if you have no clue where your first and last eigenvalues are, you may use EPSSolve() calls with EPS_LARGEST_REAL and EPS_SMALLEST_REAL.
> 
> Jose
> 
> > El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > 
> > Thank you for the fast reply. 
> > 
> > I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:
> > 
> > -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000
> > 
> > and I get the following error message:
> > 
> > [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
> > ....
> > [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> > [0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> > [0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> > [0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
> > 
> > 
> > 
> > Do you have a clue what I've missed?
> > 
> > 
> > Moritz
> > 
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: Wednesday, October 10, 2018 2:30 PM
> > To: Moritz Cygorek
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] STFILTER in slepc
> >  
> > 
> > > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > > 
> > > Hi,
> > > 
> > > in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER. 
> > > 
> > > Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> > > 
> > > Does anybody have an example of how to use STFILTER or any documentation about it?
> > > 
> > > Thanks in advance,
> > > Moritz
> > 
> > There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here:http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
> > 
> > In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.
> > 
> > Jose
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From yjwu16 at gmail.com  Thu Oct 11 10:37:41 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Thu, 11 Oct 2018 23:37:41 +0800
Subject: [petsc-users] Problems about Matrix-Free and DM object
Message-ID: <CAAMEhBgmLx63Bg6+3vsnj=i8RFivp2iwu+=P-dxSkCAmcbF62w@mail.gmail.com>

Dear Petsc developer:
Hi,
Thank you very much for your previous reply. I feel that I have learned a
lot about the matrix-free method.
I was studying the example /src/snes/example/tutorials/ex28.c recently. One
of its tests is as follows:

-u_da_grid_x 20 -snes_converged_reason -snes_monitor_short
-ksp_monitor_short -problem_type 2 -snes_mf_operator -pack_dm_mat_type
{{aij nest}} -pc_type fieldsplit -pc_fieldsplit_dm_splits
-pc_fieldsplit_type additive -fieldsplit_u_ksp_type gmres
-fieldsplit_k_pc_type jacobi

The example is calculated by the Matrix-free method, and the precondition
matrix is divided by FieldSplit. There are several questions in the
procedure that make me feel uncertain.
1. The program uses the Matrixfree method by the command
"-snes_mf_operator", but the form of Jacobian matrix is not defined as
MATMFFD in the program. Does the program automatically set the Jacobian
matrix to the form of MATMFFD after opening the command?

Both Jacobian matrix and precondition matrix are defined as B in the
program, and matrix elements are provided by Jacobian evaluation program.
But the Jacobian matrix in the Matrixfree method is in the form of MATMFFD.
You mentioned earlier that adding elements to the matrix with type MATMFFD
can cause program errors, but why does the program work in this example?
The procedures are as follows:

ierr = SNESSetFunction(snes,F,FormFunction_All,user);CHKERRQ(ierr);
ierr = SNESSetJacobian(snes,B,B,FormJacobian_All,user);CHKERRQ(ierr);


2.Can we use DMCompositeScatter to scatter ghost value into local vectors?
I didn't seem to mention the documentation. The program only uses the
following functions, but can use ghost value.
    ierr = DMCompositeScatter(user->pack,X,Uloc,Kloc);CHKERRQ(ierr);
    ierr = DMDAVecGetArray(dau,Uloc,&u);CHKERRQ(ierr);
    ierr = DMDAVecGetArray(dak,Kloc,&k);CHKERRQ(ierr);

Greatly appreciated your reply and attention.
Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/12704c61/attachment-0001.html>

From knepley at gmail.com  Thu Oct 11 10:54:33 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Oct 2018 11:54:33 -0400
Subject: [petsc-users] Problems about Matrix-Free and DM object
In-Reply-To: <CAAMEhBgmLx63Bg6+3vsnj=i8RFivp2iwu+=P-dxSkCAmcbF62w@mail.gmail.com>
References: <CAAMEhBgmLx63Bg6+3vsnj=i8RFivp2iwu+=P-dxSkCAmcbF62w@mail.gmail.com>
Message-ID: <CAMYG4GkdWCh0-aBuG-cC_DNaWhVEqdEwDrtOof5V5CkEzADhcg@mail.gmail.com>

On Thu, Oct 11, 2018 at 11:38 AM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Dear Petsc developer:
> Hi,
> Thank you very much for your previous reply. I feel that I have learned a
> lot about the matrix-free method.
> I was studying the example /src/snes/example/tutorials/ex28.c recently.
> One of its tests is as follows:
>
> -u_da_grid_x 20 -snes_converged_reason -snes_monitor_short
> -ksp_monitor_short -problem_type 2 -snes_mf_operator -pack_dm_mat_type
> {{aij nest}} -pc_type fieldsplit -pc_fieldsplit_dm_splits
> -pc_fieldsplit_type additive -fieldsplit_u_ksp_type gmres
> -fieldsplit_k_pc_type jacobi
>
> The example is calculated by the Matrix-free method, and the precondition
> matrix is divided by FieldSplit. There are several questions in the
> procedure that make me feel uncertain.
> 1. The program uses the Matrixfree method by the command
> "-snes_mf_operator", but the form of Jacobian matrix is not defined as
> MATMFFD in the program. Does the program automatically set the Jacobian
> matrix to the form of MATMFFD after opening the command?
>

If you give that option, PETSc will automatically create a MFFD Mat object
and stick your nonlinear residual evaluation in it.


> Both Jacobian matrix and precondition matrix are defined as B in the
> program, and matrix elements are provided by Jacobian evaluation program.
> But the Jacobian matrix in the Matrixfree method is in the form of MATMFFD.
> You mentioned earlier that adding elements to the matrix with type MATMFFD
> can cause program errors, but why does the program work in this example?
> The procedures are as follows:
>
> ierr = SNESSetFunction(snes,F,FormFunction_All,user);CHKERRQ(ierr);
> ierr = SNESSetJacobian(snes,B,B,FormJacobian_All,user);CHKERRQ(ierr);
>
>
Because the FoirmJacobian() function in ex28 only ever puts values into the
preconditioning matrix. This is generally the right way to proceed.


> 2.Can we use DMCompositeScatter to scatter ghost value into local vectors?
> I didn't seem to mention the documentation. The program only uses the
> following functions, but can use ghost value.
>     ierr = DMCompositeScatter(user->pack,X,Uloc,Kloc);CHKERRQ(ierr);
>     ierr = DMDAVecGetArray(dau,Uloc,&u);CHKERRQ(ierr);
>     ierr = DMDAVecGetArray(dak,Kloc,&k);CHKERRQ(ierr);
>

DMCompositeScatter splits a vector up into the smaller vectors for each
system. DMComposite is very specialized, and we do not
recommend using it except in special cases.

DMGlobalToLocal() and DMLocalToGlobal() communicate ghost values.

  Thanks,

     Matt


> Greatly appreciated your reply and attention.
> Thanks,
> Yingjie
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/b76ab3d1/attachment.html>

From mcygorek at uottawa.ca  Thu Oct 11 13:13:48 2018
From: mcygorek at uottawa.ca (Moritz Cygorek)
Date: Thu, 11 Oct 2018 18:13:48 +0000
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <61835618-52E0-42BC-BDAF-52FDF1662905@dsic.upv.es>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>
	<61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
	<0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>
	<CAMYG4G=9s03652Dzx92FzC4KiWS_iy5RTwWkMsK0uYZ0ahS5=w@mail.gmail.com>,
	<61835618-52E0-42BC-BDAF-52FDF1662905@dsic.upv.es>
Message-ID: <63a8350e7e5941d9951ee18f9a87442c@uottawa.ca>

Thank you for the response. This example is exactly the kind of thing I thought about.

But, as you say, convergence is indeed not better.



What's the best format to send the matrix to slepc-maint?

My small test case is a sparse matrix of dimension 30720. I currently read it from three files (containing IA, JA, and the values in CSR format). Because everything is contained in a larger framework, it's not very easy to extract the reader.  Is there a generic way to read in sparse matrices from files for PETSc?




Maybe you can already give me a hint about the best method if I tell you the properties of the spectrum:

The problem is the calculation of electronic states in a quantum dot.  There is a band gap from around 0 to 2 and I am interested in the first 20 or 40 eigenvectors above and below the gap.

The first few states are separated by about 0.1 but for higher states the energies come closer and closer together.

Additionally, there are a number of states with energies around 1000, which are artificial and originate from the way we treat the boundary conditions. Also, I know that all states come in pairs with the same energy (Kramers degeneracy).


For the small test case, the conduction band states, i.e. eigenvectors with energies close to 2, converge very fast (about 5 min on a laptop computer). However, the states with energies around 0 converge much more slowly and that's one of my major problems.

For those states harmonic extraction seems to be better suited, but I have the impression that it is not extremely stable. For example, applied to the states close to 2 I see that some states are skipped, which can be seen by the fact that the degeneracies are sometimes wrong.

Also, with harmonic extraction, the program sometimes stops claiming the number of requested eigenpairs are reached, but the calculated relative error of most states is way larger than the tolerance.


Maybe you know from experience which method is better suited to tackle these kinds of problems?



Eventually, I intend to do calculates with dimensions of ~ 10 million distributed of a few 100 CPUs.


Regards,
Moritz




________________________________
From: Jose E. Roman <jroman at dsic.upv.es>
Sent: Thursday, October 11, 2018 5:55 AM
To: Matthew Knepley; Moritz Cygorek
Cc: PETSc; Carmen Campos
Subject: Re: [petsc-users] STFILTER in slepc

The filter technique must be used with an interval in EPS, e.g.
  -eps_interval 2.,2.7
(This interval will be passed to STFILTER, so no need to specify -st_filter_interval).
Therefore, it does not require -eps_target_magnitude or similar.

Regarding the simpler (A-tau*I)^2, we call it "spectrum folding" (see section 3.4.6 of SLEPc's manual) and it is implemented in ex24.c
http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex24.c.html

My experience is that spectrum folding will not give good convergence except for easy problems. For more difficult problems with clustered eigenvalues you need a polynomial filter. The technique of polynomial filter trades iterations of the Krylov subspace for matrix-vector products required by a high-degree polynomial.

If you want, send us the matrix to slepc-maint and we will have a try.

Jose


> El 11 oct 2018, a las 2:43, Matthew Knepley <knepley at gmail.com> escribi?:
>
> On Wed, Oct 10, 2018 at 8:41 PM Moritz Cygorek <mcygorek at uottawa.ca> wrote:
> Thank you very much. Apparently, I've misunderstood what the filter actually does. I thought about the much simpler process, where you diagonalize
>
>
>
> -(A- tau*I)^2 +offset*I
>
>
>
> where tau is my target an offset is large enough so that the global maximum is reached for eigenvalues around tau.
>
>
> Is this different from -eps_target_magnitude?
>
>   Thanks,
>
>     Matt
>
> Then you look for the largest eigenvalue of the modified problem and either calculate the Ritz value of the original matrix or calculate back from the eigenvalues of the modified problem.
>
>
>
> Now, it looks to me like -st_type filter activates something like the package FILTLAN.
>
>
>
> I guess I can define a MatShell to do the thing I intended in the first place.
>
> But, I guess, this is a common thing, so I am wondering whether it is already implemented somewhere and I just didn't find it in the documentation.  Can you say something about this?
>
>
>
> Regards,
>
> Moritz
>
>
>
>
>
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: Wednesday, October 10, 2018 3:48 PM
> To: Moritz Cygorek
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] STFILTER in slepc
>
> This type of method requires a very high degree polynomial; suggest using degree=100 at least (this is the default value), but larger values may be necessary. Also, for this particular filter the "range" must be approximately equal to the numerical range; if you have no clue where your first and last eigenvalues are, you may use EPSSolve() calls with EPS_LARGEST_REAL and EPS_SMALLEST_REAL.
>
> Jose
>
> > El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> >
> > Thank you for the fast reply.
> >
> > I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:
> >
> > -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000
> >
> > and I get the following error message:
> >
> > [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
> > ....
> > [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> > [0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> > [0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> > [0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
> >
> >
> >
> > Do you have a clue what I've missed?
> >
> >
> > Moritz
> >
> >
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: Wednesday, October 10, 2018 2:30 PM
> > To: Moritz Cygorek
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] STFILTER in slepc
> >
> >
> > > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > >
> > > Hi,
> > >
> > > in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER.
> > >
> > > Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> > >
> > > Does anybody have an example of how to use STFILTER or any documentation about it?
> > >
> > > Thanks in advance,
> > > Moritz
> >
> > There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here:http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
> >
> > In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.
> >
> > Jose
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/3e7420e2/attachment.html>

From jroman at dsic.upv.es  Thu Oct 11 14:15:46 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 11 Oct 2018 21:15:46 +0200
Subject: [petsc-users] STFILTER in slepc
In-Reply-To: <63a8350e7e5941d9951ee18f9a87442c@uottawa.ca>
References: <e1b60698d02e4382801e8aa46ae06772@uottawa.ca>
	<278D7A53-FA81-4980-8392-9227C3437889@dsic.upv.es>
	<867180c290734d1da494a13d49fcbc7e@uottawa.ca>
	<61D97868-F6C6-4C4D-8DF5-38F2BB3E283F@dsic.upv.es>
	<0c920e55c6f946ea96359db9b0adb78a@uottawa.ca>
	<CAMYG4G=9s03652Dzx92FzC4KiWS_iy5RTwWkMsK0uYZ0ahS5=w@mail.gmail.com>
	<61835618-52E0-42BC-BDAF-52FDF1662905@dsic.upv.es>
	<63a8350e7e5941d9951ee18f9a87442c@uottawa.ca>
Message-ID: <BD6139C7-629E-4C16-8A15-22B63692CB27@dsic.upv.es>

Run your code with -eps_view_mat0 binary:amatrix.bin
This will save the A matrix of the EPS solver in file "amatrix.bin" at the end of EPSSolve().
I cannot say which is the best method, it depends on the distribution of the spectrum. Let's see how it goes.

Jose


> El 11 oct 2018, a las 20:13, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> 
> Thank you for the response. This example is exactly the kind of thing I thought about.
> But, as you say, convergence is indeed not better. 
> 
> 
> What's the best format to send the matrix to slepc-maint? 
> My small test case is a sparse matrix of dimension 30720. I currently read it from three files (containing IA, JA, and the values in CSR format). Because everything is contained in a larger framework, it's not very easy to extract the reader.  Is there a generic way to read in sparse matrices from files for PETSc?
> 
> 
> 
> Maybe you can already give me a hint about the best method if I tell you the properties of the spectrum:
> The problem is the calculation of electronic states in a quantum dot.  There is a band gap from around 0 to 2 and I am interested in the first 20 or 40 eigenvectors above and below the gap.
> The first few states are separated by about 0.1 but for higher states the energies come closer and closer together.
> Additionally, there are a number of states with energies around 1000, which are artificial and originate from the way we treat the boundary conditions. Also, I know that all states come in pairs with the same energy (Kramers degeneracy).
> 
> For the small test case, the conduction band states, i.e. eigenvectors with energies close to 2, converge very fast (about 5 min on a laptop computer). However, the states with energies around 0 converge much more slowly and that's one of my major problems. 
> For those states harmonic extraction seems to be better suited, but I have the impression that it is not extremely stable. For example, applied to the states close to 2 I see that some states are skipped, which can be seen by the fact that the degeneracies are sometimes wrong. 
> Also, with harmonic extraction, the program sometimes stops claiming the number of requested eigenpairs are reached, but the calculated relative error of most states is way larger than the tolerance.
> 
> Maybe you know from experience which method is better suited to tackle these kinds of problems?
> 
> 
> Eventually, I intend to do calculates with dimensions of ~ 10 million distributed of a few 100 CPUs.
> 
> Regards,
> Moritz
> 
> 
> 
> 
> From: Jose E. Roman <jroman at dsic.upv.es>
> Sent: Thursday, October 11, 2018 5:55 AM
> To: Matthew Knepley; Moritz Cygorek
> Cc: PETSc; Carmen Campos
> Subject: Re: [petsc-users] STFILTER in slepc
>  
> The filter technique must be used with an interval in EPS, e.g.
>   -eps_interval 2.,2.7
> (This interval will be passed to STFILTER, so no need to specify -st_filter_interval).
> Therefore, it does not require -eps_target_magnitude or similar.
> 
> Regarding the simpler (A-tau*I)^2, we call it "spectrum folding" (see section 3.4.6 of SLEPc's manual) and it is implemented in ex24.c
> http://slepc.upv.es/documentation/current/src/eps/examples/tutorials/ex24.c.html
> 
> My experience is that spectrum folding will not give good convergence except for easy problems. For more difficult problems with clustered eigenvalues you need a polynomial filter. The technique of polynomial filter trades iterations of the Krylov subspace for matrix-vector products required by a high-degree polynomial.
> 
> If you want, send us the matrix to slepc-maint and we will have a try.
> 
> Jose
>  
> 
> > El 11 oct 2018, a las 2:43, Matthew Knepley <knepley at gmail.com> escribi?:
> > 
> > On Wed, Oct 10, 2018 at 8:41 PM Moritz Cygorek <mcygorek at uottawa.ca> wrote:
> > Thank you very much. Apparently, I've misunderstood what the filter actually does. I thought about the much simpler process, where you diagonalize
> > 
> > 
> > 
> > -(A- tau*I)^2 +offset*I
> > 
> > 
> > 
> > where tau is my target an offset is large enough so that the global maximum is reached for eigenvalues around tau.
> > 
> > 
> > Is this different from -eps_target_magnitude?
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > Then you look for the largest eigenvalue of the modified problem and either calculate the Ritz value of the original matrix or calculate back from the eigenvalues of the modified problem.
> > 
> > 
> > 
> > Now, it looks to me like -st_type filter activates something like the package FILTLAN.
> > 
> > 
> > 
> > I guess I can define a MatShell to do the thing I intended in the first place.
> > 
> > But, I guess, this is a common thing, so I am wondering whether it is already implemented somewhere and I just didn't find it in the documentation.  Can you say something about this?
> > 
> > 
> > 
> > Regards,
> > 
> > Moritz
> > 
> > 
> > 
> > 
> > 
> > From: Jose E. Roman <jroman at dsic.upv.es>
> > Sent: Wednesday, October 10, 2018 3:48 PM
> > To: Moritz Cygorek
> > Cc: petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] STFILTER in slepc
> >  
> > This type of method requires a very high degree polynomial; suggest using degree=100 at least (this is the default value), but larger values may be necessary. Also, for this particular filter the "range" must be approximately equal to the numerical range; if you have no clue where your first and last eigenvalues are, you may use EPSSolve() calls with EPS_LARGEST_REAL and EPS_SMALLEST_REAL.
> > 
> > Jose
> > 
> > > El 10 oct 2018, a las 21:10, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > > 
> > > Thank you for the fast reply. 
> > > 
> > > I've tried running my program (using the defaul Krylov-Schur method for sparse MPI matrices) with the additional options:
> > > 
> > > -st_type filter -st_filter_degree 2 -st_filter_interval 2.,2.7  -st_filter_range -2000,2000
> > > 
> > > and I get the following error message:
> > > 
> > > [0]PETSC ERROR: STFILTER cannot get the filter specified; please adjust your filter parameters (e.g. increasing the polynomial degree)
> > > ....
> > > [0]PETSC ERROR: #1 FILTLAN_GetIntervals() line 451 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > > [0]PETSC ERROR: #2 STFilter_FILTLAN_setFilter() line 1016 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filtlan.c
> > > [0]PETSC ERROR: #3 STSetUp_Filter() line 42 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/impls/filter/filter.c
> > > [0]PETSC ERROR: #4 STSetUp() line 271 in /home/applications/sources/libraries/slepc-3.9.2/src/sys/classes/st/interface/stsolve.c
> > > [0]PETSC ERROR: #5 EPSSetUp() line 263 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssetup.c
> > > [0]PETSC ERROR: #6 EPSSolve() line 135 in /home/applications/sources/libraries/slepc-3.9.2/src/eps/interface/epssolve.c
> > > 
> > > 
> > > 
> > > Do you have a clue what I've missed?
> > > 
> > > 
> > > Moritz
> > > 
> > > 
> > > From: Jose E. Roman <jroman at dsic.upv.es>
> > > Sent: Wednesday, October 10, 2018 2:30 PM
> > > To: Moritz Cygorek
> > > Cc: petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] STFILTER in slepc
> > >  
> > > 
> > > > El 10 oct 2018, a las 19:54, Moritz Cygorek <mcygorek at uottawa.ca> escribi?:
> > > > 
> > > > Hi,
> > > > 
> > > > in the list of changes to SLEPc version 3.8, it is stated that there is a preliminary implementation of polynomial filtering using STFILTER. 
> > > > 
> > > > Because I am struggling to obtain interior eigenvalues and harmonic extraction seems not to be stable enough in my case, I wanted to give it a try, but I could not find any documentation yet.
> > > > 
> > > > Does anybody have an example of how to use STFILTER or any documentation about it?
> > > > 
> > > > Thanks in advance,
> > > > Moritz
> > > 
> > > There are no examples. You just set the type to STFILTER and set some parameters such as the interval of interest or the polynomial degree. See functions starting with STFilter here:http://slepc.upv.es/documentation/current/docs/manualpages/ST/index.html
> > > 
> > > In some problems it works well, but don't expect too much. It is still in our to-do list to make it more usable. It will be good to have your feedback. If you want, send results to slepc-maint, maybe we can help tuning the parameters.
> > > 
> > > Jose
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/


From michael.wick.1980 at gmail.com  Thu Oct 11 14:26:35 2018
From: michael.wick.1980 at gmail.com (Michael Wick)
Date: Thu, 11 Oct 2018 12:26:35 -0700
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
	<CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>
Message-ID: <CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>

Thanks for all the suggestions!

Increasing the value of icntl_14 in MUMPS helps a lot for my case.

Do you have any suggestions for higher-order methods in saddle-point
problems?

Mike

Dave May <dave.mayhem23 at gmail.com> ?2018?10?11??? ??1:50???

>
>
> On Sat, 6 Oct 2018 at 12:42, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Oct 5, 2018 at 9:08 PM Mike Wick <michael.wick.1980 at gmail.com>
>> wrote:
>>
>>> Hello PETSc team:
>>>
>>> I am trying to solve a PDE problem with high-order finite elements. The
>>> matrix is getting denser and my experience is that MUMPS just outperforms
>>> iterative solvers.
>>>
>>
>> If the problem is elliptic, there is a lot of evidence that the P1
>> preconditioner is descent for the system. Some people
>> just project the system to P1, invert that with multigrid, and use that
>> as the PC for Krylov. It should be worth trying.
>>
>
> Matt means project to P1 directly from your high order function space in
> one step. It is definitely worth trying.
> For those interested, this approach is first described and discussed (to
> my knowledge) in this paper:
>
> Persson, Per-Olof, and Jaime Peraire. "An efficient low memory implicit DG
> algorithm for time dependent problems." *44th AIAA Aerospace Sciences
> Meeting and Exhibit*. 2006.
>
>
>> Moreover, as Jed will tell you, forming matrices for higher order is
>> counterproductive. You should apply those matrix-free.
>>
>
> I definitely agree with that.
>
> Cheers,
>   Dave
>
>
>
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> For certain problems, MUMPS just fail in the middle for no clear reason.
>>> I just wander if there is any suggestion to improve the robustness of
>>> MUMPS? Or in general, any suggestion for interative solver with very
>>> high-order finite elements?
>>>
>>> Thanks!
>>>
>>> Mike
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/aca6cbcb/attachment-0001.html>

From ellen.price at cfa.harvard.edu  Thu Oct 11 15:38:50 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Thu, 11 Oct 2018 16:38:50 -0400
Subject: [petsc-users] DMPLEX project function error
Message-ID: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>

I was working with a DMPLEX and FEM, following SNES example 12. I get
the following error when I call DMProjectFunction, but I don't know what
it means. Can anyone explain where I might have gone wrong, or at least
what this error is telling me? I think the point closure size is
correct, since my mesh is 3d simplex, but what is the dual space
dimension, and where might I have set it incorrectly?

[0]PETSC ERROR: Nonconforming object sizes
[0]PETSC ERROR: The section point closure size 4 != dual space dimension 1
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
...
[0]PETSC ERROR: Configure options
--prefix=/home/eprice/software/petsc-opt --with-hdf5=1
--with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1
--with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native -mtune=native"
CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
-march=native -mtune=native" --with-mpi=1
--with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
--with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
--with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1
--with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1
--with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1
--with-scalapack-dir=/home/eprice/software/scalapack
[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
[0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
[0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
...

(I know this is an optimized PETSc build, but I get the same error from
my debug build, it's just much slower.)

Ellen

From knepley at gmail.com  Thu Oct 11 16:16:45 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Oct 2018 17:16:45 -0400
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
	<CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>
	<CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>
Message-ID: <CAMYG4Gk2RockG9Xyipceji3Vw0ZmPVEUSfdX2+dfsAsV_kWd9Q@mail.gmail.com>

On Thu, Oct 11, 2018 at 3:26 PM Michael Wick <michael.wick.1980 at gmail.com>
wrote:

> Thanks for all the suggestions!
>
> Increasing the value of icntl_14 in MUMPS helps a lot for my case.
>
> Do you have any suggestions for higher-order methods in saddle-point
> problems?
>

It depends on what the Schur complement looks like.

   Matt


> Mike
>
> Dave May <dave.mayhem23 at gmail.com> ?2018?10?11??? ??1:50???
>
>>
>>
>> On Sat, 6 Oct 2018 at 12:42, Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Fri, Oct 5, 2018 at 9:08 PM Mike Wick <michael.wick.1980 at gmail.com>
>>> wrote:
>>>
>>>> Hello PETSc team:
>>>>
>>>> I am trying to solve a PDE problem with high-order finite elements. The
>>>> matrix is getting denser and my experience is that MUMPS just outperforms
>>>> iterative solvers.
>>>>
>>>
>>> If the problem is elliptic, there is a lot of evidence that the P1
>>> preconditioner is descent for the system. Some people
>>> just project the system to P1, invert that with multigrid, and use that
>>> as the PC for Krylov. It should be worth trying.
>>>
>>
>> Matt means project to P1 directly from your high order function space in
>> one step. It is definitely worth trying.
>> For those interested, this approach is first described and discussed (to
>> my knowledge) in this paper:
>>
>> Persson, Per-Olof, and Jaime Peraire. "An efficient low memory implicit
>> DG algorithm for time dependent problems." *44th AIAA Aerospace Sciences
>> Meeting and Exhibit*. 2006.
>>
>>
>>> Moreover, as Jed will tell you, forming matrices for higher order is
>>> counterproductive. You should apply those matrix-free.
>>>
>>
>> I definitely agree with that.
>>
>> Cheers,
>>   Dave
>>
>>
>>
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> For certain problems, MUMPS just fail in the middle for no clear
>>>> reason. I just wander if there is any suggestion to improve the robustness
>>>> of MUMPS? Or in general, any suggestion for interative solver with very
>>>> high-order finite elements?
>>>>
>>>> Thanks!
>>>>
>>>> Mike
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/3dbde352/attachment.html>

From knepley at gmail.com  Thu Oct 11 16:41:51 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 11 Oct 2018 17:41:51 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
Message-ID: <CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>

On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price <ellen.price at cfa.harvard.edu>
wrote:

> I was working with a DMPLEX and FEM, following SNES example 12. I get
> the following error when I call DMProjectFunction, but I don't know what
> it means. Can anyone explain where I might have gone wrong, or at least
> what this error is telling me? I think the point closure size is
> correct, since my mesh is 3d simplex,


Yes, if you have 3D SIMPLEX mesh and are using P1 elements, then you would
have
4 dofs in the closure of a cell. The dual space dimension is the number of
dual space
basis vectors assigned to points in the closure. Since it is 1, it looks
like you have a P0
dual space. I assume you changed something in ex12?

  Thanks,

     Matt


> but what is the dual space
> dimension, and where might I have set it incorrectly?
>
> [0]PETSC ERROR: Nonconforming object sizes
> [0]PETSC ERROR: The section point closure size 4 != dual space dimension 1
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
> ...
> [0]PETSC ERROR: Configure options
> --prefix=/home/eprice/software/petsc-opt --with-hdf5=1
> --with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1
> --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
> LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native -mtune=native"
> CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
> -march=native -mtune=native" --with-mpi=1
> --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
> --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
> --with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1
> --with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1
> --with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1
> --with-scalapack-dir=/home/eprice/software/scalapack
> [0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
> /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> [0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
> /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> [0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
> /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> [0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
> /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> ...
>
> (I know this is an optimized PETSc build, but I get the same error from
> my debug build, it's just much slower.)
>
> Ellen
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181011/c3e36257/attachment.html>

From dave.mayhem23 at gmail.com  Fri Oct 12 02:33:44 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Fri, 12 Oct 2018 08:33:44 +0100
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
	<CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>
	<CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>
Message-ID: <CAJ98EDrNPng6XLkBQ0cr_QJ9DN496aNGoif3yh2MyEie9nB4Dg@mail.gmail.com>

On Thu, 11 Oct 2018 at 20:26, Michael Wick <michael.wick.1980 at gmail.com>
wrote:

> Thanks for all the suggestions!
>
> Increasing the value of icntl_14 in MUMPS helps a lot for my case.
>
> Do you have any suggestions for higher-order methods in saddle-point
> problems?
>

If the saddle point system arises from Stokes or an incompressible
elasticity formulation, then the standard block factorizations of
Silvester, Elman, Wathan will work very well for high-order - assuming of
course you use inf-sup stable basis for u/p. For Stokes/elasticity, the
pressure mass matrix is a decent spectrally equivalent operator for the
Schur complement.

These preconditioners are discussed here:

* Michele Benzi, Gene H. Golub, and J?rg Liesen, Numerical solution of
saddle point problems,
Acta Numerica, 14 (2005), pp. 1?137.

* Howard C. Elman, David J. Silvester, and Andrew J. Wathen, Finite
elements and fast iterative
solvers: with applications in incompressible fluid dynamics, Oxford
University Press, 2014.

High order examples can be found here:

* https://arxiv.org/abs/1607.03936

* Rudi, Johann, Georg Stadler, and Omar Ghattas. "Weighted BFBT
Preconditioner for Stokes Flow Problems with Highly Heterogeneous
Viscosity." SIAM Journal on Scientific Computing 39.5 (2017): S272-S297.

Note that in the Rudi et al papers, due the highly variable nature of the
viscosity, the authors advocate using a more complex definition of the
preconditioner for the Schur complement. Whether you need to use their
approach is dependent on the nature of the problem you are solving.

Thanks,
  Dave






>
> Mike
>
> Dave May <dave.mayhem23 at gmail.com> ?2018?10?11??? ??1:50???
>
>>
>>
>> On Sat, 6 Oct 2018 at 12:42, Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Fri, Oct 5, 2018 at 9:08 PM Mike Wick <michael.wick.1980 at gmail.com>
>>> wrote:
>>>
>>>> Hello PETSc team:
>>>>
>>>> I am trying to solve a PDE problem with high-order finite elements. The
>>>> matrix is getting denser and my experience is that MUMPS just outperforms
>>>> iterative solvers.
>>>>
>>>
>>> If the problem is elliptic, there is a lot of evidence that the P1
>>> preconditioner is descent for the system. Some people
>>> just project the system to P1, invert that with multigrid, and use that
>>> as the PC for Krylov. It should be worth trying.
>>>
>>
>> Matt means project to P1 directly from your high order function space in
>> one step. It is definitely worth trying.
>> For those interested, this approach is first described and discussed (to
>> my knowledge) in this paper:
>>
>> Persson, Per-Olof, and Jaime Peraire. "An efficient low memory implicit
>> DG algorithm for time dependent problems." *44th AIAA Aerospace Sciences
>> Meeting and Exhibit*. 2006.
>>
>>
>>> Moreover, as Jed will tell you, forming matrices for higher order is
>>> counterproductive. You should apply those matrix-free.
>>>
>>
>> I definitely agree with that.
>>
>> Cheers,
>>   Dave
>>
>>
>>
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> For certain problems, MUMPS just fail in the middle for no clear
>>>> reason. I just wander if there is any suggestion to improve the robustness
>>>> of MUMPS? Or in general, any suggestion for interative solver with very
>>>> high-order finite elements?
>>>>
>>>> Thanks!
>>>>
>>>> Mike
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/815ac8dc/attachment-0001.html>

From barrydog505 at gmail.com  Fri Oct 12 03:56:51 2018
From: barrydog505 at gmail.com (=?UTF-8?B?6Zmz5a6X6IiI?=)
Date: Fri, 12 Oct 2018 16:56:51 +0800
Subject: [petsc-users] How to get nodes list refer to unstructure mesh
 element
Message-ID: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>

Hi all,

I have import a .msh file from DMPlexCreateFromFile.
I have found DMDAGetElements in DMDA's ex5 having this function,
how can i do this from DMPlex or unstructure grid.
Are there any methods that can get nodes list refer to unstructure mesh
element?
------------------------------------
for example :
       4
     /     \
   /    1   \
 2 - - - - - 3           element [1]  >> nodes list [2 3 4]

Thanks a lot.

Best regards,
Barry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/d79933e5/attachment.html>

From mfadams at lbl.gov  Fri Oct 12 04:21:50 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Fri, 12 Oct 2018 05:21:50 -0400
Subject: [petsc-users] How to get nodes list refer to unstructure mesh
 element
In-Reply-To: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>
References: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>
Message-ID: <CADOhEh5EVfK5X4XJzMHQ6iwwvYfvpu25Pg-ttiDe=p7N++3R-g@mail.gmail.com>

You want to use DMPlexVecGetClosure:

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexVecGetClosure.html

The examples shown here don't close cells. What you
want. src/dm/impls/plex/examples/tutorials/ex6.c does iterate over cells,
and get the closure (vertices), which is what you want.

Mark



On Fri, Oct 12, 2018 at 4:58 AM ??? <barrydog505 at gmail.com> wrote:

> Hi all,
>
> I have import a .msh file from DMPlexCreateFromFile.
> I have found DMDAGetElements in DMDA's ex5 having this function,
> how can i do this from DMPlex or unstructure grid.
> Are there any methods that can get nodes list refer to unstructure mesh
> element?
> ------------------------------------
> for example :
>        4
>      /     \
>    /    1   \
>  2 - - - - - 3           element [1]  >> nodes list [2 3 4]
>
> Thanks a lot.
>
> Best regards,
> Barry
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/d5b7b7d2/attachment.html>

From barrydog505 at gmail.com  Fri Oct 12 04:55:15 2018
From: barrydog505 at gmail.com (=?UTF-8?B?6Zmz5a6X6IiI?=)
Date: Fri, 12 Oct 2018 17:55:15 +0800
Subject: [petsc-users] How to get nodes list refer to unstructure mesh
 element
In-Reply-To: <CADOhEh5EVfK5X4XJzMHQ6iwwvYfvpu25Pg-ttiDe=p7N++3R-g@mail.gmail.com>
References: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>
	<CADOhEh5EVfK5X4XJzMHQ6iwwvYfvpu25Pg-ttiDe=p7N++3R-g@mail.gmail.com>
Message-ID: <CANZ1gTZUyggy5BjiCWMSK+-0ibMLaHgrZyj+gNgHJrqJWp24BQ@mail.gmail.com>

Thanks, I'll check them out.

Mark Adams <mfadams at lbl.gov> ? 2018?10?12? ?? ??5:22???

> You want to use DMPlexVecGetClosure:
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexVecGetClosure.html
>
> The examples shown here don't close cells. What you
> want. src/dm/impls/plex/examples/tutorials/ex6.c does iterate over cells,
> and get the closure (vertices), which is what you want.
>
> Mark
>
>
>
> On Fri, Oct 12, 2018 at 4:58 AM ??? <barrydog505 at gmail.com> wrote:
>
>> Hi all,
>>
>> I have import a .msh file from DMPlexCreateFromFile.
>> I have found DMDAGetElements in DMDA's ex5 having this function,
>> how can i do this from DMPlex or unstructure grid.
>> Are there any methods that can get nodes list refer to unstructure mesh
>> element?
>> ------------------------------------
>> for example :
>>        4
>>      /     \
>>    /    1   \
>>  2 - - - - - 3           element [1]  >> nodes list [2 3 4]
>>
>> Thanks a lot.
>>
>> Best regards,
>> Barry
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/e0308331/attachment.html>

From knepley at gmail.com  Fri Oct 12 06:04:54 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Oct 2018 07:04:54 -0400
Subject: [petsc-users] How to get nodes list refer to unstructure mesh
 element
In-Reply-To: <CANZ1gTZUyggy5BjiCWMSK+-0ibMLaHgrZyj+gNgHJrqJWp24BQ@mail.gmail.com>
References: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>
	<CADOhEh5EVfK5X4XJzMHQ6iwwvYfvpu25Pg-ttiDe=p7N++3R-g@mail.gmail.com>
	<CANZ1gTZUyggy5BjiCWMSK+-0ibMLaHgrZyj+gNgHJrqJWp24BQ@mail.gmail.com>
Message-ID: <CAMYG4Gk5qVCM-picnZ9TuMHf7k9to4p3uakffq7zYitOCkTApw@mail.gmail.com>

If you only have cell-vertex meshes, then using the single-layer version
DMPlexGetCone() is faster.

  Thanks,

     Matt

On Fri, Oct 12, 2018 at 5:56 AM ??? <barrydog505 at gmail.com> wrote:

> Thanks, I'll check them out.
>
> Mark Adams <mfadams at lbl.gov> ? 2018?10?12? ?? ??5:22???
>
>> You want to use DMPlexVecGetClosure:
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexVecGetClosure.html
>>
>> The examples shown here don't close cells. What you
>> want. src/dm/impls/plex/examples/tutorials/ex6.c does iterate over cells,
>> and get the closure (vertices), which is what you want.
>>
>> Mark
>>
>>
>>
>> On Fri, Oct 12, 2018 at 4:58 AM ??? <barrydog505 at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I have import a .msh file from DMPlexCreateFromFile.
>>> I have found DMDAGetElements in DMDA's ex5 having this function,
>>> how can i do this from DMPlex or unstructure grid.
>>> Are there any methods that can get nodes list refer to unstructure mesh
>>> element?
>>> ------------------------------------
>>> for example :
>>>        4
>>>      /     \
>>>    /    1   \
>>>  2 - - - - - 3           element [1]  >> nodes list [2 3 4]
>>>
>>> Thanks a lot.
>>>
>>> Best regards,
>>> Barry
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/82069ce5/attachment.html>

From barrydog505 at gmail.com  Fri Oct 12 06:09:41 2018
From: barrydog505 at gmail.com (=?UTF-8?B?6Zmz5a6X6IiI?=)
Date: Fri, 12 Oct 2018 19:09:41 +0800
Subject: [petsc-users] How to get nodes list refer to unstructure mesh
 element
In-Reply-To: <CAMYG4Gk5qVCM-picnZ9TuMHf7k9to4p3uakffq7zYitOCkTApw@mail.gmail.com>
References: <CANZ1gTZzVDkmLjv3Yrnw3LchN4hsVtLN=cOaLxeeh_gxK3Fumg@mail.gmail.com>
	<CADOhEh5EVfK5X4XJzMHQ6iwwvYfvpu25Pg-ttiDe=p7N++3R-g@mail.gmail.com>
	<CANZ1gTZUyggy5BjiCWMSK+-0ibMLaHgrZyj+gNgHJrqJWp24BQ@mail.gmail.com>
	<CAMYG4Gk5qVCM-picnZ9TuMHf7k9to4p3uakffq7zYitOCkTApw@mail.gmail.com>
Message-ID: <CANZ1gTb4QMs6EX+t=b5ewJWvK0NORzz4nz0G2w5m6mqCy2pYWQ@mail.gmail.com>

Thanks for the tip!

Matthew Knepley <knepley at gmail.com> ? 2018?10?12? ?? ??7:05???

> If you only have cell-vertex meshes, then using the single-layer version
> DMPlexGetCone() is faster.
>
>   Thanks,
>
>      Matt
>
> On Fri, Oct 12, 2018 at 5:56 AM ??? <barrydog505 at gmail.com> wrote:
>
>> Thanks, I'll check them out.
>>
>> Mark Adams <mfadams at lbl.gov> ? 2018?10?12? ?? ??5:22???
>>
>>> You want to use DMPlexVecGetClosure:
>>>
>>>
>>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexVecGetClosure.html
>>>
>>> The examples shown here don't close cells. What you
>>> want. src/dm/impls/plex/examples/tutorials/ex6.c does iterate over cells,
>>> and get the closure (vertices), which is what you want.
>>>
>>> Mark
>>>
>>>
>>>
>>> On Fri, Oct 12, 2018 at 4:58 AM ??? <barrydog505 at gmail.com> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have import a .msh file from DMPlexCreateFromFile.
>>>> I have found DMDAGetElements in DMDA's ex5 having this function,
>>>> how can i do this from DMPlex or unstructure grid.
>>>> Are there any methods that can get nodes list refer to unstructure mesh
>>>> element?
>>>> ------------------------------------
>>>> for example :
>>>>        4
>>>>      /     \
>>>>    /    1   \
>>>>  2 - - - - - 3           element [1]  >> nodes list [2 3 4]
>>>>
>>>> Thanks a lot.
>>>>
>>>> Best regards,
>>>> Barry
>>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/15e22252/attachment-0001.html>

From Moritz.Huck at isea.rwth-aachen.de  Fri Oct 12 07:38:23 2018
From: Moritz.Huck at isea.rwth-aachen.de (Moritz.Huck at isea.rwth-aachen.de)
Date: Fri, 12 Oct 2018 12:38:23 +0000
Subject: [petsc-users] Order Adapting BDF timestepping
Message-ID: <DCDA345E0DF297469F4D8DCE22B44A25C0F1D4D1@Blitz7.isea.rwth-aachen.de>

Hi,

does the BDF alogrithm adapt the order?

If not are there method to get  error estimates for orders higher and lower
than the current one?

(or adaption strategies which dont need these estimates)?

 

Thanks,

Moritz

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/610c5355/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 7669 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/610c5355/attachment.p7s>

From emconsta at anl.gov  Fri Oct 12 09:14:28 2018
From: emconsta at anl.gov (Constantinescu, Emil M.)
Date: Fri, 12 Oct 2018 14:14:28 +0000
Subject: [petsc-users] Order Adapting BDF timestepping
In-Reply-To: <DCDA345E0DF297469F4D8DCE22B44A25C0F1D4D1@Blitz7.isea.rwth-aachen.de>
References: <DCDA345E0DF297469F4D8DCE22B44A25C0F1D4D1@Blitz7.isea.rwth-aachen.de>
Message-ID: <cefd3cbe-f877-f81e-b995-541915e6c1ab@anl.gov>

No, the current implementation does not seem to adapt the order, only 
the time step. With increasing order, you typically get less and less 
stability. There is no method of order 7 or higher. Order 6 is barely 
stable. So it's not a lot of wiggle room to adapt the order in there.

Emil

On 10/12/18 7:38 AM, Moritz.Huck at isea.rwth-aachen.de wrote:
> Hi,
> 
> does the BDF alogrithm adapt the order?
> 
> If not are there method to get? error estimates for orders higher and 
> lower than the current one?
> 
> (or adaption strategies which dont need these estimates)?
> 
> Thanks,
> 
> Moritz
> 

From jed at jedbrown.org  Fri Oct 12 10:03:20 2018
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 12 Oct 2018 09:03:20 -0600
Subject: [petsc-users] Failure of MUMPS
In-Reply-To: <CAJ98EDrNPng6XLkBQ0cr_QJ9DN496aNGoif3yh2MyEie9nB4Dg@mail.gmail.com>
References: <CAA24MXukGSMd74TpahqQMwTzo+fU4_PP-utt2f6F1gtM9zwRxg@mail.gmail.com>
	<CAMYG4G=wHb9yOkKxBLR3983n2gy+mbMXhQsFCp_JAHtVUE_iaQ@mail.gmail.com>
	<CAJ98EDokwnUbzWLkOZd+3=Wpa9zrJNCb19tt8cgtQyC+LiV1MQ@mail.gmail.com>
	<CAMH7-4kr7POcz-btfdMuF0NNaXf0+en0y8U89__+2BAr=b8EqA@mail.gmail.com>
	<CAJ98EDrNPng6XLkBQ0cr_QJ9DN496aNGoif3yh2MyEie9nB4Dg@mail.gmail.com>
Message-ID: <871s8vgqbr.fsf@jedbrown.org>

Dave May <dave.mayhem23 at gmail.com> writes:

> * Rudi, Johann, Georg Stadler, and Omar Ghattas. "Weighted BFBT
> Preconditioner for Stokes Flow Problems with Highly Heterogeneous
> Viscosity." SIAM Journal on Scientific Computing 39.5 (2017): S272-S297.

Somehow the SIAM version of the abstract says this is Q_k \times
Q_{k-1}^{disc} (an unstable element) despite both arXiv versions saying
Q_k \times P_{k-1}^{disc} (the stable element that they actually use).

  https://epubs.siam.org/doi/pdf/10.1137/16M108450X

This weighting scheme (using a lumped velocity mass matrix scaled by
viscosity) would be good work to add to PCLSC.

> Note that in the Rudi et al papers, due the highly variable nature of the
> viscosity, the authors advocate using a more complex definition of the
> preconditioner for the Schur complement. Whether you need to use their
> approach is dependent on the nature of the problem you are solving.

From ellen.price at cfa.harvard.edu  Fri Oct 12 10:12:39 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Fri, 12 Oct 2018 11:12:39 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
Message-ID: <6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>

So I found *that* problem -- turns out that using DMPlexCreateSection
was messing things up, and it seems PETSc is handling that part
internally? However, I've run into a related problem now. I need to
project some field data onto the mesh; it's a static array of
integration weights, and each one corresponds to a single point in the
mesh. Based on this answer:

https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html

I think I need DMPlexDistributeField. However, *that* function takes a
section as an argument, and I had to take out the part where I create a
section to get things working!  Calling DMGetDefaultSection or
DMGetDefaultGlobalSection doesn't seem to help matters, either, because
then I seem to get a null section?

So I either need to figure out how to set the section properly or how to
get the section properly. As always, any help is appreciated.

Ellen


On 10/11/2018 05:41 PM, Matthew Knepley wrote:
> On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
> <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>> wrote:
> 
>     I was working with a DMPLEX and FEM, following SNES example 12. I get
>     the following error when I call DMProjectFunction, but I don't know what
>     it means. Can anyone explain where I might have gone wrong, or at least
>     what this error is telling me? I think the point closure size is
>     correct, since my mesh is 3d simplex,
> 
> 
> Yes, if you have 3D SIMPLEX mesh and are using P1 elements, then you
> would have
> 4 dofs in the closure of a cell. The dual space dimension is the number
> of dual space
> basis vectors assigned to points in the closure. Since it is 1, it looks
> like you have a P0
> dual space. I assume you changed something in ex12?
> 
> ? Thanks,
> 
> ? ? ?Matt
> ?
> 
>     but what is the dual space
>     dimension, and where might I have set it incorrectly?
> 
>     [0]PETSC ERROR: Nonconforming object sizes
>     [0]PETSC ERROR: The section point closure size 4 != dual space
>     dimension 1
>     [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>     for trouble shooting.
>     [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
>     ...
>     [0]PETSC ERROR: Configure options
>     --prefix=/home/eprice/software/petsc-opt --with-hdf5=1
>     --with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1
>     --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
>     LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native -mtune=native"
>     CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>     -march=native -mtune=native" --with-mpi=1
>     --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
>     --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
>     --with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1
>     --with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1
>     --with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1
>     --with-scalapack-dir=/home/eprice/software/scalapack
>     [0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
>     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     [0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
>     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     [0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
>     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     [0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
>     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     ...
> 
>     (I know this is an optimized PETSc build, but I get the same error from
>     my debug build, it's just much slower.)
> 
>     Ellen
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

From knepley at gmail.com  Fri Oct 12 10:31:18 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Oct 2018 11:31:18 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
	<6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
Message-ID: <CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>

On Fri, Oct 12, 2018 at 11:12 AM Ellen M. Price <ellen.price at cfa.harvard.edu>
wrote:

> So I found *that* problem -- turns out that using DMPlexCreateSection
> was messing things up, and it seems PETSc is handling that part
> internally?


DMPlexSection creates the Section based on the discretization information
in the DM (stored in a PetscDS).
If no discretization info is available, then it defaults to P0 I think,
which is what you got.

After discretization info is put into the DM, if I ask for the Section, the
DM will create it automatically if it is missing.


> However, I've run into a related problem now. I need to
> project some field data onto the mesh; it's a static array of
> integration weights, and each one corresponds to a single point in the
> mesh. Based on this answer:
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html
>
> I think I need DMPlexDistributeField.


I don't think that is what you want.  That is just sending data around.

You want to define an FEM field. We use DMProject*() for this. These take
as input either
a function of the coordinates (regular function) or another FEM field.

Can you tell me how your function is defined? I mean apart from the
particular data you
have at points.

  Thanks,

     Matt


> However, *that* function takes a
> section as an argument, and I had to take out the part where I create a
> section to get things working!  Calling DMGetDefaultSection or
> DMGetDefaultGlobalSection doesn't seem to help matters, either, because
> then I seem to get a null section?
>
> So I either need to figure out how to set the section properly or how to
> get the section properly. As always, any help is appreciated.
>
> Ellen
>
>
> On 10/11/2018 05:41 PM, Matthew Knepley wrote:
> > On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
> > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>>
> wrote:
> >
> >     I was working with a DMPLEX and FEM, following SNES example 12. I get
> >     the following error when I call DMProjectFunction, but I don't know
> what
> >     it means. Can anyone explain where I might have gone wrong, or at
> least
> >     what this error is telling me? I think the point closure size is
> >     correct, since my mesh is 3d simplex,
> >
> >
> > Yes, if you have 3D SIMPLEX mesh and are using P1 elements, then you
> > would have
> > 4 dofs in the closure of a cell. The dual space dimension is the number
> > of dual space
> > basis vectors assigned to points in the closure. Since it is 1, it looks
> > like you have a P0
> > dual space. I assume you changed something in ex12?
> >
> >   Thanks,
> >
> >      Matt
> >
> >
> >     but what is the dual space
> >     dimension, and where might I have set it incorrectly?
> >
> >     [0]PETSC ERROR: Nonconforming object sizes
> >     [0]PETSC ERROR: The section point closure size 4 != dual space
> >     dimension 1
> >     [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/faq.html
> >     for trouble shooting.
> >     [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
> >     ...
> >     [0]PETSC ERROR: Configure options
> >     --prefix=/home/eprice/software/petsc-opt --with-hdf5=1
> >     --with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1
> >     --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
> >     LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native -mtune=native"
> >     CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
> >     -march=native -mtune=native" --with-mpi=1
> >     --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
> >     --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
> >     --with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1
> >     --with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1
> >     --with-ptscotch-dir=/home/eprice/software/scotch --with-scalapack=1
> >     --with-scalapack-dir=/home/eprice/software/scalapack
> >     [0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
> >     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> >     [0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
> >     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> >     [0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
> >     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> >     [0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
> >     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> >     ...
> >
> >     (I know this is an optimized PETSc build, but I get the same error
> from
> >     my debug build, it's just much slower.)
> >
> >     Ellen
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/%7Eknepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/7e7f5776/attachment.html>

From ellen.price at cfa.harvard.edu  Fri Oct 12 14:51:18 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Fri, 12 Oct 2018 15:51:18 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
	<6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
	<CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>
Message-ID: <4665f197-89a8-ae7f-3a54-a9f87200fe51@cfa.harvard.edu>

Hi Matt,

Basically, I have a simple meshing script that generates a set of points
for my FEM mesh and outputs a C file that compiles into my program.
Along with those points is a static data field that I need to be
associated with the correct data point. I don't have a function that
evaluates at each point, it's just pre-generated data.

Since I called DMPlexDistribute() early on, I think my mesh points get
re-ordered, right? So I need a way to re-order my data field to match
that ordering. Is there a good way to do this?

I tried using DMPlexDistributeField as a first attempt, but I get some
kind of error no matter what I do (presumably I'm not giving the right
input), and there aren't any examples in the docs.

Ellen


On 10/12/2018 11:31 AM, Matthew Knepley wrote:
> On Fri, Oct 12, 2018 at 11:12 AM Ellen M. Price
> <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>> wrote:
> 
>     So I found *that* problem -- turns out that using DMPlexCreateSection
>     was messing things up, and it seems PETSc is handling that part
>     internally? 
> 
> 
> DMPlexSection creates the Section based on the discretization
> information in the DM (stored in a PetscDS).
> If no discretization info is available, then it defaults to P0 I think,
> which is what you got.
> 
> After discretization info is put into the DM, if I ask for the Section,
> the DM will create it automatically if it is missing.
> ?
> 
>     However, I've run into a related problem now. I need to
>     project some field data onto the mesh; it's a static array of
>     integration weights, and each one corresponds to a single point in the
>     mesh. Based on this answer:
> 
>     https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html
> 
>     I think I need DMPlexDistributeField.
> 
> 
> I don't think that is what you want.? That is just sending data around.
> 
> You want to define an FEM field. We use DMProject*() for this. These
> take as input either
> a function of the coordinates (regular function) or another FEM field.
> 
> Can you tell me how your function is defined? I mean apart from the
> particular data you
> have at points.
> 
> ? Thanks,
> 
> ? ? ?Matt
> ?
> 
>     However, *that* function takes a
>     section as an argument, and I had to take out the part where I create a
>     section to get things working!? Calling DMGetDefaultSection or
>     DMGetDefaultGlobalSection doesn't seem to help matters, either, because
>     then I seem to get a null section?
> 
>     So I either need to figure out how to set the section properly or how to
>     get the section properly. As always, any help is appreciated.
> 
>     Ellen
> 
> 
>     On 10/11/2018 05:41 PM, Matthew Knepley wrote:
>     > On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
>     > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>
>     <mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>> wrote:
>     >
>     >? ? ?I was working with a DMPLEX and FEM, following SNES example
>     12. I get
>     >? ? ?the following error when I call DMProjectFunction, but I don't
>     know what
>     >? ? ?it means. Can anyone explain where I might have gone wrong, or
>     at least
>     >? ? ?what this error is telling me? I think the point closure size is
>     >? ? ?correct, since my mesh is 3d simplex,
>     >
>     >
>     > Yes, if you have 3D SIMPLEX mesh and are using P1 elements, then you
>     > would have
>     > 4 dofs in the closure of a cell. The dual space dimension is the
>     number
>     > of dual space
>     > basis vectors assigned to points in the closure. Since it is 1, it
>     looks
>     > like you have a P0
>     > dual space. I assume you changed something in ex12?
>     >
>     > ? Thanks,
>     >
>     > ? ? ?Matt
>     > ?
>     >
>     >? ? ?but what is the dual space
>     >? ? ?dimension, and where might I have set it incorrectly?
>     >
>     >? ? ?[0]PETSC ERROR: Nonconforming object sizes
>     >? ? ?[0]PETSC ERROR: The section point closure size 4 != dual space
>     >? ? ?dimension 1
>     >? ? ?[0]PETSC ERROR: See
>     http://www.mcs.anl.gov/petsc/documentation/faq.html
>     >? ? ?for trouble shooting.
>     >? ? ?[0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
>     >? ? ?...
>     >? ? ?[0]PETSC ERROR: Configure options
>     >? ? ?--prefix=/home/eprice/software/petsc-opt --with-hdf5=1
>     >? ? ?--with-hdf5-dir=/home/eprice/software/hdf5-parallel --with-mpe=1
>     >? ? ?--with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
>     >? ? ?LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native -mtune=native"
>     >? ? ?CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
>     >? ? ?-march=native -mtune=native" --with-mpi=1
>     >? ? ?--with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
>     >? ? ?--with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
>     >? ? ?--with-parmetis-dir=/home/eprice/software/parmetis --with-metis=1
>     >? ? ?--with-metis-dir=/home/eprice/software/parmetis --with-ptscotch=1
>     >? ? ?--with-ptscotch-dir=/home/eprice/software/scotch
>     --with-scalapack=1
>     >? ? ?--with-scalapack-dir=/home/eprice/software/scalapack
>     >? ? ?[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
>     >? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     >? ? ?[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
>     >? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     >? ? ?[0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
>     >? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     >? ? ?[0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
>     >? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     >? ? ?...
>     >
>     >? ? ?(I know this is an optimized PETSc build, but I get the same
>     error from
>     >? ? ?my debug build, it's just much slower.)
>     >
>     >? ? ?Ellen
>     >
>     >
>     >
>     > --
>     > What most experimenters take for granted before they begin their
>     > experiments is infinitely more interesting than any results to which
>     > their experiments lead.
>     > -- Norbert Wiener
>     >
>     > https://www.cse.buffalo.edu/~knepley/
>     <https://www.cse.buffalo.edu/%7Eknepley/>
>     > <http://www.cse.buffalo.edu/%7Eknepley/>
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

From knepley at gmail.com  Fri Oct 12 15:36:27 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 12 Oct 2018 16:36:27 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <4665f197-89a8-ae7f-3a54-a9f87200fe51@cfa.harvard.edu>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
	<6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
	<CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>
	<4665f197-89a8-ae7f-3a54-a9f87200fe51@cfa.harvard.edu>
Message-ID: <CAMYG4G=Nif5bo5v122SUyr0PZ8V4B-p85gaPJ6BokMo-kqqUXg@mail.gmail.com>

On Fri, Oct 12, 2018 at 3:51 PM Ellen M. Price <ellen.price at cfa.harvard.edu>
wrote:

> Hi Matt,
>
> Basically, I have a simple meshing script that generates a set of points
> for my FEM mesh and outputs a C file that compiles into my program.
> Along with those points is a static data field that I need to be
> associated with the correct data point. I don't have a function that
> evaluates at each point, it's just pre-generated data.
>

I cannot help but make editorial comments. Do not do this. It is incredibly
fragile, since
it depends on both a mesh and a discretization.


> Since I called DMPlexDistribute() early on, I think my mesh points get
> re-ordered, right? So I need a way to re-order my data field to match
> that ordering. Is there a good way to do this?
>

However, if you want to do this, it is not hard. You do use
DMPlexDistributeField (
https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeField.html),
but you need
to be careful about the arguments. You want

   DMPlexDistribute(dmOriginal, 0, &migrationSF, &dmNew);
   DMPlexDistributeField(dmOriginal, migrationSF, originalSection,
originalVec, newSection, newVec);

Your Sections are easy to make, just put 1 dof on every vertex. The
originalVec is just a Vec of your values
(you can use VecCreateSeqWithArray() to wrap one around it), and the newVec
you can get with
DMCreateLocalVector(dmNew, &newVec) if you do DMSetSection(dmNew,
newSection).

Does this make sense?

  Thanks,

     Matt


> I tried using DMPlexDistributeField as a first attempt, but I get some
> kind of error no matter what I do (presumably I'm not giving the right
> input), and there aren't any examples in the docs.
>
> Ellen
>
>
> On 10/12/2018 11:31 AM, Matthew Knepley wrote:
> > On Fri, Oct 12, 2018 at 11:12 AM Ellen M. Price
> > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>>
> wrote:
> >
> >     So I found *that* problem -- turns out that using DMPlexCreateSection
> >     was messing things up, and it seems PETSc is handling that part
> >     internally?
> >
> >
> > DMPlexSection creates the Section based on the discretization
> > information in the DM (stored in a PetscDS).
> > If no discretization info is available, then it defaults to P0 I think,
> > which is what you got.
> >
> > After discretization info is put into the DM, if I ask for the Section,
> > the DM will create it automatically if it is missing.
> >
> >
> >     However, I've run into a related problem now. I need to
> >     project some field data onto the mesh; it's a static array of
> >     integration weights, and each one corresponds to a single point in
> the
> >     mesh. Based on this answer:
> >
> >
> https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html
> >
> >     I think I need DMPlexDistributeField.
> >
> >
> > I don't think that is what you want.  That is just sending data around.
> >
> > You want to define an FEM field. We use DMProject*() for this. These
> > take as input either
> > a function of the coordinates (regular function) or another FEM field.
> >
> > Can you tell me how your function is defined? I mean apart from the
> > particular data you
> > have at points.
> >
> >   Thanks,
> >
> >      Matt
> >
> >
> >     However, *that* function takes a
> >     section as an argument, and I had to take out the part where I
> create a
> >     section to get things working!  Calling DMGetDefaultSection or
> >     DMGetDefaultGlobalSection doesn't seem to help matters, either,
> because
> >     then I seem to get a null section?
> >
> >     So I either need to figure out how to set the section properly or
> how to
> >     get the section properly. As always, any help is appreciated.
> >
> >     Ellen
> >
> >
> >     On 10/11/2018 05:41 PM, Matthew Knepley wrote:
> >     > On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
> >     > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>
> >     <mailto:ellen.price at cfa.harvard.edu
> >     <mailto:ellen.price at cfa.harvard.edu>>> wrote:
> >     >
> >     >     I was working with a DMPLEX and FEM, following SNES example
> >     12. I get
> >     >     the following error when I call DMProjectFunction, but I don't
> >     know what
> >     >     it means. Can anyone explain where I might have gone wrong, or
> >     at least
> >     >     what this error is telling me? I think the point closure size
> is
> >     >     correct, since my mesh is 3d simplex,
> >     >
> >     >
> >     > Yes, if you have 3D SIMPLEX mesh and are using P1 elements, then
> you
> >     > would have
> >     > 4 dofs in the closure of a cell. The dual space dimension is the
> >     number
> >     > of dual space
> >     > basis vectors assigned to points in the closure. Since it is 1, it
> >     looks
> >     > like you have a P0
> >     > dual space. I assume you changed something in ex12?
> >     >
> >     >   Thanks,
> >     >
> >     >      Matt
> >     >
> >     >
> >     >     but what is the dual space
> >     >     dimension, and where might I have set it incorrectly?
> >     >
> >     >     [0]PETSC ERROR: Nonconforming object sizes
> >     >     [0]PETSC ERROR: The section point closure size 4 != dual space
> >     >     dimension 1
> >     >     [0]PETSC ERROR: See
> >     http://www.mcs.anl.gov/petsc/documentation/faq.html
> >     >     for trouble shooting.
> >     >     [0]PETSC ERROR: Petsc Release Version 3.9.2, May, 20, 2018
> >     >     ...
> >     >     [0]PETSC ERROR: Configure options
> >     >     --prefix=/home/eprice/software/petsc-opt --with-hdf5=1
> >     >     --with-hdf5-dir=/home/eprice/software/hdf5-parallel
> --with-mpe=1
> >     >     --with-mpe-dir=/home/eprice/software/mpe --with-debugging=0
> >     >     LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native
> -mtune=native"
> >     >     CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3
> >     >     -march=native -mtune=native" --with-mpi=1
> >     >     --with-mpi-dir=/home/eprice/software/mpich --with-mumps=1
> >     >     --with-mumps-dir=/home/eprice/software/mumps --with-parmetis=1
> >     >     --with-parmetis-dir=/home/eprice/software/parmetis
> --with-metis=1
> >     >     --with-metis-dir=/home/eprice/software/parmetis
> --with-ptscotch=1
> >     >     --with-ptscotch-dir=/home/eprice/software/scotch
> >     --with-scalapack=1
> >     >     --with-scalapack-dir=/home/eprice/software/scalapack
> >     >     [0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex() line 347 in
> >     >     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> >     >     [0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex() line 428 in
> >     >     /h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
> >     >     [0]PETSC ERROR: #3 DMProjectFunctionLocal() line 6265 in
> >     >     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> >     >     [0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
> >     >     /h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
> >     >     ...
> >     >
> >     >     (I know this is an optimized PETSc build, but I get the same
> >     error from
> >     >     my debug build, it's just much slower.)
> >     >
> >     >     Ellen
> >     >
> >     >
> >     >
> >     > --
> >     > What most experimenters take for granted before they begin their
> >     > experiments is infinitely more interesting than any results to
> which
> >     > their experiments lead.
> >     > -- Norbert Wiener
> >     >
> >     > https://www.cse.buffalo.edu/~knepley/
> >     <https://www.cse.buffalo.edu/%7Eknepley/>
> >     > <http://www.cse.buffalo.edu/%7Eknepley/>
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which
> > their experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/%7Eknepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181012/a5ad0f34/attachment-0001.html>

From ellen.price at cfa.harvard.edu  Sat Oct 13 23:36:19 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Sun, 14 Oct 2018 00:36:19 -0400
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <CAMYG4GmnJRrYgHB+4A1fr+W0QV2jBODJ8NZrRqNJe-R4AqWr9g@mail.gmail.com>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
	<6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
	<CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>
	<4665f197-89a8-ae7f-3a54-a9f87200fe51@cfa.harvard.edu>
	<CAMYG4G=Nif5bo5v122SUyr0PZ8V4B-p85gaPJ6BokMo-kqqUXg@mail.gmail.com>
	<c5d52282-c92b-f349-68d3-e2667d03e60a@cfa.harvard.edu>
	<CAMYG4GmnJRrYgHB+4A1fr+W0QV2jBODJ8NZrRqNJe-R4AqWr9g@mail.gmail.com>
Message-ID: <0a4e98b9-4b34-570d-f375-8a6a1884c73e@cfa.harvard.edu>

Hi Matt,

I ran into a problem implementing this. My integration code relies on
knowing the points of each tetrahedron in the domain so I can turn an
integral over all space into a sum of integrals over cells. I can't find
an example of getting vertices for a given cell (I think TS ex11 comes
closest, looks like it gets faces?). Is there an elegant way to do this,
or an example I can follow?

Ellen


On 10/13/2018 04:26 AM, Matthew Knepley wrote:
> On Fri, Oct 12, 2018 at 6:00 PM Ellen M. Price
> <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>> wrote:
> 
>     Hi Matt,
> 
>     Thanks for the feedback! I'm always open to editorial comments. Do you
>     have a suggestion for a better way? I'm basically just trying to do an
>     integral over all space to get my boundary values for the FEM solve. I
>     don't know if PETSc has a way to do something like this?
> 
> 
> So you are defining your BC by a boundary integral. I would just stick
> that integration code
> into a function and let that be called as your BC function. It will not
> be called "extra" times,
> just once for each dual vector, which in the case of P1 is every vertex.
> This way it will work
> for any element.
> 
> ? Thanks,
> 
> ? ? ?Matt
> ?
> 
>     Ellen
> 
> 
>     On 10/12/2018 04:36 PM, Matthew Knepley wrote:
>     > On Fri, Oct 12, 2018 at 3:51 PM Ellen M. Price
>     > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>
>     <mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>> wrote:
>     >
>     >? ? ?Hi Matt,
>     >
>     >? ? ?Basically, I have a simple meshing script that generates a set
>     of points
>     >? ? ?for my FEM mesh and outputs a C file that compiles into my
>     program.
>     >? ? ?Along with those points is a static data field that I need to be
>     >? ? ?associated with the correct data point. I don't have a
>     function that
>     >? ? ?evaluates at each point, it's just pre-generated data.
>     >
>     >
>     > I cannot help but make editorial comments. Do not do this. It is
>     > incredibly fragile, since
>     > it depends on both a mesh and a discretization.
>     > ?
>     >
>     >? ? ?Since I called DMPlexDistribute() early on, I think my mesh
>     points get
>     >? ? ?re-ordered, right? So I need a way to re-order my data field
>     to match
>     >? ? ?that ordering. Is there a good way to do this?
>     >
>     >
>     > However, if you want to do this, it is not hard. You do use
>     > DMPlexDistributeField
>     >
>     (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeField.html),
>     > but you need
>     > to be careful about the arguments. You want
>     >
>     > ? ?DMPlexDistribute(dmOriginal, 0, &migrationSF, &dmNew);
>     > ???DMPlexDistributeField(dmOriginal, migrationSF, originalSection,
>     > originalVec, newSection, newVec);
>     >
>     > Your Sections are easy to make, just put 1 dof on every vertex. The
>     > originalVec is just a Vec of your values
>     > (you can use VecCreateSeqWithArray() to wrap one around it), and the
>     > newVec you can get with
>     > DMCreateLocalVector(dmNew, &newVec) if you do DMSetSection(dmNew,
>     > newSection).
>     >
>     > Does this make sense?
>     >
>     > ? Thanks,
>     >
>     > ? ? ?Matt
>     > ?
>     >
>     >? ? ?I tried using DMPlexDistributeField as a first attempt, but I
>     get some
>     >? ? ?kind of error no matter what I do (presumably I'm not giving
>     the right
>     >? ? ?input), and there aren't any examples in the docs.
>     >
>     >? ? ?Ellen
>     >
>     >
>     >? ? ?On 10/12/2018 11:31 AM, Matthew Knepley wrote:
>     >? ? ?> On Fri, Oct 12, 2018 at 11:12 AM Ellen M. Price
>     >? ? ?> <ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     <mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>>> wrote:
>     >? ? ?>
>     >? ? ?>? ? ?So I found *that* problem -- turns out that using
>     >? ? ?DMPlexCreateSection
>     >? ? ?>? ? ?was messing things up, and it seems PETSc is handling
>     that part
>     >? ? ?>? ? ?internally?
>     >? ? ?>
>     >? ? ?>
>     >? ? ?> DMPlexSection creates the Section based on the discretization
>     >? ? ?> information in the DM (stored in a PetscDS).
>     >? ? ?> If no discretization info is available, then it defaults to P0 I
>     >? ? ?think,
>     >? ? ?> which is what you got.
>     >? ? ?>
>     >? ? ?> After discretization info is put into the DM, if I ask for the
>     >? ? ?Section,
>     >? ? ?> the DM will create it automatically if it is missing.
>     >? ? ?> ?
>     >? ? ?>
>     >? ? ?>? ? ?However, I've run into a related problem now. I need to
>     >? ? ?>? ? ?project some field data onto the mesh; it's a static
>     array of
>     >? ? ?>? ? ?integration weights, and each one corresponds to a single
>     >? ? ?point in the
>     >? ? ?>? ? ?mesh. Based on this answer:
>     >? ? ?>
>     >? ? ?>? ?
>     >? ?
>     ??https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html
>     >? ? ?>
>     >? ? ?>? ? ?I think I need DMPlexDistributeField.
>     >? ? ?>
>     >? ? ?>
>     >? ? ?> I don't think that is what you want.? That is just sending data
>     >? ? ?around.
>     >? ? ?>
>     >? ? ?> You want to define an FEM field. We use DMProject*() for
>     this. These
>     >? ? ?> take as input either
>     >? ? ?> a function of the coordinates (regular function) or another
>     FEM field.
>     >? ? ?>
>     >? ? ?> Can you tell me how your function is defined? I mean apart
>     from the
>     >? ? ?> particular data you
>     >? ? ?> have at points.
>     >? ? ?>
>     >? ? ?> ? Thanks,
>     >? ? ?>
>     >? ? ?> ? ? ?Matt
>     >? ? ?> ?
>     >? ? ?>
>     >? ? ?>? ? ?However, *that* function takes a
>     >? ? ?>? ? ?section as an argument, and I had to take out the part
>     where I
>     >? ? ?create a
>     >? ? ?>? ? ?section to get things working!? Calling
>     DMGetDefaultSection or
>     >? ? ?>? ? ?DMGetDefaultGlobalSection doesn't seem to help matters,
>     >? ? ?either, because
>     >? ? ?>? ? ?then I seem to get a null section?
>     >? ? ?>
>     >? ? ?>? ? ?So I either need to figure out how to set the section
>     properly
>     >? ? ?or how to
>     >? ? ?>? ? ?get the section properly. As always, any help is
>     appreciated.
>     >? ? ?>
>     >? ? ?>? ? ?Ellen
>     >? ? ?>
>     >? ? ?>
>     >? ? ?>? ? ?On 10/11/2018 05:41 PM, Matthew Knepley wrote:
>     >? ? ?>? ? ?> On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
>     >? ? ?>? ? ?> <ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>>
>     >? ? ?>? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>
>     >? ? ?>? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>
>     >? ? ?<mailto:ellen.price at cfa.harvard.edu
>     <mailto:ellen.price at cfa.harvard.edu>>>>> wrote:
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>? ? ?I was working with a DMPLEX and FEM, following
>     SNES example
>     >? ? ?>? ? ?12. I get
>     >? ? ?>? ? ?>? ? ?the following error when I call DMProjectFunction,
>     but I
>     >? ? ?don't
>     >? ? ?>? ? ?know what
>     >? ? ?>? ? ?>? ? ?it means. Can anyone explain where I might have gone
>     >? ? ?wrong, or
>     >? ? ?>? ? ?at least
>     >? ? ?>? ? ?>? ? ?what this error is telling me? I think the point
>     closure
>     >? ? ?size is
>     >? ? ?>? ? ?>? ? ?correct, since my mesh is 3d simplex,
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?> Yes, if you have 3D SIMPLEX mesh and are using P1
>     elements,
>     >? ? ?then you
>     >? ? ?>? ? ?> would have
>     >? ? ?>? ? ?> 4 dofs in the closure of a cell. The dual space
>     dimension is the
>     >? ? ?>? ? ?number
>     >? ? ?>? ? ?> of dual space
>     >? ? ?>? ? ?> basis vectors assigned to points in the closure. Since
>     it is
>     >? ? ?1, it
>     >? ? ?>? ? ?looks
>     >? ? ?>? ? ?> like you have a P0
>     >? ? ?>? ? ?> dual space. I assume you changed something in ex12?
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?> ? Thanks,
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?> ? ? ?Matt
>     >? ? ?>? ? ?> ?
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>? ? ?but what is the dual space
>     >? ? ?>? ? ?>? ? ?dimension, and where might I have set it incorrectly?
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Nonconforming object sizes
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: The section point closure size 4
>     != dual
>     >? ? ?space
>     >? ? ?>? ? ?>? ? ?dimension 1
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: See
>     >? ? ?>? ? ?http://www.mcs.anl.gov/petsc/documentation/faq.html
>     >? ? ?>? ? ?>? ? ?for trouble shooting.
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Petsc Release Version 3.9.2, May,
>     20, 2018
>     >? ? ?>? ? ?>? ? ?...
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Configure options
>     >? ? ?>? ? ?>? ? ?--prefix=/home/eprice/software/petsc-opt --with-hdf5=1
>     >? ? ?>? ? ?>? ? ?--with-hdf5-dir=/home/eprice/software/hdf5-parallel
>     >? ? ?--with-mpe=1
>     >? ? ?>? ? ?>? ? ?--with-mpe-dir=/home/eprice/software/mpe
>     --with-debugging=0
>     >? ? ?>? ? ?>? ? ?LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native
>     >? ? ?-mtune=native"
>     >? ? ?>? ? ?>? ? ?CXXOPTFLAGS="-O3 -march=native -mtune=native"
>     FOPTFLAGS="-O3
>     >? ? ?>? ? ?>? ? ?-march=native -mtune=native" --with-mpi=1
>     >? ? ?>? ? ?>? ? ?--with-mpi-dir=/home/eprice/software/mpich
>     --with-mumps=1
>     >? ? ?>? ? ?>? ? ?--with-mumps-dir=/home/eprice/software/mumps
>     >? ? ?--with-parmetis=1
>     >? ? ?>? ? ?>? ? ?--with-parmetis-dir=/home/eprice/software/parmetis
>     >? ? ?--with-metis=1
>     >? ? ?>? ? ?>? ? ?--with-metis-dir=/home/eprice/software/parmetis
>     >? ? ?--with-ptscotch=1
>     >? ? ?>? ? ?>? ? ?--with-ptscotch-dir=/home/eprice/software/scotch
>     >? ? ?>? ? ?--with-scalapack=1
>     >? ? ?>? ? ?>? ? ?--with-scalapack-dir=/home/eprice/software/scalapack
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex()
>     line 347 in
>     >? ? ?>? ? ?>? ?
>     ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex()
>     line 428 in
>     >? ? ?>? ? ?>? ?
>     ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #3 DMProjectFunctionLocal() line
>     6265 in
>     >? ? ?>? ? ?>? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
>     >? ? ?>? ? ?>? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>     >? ? ?>? ? ?>? ? ?...
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>? ? ?(I know this is an optimized PETSc build, but I
>     get the same
>     >? ? ?>? ? ?error from
>     >? ? ?>? ? ?>? ? ?my debug build, it's just much slower.)
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>? ? ?Ellen
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?> --
>     >? ? ?>? ? ?> What most experimenters take for granted before they
>     begin their
>     >? ? ?>? ? ?> experiments is infinitely more interesting than any
>     results
>     >? ? ?to which
>     >? ? ?>? ? ?> their experiments lead.
>     >? ? ?>? ? ?> -- Norbert Wiener
>     >? ? ?>? ? ?>
>     >? ? ?>? ? ?> https://www.cse.buffalo.edu/~knepley/
>     <https://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?>? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?>? ? ?> <http://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?>
>     >? ? ?>
>     >? ? ?>
>     >? ? ?> --
>     >? ? ?> What most experimenters take for granted before they begin their
>     >? ? ?> experiments is infinitely more interesting than any results
>     to which
>     >? ? ?> their experiments lead.
>     >? ? ?> -- Norbert Wiener
>     >? ? ?>
>     >? ? ?> https://www.cse.buffalo.edu/~knepley/
>     <https://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>     >? ? ?> <http://www.cse.buffalo.edu/%7Eknepley/>
>     >
>     >
>     >
>     > --
>     > What most experimenters take for granted before they begin their
>     > experiments is infinitely more interesting than any results to which
>     > their experiments lead.
>     > -- Norbert Wiener
>     >
>     > https://www.cse.buffalo.edu/~knepley/
>     <https://www.cse.buffalo.edu/%7Eknepley/>
>     > <http://www.cse.buffalo.edu/%7Eknepley/>
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

From yann.jobic at univ-amu.fr  Sun Oct 14 02:07:51 2018
From: yann.jobic at univ-amu.fr (yann JOBIC)
Date: Sun, 14 Oct 2018 09:07:51 +0200
Subject: [petsc-users] DMPLEX project function error
In-Reply-To: <0a4e98b9-4b34-570d-f375-8a6a1884c73e@cfa.harvard.edu>
References: <d04f690d-a799-0151-94bf-f6a243950295@cfa.harvard.edu>
	<CAMYG4GniuY74z9LCh7aidec_VZybFv4DZXH3AZRLDVpV7oLm4Q@mail.gmail.com>
	<6db4ff08-a00d-7223-c533-d9c7729a96d4@cfa.harvard.edu>
	<CAMYG4G=OCBHwRsXzCiErGV4U7CoBGPW41ss80Px67ueLp+hB1g@mail.gmail.com>
	<4665f197-89a8-ae7f-3a54-a9f87200fe51@cfa.harvard.edu>
	<CAMYG4G=Nif5bo5v122SUyr0PZ8V4B-p85gaPJ6BokMo-kqqUXg@mail.gmail.com>
	<c5d52282-c92b-f349-68d3-e2667d03e60a@cfa.harvard.edu>
	<CAMYG4GmnJRrYgHB+4A1fr+W0QV2jBODJ8NZrRqNJe-R4AqWr9g@mail.gmail.com>
	<0a4e98b9-4b34-570d-f375-8a6a1884c73e@cfa.harvard.edu>
Message-ID: <d4e27956-0bad-d474-cb19-5d5e67afd21f@univ-amu.fr>

Hi Ellen,

I've made an old code (i forgot in which version it's working) that 
gives local coordinates and connectivity of a distributed DM.

I didn't check that it's working again. I'll also work on that on the 
future.

If it can help, i send you my ugly code (i didn't clean it).

Yann


On 14/10/2018 06:36, Ellen M. Price wrote:
> Hi Matt,
>
> I ran into a problem implementing this. My integration code relies on
> knowing the points of each tetrahedron in the domain so I can turn an
> integral over all space into a sum of integrals over cells. I can't find
> an example of getting vertices for a given cell (I think TS ex11 comes
> closest, looks like it gets faces?). Is there an elegant way to do this,
> or an example I can follow?
>
> Ellen
>
>
> On 10/13/2018 04:26 AM, Matthew Knepley wrote:
>> On Fri, Oct 12, 2018 at 6:00 PM Ellen M. Price
>> <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>> wrote:
>>
>>      Hi Matt,
>>
>>      Thanks for the feedback! I'm always open to editorial comments. Do you
>>      have a suggestion for a better way? I'm basically just trying to do an
>>      integral over all space to get my boundary values for the FEM solve. I
>>      don't know if PETSc has a way to do something like this?
>>
>>
>> So you are defining your BC by a boundary integral. I would just stick
>> that integration code
>> into a function and let that be called as your BC function. It will not
>> be called "extra" times,
>> just once for each dual vector, which in the case of P1 is every vertex.
>> This way it will work
>> for any element.
>>
>>  ? Thanks,
>>
>>  ? ? ?Matt
>>   
>>
>>      Ellen
>>
>>
>>      On 10/12/2018 04:36 PM, Matthew Knepley wrote:
>>      > On Fri, Oct 12, 2018 at 3:51 PM Ellen M. Price
>>      > <ellen.price at cfa.harvard.edu <mailto:ellen.price at cfa.harvard.edu>
>>      <mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>> wrote:
>>      >
>>      >? ? ?Hi Matt,
>>      >
>>      >? ? ?Basically, I have a simple meshing script that generates a set
>>      of points
>>      >? ? ?for my FEM mesh and outputs a C file that compiles into my
>>      program.
>>      >? ? ?Along with those points is a static data field that I need to be
>>      >? ? ?associated with the correct data point. I don't have a
>>      function that
>>      >? ? ?evaluates at each point, it's just pre-generated data.
>>      >
>>      >
>>      > I cannot help but make editorial comments. Do not do this. It is
>>      > incredibly fragile, since
>>      > it depends on both a mesh and a discretization.
>>      >
>>      >
>>      >? ? ?Since I called DMPlexDistribute() early on, I think my mesh
>>      points get
>>      >? ? ?re-ordered, right? So I need a way to re-order my data field
>>      to match
>>      >? ? ?that ordering. Is there a good way to do this?
>>      >
>>      >
>>      > However, if you want to do this, it is not hard. You do use
>>      > DMPlexDistributeField
>>      >
>>      (https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexDistributeField.html),
>>      > but you need
>>      > to be careful about the arguments. You want
>>      >
>>      > ? ?DMPlexDistribute(dmOriginal, 0, &migrationSF, &dmNew);
>>      > ???DMPlexDistributeField(dmOriginal, migrationSF, originalSection,
>>      > originalVec, newSection, newVec);
>>      >
>>      > Your Sections are easy to make, just put 1 dof on every vertex. The
>>      > originalVec is just a Vec of your values
>>      > (you can use VecCreateSeqWithArray() to wrap one around it), and the
>>      > newVec you can get with
>>      > DMCreateLocalVector(dmNew, &newVec) if you do DMSetSection(dmNew,
>>      > newSection).
>>      >
>>      > Does this make sense?
>>      >
>>      > ? Thanks,
>>      >
>>      > ? ? ?Matt
>>      >
>>      >
>>      >? ? ?I tried using DMPlexDistributeField as a first attempt, but I
>>      get some
>>      >? ? ?kind of error no matter what I do (presumably I'm not giving
>>      the right
>>      >? ? ?input), and there aren't any examples in the docs.
>>      >
>>      >? ? ?Ellen
>>      >
>>      >
>>      >? ? ?On 10/12/2018 11:31 AM, Matthew Knepley wrote:
>>      >? ? ?> On Fri, Oct 12, 2018 at 11:12 AM Ellen M. Price
>>      >? ? ?> <ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      <mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>>> wrote:
>>      >? ? ?>
>>      >? ? ?>? ? ?So I found *that* problem -- turns out that using
>>      >? ? ?DMPlexCreateSection
>>      >? ? ?>? ? ?was messing things up, and it seems PETSc is handling
>>      that part
>>      >? ? ?>? ? ?internally?
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?> DMPlexSection creates the Section based on the discretization
>>      >? ? ?> information in the DM (stored in a PetscDS).
>>      >? ? ?> If no discretization info is available, then it defaults to P0 I
>>      >? ? ?think,
>>      >? ? ?> which is what you got.
>>      >? ? ?>
>>      >? ? ?> After discretization info is put into the DM, if I ask for the
>>      >? ? ?Section,
>>      >? ? ?> the DM will create it automatically if it is missing.
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?>? ? ?However, I've run into a related problem now. I need to
>>      >? ? ?>? ? ?project some field data onto the mesh; it's a static
>>      array of
>>      >? ? ?>? ? ?integration weights, and each one corresponds to a single
>>      >? ? ?point in the
>>      >? ? ?>? ? ?mesh. Based on this answer:
>>      >? ? ?>
>>      >? ? ?>
>>      >
>>      ??https://lists.mcs.anl.gov/pipermail/petsc-users/2015-December/027964.html
>>      >? ? ?>
>>      >? ? ?>? ? ?I think I need DMPlexDistributeField.
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?> I don't think that is what you want.? That is just sending data
>>      >? ? ?around.
>>      >? ? ?>
>>      >? ? ?> You want to define an FEM field. We use DMProject*() for
>>      this. These
>>      >? ? ?> take as input either
>>      >? ? ?> a function of the coordinates (regular function) or another
>>      FEM field.
>>      >? ? ?>
>>      >? ? ?> Can you tell me how your function is defined? I mean apart
>>      from the
>>      >? ? ?> particular data you
>>      >? ? ?> have at points.
>>      >? ? ?>
>>      >? ? ?> ? Thanks,
>>      >? ? ?>
>>      >? ? ?> ? ? ?Matt
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?>? ? ?However, *that* function takes a
>>      >? ? ?>? ? ?section as an argument, and I had to take out the part
>>      where I
>>      >? ? ?create a
>>      >? ? ?>? ? ?section to get things working!? Calling
>>      DMGetDefaultSection or
>>      >? ? ?>? ? ?DMGetDefaultGlobalSection doesn't seem to help matters,
>>      >? ? ?either, because
>>      >? ? ?>? ? ?then I seem to get a null section?
>>      >? ? ?>
>>      >? ? ?>? ? ?So I either need to figure out how to set the section
>>      properly
>>      >? ? ?or how to
>>      >? ? ?>? ? ?get the section properly. As always, any help is
>>      appreciated.
>>      >? ? ?>
>>      >? ? ?>? ? ?Ellen
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?>? ? ?On 10/11/2018 05:41 PM, Matthew Knepley wrote:
>>      >? ? ?>? ? ?> On Thu, Oct 11, 2018 at 4:39 PM Ellen M. Price
>>      >? ? ?>? ? ?> <ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>>
>>      >? ? ?>? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>
>>      >? ? ?>? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>
>>      >? ? ?<mailto:ellen.price at cfa.harvard.edu
>>      <mailto:ellen.price at cfa.harvard.edu>>>>> wrote:
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>? ? ?I was working with a DMPLEX and FEM, following
>>      SNES example
>>      >? ? ?>? ? ?12. I get
>>      >? ? ?>? ? ?>? ? ?the following error when I call DMProjectFunction,
>>      but I
>>      >? ? ?don't
>>      >? ? ?>? ? ?know what
>>      >? ? ?>? ? ?>? ? ?it means. Can anyone explain where I might have gone
>>      >? ? ?wrong, or
>>      >? ? ?>? ? ?at least
>>      >? ? ?>? ? ?>? ? ?what this error is telling me? I think the point
>>      closure
>>      >? ? ?size is
>>      >? ? ?>? ? ?>? ? ?correct, since my mesh is 3d simplex,
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?> Yes, if you have 3D SIMPLEX mesh and are using P1
>>      elements,
>>      >? ? ?then you
>>      >? ? ?>? ? ?> would have
>>      >? ? ?>? ? ?> 4 dofs in the closure of a cell. The dual space
>>      dimension is the
>>      >? ? ?>? ? ?number
>>      >? ? ?>? ? ?> of dual space
>>      >? ? ?>? ? ?> basis vectors assigned to points in the closure. Since
>>      it is
>>      >? ? ?1, it
>>      >? ? ?>? ? ?looks
>>      >? ? ?>? ? ?> like you have a P0
>>      >? ? ?>? ? ?> dual space. I assume you changed something in ex12?
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?> ? Thanks,
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?> ? ? ?Matt
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>? ? ?but what is the dual space
>>      >? ? ?>? ? ?>? ? ?dimension, and where might I have set it incorrectly?
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Nonconforming object sizes
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: The section point closure size 4
>>      != dual
>>      >? ? ?space
>>      >? ? ?>? ? ?>? ? ?dimension 1
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: See
>>      >? ? ?>? ? ?http://www.mcs.anl.gov/petsc/documentation/faq.html
>>      >? ? ?>? ? ?>? ? ?for trouble shooting.
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Petsc Release Version 3.9.2, May,
>>      20, 2018
>>      >? ? ?>? ? ?>? ? ?...
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: Configure options
>>      >? ? ?>? ? ?>? ? ?--prefix=/home/eprice/software/petsc-opt --with-hdf5=1
>>      >? ? ?>? ? ?>? ? ?--with-hdf5-dir=/home/eprice/software/hdf5-parallel
>>      >? ? ?--with-mpe=1
>>      >? ? ?>? ? ?>? ? ?--with-mpe-dir=/home/eprice/software/mpe
>>      --with-debugging=0
>>      >? ? ?>? ? ?>? ? ?LDFLAGS="-pthread -lz" COPTFLAGS="-O3 -march=native
>>      >? ? ?-mtune=native"
>>      >? ? ?>? ? ?>? ? ?CXXOPTFLAGS="-O3 -march=native -mtune=native"
>>      FOPTFLAGS="-O3
>>      >? ? ?>? ? ?>? ? ?-march=native -mtune=native" --with-mpi=1
>>      >? ? ?>? ? ?>? ? ?--with-mpi-dir=/home/eprice/software/mpich
>>      --with-mumps=1
>>      >? ? ?>? ? ?>? ? ?--with-mumps-dir=/home/eprice/software/mumps
>>      >? ? ?--with-parmetis=1
>>      >? ? ?>? ? ?>? ? ?--with-parmetis-dir=/home/eprice/software/parmetis
>>      >? ? ?--with-metis=1
>>      >? ? ?>? ? ?>? ? ?--with-metis-dir=/home/eprice/software/parmetis
>>      >? ? ?--with-ptscotch=1
>>      >? ? ?>? ? ?>? ? ?--with-ptscotch-dir=/home/eprice/software/scotch
>>      >? ? ?>? ? ?--with-scalapack=1
>>      >? ? ?>? ? ?>? ? ?--with-scalapack-dir=/home/eprice/software/scalapack
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #1 DMProjectLocal_Generic_Plex()
>>      line 347 in
>>      >? ? ?>? ? ?>
>>      ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #2 DMProjectFunctionLocal_Plex()
>>      line 428 in
>>      >? ? ?>? ? ?>
>>      ?/h/sabriel0/src/petsc-3.9.2/src/dm/impls/plex/plexproject.c
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #3 DMProjectFunctionLocal() line
>>      6265 in
>>      >? ? ?>? ? ?>? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>>      >? ? ?>? ? ?>? ? ?[0]PETSC ERROR: #4 DMProjectFunction() line 6250 in
>>      >? ? ?>? ? ?>? ? ?/h/sabriel0/src/petsc-3.9.2/src/dm/interface/dm.c
>>      >? ? ?>? ? ?>? ? ?...
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>? ? ?(I know this is an optimized PETSc build, but I
>>      get the same
>>      >? ? ?>? ? ?error from
>>      >? ? ?>? ? ?>? ? ?my debug build, it's just much slower.)
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>? ? ?Ellen
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?> --
>>      >? ? ?>? ? ?> What most experimenters take for granted before they
>>      begin their
>>      >? ? ?>? ? ?> experiments is infinitely more interesting than any
>>      results
>>      >? ? ?to which
>>      >? ? ?>? ? ?> their experiments lead.
>>      >? ? ?>? ? ?> -- Norbert Wiener
>>      >? ? ?>? ? ?>
>>      >? ? ?>? ? ?> https://www.cse.buffalo.edu/~knepley/
>>      <https://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?>? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?>? ? ?> <http://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?>
>>      >? ? ?> --
>>      >? ? ?> What most experimenters take for granted before they begin their
>>      >? ? ?> experiments is infinitely more interesting than any results
>>      to which
>>      >? ? ?> their experiments lead.
>>      >? ? ?> -- Norbert Wiener
>>      >? ? ?>
>>      >? ? ?> https://www.cse.buffalo.edu/~knepley/
>>      <https://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?<https://www.cse.buffalo.edu/%7Eknepley/>
>>      >? ? ?> <http://www.cse.buffalo.edu/%7Eknepley/>
>>      >
>>      >
>>      >
>>      > --
>>      > What most experimenters take for granted before they begin their
>>      > experiments is infinitely more interesting than any results to which
>>      > their experiments lead.
>>      > -- Norbert Wiener
>>      >
>>      > https://www.cse.buffalo.edu/~knepley/
>>      <https://www.cse.buffalo.edu/%7Eknepley/>
>>      > <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>> their experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: dmload.c
Type: text/x-csrc
Size: 15415 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181014/684c0179/attachment-0001.bin>

From zakaryah at gmail.com  Sun Oct 14 14:55:19 2018
From: zakaryah at gmail.com (zakaryah)
Date: Sun, 14 Oct 2018 15:55:19 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <CAMYG4GmNWxGMWABOx3kBwekh_Cwk5W2ikWgG9Ho=kkO3=jm6VQ@mail.gmail.com>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
	<CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
	<dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>
	<CAMYG4GmNWxGMWABOx3kBwekh_Cwk5W2ikWgG9Ho=kkO3=jm6VQ@mail.gmail.com>
Message-ID: <CABm8tB4Ox_NeS9kFaBzYDRWQbSNCRRgNY5o9wuY88MGpf_sf1w@mail.gmail.com>

Hi Matt,

> Can you explain more about this source term? It sounds like a bunch of
>> > delta functions. That
>> > would still work in this framework, but the convergence rate for a rhs
>> > with these singularities
>> > is reduced (this is a generic feature of FEM).
>>
>
Can you elaborate on this, or suggest references?  In the context of
elasticity, does this mean that convergence for problems using node forces
us generally worse than with the equivalent body forces?  Thanks!

>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181014/1cd4dffb/attachment.html>

From knepley at gmail.com  Sun Oct 14 18:13:22 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 14 Oct 2018 19:13:22 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <CABm8tB4Ox_NeS9kFaBzYDRWQbSNCRRgNY5o9wuY88MGpf_sf1w@mail.gmail.com>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
	<CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
	<dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>
	<CAMYG4GmNWxGMWABOx3kBwekh_Cwk5W2ikWgG9Ho=kkO3=jm6VQ@mail.gmail.com>
	<CABm8tB4Ox_NeS9kFaBzYDRWQbSNCRRgNY5o9wuY88MGpf_sf1w@mail.gmail.com>
Message-ID: <CAMYG4GmZZtj84VTe39Ks17AfsFvZnsjn7u0mUN9_Ka1Gog3zZw@mail.gmail.com>

On Sun, Oct 14, 2018 at 3:56 PM zakaryah <zakaryah at gmail.com> wrote:

> Hi Matt,
>
> > Can you explain more about this source term? It sounds like a bunch of
>>> > delta functions. That
>>> > would still work in this framework, but the convergence rate for a rhs
>>> > with these singularities
>>> > is reduced (this is a generic feature of FEM).
>>>
>>
> Can you elaborate on this, or suggest references?  In the context of
> elasticity, does this mean that convergence for problems using node forces
> us generally worse than with the equivalent body forces?  Thanks!
>
>>
Think of the simplest example, which is a Laplacian with a delta function
source. The solution is the Green's function 1/r.
This is singular at the origin (and not in the FEM space), and the standard
FEM bases are bad at approximating 1/r near the origin.
Babuska has a bunch of papers on this. Now what does "worse" mean in the
context of node forces vs tractions? It means that the
convergence of your computed solution, as you refine the mesh, is slower to
the true solution of the node forces problem then it is
to the true solution of the tractions problem.

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181014/cc2c4058/attachment.html>

From zakaryah at gmail.com  Sun Oct 14 20:04:47 2018
From: zakaryah at gmail.com (zakaryah)
Date: Sun, 14 Oct 2018 21:04:47 -0400
Subject: [petsc-users] Implementing FEM with PLEX/DS/SNES
In-Reply-To: <CAMYG4GmZZtj84VTe39Ks17AfsFvZnsjn7u0mUN9_Ka1Gog3zZw@mail.gmail.com>
References: <ac1e2a92-4859-697f-12a7-f1b05e29a6b2@cfa.harvard.edu>
	<CAMYG4GkqS_oGoSUHLyLdYERzu_Phagdd6QfSNYsPuNgwSDOgZg@mail.gmail.com>
	<dbc1b5d7-6d59-528c-4baa-54d2e03ec57e@cfa.harvard.edu>
	<CAMYG4GmNWxGMWABOx3kBwekh_Cwk5W2ikWgG9Ho=kkO3=jm6VQ@mail.gmail.com>
	<CABm8tB4Ox_NeS9kFaBzYDRWQbSNCRRgNY5o9wuY88MGpf_sf1w@mail.gmail.com>
	<CAMYG4GmZZtj84VTe39Ks17AfsFvZnsjn7u0mUN9_Ka1Gog3zZw@mail.gmail.com>
Message-ID: <CABm8tB4B0eXJO1=945rEbehdSge2_ozj3UE7Yqu1pYjhbB7b2Q@mail.gmail.com>

Thanks Matt - this is very helpful.

On Sun, Oct 14, 2018, 7:13 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Sun, Oct 14, 2018 at 3:56 PM zakaryah <zakaryah at gmail.com> wrote:
>
>> Hi Matt,
>>
>> > Can you explain more about this source term? It sounds like a bunch of
>>>> > delta functions. That
>>>> > would still work in this framework, but the convergence rate for a rhs
>>>> > with these singularities
>>>> > is reduced (this is a generic feature of FEM).
>>>>
>>>
>> Can you elaborate on this, or suggest references?  In the context of
>> elasticity, does this mean that convergence for problems using node forces
>> us generally worse than with the equivalent body forces?  Thanks!
>>
>>>
> Think of the simplest example, which is a Laplacian with a delta function
> source. The solution is the Green's function 1/r.
> This is singular at the origin (and not in the FEM space), and the
> standard FEM bases are bad at approximating 1/r near the origin.
> Babuska has a bunch of papers on this. Now what does "worse" mean in the
> context of node forces vs tractions? It means that the
> convergence of your computed solution, as you refine the mesh, is slower
> to the true solution of the node forces problem then it is
> to the true solution of the tractions problem.
>
>    Matt
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181014/4e0cf8fb/attachment.html>

From yjwu16 at gmail.com  Mon Oct 15 09:41:27 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Mon, 15 Oct 2018 22:41:27 +0800
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
Message-ID: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>

Dear Petsc developer:
Hi,
Thank you very much for your previous reply.
I recently wanted to modify my program to parallel version, and encountered
some problems in modifying it.
1. There are functions (read matrix) in the program that reads files,will
they affect my parallelism?  The codes are as follows:

ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
&viewer); CHKERRQ (ierr);
ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);

I read two matrix information from a binary file and wanted to use it on
each processor (in fact, my goal was to use these two matrices to give
initial values to the two field variables). The program can run in serial
time. Now I want to change it to parallel program. What do I need to
modify?
2. Following the last question, I used the following code in giving initial
value procedure FormInitialGuess():

ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);

I found this function on manualpages, and I felt that it could fulfill my
need to represent the elements of the matrix in arrays to give field
variables an initial value in each grid. The matrix A1 and A2 above are
actually the matrix information that was read from the file before. Now I
want to modify it as a parallel program. How do I use matrix information to
give initial values in parallel? (In program, field variables are
implemented through DM because parallel computing and Ghost Value are
supported)

Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/f3aae8e4/attachment.html>

From knepley at gmail.com  Mon Oct 15 09:52:39 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Oct 2018 10:52:39 -0400
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
Message-ID: <CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>

On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Dear Petsc developer:
> Hi,
> Thank you very much for your previous reply.
> I recently wanted to modify my program to parallel version, and
> encountered some problems in modifying it.
> 1. There are functions (read matrix) in the program that reads files,will
> they affect my parallelism?
>

No, MatLoad works in parallel. This will work unchanged if you want the
default layout. If you want a special partition,
you must call MatSetSizes() after MatCreate().


> The codes are as follows:
>
> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
> &viewer); CHKERRQ (ierr);
> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>
> I read two matrix information from a binary file and wanted to use it on
> each processor (in fact, my goal was to use these two matrices to give
> initial values to the two field variables). The program can run in serial
> time. Now I want to change it to parallel program. What do I need to
> modify?
> 2. Following the last question, I used the following code in giving
> initial value procedure FormInitialGuess():
>
> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>
> I found this function on manualpages, and I felt that it could fulfill my
> need to represent the elements of the matrix in arrays to give field
> variables an initial value in each grid. The matrix A1 and A2 above are
> actually the matrix information that was read from the file before. Now I
> want to modify it as a parallel program. How do I use matrix information to
> give initial values in parallel? (In program, field variables are
> implemented through DM because parallel computing and Ghost Value are
> supported)
>

I do not understand the use of matrices to initialize field values. Usually
field values are stored on Vec objects, and this is
the philosophy of DM objects.

  Thanks,

     Matt


> Thanks,
> Yingjie
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/d4e8af13/attachment-0001.html>

From yjwu16 at gmail.com  Mon Oct 15 10:12:54 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Mon, 15 Oct 2018 23:12:54 +0800
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
	<CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
Message-ID: <CAAMEhBjjqyCnGCjxhWReCVOCA86E360TXUadvVfe9EOM9g0c8Q@mail.gmail.com>

Thank you very much, Matt.

My problem is actually a two-dimensional problem, and the matrix in my
binary file is actually a two-dimensional array, each point corresponding
to a value on the two-dimensional grid.
Your suggestion is very useful to me. I can try to convert the matrix
binary file into a vector binary file to give program an initial value. So
what I should do is:
1. establish parallel vectors.
 2. reads data from binary files to vectors. (VecLoad())
3. use VecGetArray() in FormInitialGuess() to use vector elements to give
the initial values of the field variables.
I know DM can support parallel computing (domain decomposition). How can I
ensure that the part of the vector built on each processor corresponds to
the index of DM object? (Cause my field variables are established by
DMDACreate2d,  and I want to assign initial values to field variables using
read vectors )

Thanks,
Yingjie

Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??10:52???

> On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Dear Petsc developer:
>> Hi,
>> Thank you very much for your previous reply.
>> I recently wanted to modify my program to parallel version, and
>> encountered some problems in modifying it.
>> 1. There are functions (read matrix) in the program that reads files,will
>> they affect my parallelism?
>>
>
> No, MatLoad works in parallel. This will work unchanged if you want the
> default layout. If you want a special partition,
> you must call MatSetSizes() after MatCreate().
>
>
>> The codes are as follows:
>>
>> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
>> &viewer); CHKERRQ (ierr);
>> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
>> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
>> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
>> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
>> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
>> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
>> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>>
>> I read two matrix information from a binary file and wanted to use it on
>> each processor (in fact, my goal was to use these two matrices to give
>> initial values to the two field variables). The program can run in serial
>> time. Now I want to change it to parallel program. What do I need to
>> modify?
>> 2. Following the last question, I used the following code in giving
>> initial value procedure FormInitialGuess():
>>
>> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
>> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>>
>> I found this function on manualpages, and I felt that it could fulfill my
>> need to represent the elements of the matrix in arrays to give field
>> variables an initial value in each grid. The matrix A1 and A2 above are
>> actually the matrix information that was read from the file before. Now I
>> want to modify it as a parallel program. How do I use matrix information to
>> give initial values in parallel? (In program, field variables are
>> implemented through DM because parallel computing and Ghost Value are
>> supported)
>>
>
> I do not understand the use of matrices to initialize field values.
> Usually field values are stored on Vec objects, and this is
> the philosophy of DM objects.
>
>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Yingjie
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/5d60b58c/attachment.html>

From knepley at gmail.com  Mon Oct 15 10:32:09 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Oct 2018 11:32:09 -0400
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAAMEhBjjqyCnGCjxhWReCVOCA86E360TXUadvVfe9EOM9g0c8Q@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
	<CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
	<CAAMEhBjjqyCnGCjxhWReCVOCA86E360TXUadvVfe9EOM9g0c8Q@mail.gmail.com>
Message-ID: <CAMYG4Gk6dj+Lo3J66Y7tP-VjZCn1Wjaon5gK=i5BJNDNgNU57Q@mail.gmail.com>

On Mon, Oct 15, 2018 at 11:13 AM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Thank you very much, Matt.
>
> My problem is actually a two-dimensional problem, and the matrix in my
> binary file is actually a two-dimensional array, each point corresponding
> to a value on the two-dimensional grid.
>

These are still vectors. Vec store fields, which gives values to points on
your grid. Matrices map between states, so between vectors.


> Your suggestion is very useful to me. I can try to convert the matrix
> binary file into a vector binary file to give program an initial value. So
> what I should do is:
> 1. establish parallel vectors.
>  2. reads data from binary files to vectors. (VecLoad())
> 3. use VecGetArray() in FormInitialGuess() to use vector elements to give
> the initial values of the field variables.
> I know DM can support parallel computing (domain decomposition). How can I
> ensure that the part of the vector built on each processor corresponds to
> the index of DM object? (Cause my field variables are established by
> DMDACreate2d,  and I want to assign initial values to field variables using
> read vectors )
>

Yes. Many times you will want to VecLoad, then DMGlobalToLocal() to get
ghosted vectors, then DMVecGetArray() to get ghosted arrays.

   Matt


>
> Thanks,
> Yingjie
>
> Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??10:52???
>
>> On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>>
>>> Dear Petsc developer:
>>> Hi,
>>> Thank you very much for your previous reply.
>>> I recently wanted to modify my program to parallel version, and
>>> encountered some problems in modifying it.
>>> 1. There are functions (read matrix) in the program that reads
>>> files,will they affect my parallelism?
>>>
>>
>> No, MatLoad works in parallel. This will work unchanged if you want the
>> default layout. If you want a special partition,
>> you must call MatSetSizes() after MatCreate().
>>
>>
>>> The codes are as follows:
>>>
>>> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
>>> &viewer); CHKERRQ (ierr);
>>> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
>>> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
>>> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
>>> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
>>> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
>>> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
>>> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>>>
>>> I read two matrix information from a binary file and wanted to use it on
>>> each processor (in fact, my goal was to use these two matrices to give
>>> initial values to the two field variables). The program can run in serial
>>> time. Now I want to change it to parallel program. What do I need to
>>> modify?
>>> 2. Following the last question, I used the following code in giving
>>> initial value procedure FormInitialGuess():
>>>
>>> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
>>> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>>>
>>>
>>> I found this function on manualpages, and I felt that it could fulfill
>>> my need to represent the elements of the matrix in arrays to give field
>>> variables an initial value in each grid. The matrix A1 and A2 above are
>>> actually the matrix information that was read from the file before. Now I
>>> want to modify it as a parallel program. How do I use matrix information to
>>> give initial values in parallel? (In program, field variables are
>>> implemented through DM because parallel computing and Ghost Value are
>>> supported)
>>>
>>
>> I do not understand the use of matrices to initialize field values.
>> Usually field values are stored on Vec objects, and this is
>> the philosophy of DM objects.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Thanks,
>>> Yingjie
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/08b0e746/attachment.html>

From dave.mayhem23 at gmail.com  Mon Oct 15 10:38:45 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Mon, 15 Oct 2018 17:38:45 +0200
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
	<CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
Message-ID: <CAJ98EDqfj2grxL4cdMTsid8vecCa3otNgibucjC+=JHCEVfJPg@mail.gmail.com>

On Mon, 15 Oct 2018 at 16:54, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Dear Petsc developer:
>> Hi,
>> Thank you very much for your previous reply.
>> I recently wanted to modify my program to parallel version, and
>> encountered some problems in modifying it.
>> 1. There are functions (read matrix) in the program that reads files,will
>> they affect my parallelism?
>>
>
> No, MatLoad works in parallel. This will work unchanged if you want the
> default layout. If you want a special partition,
> you must call MatSetSizes() after MatCreate().
>
>
>> The codes are as follows:
>>
>> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
>> &viewer); CHKERRQ (ierr);
>> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
>> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
>> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
>> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
>> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
>> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
>> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>>
>> I read two matrix information from a binary file and wanted to use it on
>> each processor (in fact, my goal was to use these two matrices to give
>> initial values to the two field variables). The program can run in serial
>> time. Now I want to change it to parallel program. What do I need to
>> modify?
>> 2. Following the last question, I used the following code in giving
>> initial value procedure FormInitialGuess():
>>
>> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
>> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>>
>> I found this function on manualpages, and I felt that it could fulfill my
>> need to represent the elements of the matrix in arrays to give field
>> variables an initial value in each grid. The matrix A1 and A2 above are
>> actually the matrix information that was read from the file before. Now I
>> want to modify it as a parallel program. How do I use matrix information to
>> give initial values in parallel? (In program, field variables are
>> implemented through DM because parallel computing and Ghost Value are
>> supported)
>>
>
> I do not understand the use of matrices to initialize field values.
> Usually field values are stored on Vec objects, and this is
> the philosophy of DM objects.
>

Maybe this is a "matlab style" idea / way of thinking, e.g. representing 2d
meshes and fields as matrices because they "look kinda the same....". If
that is the case - don't do it! Use vectors to represent fields



>   Thanks,
>
>      Matt
>
>
>> Thanks,
>> Yingjie
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/98f74803/attachment-0001.html>

From yjwu16 at gmail.com  Mon Oct 15 11:03:03 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Tue, 16 Oct 2018 00:03:03 +0800
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAMYG4Gk6dj+Lo3J66Y7tP-VjZCn1Wjaon5gK=i5BJNDNgNU57Q@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
	<CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
	<CAAMEhBjjqyCnGCjxhWReCVOCA86E360TXUadvVfe9EOM9g0c8Q@mail.gmail.com>
	<CAMYG4Gk6dj+Lo3J66Y7tP-VjZCn1Wjaon5gK=i5BJNDNgNU57Q@mail.gmail.com>
Message-ID: <CAAMEhBgHtCdxWTAUNdSpi9J_1Gwm45a94JE8PCf=uUvgEshRMg@mail.gmail.com>

Thanks,Matt

I'm sorry I didn't describe the problem clearly.
I'm still wondering how, although VecLoad supports parallel vector reading,
how can I guarantee that the local vector on the processor corresponds to
the DM's IS?
For example, when in FormInitialGuess, I want to make sure that the array
VecGetArray() obtained by the local vector(Vectors read from binary files)
and the array  DMGlobalToLocal () DMVecGetArray () obtained by the DM
belong to the same processor, so that some errors in the mpi can be avoided.
Thanks,
Yingjie

Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??11:32???

> On Mon, Oct 15, 2018 at 11:13 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Thank you very much, Matt.
>>
>> My problem is actually a two-dimensional problem, and the matrix in my
>> binary file is actually a two-dimensional array, each point corresponding
>> to a value on the two-dimensional grid.
>>
>
> These are still vectors. Vec store fields, which gives values to points on
> your grid. Matrices map between states, so between vectors.
>
>
>> Your suggestion is very useful to me. I can try to convert the matrix
>> binary file into a vector binary file to give program an initial value. So
>> what I should do is:
>> 1. establish parallel vectors.
>>  2. reads data from binary files to vectors. (VecLoad())
>> 3. use VecGetArray() in FormInitialGuess() to use vector elements to give
>> the initial values of the field variables.
>> I know DM can support parallel computing (domain decomposition). How can
>> I ensure that the part of the vector built on each processor corresponds to
>> the index of DM object? (Cause my field variables are established by
>> DMDACreate2d,  and I want to assign initial values to field variables using
>> read vectors )
>>
>
> Yes. Many times you will want to VecLoad, then DMGlobalToLocal() to get
> ghosted vectors, then DMVecGetArray() to get ghosted arrays.
>
>    Matt
>
>
>>
>> Thanks,
>> Yingjie
>>
>> Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??10:52???
>>
>>> On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>>>
>>>> Dear Petsc developer:
>>>> Hi,
>>>> Thank you very much for your previous reply.
>>>> I recently wanted to modify my program to parallel version, and
>>>> encountered some problems in modifying it.
>>>> 1. There are functions (read matrix) in the program that reads
>>>> files,will they affect my parallelism?
>>>>
>>>
>>> No, MatLoad works in parallel. This will work unchanged if you want the
>>> default layout. If you want a special partition,
>>> you must call MatSetSizes() after MatCreate().
>>>
>>>
>>>> The codes are as follows:
>>>>
>>>> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
>>>> &viewer); CHKERRQ (ierr);
>>>> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
>>>> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
>>>> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
>>>> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
>>>> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
>>>> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
>>>> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>>>>
>>>> I read two matrix information from a binary file and wanted to use it
>>>> on each processor (in fact, my goal was to use these two matrices to give
>>>> initial values to the two field variables). The program can run in serial
>>>> time. Now I want to change it to parallel program. What do I need to
>>>> modify?
>>>> 2. Following the last question, I used the following code in giving
>>>> initial value procedure FormInitialGuess():
>>>>
>>>> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
>>>>
>>>> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>>>>
>>>>
>>>> I found this function on manualpages, and I felt that it could fulfill
>>>> my need to represent the elements of the matrix in arrays to give field
>>>> variables an initial value in each grid. The matrix A1 and A2 above are
>>>> actually the matrix information that was read from the file before. Now I
>>>> want to modify it as a parallel program. How do I use matrix information to
>>>> give initial values in parallel? (In program, field variables are
>>>> implemented through DM because parallel computing and Ghost Value are
>>>> supported)
>>>>
>>>
>>> I do not understand the use of matrices to initialize field values.
>>> Usually field values are stored on Vec objects, and this is
>>> the philosophy of DM objects.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Thanks,
>>>> Yingjie
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/afbc59e2/attachment.html>

From knepley at gmail.com  Mon Oct 15 11:47:53 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 15 Oct 2018 12:47:53 -0400
Subject: [petsc-users] Some Problems in Modifying Parallel Programs
In-Reply-To: <CAAMEhBgHtCdxWTAUNdSpi9J_1Gwm45a94JE8PCf=uUvgEshRMg@mail.gmail.com>
References: <CAAMEhBh=Pq-M0H8RM3NGWWK468_cmC9NUKLReTSQVPxXQm9KNA@mail.gmail.com>
	<CAMYG4Gk5FNCkBrt9qjzxJR7Esdi6STOxUn=Zm18pQkYTUwxROQ@mail.gmail.com>
	<CAAMEhBjjqyCnGCjxhWReCVOCA86E360TXUadvVfe9EOM9g0c8Q@mail.gmail.com>
	<CAMYG4Gk6dj+Lo3J66Y7tP-VjZCn1Wjaon5gK=i5BJNDNgNU57Q@mail.gmail.com>
	<CAAMEhBgHtCdxWTAUNdSpi9J_1Gwm45a94JE8PCf=uUvgEshRMg@mail.gmail.com>
Message-ID: <CAMYG4Gk7miDz_vER4D1DMwOY-7WqgE+EkvFsgjexLTQ2a+UFRA@mail.gmail.com>

On Mon, Oct 15, 2018 at 12:03 PM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Thanks,Matt
>
> I'm sorry I didn't describe the problem clearly.
> I'm still wondering how, although VecLoad supports parallel vector
> reading, how can I guarantee that the local vector on the processor
> corresponds to the DM's IS?
> For example, when in FormInitialGuess, I want to make sure that the array
> VecGetArray() obtained by the local vector(Vectors read from binary files)
> and the array  DMGlobalToLocal () DMVecGetArray () obtained by the DM
> belong to the same processor, so that some errors in the mpi can be avoided.
>

I think I understand your question. If you give in a Vec in natural
ordering, you can use VecLoad with a Vec you get from
DMCreateGlobalVector(), and it will automatically reorder it into the right
parallel ordering.

  Thanks,

    Matt


> Thanks,
> Yingjie
>
> Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??11:32???
>
>> On Mon, Oct 15, 2018 at 11:13 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>>
>>> Thank you very much, Matt.
>>>
>>> My problem is actually a two-dimensional problem, and the matrix in my
>>> binary file is actually a two-dimensional array, each point corresponding
>>> to a value on the two-dimensional grid.
>>>
>>
>> These are still vectors. Vec store fields, which gives values to points
>> on your grid. Matrices map between states, so between vectors.
>>
>>
>>> Your suggestion is very useful to me. I can try to convert the matrix
>>> binary file into a vector binary file to give program an initial value. So
>>> what I should do is:
>>> 1. establish parallel vectors.
>>>  2. reads data from binary files to vectors. (VecLoad())
>>> 3. use VecGetArray() in FormInitialGuess() to use vector elements to
>>> give the initial values of the field variables.
>>> I know DM can support parallel computing (domain decomposition). How can
>>> I ensure that the part of the vector built on each processor corresponds to
>>> the index of DM object? (Cause my field variables are established by
>>> DMDACreate2d,  and I want to assign initial values to field variables using
>>> read vectors )
>>>
>>
>> Yes. Many times you will want to VecLoad, then DMGlobalToLocal() to get
>> ghosted vectors, then DMVecGetArray() to get ghosted arrays.
>>
>>    Matt
>>
>>
>>>
>>> Thanks,
>>> Yingjie
>>>
>>> Matthew Knepley <knepley at gmail.com> ?2018?10?15??? ??10:52???
>>>
>>>> On Mon, Oct 15, 2018 at 10:42 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>>>>
>>>>> Dear Petsc developer:
>>>>> Hi,
>>>>> Thank you very much for your previous reply.
>>>>> I recently wanted to modify my program to parallel version, and
>>>>> encountered some problems in modifying it.
>>>>> 1. There are functions (read matrix) in the program that reads
>>>>> files,will they affect my parallelism?
>>>>>
>>>>
>>>> No, MatLoad works in parallel. This will work unchanged if you want the
>>>> default layout. If you want a special partition,
>>>> you must call MatSetSizes() after MatCreate().
>>>>
>>>>
>>>>> The codes are as follows:
>>>>>
>>>>> ierr = PetscViewerBinaryOpen (PETSC_COMM_WORLD, file, FILE_MODE_READ,
>>>>> &viewer); CHKERRQ (ierr);
>>>>> ierr = MatCreate (PETSC_COMM_WORLD, &A1); CHKERRQ (ierr);
>>>>> ierr = MatSetFromOptions (A1); CHKERRQ (ierr);
>>>>> ierr = MatCreate (PETSC_COMM_WORLD, &A2); CHKERRQ (ierr);
>>>>> ierr = MatSetFromOptions (A2); CHKERRQ (ierr);
>>>>> ierr = MatLoad (A1, viewer); CHKERRQ (ierr);
>>>>> ierr = MatLoad (A2, viewer); CHKERRQ (ierr);
>>>>> ierr = PetscViewerDestroy (&viewer); CHKERRQ (ierr);
>>>>>
>>>>> I read two matrix information from a binary file and wanted to use it
>>>>> on each processor (in fact, my goal was to use these two matrices to give
>>>>> initial values to the two field variables). The program can run in serial
>>>>> time. Now I want to change it to parallel program. What do I need to
>>>>> modify?
>>>>> 2. Following the last question, I used the following code in giving
>>>>> initial value procedure FormInitialGuess():
>>>>>
>>>>> ierr = MatSeqAIJGetArray (A1, &initial_phi1); CHKERRQ (ierr);
>>>>>
>>>>> ierr = MatSeqAIJGetArray (A2, &initial_phi2); CHKERRQ (ierr);
>>>>>
>>>>>
>>>>> I found this function on manualpages, and I felt that it could fulfill
>>>>> my need to represent the elements of the matrix in arrays to give field
>>>>> variables an initial value in each grid. The matrix A1 and A2 above are
>>>>> actually the matrix information that was read from the file before. Now I
>>>>> want to modify it as a parallel program. How do I use matrix information to
>>>>> give initial values in parallel? (In program, field variables are
>>>>> implemented through DM because parallel computing and Ghost Value are
>>>>> supported)
>>>>>
>>>>
>>>> I do not understand the use of matrices to initialize field values.
>>>> Usually field values are stored on Vec objects, and this is
>>>> the philosophy of DM objects.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Thanks,
>>>>> Yingjie
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/54d14714/attachment-0001.html>

From mvalera-w at sdsu.edu  Mon Oct 15 16:22:02 2018
From: mvalera-w at sdsu.edu (Manuel Valera)
Date: Mon, 15 Oct 2018 14:22:02 -0700
Subject: [petsc-users] Fwd:  Interpolation in staggered grid
In-Reply-To: <CAPzU9WXP1f_iMrofKSRNrAewQNvDhKxFiO76-XaZLzZNJ2O+oA@mail.gmail.com>
References: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>
	<CAMYG4G=VLfR_EfBFV9trX6VA2ETUOE2Fhc=6926rtDWiDEtRuQ@mail.gmail.com>
	<CAPzU9WXP1f_iMrofKSRNrAewQNvDhKxFiO76-XaZLzZNJ2O+oA@mail.gmail.com>
Message-ID: <CAPzU9WV5-OPh0wiBdUGm8-cHukbDMBeKPNUX3m-zzVZ9hF9ijQ@mail.gmail.com>

Thanks Matthew, i have made some progress but i am still unsure on how to
proceed to make the DMDAs work as intended, i will try to lay out what i am
trying now:

I was able to port the interpolation into the DMDA model but it works only
in serial runs, and it becomes unstable for more than one processor running,

What i do is roughly this:

   - Calculate density at a cell-centered DMDA object, using a local vector
   so i can access the ghost rows (indices -1 and max+1)
   - Interpolate into a face-centered DMDA object for velocities, using
   also a local vector.

Doing this i get the right results using the same interpolation i used for
my non-petsc implementation of the model, as long as i use only one
processor but the doubts i have are:

   - How to use the local vectors properly: Is operating on them the
   recommended course in this case?
   - How can i access the ghost indices in the global vector so i can then
   communicate GlobalToLocal? would this be a better strategy?
   - I feel is wrong to interpolate a cell-centered based vector into a
   face-centered based vector, using the indices of the latter, what strategy
   would work best in this case?

I also tried opening a global vector with a different DA layout than the
one created (opening global density on the velocities DMDA layout) and this
introduced an error in the GlobalToLocal, LocalToLocal, LocalToGlobal
communication, something didn't happen if i used the local versions of
these vectors instead,

Thanks for your help,

Manuel





On Sat, Oct 6, 2018 at 4:45 AM Matthew Knepley <knepley at gmail.com> wrote:

> On Fri, Oct 5, 2018 at 6:49 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:
>
>> Hello,
>>
>> I'm trying to do a simple variable interpolation, from a cell center to a
>> face in a staggered grid, my model data management is done with DMDAs, with
>> two different DMs one for each cell position,
>>
>> I already did this task on a fortran only version of the model using the
>> 4 closest neighbors of the scalars (cell center) to be interpolated at the
>> velocity site (cell face), i did it using a loop over the domain, somehow
>> this easy task is not translating into the DMDA framework,
>>
>
> Its not clear to me what problem you are having. We have done this before.
> For example,
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex30.c.html
>
> I would note that the development version of PETSc now has DMStag which
> supports staggered grid discretizations directly.
>
>   Thanks,
>
>     Matt
>
>
>> I'm not sure what I'm doing wrong or not aware of a easier way Petsc may
>> have for this task, if you could point out an easier strategy or an example
>> I would be grateful,
>>
>> Thanks,
>>
>>
>>
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181015/6714c1ba/attachment.html>

From popov at uni-mainz.de  Tue Oct 16 02:56:47 2018
From: popov at uni-mainz.de (Anton Popov)
Date: Tue, 16 Oct 2018 09:56:47 +0200
Subject: [petsc-users] Fwd: Interpolation in staggered grid
In-Reply-To: <CAPzU9WV5-OPh0wiBdUGm8-cHukbDMBeKPNUX3m-zzVZ9hF9ijQ@mail.gmail.com>
References: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>
	<CAMYG4G=VLfR_EfBFV9trX6VA2ETUOE2Fhc=6926rtDWiDEtRuQ@mail.gmail.com>
	<CAPzU9WXP1f_iMrofKSRNrAewQNvDhKxFiO76-XaZLzZNJ2O+oA@mail.gmail.com>
	<CAPzU9WV5-OPh0wiBdUGm8-cHukbDMBeKPNUX3m-zzVZ9hF9ijQ@mail.gmail.com>
Message-ID: <63a76b79-e51b-74df-6282-72477e11ec0e@uni-mainz.de>

Hi Manuel,

you might consider useful taking a look at this code 
https://bitbucket.org/bkaus/lamem, which is a multi-DMDA staggered grid 
implementation.

Also, as Matt pointed out, there is a direct PETSc support for staggered 
grids (DMSTAG), recently contributed by Patrick Sanan.

Thanks,

Anton


On 15.10.2018 23:22, Manuel Valera wrote:
>
> Thanks Matthew, i have made some progress but i am still unsure on how 
> to proceed to make the DMDAs work as intended, i will try to lay out 
> what i am trying now:
>
> I was able to port the interpolation into the DMDA model but it works 
> only in serial runs, and it becomes unstable for more than one 
> processor running,
>
> What i do is roughly this:
>
>   * Calculate density at a cell-centered DMDA object, using a local
>     vector so i can access the ghost rows (indices -1 and max+1)
>   * Interpolate into a face-centered DMDA object for velocities, using
>     also a local vector.
>
> Doing this i get the right results using the same interpolation i used 
> for my non-petsc implementation of the model, as long as i use only 
> one processor but the doubts i have are:
>
>   * How to use the local vectors properly: Is operating on them the
>     recommended course in this case?
>   * How can i access the ghost indices in the global vector so i can
>     then communicate GlobalToLocal? would this be a better strategy?
>   * I feel is wrong to interpolate a cell-centered based vector into a
>     face-centered based vector, using the indices of the latter, what
>     strategy would work best in this case?
>
> I also tried opening a global vector with a different DA layout than 
> the one created (opening global density on the velocities DMDA layout) 
> and this introduced an error in the GlobalToLocal, LocalToLocal, 
> LocalToGlobal communication, something didn't happen if i used the 
> local versions of these vectors instead,
>
> Thanks for your help,
>
> Manuel
>
>
>
>
>
> On Sat, Oct 6, 2018 at 4:45 AM Matthew Knepley <knepley at gmail.com 
> <mailto:knepley at gmail.com>> wrote:
>
>     On Fri, Oct 5, 2018 at 6:49 PM Manuel Valera <mvalera-w at sdsu.edu
>     <mailto:mvalera-w at sdsu.edu>> wrote:
>
>         Hello,
>
>         I'm trying to do a simple variable interpolation, from a cell
>         center to a face in a staggered grid, my model data management
>         is done with DMDAs, with two different DMs one for each cell
>         position,
>
>         I already did this task on a fortran only version of the model
>         using the 4 closest neighbors of the scalars (cell center) to
>         be interpolated at the velocity site (cell face), i did it
>         using a loop over the domain, somehow this easy task is not
>         translating into the DMDA framework,
>
>
>     Its not clear to me what problem you are having. We have done this
>     before. For example,
>
>     https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex30.c.html
>
>     I would note that the development version of PETSc now has DMStag
>     which supports staggered grid discretizations directly.
>
>     ? Thanks,
>
>     ? ? Matt
>
>         I'm not sure what I'm doing wrong or not aware of a easier way
>         Petsc may have for this task, if you could point out an easier
>         strategy or an example I would be grateful,
>
>         Thanks,
>
>
>
>
>
>
>     -- 
>     What most experimenters take for granted before they begin their
>     experiments is infinitely more interesting than any results to
>     which their experiments lead.
>     -- Norbert Wiener
>
>     https://www.cse.buffalo.edu/~knepley/
>     <http://www.cse.buffalo.edu/%7Eknepley/>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/05562079/attachment.html>

From knepley at gmail.com  Tue Oct 16 05:06:27 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Oct 2018 06:06:27 -0400
Subject: [petsc-users] Fwd: Interpolation in staggered grid
In-Reply-To: <63a76b79-e51b-74df-6282-72477e11ec0e@uni-mainz.de>
References: <CAPzU9WU8xzYAtRO30dDwutXVoe2Hc17y_0L-yDD33vpZ-6=uJw@mail.gmail.com>
	<CAMYG4G=VLfR_EfBFV9trX6VA2ETUOE2Fhc=6926rtDWiDEtRuQ@mail.gmail.com>
	<CAPzU9WXP1f_iMrofKSRNrAewQNvDhKxFiO76-XaZLzZNJ2O+oA@mail.gmail.com>
	<CAPzU9WV5-OPh0wiBdUGm8-cHukbDMBeKPNUX3m-zzVZ9hF9ijQ@mail.gmail.com>
	<63a76b79-e51b-74df-6282-72477e11ec0e@uni-mainz.de>
Message-ID: <CAMYG4GmGTLHVpVqQaP0cMk0FV922ugvZ6Xp-CA4XdALimRgmaA@mail.gmail.com>

On Tue, Oct 16, 2018 at 3:57 AM Anton Popov <popov at uni-mainz.de> wrote:

> Hi Manuel,
>
> you might consider useful taking a look at this code
> https://bitbucket.org/bkaus/lamem, which is a multi-DMDA staggered grid
> implementation.
>
> Also, as Matt pointed out, there is a direct PETSc support for staggered
> grids (DMSTAG), recently contributed by Patrick Sanan.
>
> Anton is right, LaMeM is a great example. Did you also have a look at SNES
ex30? It is staggered grid FV and works in parallel.

  Thanks,

    Matt

> Thanks,
>
> Anton
>
> On 15.10.2018 23:22, Manuel Valera wrote:
>
>
> Thanks Matthew, i have made some progress but i am still unsure on how to
> proceed to make the DMDAs work as intended, i will try to lay out what i am
> trying now:
>
> I was able to port the interpolation into the DMDA model but it works only
> in serial runs, and it becomes unstable for more than one processor running,
>
> What i do is roughly this:
>
>    - Calculate density at a cell-centered DMDA object, using a local
>    vector so i can access the ghost rows (indices -1 and max+1)
>    - Interpolate into a face-centered DMDA object for velocities, using
>    also a local vector.
>
> Doing this i get the right results using the same interpolation i used for
> my non-petsc implementation of the model, as long as i use only one
> processor but the doubts i have are:
>
>    - How to use the local vectors properly: Is operating on them the
>    recommended course in this case?
>    - How can i access the ghost indices in the global vector so i can
>    then communicate GlobalToLocal? would this be a better strategy?
>    - I feel is wrong to interpolate a cell-centered based vector into a
>    face-centered based vector, using the indices of the latter, what strategy
>    would work best in this case?
>
> I also tried opening a global vector with a different DA layout than the
> one created (opening global density on the velocities DMDA layout) and this
> introduced an error in the GlobalToLocal, LocalToLocal, LocalToGlobal
> communication, something didn't happen if i used the local versions of
> these vectors instead,
>
> Thanks for your help,
>
> Manuel
>
>
>
>
>
> On Sat, Oct 6, 2018 at 4:45 AM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Fri, Oct 5, 2018 at 6:49 PM Manuel Valera <mvalera-w at sdsu.edu> wrote:
>>
>>> Hello,
>>>
>>> I'm trying to do a simple variable interpolation, from a cell center to
>>> a face in a staggered grid, my model data management is done with DMDAs,
>>> with two different DMs one for each cell position,
>>>
>>> I already did this task on a fortran only version of the model using the
>>> 4 closest neighbors of the scalars (cell center) to be interpolated at the
>>> velocity site (cell face), i did it using a loop over the domain, somehow
>>> this easy task is not translating into the DMDA framework,
>>>
>>
>> Its not clear to me what problem you are having. We have done this
>> before. For example,
>>
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex30.c.html
>>
>> I would note that the development version of PETSc now has DMStag which
>> supports staggered grid discretizations directly.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> I'm not sure what I'm doing wrong or not aware of a easier way Petsc may
>>> have for this task, if you could point out an easier strategy or an example
>>> I would be grateful,
>>>
>>> Thanks,
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>
>>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/25b12f16/attachment-0001.html>

From ysjosh.lo at gmail.com  Tue Oct 16 11:30:15 2018
From: ysjosh.lo at gmail.com (Josh L)
Date: Tue, 16 Oct 2018 11:30:15 -0500
Subject: [petsc-users] Gauss quadrature for triangle or tetrahedral element
 and Nature ordering
Message-ID: <CA+2ebbB7r_9u9RSCShJ41DC=cB3uBQ+HxJA=c0d5W_rgBTH9Gw@mail.gmail.com>

Hi,

Does anyone use PETSc to calculate the value determinant, basis function
and the derivative of basis function at Gauss integration point?

For quadrilateral element, I use PetscDTGaussQuadrature to get the location
of integration point, and then use DMPlexComputeCellGeometryFEM to get the
value of determinant.

But I don't know how to do it for triangle element, and don't know how to
use PETSc to calculate basis function and it's derivative?

Is there anyway to get the mapping between nature ordering and global
ordering for vertex number?


Thanks,
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/b6f05e4d/attachment.html>

From knepley at gmail.com  Tue Oct 16 13:55:51 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Oct 2018 14:55:51 -0400
Subject: [petsc-users] Gauss quadrature for triangle or tetrahedral
 element and Nature ordering
In-Reply-To: <CA+2ebbB7r_9u9RSCShJ41DC=cB3uBQ+HxJA=c0d5W_rgBTH9Gw@mail.gmail.com>
References: <CA+2ebbB7r_9u9RSCShJ41DC=cB3uBQ+HxJA=c0d5W_rgBTH9Gw@mail.gmail.com>
Message-ID: <CAMYG4G=-8q2L9sa8PNM8VQCs8OEYAxZB39=C7gSwJ7ijXsMtzQ@mail.gmail.com>

On Tue, Oct 16, 2018 at 12:31 PM Josh L <ysjosh.lo at gmail.com> wrote:

> Hi,
>
> Does anyone use PETSc to calculate the value determinant, basis function
> and the derivative of basis function at Gauss integration point?
>
> For quadrilateral element, I use PetscDTGaussQuadrature to get the
> location of integration point, and then use DMPlexComputeCellGeometryFEM to
> get the value of determinant.
>
> But I don't know how to do it for triangle element, and don't know how to
> use PETSc to calculate basis function and it's derivative?
>
> Is there anyway to get the mapping between nature ordering and global
> ordering for vertex number?
>

The reason that it is a little disjointed is that we are in the process of
converting from what I had previously done:


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DMPLEX/DMPlexComputeCellGeometryFEM.html

to something which is much more flexible and nice


https://www.mcs.anl.gov/petsc/petsc-master/docs/manualpages/DM/DMFieldCreateFEGeom.html

You get the DMField object from

  PetscErrorCode DMGetCoordinateField(DM dm, DMField *field)

which is sadly not documented yet, but just gives back the right thing. If
the DMField stuff looks too new, you can just use the
old code until the documentation catches up. Otherwise, feel free to mail
DMField questions. The reason for DMField is that
you can have very general things that can evaluate themselves (particles,
Fourier bases, etc.) instead of just FEM.

  Thanks,

    Matt


> Thanks,
> Josh
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/bfd2e305/attachment.html>

From weizhuo2 at illinois.edu  Tue Oct 16 18:26:14 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Tue, 16 Oct 2018 18:26:14 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
Message-ID: <CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>

Hello again!

After some tweaking the code is giving right answers now. However it start
to disagree with MATLAB results ('traditional' way using matrix inverse)
when the grid is larger than 100*100. My PhD advisor and I suspects that
the default dimension of the Krylov subspace is 100 in the test case we are
running. If so, is there a way to increase the size of the subspace?

[image: Disagrees.png]

Thanks!

Wang Weizhuo

On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:

> To reiterate what Matt is saying, you seem to have the exact solution on a
> 10x10 grid. That makes no sense unless the solution can be represented
> exactly by your FE space (eg, u(x,y) = x + y).
>
> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> The code is attached in case anyone wants to take a look, I will try the
>>> high frequency scenario later.
>>>
>>
>> That is not the error. It is superconvergence at the vertices. The real
>> solution is trigonometric, so your
>> linear interpolants or whatever you use is not going to get the right
>> value in between mesh points. You
>> need to do a real integral over the whole interval to get the L_2 error.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>>
>>>>
>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> The first plot is the norm with the flag -pc_type lu with respect to
>>>>> number of grids in one axis (n), and the second plot is the norm without
>>>>> the flag -pc_type lu.
>>>>>
>>>>
>>>> So you are using the default PC w/o LU. The default is ILU. This will
>>>> reduce high frequency effectively but is not effective on the low frequency
>>>> error. Don't expect your algebraic error reduction to be at the same scale
>>>> as the residual reduction (what KSP measures).
>>>>
>>>>
>>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/6b1fab38/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/6b1fab38/attachment-0001.png>

From knepley at gmail.com  Tue Oct 16 20:05:47 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Oct 2018 21:05:47 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
Message-ID: <CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>

On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> Hello again!
>
> After some tweaking the code is giving right answers now. However it start
> to disagree with MATLAB results ('traditional' way using matrix inverse)
> when the grid is larger than 100*100. My PhD advisor and I suspects that
> the default dimension of the Krylov subspace is 100 in the test case we are
> running. If so, is there a way to increase the size of the subspace?
>

1) The default subspace size is 30, not 100. You can increase the subspace
size using

       -ksp_gmres_restart n

2) The problem is likely your tolerance. The default solver tolerance is
1e-5. You can change it using

       -ksp_rtol 1e-9

  Thanks,

     Matt


>
> [image: Disagrees.png]
>
> Thanks!
>
> Wang Weizhuo
>
> On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:
>
>> To reiterate what Matt is saying, you seem to have the exact solution on
>> a 10x10 grid. That makes no sense unless the solution can be represented
>> exactly by your FE space (eg, u(x,y) = x + y).
>>
>> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com> wrote:
>>
>>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> The code is attached in case anyone wants to take a look, I will try
>>>> the high frequency scenario later.
>>>>
>>>
>>> That is not the error. It is superconvergence at the vertices. The real
>>> solution is trigonometric, so your
>>> linear interpolants or whatever you use is not going to get the right
>>> value in between mesh points. You
>>> need to do a real integral over the whole interval to get the L_2 error.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> The first plot is the norm with the flag -pc_type lu with respect to
>>>>>> number of grids in one axis (n), and the second plot is the norm without
>>>>>> the flag -pc_type lu.
>>>>>>
>>>>>
>>>>> So you are using the default PC w/o LU. The default is ILU. This will
>>>>> reduce high frequency effectively but is not effective on the low frequency
>>>>> error. Don't expect your algebraic error reduction to be at the same scale
>>>>> as the residual reduction (what KSP measures).
>>>>>
>>>>>
>>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>>
>>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/edc2f3ea/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/edc2f3ea/attachment-0001.png>

From weizhuo2 at illinois.edu  Tue Oct 16 20:14:15 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Tue, 16 Oct 2018 20:14:15 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
Message-ID: <CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>

I just tried both, neither of them make a difference. I got exactly the
same curve with either combination.

Thanks!

Wang weizhuo

On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
> wrote:
>
>> Hello again!
>>
>> After some tweaking the code is giving right answers now. However it
>> start to disagree with MATLAB results ('traditional' way using matrix
>> inverse) when the grid is larger than 100*100. My PhD advisor and I
>> suspects that the default dimension of the Krylov subspace is 100 in the
>> test case we are running. If so, is there a way to increase the size of the
>> subspace?
>>
>
> 1) The default subspace size is 30, not 100. You can increase the subspace
> size using
>
>        -ksp_gmres_restart n
>
> 2) The problem is likely your tolerance. The default solver tolerance is
> 1e-5. You can change it using
>
>        -ksp_rtol 1e-9
>
>   Thanks,
>
>      Matt
>
>
>>
>> [image: Disagrees.png]
>>
>> Thanks!
>>
>> Wang Weizhuo
>>
>> On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> To reiterate what Matt is saying, you seem to have the exact solution on
>>> a 10x10 grid. That makes no sense unless the solution can be represented
>>> exactly by your FE space (eg, u(x,y) = x + y).
>>>
>>> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>> wrote:
>>>>
>>>>> The code is attached in case anyone wants to take a look, I will try
>>>>> the high frequency scenario later.
>>>>>
>>>>
>>>> That is not the error. It is superconvergence at the vertices. The real
>>>> solution is trigonometric, so your
>>>> linear interpolants or whatever you use is not going to get the right
>>>> value in between mesh points. You
>>>> need to do a real integral over the whole interval to get the L_2 error.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> The first plot is the norm with the flag -pc_type lu with respect to
>>>>>>> number of grids in one axis (n), and the second plot is the norm without
>>>>>>> the flag -pc_type lu.
>>>>>>>
>>>>>>
>>>>>> So you are using the default PC w/o LU. The default is ILU. This will
>>>>>> reduce high frequency effectively but is not effective on the low frequency
>>>>>> error. Don't expect your algebraic error reduction to be at the same scale
>>>>>> as the residual reduction (what KSP measures).
>>>>>>
>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> Wang Weizhuo
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>>>
>>>
>>
>> --
>> Wang Weizhuo
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/47918bc9/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/47918bc9/attachment-0001.png>

From dave.mayhem23 at gmail.com  Tue Oct 16 20:31:11 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Wed, 17 Oct 2018 03:31:11 +0200
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
	<CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
Message-ID: <CAJ98EDrh4VGwLNuZGOfX88HMubRDsNfSz2swHhL9PM0LjKVHrw@mail.gmail.com>

On Wed, 17 Oct 2018 at 03:15, Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> I just tried both, neither of them make a difference. I got exactly the
> same curve with either combination.
>

Try using right preconditioning.

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetPCSide.html


Use the options:

-ksp_type gmres -ksp_pc_side right -ksp_rtol 1e-12

Or;

-ksp_type fgmres  -ksp_rtol 1e-12

Fgmres does right preconditioning my default



> Thanks!
>
> Wang weizhuo
>
> On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Hello again!
>>>
>>> After some tweaking the code is giving right answers now. However it
>>> start to disagree with MATLAB results ('traditional' way using matrix
>>> inverse) when the grid is larger than 100*100. My PhD advisor and I
>>> suspects that the default dimension of the Krylov subspace is 100 in the
>>> test case we are running. If so, is there a way to increase the size of the
>>> subspace?
>>>
>>
>> 1) The default subspace size is 30, not 100. You can increase the
>> subspace size using
>>
>>        -ksp_gmres_restart n
>>
>> 2) The problem is likely your tolerance. The default solver tolerance is
>> 1e-5. You can change it using
>>
>>        -ksp_rtol 1e-9
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> [image: Disagrees.png]
>>>
>>> Thanks!
>>>
>>> Wang Weizhuo
>>>
>>> On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> To reiterate what Matt is saying, you seem to have the exact solution
>>>> on a 10x10 grid. That makes no sense unless the solution can be represented
>>>> exactly by your FE space (eg, u(x,y) = x + y).
>>>>
>>>> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> The code is attached in case anyone wants to take a look, I will try
>>>>>> the high frequency scenario later.
>>>>>>
>>>>>
>>>>> That is not the error. It is superconvergence at the vertices. The
>>>>> real solution is trigonometric, so your
>>>>> linear interpolants or whatever you use is not going to get the right
>>>>> value in between mesh points. You
>>>>> need to do a real integral over the whole interval to get the L_2
>>>>> error.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The first plot is the norm with the flag -pc_type lu with respect
>>>>>>>> to number of grids in one axis (n), and the second plot is the norm without
>>>>>>>> the flag -pc_type lu.
>>>>>>>>
>>>>>>>
>>>>>>> So you are using the default PC w/o LU. The default is ILU. This
>>>>>>> will reduce high frequency effectively but is not effective on the low
>>>>>>> frequency error. Don't expect your algebraic error reduction to be at the
>>>>>>> same scale as the residual reduction (what KSP measures).
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wang Weizhuo
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>>>>
>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Wang Weizhuo
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/0692cc4f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/0692cc4f/attachment-0001.png>

From knepley at gmail.com  Tue Oct 16 20:36:43 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 16 Oct 2018 21:36:43 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
	<CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
Message-ID: <CAMYG4GmRDKcQkRdSsiLnVKqge_A3Ff8mBu1=5-w6o5f1BKYyKA@mail.gmail.com>

On Tue, Oct 16, 2018 at 9:14 PM Weizhuo Wang <weizhuo2 at illinois.edu> wrote:

> I just tried both, neither of them make a difference. I got exactly the
> same curve with either combination.
>

I have a hard time believing you. If you make the residual tolerance much
finer, your error will definitely change.
I run tests every day that do exactly this. You can run them too, since
they are just examples.

  Thanks,

     Matt


> Thanks!
>
> Wang weizhuo
>
> On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>> wrote:
>>
>>> Hello again!
>>>
>>> After some tweaking the code is giving right answers now. However it
>>> start to disagree with MATLAB results ('traditional' way using matrix
>>> inverse) when the grid is larger than 100*100. My PhD advisor and I
>>> suspects that the default dimension of the Krylov subspace is 100 in the
>>> test case we are running. If so, is there a way to increase the size of the
>>> subspace?
>>>
>>
>> 1) The default subspace size is 30, not 100. You can increase the
>> subspace size using
>>
>>        -ksp_gmres_restart n
>>
>> 2) The problem is likely your tolerance. The default solver tolerance is
>> 1e-5. You can change it using
>>
>>        -ksp_rtol 1e-9
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> [image: Disagrees.png]
>>>
>>> Thanks!
>>>
>>> Wang Weizhuo
>>>
>>> On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> To reiterate what Matt is saying, you seem to have the exact solution
>>>> on a 10x10 grid. That makes no sense unless the solution can be represented
>>>> exactly by your FE space (eg, u(x,y) = x + y).
>>>>
>>>> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>> wrote:
>>>>>
>>>>>> The code is attached in case anyone wants to take a look, I will try
>>>>>> the high frequency scenario later.
>>>>>>
>>>>>
>>>>> That is not the error. It is superconvergence at the vertices. The
>>>>> real solution is trigonometric, so your
>>>>> linear interpolants or whatever you use is not going to get the right
>>>>> value in between mesh points. You
>>>>> need to do a real integral over the whole interval to get the L_2
>>>>> error.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> The first plot is the norm with the flag -pc_type lu with respect
>>>>>>>> to number of grids in one axis (n), and the second plot is the norm without
>>>>>>>> the flag -pc_type lu.
>>>>>>>>
>>>>>>>
>>>>>>> So you are using the default PC w/o LU. The default is ILU. This
>>>>>>> will reduce high frequency effectively but is not effective on the low
>>>>>>> frequency error. Don't expect your algebraic error reduction to be at the
>>>>>>> same scale as the residual reduction (what KSP measures).
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>
>>>>>> --
>>>>>> Wang Weizhuo
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>>>>
>>>>
>>>
>>> --
>>> Wang Weizhuo
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Wang Weizhuo
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/b043002f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/b043002f/attachment-0001.png>

From ellen.price at cfa.harvard.edu  Tue Oct 16 20:40:07 2018
From: ellen.price at cfa.harvard.edu (Ellen M. Price)
Date: Tue, 16 Oct 2018 21:40:07 -0400
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAMYG4GmRDKcQkRdSsiLnVKqge_A3Ff8mBu1=5-w6o5f1BKYyKA@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
	<CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
	<CAMYG4GmRDKcQkRdSsiLnVKqge_A3Ff8mBu1=5-w6o5f1BKYyKA@mail.gmail.com>
Message-ID: <c0d4c025-765b-35e0-0fb9-ba71d976156a@cfa.harvard.edu>

Maybe a stupid suggestion, but sometimes I forget to call the
*SetFromOptions function on my object, and then get confused when
changing the options has no effect. Just a thought from a fellow grad
student.

Ellen


On 10/16/2018 09:36 PM, Matthew Knepley wrote:
> On Tue, Oct 16, 2018 at 9:14 PM Weizhuo Wang <weizhuo2 at illinois.edu
> <mailto:weizhuo2 at illinois.edu>> wrote:
> 
>     I just tried both, neither of them make a difference. I got exactly
>     the same curve with either combination.
> 
> 
> I have a hard time believing you. If you make the residual tolerance
> much finer, your error will definitely change.
> I run tests every day that do exactly this. You can run them too, since
> they are just examples.
> 
> ? Thanks,
> 
> ? ? ?Matt
> ?
> 
>     Thanks!
> 
>     Wang weizhuo
> 
>     On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com
>     <mailto:knepley at gmail.com>> wrote:
> 
>         On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang
>         <weizhuo2 at illinois.edu <mailto:weizhuo2 at illinois.edu>> wrote:
> 
>             Hello again!
> 
>             After some tweaking the code is giving right answers now.
>             However it start to disagree with MATLAB results
>             ('traditional' way using matrix inverse) when the grid is
>             larger than 100*100. My PhD advisor and I suspects that the
>             default dimension of the Krylov subspace is 100 in the test
>             case we are running. If so, is there a way to increase the
>             size of the subspace?
> 
> 
>         1) The default subspace size is 30, not 100. You can increase
>         the subspace size using
> 
>         ? ? ? ?-ksp_gmres_restart n
> 
>         2) The problem is likely your tolerance. The default solver
>         tolerance is 1e-5. You can change it using
> 
>         ? ? ? ?-ksp_rtol 1e-9
> 
>         ? Thanks,
> 
>         ? ? ?Matt
>         ?
> 
> 
>             Disagrees.png
> 
>             Thanks!
> 
>             Wang Weizhuo
> 
>             On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov
>             <mailto:mfadams at lbl.gov>> wrote:
> 
>                 To reiterate what Matt is saying, you seem to have the
>                 exact solution on a 10x10 grid. That makes no sense
>                 unless the solution can be represented exactly by your
>                 FE space (eg, u(x,y) = x + y).
> 
>                 On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley
>                 <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
> 
>                     On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang
>                     <weizhuo2 at illinois.edu
>                     <mailto:weizhuo2 at illinois.edu>> wrote:
> 
>                         The code is attached in case anyone wants to
>                         take a look, I will try the high frequency
>                         scenario later.
> 
> 
>                     That is not the error. It is superconvergence at the
>                     vertices. The real solution is trigonometric, so your
>                     linear interpolants or whatever you use is not going
>                     to get the right value in between mesh points. You
>                     need to do a real integral over the whole interval
>                     to get the L_2 error.
> 
>                     ? Thanks,
> 
>                     ? ? ?Matt
>                     ?
> 
>                         On Mon, Oct 8, 2018 at 7:58 PM Mark Adams
>                         <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
> 
> 
> 
>                             On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang
>                             <weizhuo2 at illinois.edu
>                             <mailto:weizhuo2 at illinois.edu>> wrote:
> 
>                                 The first plot is the norm with the flag
>                                 -pc_type lu with respect to number of
>                                 grids in one axis (n), and the second
>                                 plot is the norm without the flag
>                                 -pc_type lu.?
> 
> 
>                             So you are using the default PC w/o LU. The
>                             default is ILU. This will reduce high
>                             frequency effectively but is not effective
>                             on the low frequency error. Don't expect
>                             your algebraic error reduction to be at the
>                             same scale as the residual reduction (what
>                             KSP measures).?
>                             ?
> 
> 
> 
>                         -- 
>                         Wang Weizhuo
> 
> 
> 
>                     -- 
>                     What most experimenters take for granted before they
>                     begin their experiments is infinitely more
>                     interesting than any results to which their
>                     experiments lead.
>                     -- Norbert Wiener
> 
>                     https://www.cse.buffalo.edu/~knepley/
>                     <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
> 
> 
> 
>             -- 
>             Wang Weizhuo
> 
> 
> 
>         -- 
>         What most experimenters take for granted before they begin their
>         experiments is infinitely more interesting than any results to
>         which their experiments lead.
>         -- Norbert Wiener
> 
>         https://www.cse.buffalo.edu/~knepley/
>         <http://www.cse.buffalo.edu/%7Eknepley/>
> 
> 
> 
>     -- 
>     Wang Weizhuo
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/%7Eknepley/>

From fdkong.jd at gmail.com  Tue Oct 16 20:54:52 2018
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 16 Oct 2018 19:54:52 -0600
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <c0d4c025-765b-35e0-0fb9-ba71d976156a@cfa.harvard.edu>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
	<CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
	<CAMYG4GmRDKcQkRdSsiLnVKqge_A3Ff8mBu1=5-w6o5f1BKYyKA@mail.gmail.com>
	<c0d4c025-765b-35e0-0fb9-ba71d976156a@cfa.harvard.edu>
Message-ID: <A39E416D-009B-4F5E-9A58-73057B67860B@gmail.com>

Use -ksp_view to confirm the options are actually set.

Fande 

Sent from my iPhone

> On Oct 16, 2018, at 7:40 PM, Ellen M. Price <ellen.price at cfa.harvard.edu> wrote:
> 
> Maybe a stupid suggestion, but sometimes I forget to call the
> *SetFromOptions function on my object, and then get confused when
> changing the options has no effect. Just a thought from a fellow grad
> student.
> 
> Ellen
> 
> 
>> On 10/16/2018 09:36 PM, Matthew Knepley wrote:
>> On Tue, Oct 16, 2018 at 9:14 PM Weizhuo Wang <weizhuo2 at illinois.edu
>> <mailto:weizhuo2 at illinois.edu>> wrote:
>> 
>>    I just tried both, neither of them make a difference. I got exactly
>>    the same curve with either combination.
>> 
>> 
>> I have a hard time believing you. If you make the residual tolerance
>> much finer, your error will definitely change.
>> I run tests every day that do exactly this. You can run them too, since
>> they are just examples.
>> 
>>   Thanks,
>> 
>>      Matt
>>  
>> 
>>    Thanks!
>> 
>>    Wang weizhuo
>> 
>>    On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com
>>    <mailto:knepley at gmail.com>> wrote:
>> 
>>        On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang
>>        <weizhuo2 at illinois.edu <mailto:weizhuo2 at illinois.edu>> wrote:
>> 
>>            Hello again!
>> 
>>            After some tweaking the code is giving right answers now.
>>            However it start to disagree with MATLAB results
>>            ('traditional' way using matrix inverse) when the grid is
>>            larger than 100*100. My PhD advisor and I suspects that the
>>            default dimension of the Krylov subspace is 100 in the test
>>            case we are running. If so, is there a way to increase the
>>            size of the subspace?
>> 
>> 
>>        1) The default subspace size is 30, not 100. You can increase
>>        the subspace size using
>> 
>>               -ksp_gmres_restart n
>> 
>>        2) The problem is likely your tolerance. The default solver
>>        tolerance is 1e-5. You can change it using
>> 
>>               -ksp_rtol 1e-9
>> 
>>          Thanks,
>> 
>>             Matt
>>         
>> 
>> 
>>            Disagrees.png
>> 
>>            Thanks!
>> 
>>            Wang Weizhuo
>> 
>>            On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov
>>            <mailto:mfadams at lbl.gov>> wrote:
>> 
>>                To reiterate what Matt is saying, you seem to have the
>>                exact solution on a 10x10 grid. That makes no sense
>>                unless the solution can be represented exactly by your
>>                FE space (eg, u(x,y) = x + y).
>> 
>>                On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley
>>                <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>>                    On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang
>>                    <weizhuo2 at illinois.edu
>>                    <mailto:weizhuo2 at illinois.edu>> wrote:
>> 
>>                        The code is attached in case anyone wants to
>>                        take a look, I will try the high frequency
>>                        scenario later.
>> 
>> 
>>                    That is not the error. It is superconvergence at the
>>                    vertices. The real solution is trigonometric, so your
>>                    linear interpolants or whatever you use is not going
>>                    to get the right value in between mesh points. You
>>                    need to do a real integral over the whole interval
>>                    to get the L_2 error.
>> 
>>                      Thanks,
>> 
>>                         Matt
>>                     
>> 
>>                        On Mon, Oct 8, 2018 at 7:58 PM Mark Adams
>>                        <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> 
>> 
>>                            On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang
>>                            <weizhuo2 at illinois.edu
>>                            <mailto:weizhuo2 at illinois.edu>> wrote:
>> 
>>                                The first plot is the norm with the flag
>>                                -pc_type lu with respect to number of
>>                                grids in one axis (n), and the second
>>                                plot is the norm without the flag
>>                                -pc_type lu. 
>> 
>> 
>>                            So you are using the default PC w/o LU. The
>>                            default is ILU. This will reduce high
>>                            frequency effectively but is not effective
>>                            on the low frequency error. Don't expect
>>                            your algebraic error reduction to be at the
>>                            same scale as the residual reduction (what
>>                            KSP measures). 
>>                             
>> 
>> 
>> 
>>                        -- 
>>                        Wang Weizhuo
>> 
>> 
>> 
>>                    -- 
>>                    What most experimenters take for granted before they
>>                    begin their experiments is infinitely more
>>                    interesting than any results to which their
>>                    experiments lead.
>>                    -- Norbert Wiener
>> 
>>                    https://www.cse.buffalo.edu/~knepley/
>>                    <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>> 
>> 
>> 
>>            -- 
>>            Wang Weizhuo
>> 
>> 
>> 
>>        -- 
>>        What most experimenters take for granted before they begin their
>>        experiments is infinitely more interesting than any results to
>>        which their experiments lead.
>>        -- Norbert Wiener
>> 
>>        https://www.cse.buffalo.edu/~knepley/
>>        <http://www.cse.buffalo.edu/%7Eknepley/>
>> 
>> 
>> 
>>    -- 
>>    Wang Weizhuo
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which
>> their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/%7Eknepley/>

From ysjosh.lo at gmail.com  Tue Oct 16 23:27:56 2018
From: ysjosh.lo at gmail.com (Josh L)
Date: Tue, 16 Oct 2018 23:27:56 -0500
Subject: [petsc-users] DMPlex. Apply Dirichlet boundary condition
Message-ID: <CA+2ebbA24-3Hg=g1c7ZPUeVgDBmYHKD9ojqsXQZ7uC-dqPATuw@mail.gmail.com>

Hi,

My problem has an irregular domain and for the nodes I have to apply
boundary condition, I only know their node number in nature ordering.
How do I apply BC(zero out the corresponding entries in global Vec and Mat)
after DMPlexDistribute when it is all in global ordering?

If using DMLabel, how do I add those nodes to the label If I only have
their number in nature ordering?


Thanks,
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181016/5b7885c0/attachment.html>

From knepley at gmail.com  Wed Oct 17 08:48:41 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 17 Oct 2018 09:48:41 -0400
Subject: [petsc-users] DMPlex. Apply Dirichlet boundary condition
In-Reply-To: <CA+2ebbA24-3Hg=g1c7ZPUeVgDBmYHKD9ojqsXQZ7uC-dqPATuw@mail.gmail.com>
References: <CA+2ebbA24-3Hg=g1c7ZPUeVgDBmYHKD9ojqsXQZ7uC-dqPATuw@mail.gmail.com>
Message-ID: <CAMYG4G=0HBmd_R3NGA1Oa0n1DeV_CY3bGbePBdtJ0QEo+C9++Q@mail.gmail.com>

On Wed, Oct 17, 2018 at 12:29 AM Josh L <ysjosh.lo at gmail.com> wrote:

> Hi,
>
> My problem has an irregular domain and for the nodes I have to apply
> boundary condition, I only know their node number in nature ordering.
> How do I apply BC(zero out the corresponding entries in global Vec and
> Mat) after DMPlexDistribute when it is all in global ordering?
>
> If using DMLabel, how do I add those nodes to the label If I only have
> their number in nature ordering?
>

I will help you do whatever you want to do. However, first I would like to
encourage you not to do what you are doing.

<paperclip talking>It looks like you want to add  a boundary condition to
your continuum model?</paperclip talking>

First, you need a way to describe the boundary that is invariant to mesh
transformations, such as redistribution, adaptation, and coordinate
remapping.. For this you can use DMLabel. Those transformations update the
label so that the same topological piece of the mesh is marked after the
transformation.

Second, you should also specify your boundary condition in a way that is
invariant to these transformations. This function


https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMAddBoundary.html

encourages you to write boundary conditions as pointwise functions, which
can easily be projected into a finite element space
along a given DMLabel using DMProjectFunctionLabelLocal(). You can also use
a finite element function instead using DMProjectFieldLabelLocal(). These
functions are used internally by DMPlexInsertBoundaryValues().

Could you use that kind of a scheme?

  Thanks,

     Matt


> Thanks,
> Josh
>
>
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/6d8746b8/attachment-0001.html>

From griesser.jan at googlemail.com  Wed Oct 17 10:53:15 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 17 Oct 2018 17:53:15 +0200
Subject: [petsc-users] PetscInt overflow
Message-ID: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>

Hi all,
i am using slepc4py and petsc4py to solve for the smallest real eigenvalues
and eigenvectors. For my test cases with a matrix A of the size 30k x 30k
solving for the smallest soutions works quite well, but when i increase the
dimension of my system to around A = 768000 x 768000 or 3 million x 3
million and ask for the smallest real 3000 (the number is increasing with
increasing system size) eigenvalues and eigenvectors i get the output (for
the 768000):
 The product 4001 times 768000 overflows the size of PetscInt; consider
reducing the number of columns, or use BVVECS instead
i understand that the requested number of eigenvectors and eigenvalues is
causing an overflow but i do not understand the solution of the problem
which is stated in the error message. Can someone tell me what exactly
BVVECS is and how i can use it? Or is there any other solution to my
problem ?
Thank you very much in advance,
Jan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/9ffca831/attachment.html>

From knepley at gmail.com  Wed Oct 17 10:58:26 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 17 Oct 2018 11:58:26 -0400
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
Message-ID: <CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>

On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com>
wrote:

> Hi all,
> i am using slepc4py and petsc4py to solve for the smallest real
> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
> 30k x 30k solving for the smallest soutions works quite well, but when i
> increase the dimension of my system to around A = 768000 x 768000 or 3
> million x 3 million and ask for the smallest real 3000 (the number is
> increasing with increasing system size) eigenvalues and eigenvectors i get
> the output (for the 768000):
>  The product 4001 times 768000 overflows the size of PetscInt; consider
> reducing the number of columns, or use BVVECS instead
> i understand that the requested number of eigenvectors and eigenvalues is
> causing an overflow but i do not understand the solution of the problem
> which is stated in the error message. Can someone tell me what exactly
> BVVECS is and how i can use it? Or is there any other solution to my
> problem ?
>

You can also reconfigure with 64-bit integers: --with-64-bit-indices

  Thanks,

    Matt


> Thank you very much in advance,
> Jan
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/42441192/attachment.html>

From iancclin at umich.edu  Wed Oct 17 11:05:28 2018
From: iancclin at umich.edu (Ian Lin)
Date: Wed, 17 Oct 2018 12:05:28 -0400
Subject: [petsc-users] Mac OSX 10.14 related MPI compilation issue
Message-ID: <39FFEB45-C2FD-4CEE-9F2F-F6E19218A68F@umich.edu>

Hi PETSc team,

I am using Mac computer as my local machine for testing my code linked with PETSc. Recently I just updated to OSX 10.14 (Mojave), and I found some compilation issue regarding the path of mpi.h. In my system, I used home-brew to install openmpi 3.1.2 with gcc 8.2. In the meantime, I had openmpi 1.10.7 installed and linked with gcc 8.2 and used it to compile PETSc. However it send me an error saying that 
"PETSc was configured with one OpenMPI mpi.h version but now appears to be compiling using a different OpenMPI mpi.h version?
My configure command was ./configure --with-blaslapack-dir=/usr/local/opt/openblas --with-mpi-dir=/Users/iancclin/softwares/openmpi/1.10.7/gcc8 --prefix=/Users/iancclin/softwares/petsc/3.10.2/openmpi1.10.7_gcc8.2.0 I guess this should be simple enough to avoid any mistake. However the problem still persists. Besides, when I tried to compile PETSc with home-brew installed openmpi 3.1.2, it compile without any problem.

In make.log, somehow it was searching mpi.h in /usr/local/include instead of the include directory of my self installed openmpi 1.10.7. I attached my configure.log and make.log for your reference. Did I make any mistake on compiling? Or do you have any suggestion to workaround this?

Thanks a lot,
Ian

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/fdb0f8a2/attachment-0003.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 4195141 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/fdb0f8a2/attachment-0002.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/fdb0f8a2/attachment-0004.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.log
Type: application/octet-stream
Size: 16563 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/fdb0f8a2/attachment-0003.obj>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181017/fdb0f8a2/attachment-0005.html>

From bsmith at mcs.anl.gov  Wed Oct 17 11:23:35 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 17 Oct 2018 16:23:35 +0000
Subject: [petsc-users] Mac OSX 10.14 related MPI compilation issue
In-Reply-To: <39FFEB45-C2FD-4CEE-9F2F-F6E19218A68F@umich.edu>
References: <39FFEB45-C2FD-4CEE-9F2F-F6E19218A68F@umich.edu>
Message-ID: <7D2E0DF8-D842-43B5-ACD7-E9360D42FD34@anl.gov>


  Ian,

     You need to use brew to uninstall the MPI in /usr/local before you use the other install of MPI. There is no easy systematic way to force 
the compiler and linker to find the "other" mpi.h first.


    Barry

  Having multiple MPI's installed on the system is often problematic.



> On Oct 17, 2018, at 11:05 AM, Ian Lin <iancclin at umich.edu> wrote:
> 
> Hi PETSc team,
> 
> I am using Mac computer as my local machine for testing my code linked with PETSc. Recently I just updated to OSX 10.14 (Mojave), and I found some compilation issue regarding the path of mpi.h. In my system, I used home-brew to install openmpi 3.1.2 with gcc 8.2. In the meantime, I had openmpi 1.10.7 installed and linked with gcc 8.2 and used it to compile PETSc. However it send me an error saying that 
> "PETSc was configured with one OpenMPI mpi.h version but now appears to be compiling using a different OpenMPI mpi.h version?
> My configure command was ./configure --with-blaslapack-dir=/usr/local/opt/openblas --with-mpi-dir=/Users/iancclin/softwares/openmpi/1.10.7/gcc8 --prefix=/Users/iancclin/softwares/petsc/3.10.2/openmpi1.10.7_gcc8.2.0 I guess this should be simple enough to avoid any mistake. However the problem still persists. Besides, when I tried to compile PETSc with home-brew installed openmpi 3.1.2, it compile without any problem.
> 
> In make.log, somehow it was searching mpi.h in /usr/local/include instead of the include directory of my self installed openmpi 1.10.7. I attached my configure.log and make.log for your reference. Did I make any mistake on compiling? Or do you have any suggestion to workaround this?
> 
> Thanks a lot,
> Ian
> 
> <configure.log><make.log>


From jroman at dsic.upv.es  Wed Oct 17 11:24:48 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 17 Oct 2018 18:24:48 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
Message-ID: <97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>

To use BVVECS just add the command-line option -bv_type vecs
This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.

Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.

Jose


> El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com> escribi?:
> 
> On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> Hi all,
> i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): 
>  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
> 
> You can also reconfigure with 64-bit integers: --with-64-bit-indices
> 
>   Thanks,
> 
>     Matt
>  
> Thank you very much in advance,
> Jan 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From mailinglists at xgm.de  Thu Oct 18 10:56:57 2018
From: mailinglists at xgm.de (Florian Lindner)
Date: Thu, 18 Oct 2018 17:56:57 +0200
Subject: [petsc-users] KSP and matrix-free matrix (shell)
Message-ID: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>

Hello,

I try to use the KSP solver package together with a shell matrix:


  MyContext mycontext; // an empty struct, not sure it it's needed?
  Mat s;
  ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE, PETSC_DECIDE, &mycontext, &s);
  ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult); CHKERRQ(ierr);

To simulate a meaningfull usermult, I use MatMult on an actual existing matrix of same dimensions:

extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
{
  PetscErrorCode ierr = 0;
  ierr = MatMult(matrix, x, y);
  printf("Call\n");
  return ierr;
}

Btw, what is the significance of the Mat m argument here?

matrix is created like:

  ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
  ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);
  ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
  ierr = MatSetUp(matrix); CHKERRQ(ierr);


  MatMult(s, b, x);

works. The usermult function is called.

But trying to use a KSP gives an error:

  KSP solver;
  KSPCreate(PETSC_COMM_WORLD, &solver);
  KSPSetFromOptions(solver);
  KSPSetOperators(solver, s, s);


error:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3, unknown 
[0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn Thu Oct 18 17:39:52 2018
[0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py --download-mpi4py --with-mpi-dir=/opt/mpich
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/lindnefn/software/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 923 in /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c

Do I need to MatShellSetOperations additional operations? Like MATOP_ILUFACTOR? How can I know what operations to implement?

Best Thanks,
Florian



From hzhang at mcs.anl.gov  Thu Oct 18 11:07:27 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 18 Oct 2018 16:07:27 +0000
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
Message-ID: <CAGCphBsDFcowMe0kgYxWErYUaX-wo=zRzL-vJmSbMD2N2j5R=g@mail.gmail.com>

When you use shell matrix, i.e., matrix-free, you cannot it as a preconditioner. Add option '-pc_type none'.
Hong

On Thu, Oct 18, 2018 at 10:57 AM Florian Lindner <mailinglists at xgm.de<mailto:mailinglists at xgm.de>> wrote:
Hello,

I try to use the KSP solver package together with a shell matrix:


  MyContext mycontext; // an empty struct, not sure it it's needed?
  Mat s;
  ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE, PETSC_DECIDE, &mycontext, &s);
  ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult); CHKERRQ(ierr);

To simulate a meaningfull usermult, I use MatMult on an actual existing matrix of same dimensions:

extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
{
  PetscErrorCode ierr = 0;
  ierr = MatMult(matrix, x, y);
  printf("Call\n");
  return ierr;
}

Btw, what is the significance of the Mat m argument here?

matrix is created like:

  ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
  ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);
  ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
  ierr = MatSetUp(matrix); CHKERRQ(ierr);


  MatMult(s, b, x);

works. The usermult function is called.

But trying to use a KSP gives an error:

  KSP solver;
  KSPCreate(PETSC_COMM_WORLD, &solver);
  KSPSetFromOptions(solver);
  KSPSetOperators(solver, s, s);


error:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
[0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn Thu Oct 18 17:39:52 2018
[0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py --download-mpi4py --with-mpi-dir=/opt/mpich
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/lindnefn/software/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 923 in /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c

Do I need to MatShellSetOperations additional operations? Like MATOP_ILUFACTOR? How can I know what operations to implement?

Best Thanks,
Florian


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181018/2b36eb1e/attachment.html>

From hzhang at mcs.anl.gov  Thu Oct 18 11:07:28 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Thu, 18 Oct 2018 16:07:28 +0000
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
Message-ID: <CAGCphBsDFcowMe0kgYxWErYUaX-wo=zRzL-vJmSbMD2N2j5R=g@mail.gmail.com>

When you use shell matrix, i.e., matrix-free, you cannot it as a preconditioner. Add option '-pc_type none'.
Hong

On Thu, Oct 18, 2018 at 10:57 AM Florian Lindner <mailinglists at xgm.de<mailto:mailinglists at xgm.de>> wrote:
Hello,

I try to use the KSP solver package together with a shell matrix:


  MyContext mycontext; // an empty struct, not sure it it's needed?
  Mat s;
  ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE, PETSC_DECIDE, &mycontext, &s);
  ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult); CHKERRQ(ierr);

To simulate a meaningfull usermult, I use MatMult on an actual existing matrix of same dimensions:

extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
{
  PetscErrorCode ierr = 0;
  ierr = MatMult(matrix, x, y);
  printf("Call\n");
  return ierr;
}

Btw, what is the significance of the Mat m argument here?

matrix is created like:

  ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
  ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);
  ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
  ierr = MatSetUp(matrix); CHKERRQ(ierr);


  MatMult(s, b, x);

works. The usermult function is called.

But trying to use a KSP gives an error:

  KSP solver;
  KSPCreate(PETSC_COMM_WORLD, &solver);
  KSPSetFromOptions(solver);
  KSPSetOperators(solver, s, s);


error:

[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
[0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
[0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn Thu Oct 18 17:39:52 2018
[0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py --download-mpi4py --with-mpi-dir=/opt/mpich
[0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/lindnefn/software/petsc/src/mat/interface/matrix.c
[0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
[0]PETSC ERROR: #3 PCSetUp() line 923 in /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c

Do I need to MatShellSetOperations additional operations? Like MATOP_ILUFACTOR? How can I know what operations to implement?

Best Thanks,
Florian


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181018/2b36eb1e/attachment-0001.html>

From dave.mayhem23 at gmail.com  Thu Oct 18 12:35:40 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Thu, 18 Oct 2018 19:35:40 +0200
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
Message-ID: <CAJ98EDpFg0Q7Gx1fV_Z2VCGuJSFTdcvH_8zpkXULzrPReqW4eQ@mail.gmail.com>

On Thu, 18 Oct 2018 at 17:57, Florian Lindner <mailinglists at xgm.de> wrote:

> Hello,
>
> I try to use the KSP solver package together with a shell matrix:
>
>
>   MyContext mycontext; // an empty struct, not sure it it's needed?
>   Mat s;
>   ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE,
> PETSC_DECIDE, &mycontext, &s);
>   ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult);
> CHKERRQ(ierr);
>
> To simulate a meaningfull usermult, I use MatMult on an actual existing
> matrix of same dimensions:
>
> extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
> {
>   PetscErrorCode ierr = 0;
>   ierr = MatMult(matrix, x, y);
>   printf("Call\n");
>   return ierr;
> }
>
> Btw, what is the significance of the Mat m argument here?


m is your shell matrix. You should be calling

MatShellGetContext(m,(void**)&myctx);

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatShellGetContext.html

to get your user context inside usermult to retrieve any data you need for
the mat-vec product. Where the hell is the variable "matrix"? Is it a
global variable?? If yes - don't do that.


>
> matrix is created like:
>
>   ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
>   ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE);
> CHKERRQ(ierr);
>   ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
>   ierr = MatSetUp(matrix); CHKERRQ(ierr);
>
>
>   MatMult(s, b, x);
>
> works. The usermult function is called.
>
> But trying to use a KSP gives an error:
>
>   KSP solver;
>   KSPCreate(PETSC_COMM_WORLD, &solver);
>   KSPSetFromOptions(solver);
>   KSPSetOperators(solver, s, s);
>
>
> error:
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: See
> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
> possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
> ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
> [0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn Thu
> Oct 18 17:39:52 2018
> [0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3
> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
> FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py
> --download-mpi4py --with-mpi-dir=/opt/mpich
> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in
> /home/lindnefn/software/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
> /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in
> /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 381 in
> /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 612 in
> /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
>
> Do I need to MatShellSetOperations additional operations? Like
> MATOP_ILUFACTOR? How can I know what operations to implement?
>
> Best Thanks,
> Florian
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181018/ac4733d4/attachment.html>

From knepley at gmail.com  Thu Oct 18 17:22:42 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 18 Oct 2018 18:22:42 -0400
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <CAGCphBsDFcowMe0kgYxWErYUaX-wo=zRzL-vJmSbMD2N2j5R=g@mail.gmail.com>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
	<CAGCphBsDFcowMe0kgYxWErYUaX-wo=zRzL-vJmSbMD2N2j5R=g@mail.gmail.com>
Message-ID: <CAMYG4GkTGoU7Vo8w6gqRxpe42go4YGjatECaokBRvowyZ_bgrQ@mail.gmail.com>

On Thu, Oct 18, 2018 at 12:07 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> When you use shell matrix, i.e., matrix-free, you cannot it as a
> preconditioner. Add option '-pc_type none'.
>

There are some you can use, like -pc_type cheby

   MAtt


> Hong
>
> On Thu, Oct 18, 2018 at 10:57 AM Florian Lindner <mailinglists at xgm.de>
> wrote:
>
>> Hello,
>>
>> I try to use the KSP solver package together with a shell matrix:
>>
>>
>>   MyContext mycontext; // an empty struct, not sure it it's needed?
>>   Mat s;
>>   ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE,
>> PETSC_DECIDE, &mycontext, &s);
>>   ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult);
>> CHKERRQ(ierr);
>>
>> To simulate a meaningfull usermult, I use MatMult on an actual existing
>> matrix of same dimensions:
>>
>> extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
>> {
>>   PetscErrorCode ierr = 0;
>>   ierr = MatMult(matrix, x, y);
>>   printf("Call\n");
>>   return ierr;
>> }
>>
>> Btw, what is the significance of the Mat m argument here?
>>
>> matrix is created like:
>>
>>   ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
>>   ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE);
>> CHKERRQ(ierr);
>>   ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
>>   ierr = MatSetUp(matrix); CHKERRQ(ierr);
>>
>>
>>   MatMult(s, b, x);
>>
>> works. The usermult function is called.
>>
>> But trying to use a KSP gives an error:
>>
>>   KSP solver;
>>   KSPCreate(PETSC_COMM_WORLD, &solver);
>>   KSPSetFromOptions(solver);
>>   KSPSetOperators(solver, s, s);
>>
>>
>> error:
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for
>> possible LU and Cholesky solvers
>> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must
>> ./configure with --download-<package>
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.9.3, unknown
>> [0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn
>> Thu Oct 18 17:39:52 2018
>> [0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3
>> -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native"
>> FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py
>> --download-mpi4py --with-mpi-dir=/opt/mpich
>> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in
>> /home/lindnefn/software/petsc/src/mat/interface/matrix.c
>> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in
>> /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
>> [0]PETSC ERROR: #3 PCSetUp() line 923 in
>> /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
>> [0]PETSC ERROR: #4 KSPSetUp() line 381 in
>> /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
>> [0]PETSC ERROR: #5 KSPSolve() line 612 in
>> /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
>>
>> Do I need to MatShellSetOperations additional operations? Like
>> MATOP_ILUFACTOR? How can I know what operations to implement?
>>
>> Best Thanks,
>> Florian
>>
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181018/7d6cc19f/attachment.html>

From jed at jedbrown.org  Thu Oct 18 21:10:30 2018
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 18 Oct 2018 20:10:30 -0600
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <CAMYG4GkTGoU7Vo8w6gqRxpe42go4YGjatECaokBRvowyZ_bgrQ@mail.gmail.com>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
	<CAGCphBsDFcowMe0kgYxWErYUaX-wo=zRzL-vJmSbMD2N2j5R=g@mail.gmail.com>
	<CAMYG4GkTGoU7Vo8w6gqRxpe42go4YGjatECaokBRvowyZ_bgrQ@mail.gmail.com>
Message-ID: <87bm7qsn3d.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, Oct 18, 2018 at 12:07 PM Zhang, Hong <hzhang at mcs.anl.gov> wrote:
>
>> When you use shell matrix, i.e., matrix-free, you cannot it as a
>> preconditioner. Add option '-pc_type none'.
>>
>
> There are some you can use, like -pc_type cheby

That doesn't exist.  You can do the nested iteration that Matt has in
mind with

  -pc_type ksp -ksp_ksp_type chebyshev -ksp_pc_type none

which can be useful if GMRES restarts are killing you, for example.  But
most people using matrix-free are either using geometric multigrid,
defining the diagonal using

  MatShellSetOperation(A, MATOP_GET_DIAGONAL, f)

and/or using PCShell for preconditioning.

From mailinglists at xgm.de  Fri Oct 19 03:39:12 2018
From: mailinglists at xgm.de (Florian Lindner)
Date: Fri, 19 Oct 2018 10:39:12 +0200
Subject: [petsc-users] KSP and matrix-free matrix (shell)
In-Reply-To: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
References: <9c691647-b1a7-02ba-6c0c-d25b049208c1@xgm.de>
Message-ID: <348a13d0-8f92-424f-7a9b-41df637d4a11@xgm.de>

Thanks for your help! Using -pc_type none makes it working so far.

>  Where the hell is the variable "matrix"? Is it a global variable?? If yes - don't do that.

Yes, it's a global matrix and as I said, I just use it simulate a meaningful matrix-vector product in my 30 lines test program.

Best Thanks,
Florian

Am 18.10.18 um 17:56 schrieb Florian Lindner:
> Hello,
> 
> I try to use the KSP solver package together with a shell matrix:
> 
> 
>   MyContext mycontext; // an empty struct, not sure it it's needed?
>   Mat s;
>   ierr = MatCreateShell(PETSC_COMM_WORLD, size, size, PETSC_DECIDE, PETSC_DECIDE, &mycontext, &s);
>   ierr = MatShellSetOperation(s, MATOP_MULT, (void(*)(void))usermult); CHKERRQ(ierr);
> 
> To simulate a meaningfull usermult, I use MatMult on an actual existing matrix of same dimensions:
> 
> extern PetscErrorCode usermult(Mat m ,Vec x, Vec y)
> {
>   PetscErrorCode ierr = 0;
>   ierr = MatMult(matrix, x, y);
>   printf("Call\n");
>   return ierr;
> }
> 
> Btw, what is the significance of the Mat m argument here?
> 
> matrix is created like:
> 
>   ierr = MatCreate(PETSC_COMM_WORLD, &matrix); CHKERRQ(ierr);
>   ierr = MatSetSizes(matrix, size, size, PETSC_DECIDE, PETSC_DECIDE); CHKERRQ(ierr);
>   ierr = MatSetFromOptions(matrix); CHKERRQ(ierr);
>   ierr = MatSetUp(matrix); CHKERRQ(ierr);
> 
> 
>   MatMult(s, b, x);
> 
> works. The usermult function is called.
> 
> But trying to use a KSP gives an error:
> 
>   KSP solver;
>   KSPCreate(PETSC_COMM_WORLD, &solver);
>   KSPSetFromOptions(solver);
>   KSPSetOperators(solver, s, s);
> 
> 
> error:
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/linearsolvertable.html for possible LU and Cholesky solvers
> [0]PETSC ERROR: Could not locate a solver package. Perhaps you must ./configure with --download-<package>
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.9.3, unknown 
> [0]PETSC ERROR: ./a.out on a arch-linux2-c-opt named asaru by lindnefn Thu Oct 18 17:39:52 2018
> [0]PETSC ERROR: Configure options --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --download-petsc4py --download-mpi4py --with-mpi-dir=/opt/mpich
> [0]PETSC ERROR: #1 MatGetFactor() line 4328 in /home/lindnefn/software/petsc/src/mat/interface/matrix.c
> [0]PETSC ERROR: #2 PCSetUp_ILU() line 142 in /home/lindnefn/software/petsc/src/ksp/pc/impls/factor/ilu/ilu.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /home/lindnefn/software/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #4 KSPSetUp() line 381 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #5 KSPSolve() line 612 in /home/lindnefn/software/petsc/src/ksp/ksp/interface/itfunc.c
> 
> Do I need to MatShellSetOperations additional operations? Like MATOP_ILUFACTOR? How can I know what operations to implement?
> 
> Best Thanks,
> Florian
> 
> 

From C.Klaij at marin.nl  Fri Oct 19 03:51:30 2018
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Fri, 19 Oct 2018 08:51:30 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault after
 upgrade from 3.8.4 to 3.10.2
Message-ID: <1539939090363.13492@marin.nl>

I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
surprised to find a number of segmentation faults in my test
cases. These turned out to be related to user-defined MatMult and
PCApply for shell matrices. For example:

SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
  Mat, INTENT(in) :: aa_system
  Vec, INTENT(in) :: xx_system
  Vec, INTENT(out) :: rr_system
  PetscErrorCode, INTENT(out) :: ierr
  ...
END

where aa_system is the shell matrix. This code works fine with
3.8.4 and older, but fails with 3.10.2 due to invalid
pointers (gdb backtrace shows failure of VecSetValues due to
invalid first argument). After replacing by:

SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
  Mat :: aa_system
  Vec :: xx_system
  Vec :: rr_system
  PetscErrorCode :: ierr
  ...
END

everything's fine again. So clearly something has changed since
3.8.4 that now prevents the use of INTENT in Fortran (at least
using intel 17.0.1 compilers). Any reason for this?

Chris


dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm


From griesser.jan at googlemail.com  Fri Oct 19 04:01:05 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Fri, 19 Oct 2018 11:01:05 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
Message-ID: <CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>

With more than 1 MPI process you mean i should use spectrum slicing in
divide the full problem in smaller subproblems?
The --with-64-bit-indices is not a possibility for me since i configured
petsc with mumps, which does not allow to use the 64-bit version (At least
this was the error message when i tried to configure PETSc )

Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es
>:

> To use BVVECS just add the command-line option -bv_type vecs
> This causes to use a separate Vec for each column, instead of a single
> long Vec of size n*m. But it is considerably slower than the default.
>
> Anyway, for such large problems you should consider using more than 1 MPI
> process. In that case the error may disappear because the local size is
> smaller than 768000.
>
> Jose
>
>
> > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
> escribi?:
> >
> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
> griesser.jan at googlemail.com> wrote:
> > Hi all,
> > i am using slepc4py and petsc4py to solve for the smallest real
> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
> 30k x 30k solving for the smallest soutions works quite well, but when i
> increase the dimension of my system to around A = 768000 x 768000 or 3
> million x 3 million and ask for the smallest real 3000 (the number is
> increasing with increasing system size) eigenvalues and eigenvectors i get
> the output (for the 768000):
> >  The product 4001 times 768000 overflows the size of PetscInt; consider
> reducing the number of columns, or use BVVECS instead
> > i understand that the requested number of eigenvectors and eigenvalues
> is causing an overflow but i do not understand the solution of the problem
> which is stated in the error message. Can someone tell me what exactly
> BVVECS is and how i can use it? Or is there any other solution to my
> problem ?
> >
> > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> >
> >   Thanks,
> >
> >     Matt
> >
> > Thank you very much in advance,
> > Jan
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/4326e724/attachment.html>

From jroman at dsic.upv.es  Fri Oct 19 04:07:58 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 19 Oct 2018 11:07:58 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
Message-ID: <9F6535EC-BABB-4CB7-AD26-4B4A6C735E4F@dsic.upv.es>

No, I mean to run in parallel:

$ mpiexec -n 8 python ex1.py 

Jose


> El 19 oct 2018, a las 11:01, Jan Grie?er <griesser.jan at googlemail.com> escribi?:
> 
> With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? 
> The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )
> 
> Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es>:
> To use BVVECS just add the command-line option -bv_type vecs
> This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.
> 
> Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.
> 
> Jose
> 
> 
> > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com> escribi?:
> > 
> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> > Hi all,
> > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): 
> >  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
> > 
> > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > Thank you very much in advance,
> > Jan 
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> 


From griesser.jan at googlemail.com  Fri Oct 19 06:12:00 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Fri, 19 Oct 2018 13:12:00 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyUz7opATqNjc56+L7zVvt4WMnzfJmDs8NhcKS-kgxDjtA@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<9F6535EC-BABB-4CB7-AD26-4B4A6C735E4F@dsic.upv.es>
	<CAB_-EyUz7opATqNjc56+L7zVvt4WMnzfJmDs8NhcKS-kgxDjtA@mail.gmail.com>
Message-ID: <CAB_-EyWpP52F_3ZFj=99rZvMhyNao-GPR-hdz=KQMjS7peobmQ@mail.gmail.com>

Thiis i already did with mpiexec -n 20 ... and there the error occured. I
was also a little bit surprised that this error occured. Our computation
nodes have 20 cores with 6GB RAM.
Is PETSc/ SLEPc saving the dense eigenvector error in one core ?


Am Fr., 19. Okt. 2018 um 12:52 Uhr schrieb Jan Grie?er <
griesser.jan at googlemail.com>:

> Thiis i already did with mpiexec -n 20 ... and there the error occured. I
> was also a little bit surprised that this error occured. Our computation
> nodes have 20 cores with 6GB RAM.
> Is PETSc/ SLEPc saving the dense eigenvector error in one core ?
>
> Am Fr., 19. Okt. 2018 um 11:08 Uhr schrieb Jose E. Roman <
> jroman at dsic.upv.es>:
>
>> No, I mean to run in parallel:
>>
>> $ mpiexec -n 8 python ex1.py
>>
>> Jose
>>
>>
>> > El 19 oct 2018, a las 11:01, Jan Grie?er <griesser.jan at googlemail.com>
>> escribi?:
>> >
>> > With more than 1 MPI process you mean i should use spectrum slicing in
>> divide the full problem in smaller subproblems?
>> > The --with-64-bit-indices is not a possibility for me since i
>> configured petsc with mumps, which does not allow to use the 64-bit version
>> (At least this was the error message when i tried to configure PETSc )
>> >
>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>> jroman at dsic.upv.es>:
>> > To use BVVECS just add the command-line option -bv_type vecs
>> > This causes to use a separate Vec for each column, instead of a single
>> long Vec of size n*m. But it is considerably slower than the default.
>> >
>> > Anyway, for such large problems you should consider using more than 1
>> MPI process. In that case the error may disappear because the local size is
>> smaller than 768000.
>> >
>> > Jose
>> >
>> >
>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>> escribi?:
>> > >
>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>> griesser.jan at googlemail.com> wrote:
>> > > Hi all,
>> > > i am using slepc4py and petsc4py to solve for the smallest real
>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>> 30k x 30k solving for the smallest soutions works quite well, but when i
>> increase the dimension of my system to around A = 768000 x 768000 or 3
>> million x 3 million and ask for the smallest real 3000 (the number is
>> increasing with increasing system size) eigenvalues and eigenvectors i get
>> the output (for the 768000):
>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>> consider reducing the number of columns, or use BVVECS instead
>> > > i understand that the requested number of eigenvectors and
>> eigenvalues is causing an overflow but i do not understand the solution of
>> the problem which is stated in the error message. Can someone tell me what
>> exactly BVVECS is and how i can use it? Or is there any other solution to
>> my problem ?
>> > >
>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>> > >
>> > >   Thanks,
>> > >
>> > >     Matt
>> > >
>> > > Thank you very much in advance,
>> > > Jan
>> > >
>> > >
>> > >
>> > > --
>> > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > -- Norbert Wiener
>> > >
>> > > https://www.cse.buffalo.edu/~knepley/
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/6f60a742/attachment.html>

From jroman at dsic.upv.es  Fri Oct 19 06:18:56 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Fri, 19 Oct 2018 13:18:56 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyWpP52F_3ZFj=99rZvMhyNao-GPR-hdz=KQMjS7peobmQ@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<9F6535EC-BABB-4CB7-AD26-4B4A6C735E4F@dsic.upv.es>
	<CAB_-EyUz7opATqNjc56+L7zVvt4WMnzfJmDs8NhcKS-kgxDjtA@mail.gmail.com>
	<CAB_-EyWpP52F_3ZFj=99rZvMhyNao-GPR-hdz=KQMjS7peobmQ@mail.gmail.com>
Message-ID: <6FE29125-347F-4FBB-814F-454667F044BB@dsic.upv.es>

Sorry, running in parallel does not change the thing. I was wrong, the limitation is for the global size and not the local size. So what you have to do is use -bv_type vecs or also -bv_type mat
Let me know how this works.

Jose


> El 19 oct 2018, a las 13:12, Jan Grie?er <griesser.jan at googlemail.com> escribi?:
> 
> Thiis i already did with mpiexec -n 20 ... and there the error occured. I was also a little bit surprised that this error occured. Our computation nodes have 20 cores with 6GB RAM. 
> Is PETSc/ SLEPc saving the dense eigenvector error in one core ? 
> 
> 
> Am Fr., 19. Okt. 2018 um 12:52 Uhr schrieb Jan Grie?er <griesser.jan at googlemail.com>:
> Thiis i already did with mpiexec -n 20 ... and there the error occured. I was also a little bit surprised that this error occured. Our computation nodes have 20 cores with 6GB RAM. 
> Is PETSc/ SLEPc saving the dense eigenvector error in one core ? 
> 
> Am Fr., 19. Okt. 2018 um 11:08 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es>:
> No, I mean to run in parallel:
> 
> $ mpiexec -n 8 python ex1.py 
> 
> Jose
> 
> 
> > El 19 oct 2018, a las 11:01, Jan Grie?er <griesser.jan at googlemail.com> escribi?:
> > 
> > With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? 
> > The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )
> > 
> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es>:
> > To use BVVECS just add the command-line option -bv_type vecs
> > This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.
> > 
> > Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.
> > 
> > Jose
> > 
> > 
> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com> escribi?:
> > > 
> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> > > Hi all,
> > > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): 
> > >  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> > > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
> > > 
> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> > > 
> > >   Thanks,
> > > 
> > >     Matt
> > >  
> > > Thank you very much in advance,
> > > Jan 
> > > 
> > > 
> > > 
> > > -- 
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > > 
> > > https://www.cse.buffalo.edu/~knepley/
> > 
> 


From bsmith at mcs.anl.gov  Fri Oct 19 07:32:21 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 19 Oct 2018 12:32:21 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <1539939090363.13492@marin.nl>
References: <1539939090363.13492@marin.nl>
Message-ID: <BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>


   Hmm, the intent of all three first arguments should be in since they are passed in from the routine above. Does it work if you replace

>  Vec, INTENT(out) :: rr_system

with

>  Vec, INTENT(in) :: rr_system

?

    Barry


> On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> 
> I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
> surprised to find a number of segmentation faults in my test
> cases. These turned out to be related to user-defined MatMult and
> PCApply for shell matrices. For example:
> 
> SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
>  Mat, INTENT(in) :: aa_system
>  Vec, INTENT(in) :: xx_system
>  Vec, INTENT(out) :: rr_system
>  PetscErrorCode, INTENT(out) :: ierr
>  ...
> END
> 
> where aa_system is the shell matrix. This code works fine with
> 3.8.4 and older, but fails with 3.10.2 due to invalid
> pointers (gdb backtrace shows failure of VecSetValues due to
> invalid first argument). After replacing by:
> 
> SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
>  Mat :: aa_system
>  Vec :: xx_system
>  Vec :: rr_system
>  PetscErrorCode :: ierr
>  ...
> END
> 
> everything's fine again. So clearly something has changed since
> 3.8.4 that now prevents the use of INTENT in Fortran (at least
> using intel 17.0.1 compilers). Any reason for this?
> 
> Chris
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
> 
> MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
> 


From knepley at gmail.com  Fri Oct 19 07:44:00 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 19 Oct 2018 08:44:00 -0400
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
Message-ID: <CAMYG4GkuPPu274j262ff+uh2f+C=uyiJycuTu4wP76THg8myyw@mail.gmail.com>

On Fri, Oct 19, 2018 at 5:01 AM Jan Grie?er <griesser.jan at googlemail.com>
wrote:

> With more than 1 MPI process you mean i should use spectrum slicing in
> divide the full problem in smaller subproblems?
> The --with-64-bit-indices is not a possibility for me since i configured
> petsc with mumps, which does not allow to use the 64-bit version (At least
> this was the error message when i tried to configure PETSc )
>

I believe you can replace MUMPS with SuperLU_dist for 64-bit ints.

   Matt


> Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
> jroman at dsic.upv.es>:
>
>> To use BVVECS just add the command-line option -bv_type vecs
>> This causes to use a separate Vec for each column, instead of a single
>> long Vec of size n*m. But it is considerably slower than the default.
>>
>> Anyway, for such large problems you should consider using more than 1 MPI
>> process. In that case the error may disappear because the local size is
>> smaller than 768000.
>>
>> Jose
>>
>>
>> > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>> escribi?:
>> >
>> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>> griesser.jan at googlemail.com> wrote:
>> > Hi all,
>> > i am using slepc4py and petsc4py to solve for the smallest real
>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>> 30k x 30k solving for the smallest soutions works quite well, but when i
>> increase the dimension of my system to around A = 768000 x 768000 or 3
>> million x 3 million and ask for the smallest real 3000 (the number is
>> increasing with increasing system size) eigenvalues and eigenvectors i get
>> the output (for the 768000):
>> >  The product 4001 times 768000 overflows the size of PetscInt; consider
>> reducing the number of columns, or use BVVECS instead
>> > i understand that the requested number of eigenvectors and eigenvalues
>> is causing an overflow but i do not understand the solution of the problem
>> which is stated in the error message. Can someone tell me what exactly
>> BVVECS is and how i can use it? Or is there any other solution to my
>> problem ?
>> >
>> > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>> >
>> >   Thanks,
>> >
>> >     Matt
>> >
>> > Thank you very much in advance,
>> > Jan
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/b04227d2/attachment.html>

From jczhang at mcs.anl.gov  Fri Oct 19 07:56:26 2018
From: jczhang at mcs.anl.gov (Zhang, Junchao)
Date: Fri, 19 Oct 2018 12:56:26 +0000
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
Message-ID: <CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>


On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <griesser.jan at googlemail.com<mailto:griesser.jan at googlemail.com>> wrote:
With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems?
The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )

MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit integer feature" and "full 64-bit integer version" as well.

Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es<mailto:jroman at dsic.upv.es>>:
To use BVVECS just add the command-line option -bv_type vecs
This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.

Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.

Jose


> El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> escribi?:
>
> On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com<mailto:griesser.jan at googlemail.com>> wrote:
> Hi all,
> i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000):
>  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
>
> You can also reconfigure with 64-bit integers: --with-64-bit-indices
>
>   Thanks,
>
>     Matt
>
> Thank you very much in advance,
> Jan
>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/ecc5ee5c/attachment.html>

From C.Klaij at marin.nl  Fri Oct 19 09:37:58 2018
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Fri, 19 Oct 2018 14:37:58 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
References: <1539939090363.13492@marin.nl>,
	<BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
Message-ID: <1539959878384.42597@marin.nl>

As far as I (mis)understand fortran, this is a data protection
thing: all arguments are passed in from above but the subroutine
is only allowed to change rr and ierr, not aa and xx (if you try,
you get a compiler warning). That's why I find it very odd to
give an intent(in) to rr. But I've tried your suggestion anyway:
both intent(in) and intent(inout) for rr do work! Can't say I
understand though.

Below's a small example of what I was expecting. Change rr to
intent(in) and the compiler complains.

Chris

$ cat intent.f90
program intent

  implicit none

  real, allocatable :: aa(:), xx(:), rr(:)
  integer :: ierr

  allocate(aa(10),xx(10),rr(10))

  aa = 1.0
  xx = 2.0

  call matmult(aa,xx,rr,ierr)

  print *, rr(1)
  print *, ierr

  deallocate(aa,xx,rr)

  contains

    subroutine matmult(aa,xx,rr,ierr)
      real, intent(in) :: aa(:), xx(:)
      real, intent(out):: rr(:)
      integer, intent(out) :: ierr
      rr=aa*xx
      ierr=0
    end subroutine matmult

end program intent
$ ./a.out
   2.000000
           0




dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm

________________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Friday, October 19, 2018 2:32 PM
To: Klaij, Christiaan
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2

   Hmm, the intent of all three first arguments should be in since they are passed in from the routine above. Does it work if you replace

>  Vec, INTENT(out) :: rr_system

with

>  Vec, INTENT(in) :: rr_system

?

    Barry


> On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>
> I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
> surprised to find a number of segmentation faults in my test
> cases. These turned out to be related to user-defined MatMult and
> PCApply for shell matrices. For example:
>
> SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
>  Mat, INTENT(in) :: aa_system
>  Vec, INTENT(in) :: xx_system
>  Vec, INTENT(out) :: rr_system
>  PetscErrorCode, INTENT(out) :: ierr
>  ...
> END
>
> where aa_system is the shell matrix. This code works fine with
> 3.8.4 and older, but fails with 3.10.2 due to invalid
> pointers (gdb backtrace shows failure of VecSetValues due to
> invalid first argument). After replacing by:
>
> SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
>  Mat :: aa_system
>  Vec :: xx_system
>  Vec :: rr_system
>  PetscErrorCode :: ierr
>  ...
> END
>
> everything's fine again. So clearly something has changed since
> 3.8.4 that now prevents the use of INTENT in Fortran (at least
> using intel 17.0.1 compilers). Any reason for this?
>
> Chris
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>
> MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
>


From knepley at gmail.com  Fri Oct 19 13:09:20 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 19 Oct 2018 14:09:20 -0400
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <1539959878384.42597@marin.nl>
References: <1539939090363.13492@marin.nl>
	<BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
	<1539959878384.42597@marin.nl>
Message-ID: <CAMYG4G=7WgObZQwhXSSpetiZ9rPTXZv2iDMkSBZFCfGnZ3u3dA@mail.gmail.com>

On Fri, Oct 19, 2018 at 10:38 AM Klaij, Christiaan <C.Klaij at marin.nl> wrote:

> As far as I (mis)understand fortran, this is a data protection
> thing: all arguments are passed in from above but the subroutine
> is only allowed to change rr and ierr, not aa and xx (if you try,
> you get a compiler warning). That's why I find it very odd to
> give an intent(in) to rr.


rr is not changed, and it cannot be. You pass in a pointer to the object
and we
fill up the object with values. We cannot change that pointer.

   Matt


> But I've tried your suggestion anyway:
> both intent(in) and intent(inout) for rr do work! Can't say I
> understand though.
>
> Below's a small example of what I was expecting. Change rr to
> intent(in) and the compiler complains.
>
> Chris
>
> $ cat intent.f90
> program intent
>
>   implicit none
>
>   real, allocatable :: aa(:), xx(:), rr(:)
>   integer :: ierr
>
>   allocate(aa(10),xx(10),rr(10))
>
>   aa = 1.0
>   xx = 2.0
>
>   call matmult(aa,xx,rr,ierr)
>
>   print *, rr(1)
>   print *, ierr
>
>   deallocate(aa,xx,rr)
>
>   contains
>
>     subroutine matmult(aa,xx,rr,ierr)
>       real, intent(in) :: aa(:), xx(:)
>       real, intent(out):: rr(:)
>       integer, intent(out) :: ierr
>       rr=aa*xx
>       ierr=0
>     end subroutine matmult
>
> end program intent
> $ ./a.out
>    2.000000
>            0
>
>
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>
> MARIN news:
> http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm
>
> ________________________________________
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Friday, October 19, 2018 2:32 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault
> after upgrade from 3.8.4 to 3.10.2
>
>    Hmm, the intent of all three first arguments should be in since they
> are passed in from the routine above. Does it work if you replace
>
> >  Vec, INTENT(out) :: rr_system
>
> with
>
> >  Vec, INTENT(in) :: rr_system
>
> ?
>
>     Barry
>
>
> > On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> >
> > I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
> > surprised to find a number of segmentation faults in my test
> > cases. These turned out to be related to user-defined MatMult and
> > PCApply for shell matrices. For example:
> >
> > SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
> >  Mat, INTENT(in) :: aa_system
> >  Vec, INTENT(in) :: xx_system
> >  Vec, INTENT(out) :: rr_system
> >  PetscErrorCode, INTENT(out) :: ierr
> >  ...
> > END
> >
> > where aa_system is the shell matrix. This code works fine with
> > 3.8.4 and older, but fails with 3.10.2 due to invalid
> > pointers (gdb backtrace shows failure of VecSetValues due to
> > invalid first argument). After replacing by:
> >
> > SUBROUTINE
> mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
> >  Mat :: aa_system
> >  Vec :: xx_system
> >  Vec :: rr_system
> >  PetscErrorCode :: ierr
> >  ...
> > END
> >
> > everything's fine again. So clearly something has changed since
> > 3.8.4 that now prevents the use of INTENT in Fortran (at least
> > using intel 17.0.1 compilers). Any reason for this?
> >
> > Chris
> >
> >
> > dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> > MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl |
> http://www.marin.nl
> >
> > MARIN news:
> http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
> >
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/d7370806/attachment.html>

From bsmith at mcs.anl.gov  Fri Oct 19 14:00:20 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 19 Oct 2018 19:00:20 +0000
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
Message-ID: <C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>



> On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
> 
> 
> On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? 
> The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )
>  
> MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit integer feature" and "full 64-bit integer version" as well. 

    They use to achieve this by compiling with special Fortran flags to promote integers to 64 bit; this is too fragile for our taste so we never hooked PETSc up wit it. If they have a dependable way of using 64 bit integers we should add that to our mumps.py and test it.

   Barry

> 
> Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es>:
> To use BVVECS just add the command-line option -bv_type vecs
> This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.
> 
> Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.
> 
> Jose
> 
> 
> > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com> escribi?:
> > 
> > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> > Hi all,
> > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): 
> >  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
> > 
> > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> > 
> >   Thanks,
> > 
> >     Matt
> >  
> > Thank you very much in advance,
> > Jan 
> > 
> > 
> > 
> > -- 
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> > 
> > https://www.cse.buffalo.edu/~knepley/
> 


From mcygorek at uottawa.ca  Fri Oct 19 14:08:26 2018
From: mcygorek at uottawa.ca (Moritz Cygorek)
Date: Fri, 19 Oct 2018 19:08:26 +0000
Subject: [petsc-users] random SLEPc segfault using openmpi-3.0.1
Message-ID: <b5427a1316754a64bee9ff1f6cbe4207@uottawa.ca>

Hi,


I'm using SLEPc to diagonalize a huge sparse matrix and I've encountered random segmentation faults.



I'm actually using a the slepc example 4 without modifications to rule out errors due to coding.

Concretely, I use the command line


ompirun -n 28 ex4 \
-file amatrix.bin -eps_tol 1e-6 -eps_target 0 -eps_nev 18 \
-eps_harmonic -eps_ncv 40 -eps_max_it 100000 \
-eps_monitor -eps_view  -eps_view_values -eps_view_vectors 2>&1 |tee -a $LOGFILE




The program runs for some time (about half a day) and then stops with the error message



[13]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range

There is definitely enough memory, because I'm using less than 4% of the available 128GB.




Since everything worked fine on a slower computer with a different setup and from previous mailing list comments, I have the feeling that this might be due to some issues with MPI.


Unfortunately, I have to share the computer with other people and can not uninstall the current MPI implementation and I've also heard that there are issues if you install more than one MPI implementation.


For your information: I've configured PETSc with


./configure  --with-mpi-dir=/home/applications/builds/intel_2018/openmpi-3.0.1/ --with-scalar-type=complex --download-mumps --download-scalapack --with-blas-lapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl





I wanted to ask a few things:


- Is there a known issue with openmpi causing random segmentation faults?


- I've also tried to install everything needed by configuring PETSc with

./configure \
--with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-scalar-type=complex \
--download-mumps --download-scalapack --download-mpich --download-fblaslapack

Here, the problem is that performing the checks after "make" stops after the check with 1 MPI process, i.e., the check using 2 MPI just never finishes.
Is that a known issue of conflict between the downloaded mpich and the installed openmpi?
Do you know a way to install mpich without conflicts with openmpi without actually removing openmpi?


- Some time ago a posted a question in the mailing list about how to compile SLEPc/PETSc with OpenMP only instead of MPI. After some time, I was able to get MPI to work on a different computer,
but I was never really able to use OpenMP with slepc, but it would be very useful in the present situation. The programs compile but they never take more than 100% CPU load as displayed by top.
The answers to my question contained the recommendations that I should configure with --download-openblas and have the OMP_NUM_THREADS variable set when executing the program. I did it, but it didn't help either.
So, my question: has someone ever managed to find a configure line that disables MPI but enables the usage of OpenMP so that the slepc ex4 program uses significantly more than 100% CPU usage when executing the standard Krylov-Schur method?





Regards,

Moritz

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/bf898ec7/attachment.html>

From knepley at gmail.com  Fri Oct 19 14:35:05 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 19 Oct 2018 15:35:05 -0400
Subject: [petsc-users] random SLEPc segfault using openmpi-3.0.1
In-Reply-To: <b5427a1316754a64bee9ff1f6cbe4207@uottawa.ca>
References: <b5427a1316754a64bee9ff1f6cbe4207@uottawa.ca>
Message-ID: <CAMYG4GmJ8=nghs-h6U9GSDaYvfZFvQP1rsREkXeiiP1Fg52tAw@mail.gmail.com>

On Fri, Oct 19, 2018 at 3:09 PM Moritz Cygorek <mcygorek at uottawa.ca> wrote:

> Hi,
>
> I'm using SLEPc to diagonalize a huge sparse matrix and I've encountered
> random segmentation faults.
>
> I'm actually using a the slepc example 4 without modifications to rule out
> errors due to coding.
>
> Concretely, I use the command line
>
>
> ompirun -n 28 ex4 \
> -file amatrix.bin -eps_tol 1e-6 -eps_target 0 -eps_nev 18 \
> -eps_harmonic -eps_ncv 40 -eps_max_it 100000 \
> -eps_monitor -eps_view  -eps_view_values -eps_view_vectors 2>&1 |tee -a
> $LOGFILE
>
>
>
> The program runs for some time (about half a day) and then stops with the
> error message
>
>
>
> [13]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
>
> There is definitely enough memory, because I'm using less than 4% of the
> available 128GB.
>
>
>
>
> Since everything worked fine on a slower computer with a different setup
> and from previous mailing list comments, I have the feeling that this might
> be due to some issues with MPI.
>
>
> Unfortunately, I have to share the computer with other people and can not
> uninstall the current MPI implementation and I've also heard that there are
> issues if you install more than one MPI implementation.
>
>
> For your information: I've configured PETSc with
>
>
> ./configure
> --with-mpi-dir=/home/applications/builds/intel_2018/openmpi-3.0.1/
> --with-scalar-type=complex --download-mumps --download-scalapack
> --with-blas-lapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl
>
> I wanted to ask a few things:
>
> - Is there a known issue with openmpi causing random segmentation faults?
>
OpenMPI certainly has had bugs, but this is not a constrained enough
question to pin the fault on any one of those.

> - I've also tried to install everything needed by configuring PETSc with
> ./configure \
> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-scalar-type=complex
> \
> --download-mumps --download-scalapack --download-mpich
> --download-fblaslapack
>
> Here, the problem is that performing the checks after "make" stops after
> the check with 1 MPI process, i.e., the check using 2 MPI just never
> finishes.
> Is that a known issue of conflict between the downloaded mpich and the
> installed openmpi?
>

No, it likely has to do with the network configuration, that is mpiexec is
waiting for gethostbyname() for your machine, which is failing.


> Do you know a way to install mpich without conflicts with openmpi without
> actually removing openmpi?
>

The above can work as long as OpenMPI is not in default compiler paths like
/usr/lib.


> - Some time ago a posted a question in the mailing list about how to
> compile SLEPc/PETSc with OpenMP only instead of MPI. After some time, I was
> able to get MPI to work on a different computer,
> but I was never really able to use OpenMP with slepc, but it would be very
> useful in the present situation. The
>

Why do you think so?


> programs compile but they never take more than 100% CPU load as displayed
> by top.
>

That is perfectly understandable since the memory bandwidth can be maxed
out with fewer cores than are present. OpenMP will not help this.


> The answers to my question contained the recommendations that I should
> configure with --download-openblas and have the OMP_NUM_THREADS variable
> set when executing the program. I did it, but it didn't help either.
>

Yep.


> So, my question: has someone ever managed to find a configure line that
> disables MPI but enables the usage of OpenMP so that the slepc ex4 program
> uses significantly more than 100% CPU usage when executing the standard
> Krylov-Schur method?
>

As I said, this is likely to be impossible for architecture reasons:
https://www.mcs.anl.gov/petsc/documentation/faq.html#computers

  Thanks,

     Matt

> Regards,
>
> Moritz
>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181019/0085ce6d/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri Oct 19 15:26:57 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 19 Oct 2018 20:26:57 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <1539959878384.42597@marin.nl>
References: <1539939090363.13492@marin.nl>
	<BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
	<1539959878384.42597@marin.nl>
Message-ID: <380804AB-7723-4539-9722-9B2D689D6A05@mcs.anl.gov>



> On Oct 19, 2018, at 9:37 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> 
> As far as I (mis)understand fortran, this is a data protection
> thing: all arguments are passed in from above but the subroutine
> is only allowed to change rr and ierr, not aa and xx (if you try,
> you get a compiler warning).

      The routine is not allowed to change rr (it is only allowed to change the values "inside" rr) that is why it needs to be intent in or inout. Otherwise the compiler can optimize and not pass down the value of the rr pointer to the subroutine since by declaring as it as out the compiler thinks your subroutine is going to set is value. 

    Barry


> That's why I find it very odd to
> give an intent(in) to rr. But I've tried your suggestion anyway:
> both intent(in) and intent(inout) for rr do work! Can't say I
> understand though.
> 
> Below's a small example of what I was expecting. Change rr to
> intent(in) and the compiler complains.
> 
> Chris
> 
> $ cat intent.f90
> program intent
> 
>  implicit none
> 
>  real, allocatable :: aa(:), xx(:), rr(:)
>  integer :: ierr
> 
>  allocate(aa(10),xx(10),rr(10))
> 
>  aa = 1.0
>  xx = 2.0
> 
>  call matmult(aa,xx,rr,ierr)
> 
>  print *, rr(1)
>  print *, ierr
> 
>  deallocate(aa,xx,rr)
> 
>  contains
> 
>    subroutine matmult(aa,xx,rr,ierr)
>      real, intent(in) :: aa(:), xx(:)
>      real, intent(out):: rr(:)
>      integer, intent(out) :: ierr
>      rr=aa*xx
>      ierr=0
>    end subroutine matmult
> 
> end program intent
> $ ./a.out
>   2.000000
>           0
> 
> 
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
> 
> MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm
> 
> ________________________________________
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Friday, October 19, 2018 2:32 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2
> 
>   Hmm, the intent of all three first arguments should be in since they are passed in from the routine above. Does it work if you replace
> 
>> Vec, INTENT(out) :: rr_system
> 
> with
> 
>> Vec, INTENT(in) :: rr_system
> 
> ?
> 
>    Barry
> 
> 
>> On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>> 
>> I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
>> surprised to find a number of segmentation faults in my test
>> cases. These turned out to be related to user-defined MatMult and
>> PCApply for shell matrices. For example:
>> 
>> SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
>> Mat, INTENT(in) :: aa_system
>> Vec, INTENT(in) :: xx_system
>> Vec, INTENT(out) :: rr_system
>> PetscErrorCode, INTENT(out) :: ierr
>> ...
>> END
>> 
>> where aa_system is the shell matrix. This code works fine with
>> 3.8.4 and older, but fails with 3.10.2 due to invalid
>> pointers (gdb backtrace shows failure of VecSetValues due to
>> invalid first argument). After replacing by:
>> 
>> SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
>> Mat :: aa_system
>> Vec :: xx_system
>> Vec :: rr_system
>> PetscErrorCode :: ierr
>> ...
>> END
>> 
>> everything's fine again. So clearly something has changed since
>> 3.8.4 that now prevents the use of INTENT in Fortran (at least
>> using intel 17.0.1 compilers). Any reason for this?
>> 
>> Chris
>> 
>> 
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>> 
>> MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
>> 
> 


From bsmith at mcs.anl.gov  Fri Oct 19 16:21:44 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Fri, 19 Oct 2018 21:21:44 +0000
Subject: [petsc-users] random SLEPc segfault using openmpi-3.0.1
In-Reply-To: <b5427a1316754a64bee9ff1f6cbe4207@uottawa.ca>
References: <b5427a1316754a64bee9ff1f6cbe4207@uottawa.ca>
Message-ID: <6E2E99DF-1525-4F99-A787-89E13151A86B@anl.gov>



> On Oct 19, 2018, at 2:08 PM, Moritz Cygorek <mcygorek at uottawa.ca> wrote:
> 
> Hi,
> 
> I'm using SLEPc to diagonalize a huge sparse matrix and I've encountered random segmentation faults. 

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind


> 
> 
> I'm actually using a the slepc example 4 without modifications to rule out errors due to coding.
> Concretely, I use the command line 
> 
> ompirun -n 28 ex4 \
> -file amatrix.bin -eps_tol 1e-6 -eps_target 0 -eps_nev 18 \
> -eps_harmonic -eps_ncv 40 -eps_max_it 100000 \
> -eps_monitor -eps_view  -eps_view_values -eps_view_vectors 2>&1 |tee -a $LOGFILE
> 
> 
> 
> The program runs for some time (about half a day) and then stops with the error message
> 
> 
> [13]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> 
> There is definitely enough memory, because I'm using less than 4% of the available 128GB.
> 
> 
> 
> Since everything worked fine on a slower computer with a different setup and from previous mailing list comments, I have the feeling that this might be due to some issues with MPI.
> 
> Unfortunately, I have to share the computer with other people and can not uninstall the current MPI implementation and I've also heard that there are issues if you install more than one MPI implementation. 
> 
> For your information: I've configured PETSc with
> 
> ./configure  --with-mpi-dir=/home/applications/builds/intel_2018/openmpi-3.0.1/ --with-scalar-type=complex --download-mumps --download-scalapack --with-blas-lapack-dir=/opt/intel/compilers_and_libraries_2018.2.199/linux/mkl
> 
> 
> 
> 
> I wanted to ask a few things:
> 
> - Is there a known issue with openmpi causing random segmentation faults?
> 
> - I've also tried to install everything needed by configuring PETSc with
> 
> ./configure \
> --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --with-scalar-type=complex \
> --download-mumps --download-scalapack --download-mpich --download-fblaslapack
> 
> Here, the problem is that performing the checks after "make" stops after the check with 1 MPI process, i.e., the check using 2 MPI just never finishes. 
> Is that a known issue of conflict between the downloaded mpich and the installed openmpi?
> Do you know a way to install mpich without conflicts with openmpi without actually removing openmpi?
> 
> 
> - Some time ago a posted a question in the mailing list about how to compile SLEPc/PETSc with OpenMP only instead of MPI. After some time, I was able to get MPI to work on a different computer, 
> but I was never really able to use OpenMP with slepc, but it would be very useful in the present situation. The programs compile but they never take more than 100% CPU load as displayed by top.
> The answers to my question contained the recommendations that I should configure with --download-openblas and have the OMP_NUM_THREADS variable set when executing the program. I did it, but it didn't help either.
> So, my question: has someone ever managed to find a configure line that disables MPI but enables the usage of OpenMP so that the slepc ex4 program uses significantly more than 100% CPU usage when executing the standard Krylov-Schur method?
>  
> 
> 
> 
> Regards,
> Moritz


From caleb at janmager.keurslager.nl  Sat Oct 20 13:02:41 2018
From: caleb at janmager.keurslager.nl (Sherry)
Date: Sat, 20 Oct 2018 11:02:41 -0700 (UTC)
Subject: [petsc-users] Realize your dormant temper. Viagra Soft.
Message-ID: <79749962536.7n3U.86714@sender.gy6.janmager.keurslager.nl>

A non-text attachment was scrubbed...
Name: not available
Type: text/html:332933
Size: 262784 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181020/6f025bfd/attachment-0001.bin>

From mfadams at lbl.gov  Sat Oct 20 15:52:54 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 20 Oct 2018 16:52:54 -0400
Subject: [petsc-users] p4est error
Message-ID: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>

I am getting this error. This was a clean build (nuked the arch directory).

13:50 nid12922 master= ~/petsc_install/petsc$ make
PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
PETSC_ARCH="" test
Running test examples to verify correct installation
Using
PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
and PETSC_ARCH=
*******************Error detected during compile or link!*******************
See http://www.mcs.anl.gov/petsc/documentation/faq.html
/global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials ex19
*********************************************************************************
cc -o ex19.o -c -g -fp-model fast
 -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
  `pwd`/ex19.c
cc -g -fp-model fast  -o ex19 ex19.o
-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
-Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
-lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
-lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
-lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis -lmetis
-lstdc++ -ldl
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
In function `PetscDLOpen':
/global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108: warning:
Using 'dlopen' in statically linked applications requires at runtime the
shared libraries from the glibc version used for linking
/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
Warning: alignment 8 of symbol `ARCHITECTURE' in
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
is smaller than 32 in
/opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
In function `PetscOpenSocket':
/global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
warning: Using 'gethostbyname' in statically linked applications requires
at runtime the shared libraries from the glibc version used for linking
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
In function `p4est_checksum':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
undefined reference to `adler32'
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
undefined reference to `adler32_combine'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
In function `p4est_comm_checksum':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
undefined reference to `adler32_combine'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
In function `p8est_checksum':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
undefined reference to `adler32'
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
undefined reference to `adler32_combine'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
In function `p8est_comm_checksum':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
undefined reference to `adler32_combine'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
In function `sc_array_checksum':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
undefined reference to `adler32'
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
undefined reference to `adler32'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
In function `sc_vtk_write_compressed':
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
undefined reference to `compress2'
/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
undefined reference to `compress2'
/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
recipe for target 'ex19' failed
gmake[3]: [ex19] Error 1 (ignored)
true ex19
rm ex19.o
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181020/d8c43d0f/attachment.html>

From tisaac at cc.gatech.edu  Sat Oct 20 16:14:03 2018
From: tisaac at cc.gatech.edu (Isaac, Tobin G)
Date: Sat, 20 Oct 2018 21:14:03 +0000
Subject: [petsc-users] p4est error
In-Reply-To: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
Message-ID: <D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>

I think adler32 is in zlib: p4est typically finds it in configure.  Can you send along config.log from externalpackages/git.p4est?

On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov> wrote:
>I am getting this error. This was a clean build (nuked the arch
>directory).
>
>13:50 nid12922 master= ~/petsc_install/petsc$ make
>PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
>PETSC_ARCH="" test
>Running test examples to verify correct installation
>Using
>PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
>and PETSC_ARCH=
>*******************Error detected during compile or
>link!*******************
>See http://www.mcs.anl.gov/petsc/documentation/faq.html
>/global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
>ex19
>*********************************************************************************
>cc -o ex19.o -c -g -fp-model fast
>-I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
>  `pwd`/ex19.c
>cc -g -fp-model fast  -o ex19 ex19.o
>-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>-Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>-lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
>-lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
>-lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
>-lmetis
>-lstdc++ -ldl
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
>In function `PetscDLOpen':
>/global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
>warning:
>Using 'dlopen' in statically linked applications requires at runtime
>the
>shared libraries from the glibc version used for linking
>/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
>Warning: alignment 8 of symbol `ARCHITECTURE' in
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
>is smaller than 32 in
>/opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
>In function `PetscOpenSocket':
>/global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>warning: Using 'gethostbyname' in statically linked applications
>requires
>at runtime the shared libraries from the glibc version used for linking
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
>In function `p4est_checksum':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
>undefined reference to `adler32'
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
>undefined reference to `adler32_combine'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
>In function `p4est_comm_checksum':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
>undefined reference to `adler32_combine'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
>In function `p8est_checksum':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
>undefined reference to `adler32'
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
>undefined reference to `adler32_combine'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
>In function `p8est_comm_checksum':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
>undefined reference to `adler32_combine'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
>In function `sc_array_checksum':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
>undefined reference to `adler32'
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
>undefined reference to `adler32'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
>In function `sc_vtk_write_compressed':
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
>undefined reference to `compress2'
>/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
>undefined reference to `compress2'
>/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
>recipe for target 'ex19' failed
>gmake[3]: [ex19] Error 1 (ignored)
>true ex19
>rm ex19.o

From bsmith at mcs.anl.gov  Sat Oct 20 16:21:56 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 20 Oct 2018 21:21:56 +0000
Subject: [petsc-users] p4est error
In-Reply-To: <D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
Message-ID: <E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>



> On Oct 20, 2018, at 4:14 PM, Isaac, Tobin G <tisaac at cc.gatech.edu> wrote:
> 
> I think adler32 is in zlib: p4est typically finds it in configure.  Can you send along config.log from externalpackages/git.p4est?

   Is there a way to pass the zlib information to ./configure? We should add that to p4est.py Reason: we don't like external packages making their own decisions, we want PETSc's configure making the decisions and passing the decisions down to the packages.

    Barry

   This is especially true for hairy machines that Mark likes to build on, where standard configure sometimes makes the wrong decision or cannot figure out something.

> 
> On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov> wrote:
>> I am getting this error. This was a clean build (nuked the arch
>> directory).
>> 
>> 13:50 nid12922 master= ~/petsc_install/petsc$ make
>> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
>> PETSC_ARCH="" test
>> Running test examples to verify correct installation
>> Using
>> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
>> and PETSC_ARCH=
>> *******************Error detected during compile or
>> link!*******************
>> See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
>> ex19
>> *********************************************************************************
>> cc -o ex19.o -c -g -fp-model fast
>> -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
>> `pwd`/ex19.c
>> cc -g -fp-model fast  -o ex19 ex19.o
>> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>> -Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>> -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
>> -lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
>> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
>> -lmetis
>> -lstdc++ -ldl
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
>> In function `PetscDLOpen':
>> /global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
>> warning:
>> Using 'dlopen' in statically linked applications requires at runtime
>> the
>> shared libraries from the glibc version used for linking
>> /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
>> Warning: alignment 8 of symbol `ARCHITECTURE' in
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
>> is smaller than 32 in
>> /opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
>> In function `PetscOpenSocket':
>> /global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
>> warning: Using 'gethostbyname' in statically linked applications
>> requires
>> at runtime the shared libraries from the glibc version used for linking
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
>> In function `p4est_checksum':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
>> undefined reference to `adler32'
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
>> undefined reference to `adler32_combine'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
>> In function `p4est_comm_checksum':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
>> undefined reference to `adler32_combine'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
>> In function `p8est_checksum':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
>> undefined reference to `adler32'
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
>> undefined reference to `adler32_combine'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
>> In function `p8est_comm_checksum':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
>> undefined reference to `adler32_combine'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
>> In function `sc_array_checksum':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
>> undefined reference to `adler32'
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
>> undefined reference to `adler32'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
>> In function `sc_vtk_write_compressed':
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
>> undefined reference to `compress2'
>> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
>> undefined reference to `compress2'
>> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
>> recipe for target 'ex19' failed
>> gmake[3]: [ex19] Error 1 (ignored)
>> true ex19
>> rm ex19.o


From balay at mcs.anl.gov  Sat Oct 20 16:31:42 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sat, 20 Oct 2018 21:31:42 +0000
Subject: [petsc-users] p4est error
In-Reply-To: <E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
	<E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>
Message-ID: <alpine.LFD.2.21.1810201626310.16662@asterix>

I don't see an option for specifying zlib explicitly - so perhaps it can use it from LIBS?

[untested patch]

Satish

--------

diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py
index 03d1e2fcb0..31d2307f2b 100644
--- a/config/BuildSystem/config/packages/p4est.py
+++ b/config/BuildSystem/config/packages/p4est.py
@@ -21,8 +21,9 @@ class Configure(config.package.GNUPackage):
   def setupDependencies(self, framework):
     config.package.GNUPackage.setupDependencies(self, framework)
     self.mpi        = framework.require('config.packages.MPI',self)
-    self.blasLapack = self.framework.require('config.packages.BlasLapack',self)
-    self.deps = [self.mpi,self.blasLapack]
+    self.blasLapack = framework.require('config.packages.BlasLapack',self)
+    self.zlib       = framework.require('config.packages.zlib',self)
+    self.deps = [self.mpi,self.blasLapack,self.zlib]
     return
 
   def formGNUConfigureArgs(self):
@@ -30,7 +31,7 @@ class Configure(config.package.GNUPackage):
     if self.argDB['with-p4est-debugging']:
       args.append('--enable-debug')
     args.append('--enable-mpi')
-    args.append('LIBS="'+self.libraries.toString(self.blasLapack.dlib)+'"')
+    args.append('LIBS="'+self.libraries.toString(self.dlib)+'"')
     return args
 
   def updateGitDir(self):


On Sat, 20 Oct 2018, Smith, Barry F. wrote:

> 
> 
> > On Oct 20, 2018, at 4:14 PM, Isaac, Tobin G <tisaac at cc.gatech.edu> wrote:
> > 
> > I think adler32 is in zlib: p4est typically finds it in configure.  Can you send along config.log from externalpackages/git.p4est?
> 
>    Is there a way to pass the zlib information to ./configure? We should add that to p4est.py Reason: we don't like external packages making their own decisions, we want PETSc's configure making the decisions and passing the decisions down to the packages.
> 
>     Barry
> 
>    This is especially true for hairy machines that Mark likes to build on, where standard configure sometimes makes the wrong decision or cannot figure out something.
> 
> > 
> > On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov> wrote:
> >> I am getting this error. This was a clean build (nuked the arch
> >> directory).
> >> 
> >> 13:50 nid12922 master= ~/petsc_install/petsc$ make
> >> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> >> PETSC_ARCH="" test
> >> Running test examples to verify correct installation
> >> Using
> >> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> >> and PETSC_ARCH=
> >> *******************Error detected during compile or
> >> link!*******************
> >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >> /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
> >> ex19
> >> *********************************************************************************
> >> cc -o ex19.o -c -g -fp-model fast
> >> -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
> >> `pwd`/ex19.c
> >> cc -g -fp-model fast  -o ex19 ex19.o
> >> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> >> -Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> >> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> >> -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
> >> -lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
> >> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
> >> -lmetis
> >> -lstdc++ -ldl
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
> >> In function `PetscDLOpen':
> >> /global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
> >> warning:
> >> Using 'dlopen' in statically linked applications requires at runtime
> >> the
> >> shared libraries from the glibc version used for linking
> >> /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
> >> Warning: alignment 8 of symbol `ARCHITECTURE' in
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
> >> is smaller than 32 in
> >> /opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
> >> In function `PetscOpenSocket':
> >> /global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> >> warning: Using 'gethostbyname' in statically linked applications
> >> requires
> >> at runtime the shared libraries from the glibc version used for linking
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
> >> In function `p4est_checksum':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> >> undefined reference to `adler32'
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> >> undefined reference to `adler32_combine'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
> >> In function `p4est_comm_checksum':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> >> undefined reference to `adler32_combine'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
> >> In function `p8est_checksum':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> >> undefined reference to `adler32'
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> >> undefined reference to `adler32_combine'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
> >> In function `p8est_comm_checksum':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> >> undefined reference to `adler32_combine'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
> >> In function `sc_array_checksum':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
> >> undefined reference to `adler32'
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
> >> undefined reference to `adler32'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
> >> In function `sc_vtk_write_compressed':
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
> >> undefined reference to `compress2'
> >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
> >> undefined reference to `compress2'
> >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
> >> recipe for target 'ex19' failed
> >> gmake[3]: [ex19] Error 1 (ignored)
> >> true ex19
> >> rm ex19.o
> 


From balay at mcs.anl.gov  Sat Oct 20 16:39:22 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sat, 20 Oct 2018 21:39:22 +0000
Subject: [petsc-users] p4est error
In-Reply-To: <alpine.LFD.2.21.1810201626310.16662@asterix>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
	<E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>
	<alpine.LFD.2.21.1810201626310.16662@asterix>
Message-ID: <alpine.LFD.2.21.1810201638270.16662@asterix>

Pushed this change to branch balay/p4est-add-zlib-dep/maint [it works as desired on my laptop]

Satish

On Sat, 20 Oct 2018, Balay, Satish wrote:

> I don't see an option for specifying zlib explicitly - so perhaps it can use it from LIBS?
> 
> [untested patch]
> 
> Satish
> 
> --------
> 
> diff --git a/config/BuildSystem/config/packages/p4est.py b/config/BuildSystem/config/packages/p4est.py
> index 03d1e2fcb0..31d2307f2b 100644
> --- a/config/BuildSystem/config/packages/p4est.py
> +++ b/config/BuildSystem/config/packages/p4est.py
> @@ -21,8 +21,9 @@ class Configure(config.package.GNUPackage):
>    def setupDependencies(self, framework):
>      config.package.GNUPackage.setupDependencies(self, framework)
>      self.mpi        = framework.require('config.packages.MPI',self)
> -    self.blasLapack = self.framework.require('config.packages.BlasLapack',self)
> -    self.deps = [self.mpi,self.blasLapack]
> +    self.blasLapack = framework.require('config.packages.BlasLapack',self)
> +    self.zlib       = framework.require('config.packages.zlib',self)
> +    self.deps = [self.mpi,self.blasLapack,self.zlib]
>      return
>  
>    def formGNUConfigureArgs(self):
> @@ -30,7 +31,7 @@ class Configure(config.package.GNUPackage):
>      if self.argDB['with-p4est-debugging']:
>        args.append('--enable-debug')
>      args.append('--enable-mpi')
> -    args.append('LIBS="'+self.libraries.toString(self.blasLapack.dlib)+'"')
> +    args.append('LIBS="'+self.libraries.toString(self.dlib)+'"')
>      return args
>  
>    def updateGitDir(self):
> 
> 
> On Sat, 20 Oct 2018, Smith, Barry F. wrote:
> 
> > 
> > 
> > > On Oct 20, 2018, at 4:14 PM, Isaac, Tobin G <tisaac at cc.gatech.edu> wrote:
> > > 
> > > I think adler32 is in zlib: p4est typically finds it in configure.  Can you send along config.log from externalpackages/git.p4est?
> > 
> >    Is there a way to pass the zlib information to ./configure? We should add that to p4est.py Reason: we don't like external packages making their own decisions, we want PETSc's configure making the decisions and passing the decisions down to the packages.
> > 
> >     Barry
> > 
> >    This is especially true for hairy machines that Mark likes to build on, where standard configure sometimes makes the wrong decision or cannot figure out something.
> > 
> > > 
> > > On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov> wrote:
> > >> I am getting this error. This was a clean build (nuked the arch
> > >> directory).
> > >> 
> > >> 13:50 nid12922 master= ~/petsc_install/petsc$ make
> > >> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > >> PETSC_ARCH="" test
> > >> Running test examples to verify correct installation
> > >> Using
> > >> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > >> and PETSC_ARCH=
> > >> *******************Error detected during compile or
> > >> link!*******************
> > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > >> /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
> > >> ex19
> > >> *********************************************************************************
> > >> cc -o ex19.o -c -g -fp-model fast
> > >> -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
> > >> `pwd`/ex19.c
> > >> cc -g -fp-model fast  -o ex19 ex19.o
> > >> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > >> -Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > >> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > >> -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
> > >> -lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
> > >> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
> > >> -lmetis
> > >> -lstdc++ -ldl
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
> > >> In function `PetscDLOpen':
> > >> /global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
> > >> warning:
> > >> Using 'dlopen' in statically linked applications requires at runtime
> > >> the
> > >> shared libraries from the glibc version used for linking
> > >> /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
> > >> Warning: alignment 8 of symbol `ARCHITECTURE' in
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
> > >> is smaller than 32 in
> > >> /opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
> > >> In function `PetscOpenSocket':
> > >> /global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> > >> warning: Using 'gethostbyname' in statically linked applications
> > >> requires
> > >> at runtime the shared libraries from the glibc version used for linking
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
> > >> In function `p4est_checksum':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > >> undefined reference to `adler32'
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > >> undefined reference to `adler32_combine'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
> > >> In function `p4est_comm_checksum':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > >> undefined reference to `adler32_combine'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
> > >> In function `p8est_checksum':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > >> undefined reference to `adler32'
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > >> undefined reference to `adler32_combine'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
> > >> In function `p8est_comm_checksum':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > >> undefined reference to `adler32_combine'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
> > >> In function `sc_array_checksum':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
> > >> undefined reference to `adler32'
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
> > >> undefined reference to `adler32'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
> > >> In function `sc_vtk_write_compressed':
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
> > >> undefined reference to `compress2'
> > >> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
> > >> undefined reference to `compress2'
> > >> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
> > >> recipe for target 'ex19' failed
> > >> gmake[3]: [ex19] Error 1 (ignored)
> > >> true ex19
> > >> rm ex19.o
> > 
> 


From mfadams at lbl.gov  Sat Oct 20 18:25:10 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 20 Oct 2018 19:25:10 -0400
Subject: [petsc-users] p4est error
In-Reply-To: <D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
Message-ID: <CADOhEh6uCW8cokCKn5dJwDYYPS6zN5EV-LggG+iJ4ErctoPxoQ@mail.gmail.com>

There is something going on with zlib. When I get on a compute node I get a
message that zlib (and cmake) module is not found. But I add it manually
and I see it, but still get the same error.

I nuked p4est and am testing Satish's fix. I'll send the log file if it is
not working.

Thanks,

On Sat, Oct 20, 2018 at 5:14 PM Isaac, Tobin G <tisaac at cc.gatech.edu> wrote:

> I think adler32 is in zlib: p4est typically finds it in configure.  Can
> you send along config.log from externalpackages/git.p4est?
>
> On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov> wrote:
> >I am getting this error. This was a clean build (nuked the arch
> >directory).
> >
> >13:50 nid12922 master= ~/petsc_install/petsc$ make
>
> >PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> >PETSC_ARCH="" test
> >Running test examples to verify correct installation
> >Using
>
> >PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> >and PETSC_ARCH=
> >*******************Error detected during compile or
> >link!*******************
> >See http://www.mcs.anl.gov/petsc/documentation/faq.html
> >/global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
> >ex19
>
> >*********************************************************************************
> >cc -o ex19.o -c -g -fp-model fast
>
> >-I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
> >  `pwd`/ex19.c
> >cc -g -fp-model fast  -o ex19 ex19.o
> >-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
>
> >-Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> >-L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> >-lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
> >-lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
> >-lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
> >-lmetis
> >-lstdc++ -ldl
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
> >In function `PetscDLOpen':
> >/global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
> >warning:
> >Using 'dlopen' in statically linked applications requires at runtime
> >the
> >shared libraries from the glibc version used for linking
> >/usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
> >Warning: alignment 8 of symbol `ARCHITECTURE' in
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
> >is smaller than 32 in
> >/opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
> >In function `PetscOpenSocket':
>
> >/global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> >warning: Using 'gethostbyname' in statically linked applications
> >requires
> >at runtime the shared libraries from the glibc version used for linking
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
> >In function `p4est_checksum':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> >undefined reference to `adler32'
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> >undefined reference to `adler32_combine'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
> >In function `p4est_comm_checksum':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> >undefined reference to `adler32_combine'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
> >In function `p8est_checksum':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> >undefined reference to `adler32'
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> >undefined reference to `adler32_combine'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
> >In function `p8est_comm_checksum':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> >undefined reference to `adler32_combine'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
> >In function `sc_array_checksum':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
> >undefined reference to `adler32'
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
> >undefined reference to `adler32'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
> >In function `sc_vtk_write_compressed':
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
> >undefined reference to `compress2'
>
> >/global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
> >undefined reference to `compress2'
>
> >/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
> >recipe for target 'ex19' failed
> >gmake[3]: [ex19] Error 1 (ignored)
> >true ex19
> >rm ex19.o
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181020/0e9deb03/attachment-0001.html>

From mfadams at lbl.gov  Sat Oct 20 19:44:16 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Sat, 20 Oct 2018 20:44:16 -0400
Subject: [petsc-users] p4est error
In-Reply-To: <alpine.LFD.2.21.1810201638270.16662@asterix>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
	<E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>
	<alpine.LFD.2.21.1810201626310.16662@asterix>
	<alpine.LFD.2.21.1810201638270.16662@asterix>
Message-ID: <CADOhEh5gQOFHHrEev55r7b9wNRgLJMfum1EBgQcHmMf3j5HxWw@mail.gmail.com>

Thanks Satish.

I built this and it is working.

On Sat, Oct 20, 2018 at 5:39 PM Balay, Satish <balay at mcs.anl.gov> wrote:

> Pushed this change to branch balay/p4est-add-zlib-dep/maint [it works as
> desired on my laptop]
>
> Satish
>
> On Sat, 20 Oct 2018, Balay, Satish wrote:
>
> > I don't see an option for specifying zlib explicitly - so perhaps it can
> use it from LIBS?
> >
> > [untested patch]
> >
> > Satish
> >
> > --------
> >
> > diff --git a/config/BuildSystem/config/packages/p4est.py
> b/config/BuildSystem/config/packages/p4est.py
> > index 03d1e2fcb0..31d2307f2b 100644
> > --- a/config/BuildSystem/config/packages/p4est.py
> > +++ b/config/BuildSystem/config/packages/p4est.py
> > @@ -21,8 +21,9 @@ class Configure(config.package.GNUPackage):
> >    def setupDependencies(self, framework):
> >      config.package.GNUPackage.setupDependencies(self, framework)
> >      self.mpi        = framework.require('config.packages.MPI',self)
> > -    self.blasLapack =
> self.framework.require('config.packages.BlasLapack',self)
> > -    self.deps = [self.mpi,self.blasLapack]
> > +    self.blasLapack =
> framework.require('config.packages.BlasLapack',self)
> > +    self.zlib       = framework.require('config.packages.zlib',self)
> > +    self.deps = [self.mpi,self.blasLapack,self.zlib]
> >      return
> >
> >    def formGNUConfigureArgs(self):
> > @@ -30,7 +31,7 @@ class Configure(config.package.GNUPackage):
> >      if self.argDB['with-p4est-debugging']:
> >        args.append('--enable-debug')
> >      args.append('--enable-mpi')
> > -
> args.append('LIBS="'+self.libraries.toString(self.blasLapack.dlib)+'"')
> > +    args.append('LIBS="'+self.libraries.toString(self.dlib)+'"')
> >      return args
> >
> >    def updateGitDir(self):
> >
> >
> > On Sat, 20 Oct 2018, Smith, Barry F. wrote:
> >
> > >
> > >
> > > > On Oct 20, 2018, at 4:14 PM, Isaac, Tobin G <tisaac at cc.gatech.edu>
> wrote:
> > > >
> > > > I think adler32 is in zlib: p4est typically finds it in configure.
> Can you send along config.log from externalpackages/git.p4est?
> > >
> > >    Is there a way to pass the zlib information to ./configure? We
> should add that to p4est.py Reason: we don't like external packages making
> their own decisions, we want PETSc's configure making the decisions and
> passing the decisions down to the packages.
> > >
> > >     Barry
> > >
> > >    This is especially true for hairy machines that Mark likes to build
> on, where standard configure sometimes makes the wrong decision or cannot
> figure out something.
> > >
> > > >
> > > > On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov>
> wrote:
> > > >> I am getting this error. This was a clean build (nuked the arch
> > > >> directory).
> > > >>
> > > >> 13:50 nid12922 master= ~/petsc_install/petsc$ make
> > > >>
> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > > >> PETSC_ARCH="" test
> > > >> Running test examples to verify correct installation
> > > >> Using
> > > >>
> PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > > >> and PETSC_ARCH=
> > > >> *******************Error detected during compile or
> > > >> link!*******************
> > > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > > >>
> /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
> > > >> ex19
> > > >>
> *********************************************************************************
> > > >> cc -o ex19.o -c -g -fp-model fast
> > > >>
> -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
> > > >> `pwd`/ex19.c
> > > >> cc -g -fp-model fast  -o ex19 ex19.o
> > > >>
> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > >>
> -Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > >>
> -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > >> -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
> > > >> -lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
> > > >> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
> > > >> -lmetis
> > > >> -lstdc++ -ldl
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
> > > >> In function `PetscDLOpen':
> > > >> /global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
> > > >> warning:
> > > >> Using 'dlopen' in statically linked applications requires at runtime
> > > >> the
> > > >> shared libraries from the glibc version used for linking
> > > >>
> /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
> > > >> Warning: alignment 8 of symbol `ARCHITECTURE' in
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
> > > >> is smaller than 32 in
> > > >> /opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
> > > >> In function `PetscOpenSocket':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> > > >> warning: Using 'gethostbyname' in statically linked applications
> > > >> requires
> > > >> at runtime the shared libraries from the glibc version used for
> linking
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
> > > >> In function `p4est_checksum':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > > >> undefined reference to `adler32'
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > > >> undefined reference to `adler32_combine'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
> > > >> In function `p4est_comm_checksum':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > > >> undefined reference to `adler32_combine'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
> > > >> In function `p8est_checksum':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > > >> undefined reference to `adler32'
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > > >> undefined reference to `adler32_combine'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
> > > >> In function `p8est_comm_checksum':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > > >> undefined reference to `adler32_combine'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
> > > >> In function `sc_array_checksum':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
> > > >> undefined reference to `adler32'
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
> > > >> undefined reference to `adler32'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
> > > >> In function `sc_vtk_write_compressed':
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
> > > >> undefined reference to `compress2'
> > > >>
> /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
> > > >> undefined reference to `compress2'
> > > >>
> /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
> > > >> recipe for target 'ex19' failed
> > > >> gmake[3]: [ex19] Error 1 (ignored)
> > > >> true ex19
> > > >> rm ex19.o
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181020/06ce31ce/attachment.html>

From balay at mcs.anl.gov  Sat Oct 20 20:39:26 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sun, 21 Oct 2018 01:39:26 +0000
Subject: [petsc-users] p4est error
In-Reply-To: <CADOhEh5gQOFHHrEev55r7b9wNRgLJMfum1EBgQcHmMf3j5HxWw@mail.gmail.com>
References: <CADOhEh5OWQPM40dXrWmH6JsWv9pid2r6eH43ird14TGzwtHr7w@mail.gmail.com>
	<D2C8D2B2-415F-48C7-933E-EDF0E857047A@cc.gatech.edu>
	<E39FC1A8-3A51-46FC-BA07-9B9DF92C2F71@anl.gov>
	<alpine.LFD.2.21.1810201626310.16662@asterix>
	<alpine.LFD.2.21.1810201638270.16662@asterix>
	<CADOhEh5gQOFHHrEev55r7b9wNRgLJMfum1EBgQcHmMf3j5HxWw@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1810202039010.16662@asterix>

Thanks! merged to next.

Satish

On Sat, 20 Oct 2018, Mark Adams wrote:

> Thanks Satish.
> 
> I built this and it is working.
> 
> On Sat, Oct 20, 2018 at 5:39 PM Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> > Pushed this change to branch balay/p4est-add-zlib-dep/maint [it works as
> > desired on my laptop]
> >
> > Satish
> >
> > On Sat, 20 Oct 2018, Balay, Satish wrote:
> >
> > > I don't see an option for specifying zlib explicitly - so perhaps it can
> > use it from LIBS?
> > >
> > > [untested patch]
> > >
> > > Satish
> > >
> > > --------
> > >
> > > diff --git a/config/BuildSystem/config/packages/p4est.py
> > b/config/BuildSystem/config/packages/p4est.py
> > > index 03d1e2fcb0..31d2307f2b 100644
> > > --- a/config/BuildSystem/config/packages/p4est.py
> > > +++ b/config/BuildSystem/config/packages/p4est.py
> > > @@ -21,8 +21,9 @@ class Configure(config.package.GNUPackage):
> > >    def setupDependencies(self, framework):
> > >      config.package.GNUPackage.setupDependencies(self, framework)
> > >      self.mpi        = framework.require('config.packages.MPI',self)
> > > -    self.blasLapack =
> > self.framework.require('config.packages.BlasLapack',self)
> > > -    self.deps = [self.mpi,self.blasLapack]
> > > +    self.blasLapack =
> > framework.require('config.packages.BlasLapack',self)
> > > +    self.zlib       = framework.require('config.packages.zlib',self)
> > > +    self.deps = [self.mpi,self.blasLapack,self.zlib]
> > >      return
> > >
> > >    def formGNUConfigureArgs(self):
> > > @@ -30,7 +31,7 @@ class Configure(config.package.GNUPackage):
> > >      if self.argDB['with-p4est-debugging']:
> > >        args.append('--enable-debug')
> > >      args.append('--enable-mpi')
> > > -
> > args.append('LIBS="'+self.libraries.toString(self.blasLapack.dlib)+'"')
> > > +    args.append('LIBS="'+self.libraries.toString(self.dlib)+'"')
> > >      return args
> > >
> > >    def updateGitDir(self):
> > >
> > >
> > > On Sat, 20 Oct 2018, Smith, Barry F. wrote:
> > >
> > > >
> > > >
> > > > > On Oct 20, 2018, at 4:14 PM, Isaac, Tobin G <tisaac at cc.gatech.edu>
> > wrote:
> > > > >
> > > > > I think adler32 is in zlib: p4est typically finds it in configure.
> > Can you send along config.log from externalpackages/git.p4est?
> > > >
> > > >    Is there a way to pass the zlib information to ./configure? We
> > should add that to p4est.py Reason: we don't like external packages making
> > their own decisions, we want PETSc's configure making the decisions and
> > passing the decisions down to the packages.
> > > >
> > > >     Barry
> > > >
> > > >    This is especially true for hairy machines that Mark likes to build
> > on, where standard configure sometimes makes the wrong decision or cannot
> > figure out something.
> > > >
> > > > >
> > > > > On October 20, 2018 4:52:54 PM EDT, Mark Adams <mfadams at lbl.gov>
> > wrote:
> > > > >> I am getting this error. This was a clean build (nuked the arch
> > > > >> directory).
> > > > >>
> > > > >> 13:50 nid12922 master= ~/petsc_install/petsc$ make
> > > > >>
> > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > > > >> PETSC_ARCH="" test
> > > > >> Running test examples to verify correct installation
> > > > >> Using
> > > > >>
> > PETSC_DIR=/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel
> > > > >> and PETSC_ARCH=
> > > > >> *******************Error detected during compile or
> > > > >> link!*******************
> > > > >> See http://www.mcs.anl.gov/petsc/documentation/faq.html
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc/src/snes/examples/tutorials
> > > > >> ex19
> > > > >>
> > *********************************************************************************
> > > > >> cc -o ex19.o -c -g -fp-model fast
> > > > >>
> > -I/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/include
> > > > >> `pwd`/ex19.c
> > > > >> cc -g -fp-model fast  -o ex19 ex19.o
> > > > >>
> > -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > > >>
> > -Wl,-rpath,/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > > >>
> > -L/global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib
> > > > >> -lpetsc -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord
> > > > >> -lscalapack -lfftw3_mpi -lfftw3 -lp4est -lsc -lflapack -lfblas
> > > > >> -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5 -lchaco -lparmetis
> > > > >> -lmetis
> > > > >> -lstdc++ -ldl
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(dlimpl.o):
> > > > >> In function `PetscDLOpen':
> > > > >> /global/u2/m/madams/petsc_install/petsc/src/sys/dll/dlimpl.c:108:
> > > > >> warning:
> > > > >> Using 'dlopen' in statically linked applications requires at runtime
> > > > >> the
> > > > >> shared libraries from the glibc version used for linking
> > > > >>
> > /usr/lib64/gcc/x86_64-suse-linux/4.8/../../../../x86_64-suse-linux/bin/ld:
> > > > >> Warning: alignment 8 of symbol `ARCHITECTURE' in
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libchaco.a(user_params.o)
> > > > >> is smaller than 32 in
> > > > >> /opt/cray/pe/pmi/5.0.13/lib64/libpmi.a(_pmi_topology_discovery.o)
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libpetsc.a(send.o):
> > > > >> In function `PetscOpenSocket':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/src/sys/classes/viewer/impls/socket/send.c:108:
> > > > >> warning: Using 'gethostbyname' in statically linked applications
> > > > >> requires
> > > > >> at runtime the shared libraries from the glibc version used for
> > linking
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est.o):
> > > > >> In function `p4est_checksum':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > > > >> undefined reference to `adler32'
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > > > >> undefined reference to `adler32_combine'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p4est_communication.o):
> > > > >> In function `p4est_comm_checksum':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > > > >> undefined reference to `adler32_combine'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est.o):
> > > > >> In function `p8est_checksum':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3228:
> > > > >> undefined reference to `adler32'
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est.c:3236:
> > > > >> undefined reference to `adler32_combine'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libp4est.a(src_libp4est_la-p8est_communication.o):
> > > > >> In function `p8est_comm_checksum':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/src/p4est_communication.c:903:
> > > > >> undefined reference to `adler32_combine'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_containers.o):
> > > > >> In function `sc_array_checksum':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:565:
> > > > >> undefined reference to `adler32'
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_containers.c:571:
> > > > >> undefined reference to `adler32'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/libsc.a(src_libsc_la-sc_io.o):
> > > > >> In function `sc_vtk_write_compressed':
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:492:
> > > > >> undefined reference to `compress2'
> > > > >>
> > /global/u2/m/madams/petsc_install/petsc/arch-cori-haswell-opt-intel/externalpackages/git.p4est/sc/src/sc_io.c:507:
> > > > >> undefined reference to `compress2'
> > > > >>
> > /global/homes/m/madams/petsc_install/petsc-cori-haswell-opt-intel/lib/petsc/conf/test:13:
> > > > >> recipe for target 'ex19' failed
> > > > >> gmake[3]: [ex19] Error 1 (ignored)
> > > > >> true ex19
> > > > >> rm ex19.o
> > > >
> > >
> >
> >
> 


From C.Klaij at marin.nl  Mon Oct 22 04:33:14 2018
From: C.Klaij at marin.nl (Klaij, Christiaan)
Date: Mon, 22 Oct 2018 09:33:14 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <380804AB-7723-4539-9722-9B2D689D6A05@mcs.anl.gov>
References: <1539939090363.13492@marin.nl>
	<BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
	<1539959878384.42597@marin.nl>,
	<380804AB-7723-4539-9722-9B2D689D6A05@mcs.anl.gov>
Message-ID: <1540200794411.69211@marin.nl>

Thanks Barry and Matt, it makes sense if rr is a pointer instead
of an allocatable. So:

  Vec, POINTER, INTENT(in) :: rr_system

would be the proper way, right?

And out of curiosity, why did petsc-3.8.4 tolerate my wrong
INTENT(out)?

Chris



dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl

MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm

________________________________________
From: Smith, Barry F. <bsmith at mcs.anl.gov>
Sent: Friday, October 19, 2018 10:26 PM
To: Klaij, Christiaan
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2

> On Oct 19, 2018, at 9:37 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>
> As far as I (mis)understand fortran, this is a data protection
> thing: all arguments are passed in from above but the subroutine
> is only allowed to change rr and ierr, not aa and xx (if you try,
> you get a compiler warning).

      The routine is not allowed to change rr (it is only allowed to change the values "inside" rr) that is why it needs to be intent in or inout. Otherwise the compiler can optimize and not pass down the value of the rr pointer to the subroutine since by declaring as it as out the compiler thinks your subroutine is going to set is value.

    Barry


> That's why I find it very odd to
> give an intent(in) to rr. But I've tried your suggestion anyway:
> both intent(in) and intent(inout) for rr do work! Can't say I
> understand though.
>
> Below's a small example of what I was expecting. Change rr to
> intent(in) and the compiler complains.
>
> Chris
>
> $ cat intent.f90
> program intent
>
>  implicit none
>
>  real, allocatable :: aa(:), xx(:), rr(:)
>  integer :: ierr
>
>  allocate(aa(10),xx(10),rr(10))
>
>  aa = 1.0
>  xx = 2.0
>
>  call matmult(aa,xx,rr,ierr)
>
>  print *, rr(1)
>  print *, ierr
>
>  deallocate(aa,xx,rr)
>
>  contains
>
>    subroutine matmult(aa,xx,rr,ierr)
>      real, intent(in) :: aa(:), xx(:)
>      real, intent(out):: rr(:)
>      integer, intent(out) :: ierr
>      rr=aa*xx
>      ierr=0
>    end subroutine matmult
>
> end program intent
> $ ./a.out
>   2.000000
>           0
>
>
>
>
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>
> MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm
>
> ________________________________________
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Friday, October 19, 2018 2:32 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2
>
>   Hmm, the intent of all three first arguments should be in since they are passed in from the routine above. Does it work if you replace
>
>> Vec, INTENT(out) :: rr_system
>
> with
>
>> Vec, INTENT(in) :: rr_system
>
> ?
>
>    Barry
>
>
>> On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>>
>> I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
>> surprised to find a number of segmentation faults in my test
>> cases. These turned out to be related to user-defined MatMult and
>> PCApply for shell matrices. For example:
>>
>> SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
>> Mat, INTENT(in) :: aa_system
>> Vec, INTENT(in) :: xx_system
>> Vec, INTENT(out) :: rr_system
>> PetscErrorCode, INTENT(out) :: ierr
>> ...
>> END
>>
>> where aa_system is the shell matrix. This code works fine with
>> 3.8.4 and older, but fails with 3.10.2 due to invalid
>> pointers (gdb backtrace shows failure of VecSetValues due to
>> invalid first argument). After replacing by:
>>
>> SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
>> Mat :: aa_system
>> Vec :: xx_system
>> Vec :: rr_system
>> PetscErrorCode :: ierr
>> ...
>> END
>>
>> everything's fine again. So clearly something has changed since
>> 3.8.4 that now prevents the use of INTENT in Fortran (at least
>> using intel 17.0.1 compilers). Any reason for this?
>>
>> Chris
>>
>>
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>>
>> MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
>>
>


From bsmith at mcs.anl.gov  Mon Oct 22 11:37:27 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 22 Oct 2018 16:37:27 +0000
Subject: [petsc-users] fortran INTENT with petsc object gives segfault
 after upgrade from 3.8.4 to 3.10.2
In-Reply-To: <1540200794411.69211@marin.nl>
References: <1539939090363.13492@marin.nl>
	<BECF7F26-4C36-44B7-919C-334FEB9755FD@anl.gov>
	<1539959878384.42597@marin.nl>
	<380804AB-7723-4539-9722-9B2D689D6A05@mcs.anl.gov>
	<1540200794411.69211@marin.nl>
Message-ID: <E95DCA0A-D541-4C85-B1DC-D11546BC1389@mcs.anl.gov>



> On Oct 22, 2018, at 4:33 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
> 
> Thanks Barry and Matt, it makes sense if rr is a pointer instead
> of an allocatable. So:
> 
>  Vec, POINTER, INTENT(in) :: rr_system
> 
> would be the proper way, right?

   Yes

> 
> And out of curiosity, why did petsc-3.8.4 tolerate my wrong
> INTENT(out)?

   I think it is a compiler optimization that doesn't pass in the value for intent(out) hence the crash when it is used.
> 
> Chris
> 
> 
> 
> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
> 
> MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm
> 
> ________________________________________
> From: Smith, Barry F. <bsmith at mcs.anl.gov>
> Sent: Friday, October 19, 2018 10:26 PM
> To: Klaij, Christiaan
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2
> 
>> On Oct 19, 2018, at 9:37 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>> 
>> As far as I (mis)understand fortran, this is a data protection
>> thing: all arguments are passed in from above but the subroutine
>> is only allowed to change rr and ierr, not aa and xx (if you try,
>> you get a compiler warning).
> 
>      The routine is not allowed to change rr (it is only allowed to change the values "inside" rr) that is why it needs to be intent in or inout. Otherwise the compiler can optimize and not pass down the value of the rr pointer to the subroutine since by declaring as it as out the compiler thinks your subroutine is going to set is value.
> 
>    Barry
> 
> 
>> That's why I find it very odd to
>> give an intent(in) to rr. But I've tried your suggestion anyway:
>> both intent(in) and intent(inout) for rr do work! Can't say I
>> understand though.
>> 
>> Below's a small example of what I was expecting. Change rr to
>> intent(in) and the compiler complains.
>> 
>> Chris
>> 
>> $ cat intent.f90
>> program intent
>> 
>> implicit none
>> 
>> real, allocatable :: aa(:), xx(:), rr(:)
>> integer :: ierr
>> 
>> allocate(aa(10),xx(10),rr(10))
>> 
>> aa = 1.0
>> xx = 2.0
>> 
>> call matmult(aa,xx,rr,ierr)
>> 
>> print *, rr(1)
>> print *, ierr
>> 
>> deallocate(aa,xx,rr)
>> 
>> contains
>> 
>>   subroutine matmult(aa,xx,rr,ierr)
>>     real, intent(in) :: aa(:), xx(:)
>>     real, intent(out):: rr(:)
>>     integer, intent(out) :: ierr
>>     rr=aa*xx
>>     ierr=0
>>   end subroutine matmult
>> 
>> end program intent
>> $ ./a.out
>>  2.000000
>>          0
>> 
>> 
>> 
>> 
>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>> 
>> MARIN news: http://www.marin.nl/web/News/News-items/Seminar-Scheepsbrandstof-en-de-mondiale-zwavelnorm-2020.htm
>> 
>> ________________________________________
>> From: Smith, Barry F. <bsmith at mcs.anl.gov>
>> Sent: Friday, October 19, 2018 2:32 PM
>> To: Klaij, Christiaan
>> Cc: petsc-users at mcs.anl.gov
>> Subject: Re: [petsc-users] fortran INTENT with petsc object gives segfault after upgrade from 3.8.4 to 3.10.2
>> 
>>  Hmm, the intent of all three first arguments should be in since they are passed in from the routine above. Does it work if you replace
>> 
>>> Vec, INTENT(out) :: rr_system
>> 
>> with
>> 
>>> Vec, INTENT(in) :: rr_system
>> 
>> ?
>> 
>>   Barry
>> 
>> 
>>> On Oct 19, 2018, at 3:51 AM, Klaij, Christiaan <C.Klaij at marin.nl> wrote:
>>> 
>>> I've recently upgraded from petsc-3.8.4 to petsc-3.10.2 and was
>>> surprised to find a number of segmentation faults in my test
>>> cases. These turned out to be related to user-defined MatMult and
>>> PCApply for shell matrices. For example:
>>> 
>>> SUBROUTINE systemMatMult(aa_system,xx_system,rr_system,ierr)
>>> Mat, INTENT(in) :: aa_system
>>> Vec, INTENT(in) :: xx_system
>>> Vec, INTENT(out) :: rr_system
>>> PetscErrorCode, INTENT(out) :: ierr
>>> ...
>>> END
>>> 
>>> where aa_system is the shell matrix. This code works fine with
>>> 3.8.4 and older, but fails with 3.10.2 due to invalid
>>> pointers (gdb backtrace shows failure of VecSetValues due to
>>> invalid first argument). After replacing by:
>>> 
>>> SUBROUTINE mass_momentum_systemMatMult(aa_system,xx_system,rr_system,ierr)
>>> Mat :: aa_system
>>> Vec :: xx_system
>>> Vec :: rr_system
>>> PetscErrorCode :: ierr
>>> ...
>>> END
>>> 
>>> everything's fine again. So clearly something has changed since
>>> 3.8.4 that now prevents the use of INTENT in Fortran (at least
>>> using intel 17.0.1 compilers). Any reason for this?
>>> 
>>> Chris
>>> 
>>> 
>>> dr. ir. Christiaan Klaij  | Senior Researcher | Research & Development
>>> MARIN | T +31 317 49 33 44 | mailto:C.Klaij at marin.nl | http://www.marin.nl
>>> 
>>> MARIN news: http://www.marin.nl/web/News/News-items/ReFRESCO-successfully-coupled-to-ParaView-Catalyst-for-insitu-analysis-1.htm
>>> 
>> 
> 


From bhatiamanav at gmail.com  Mon Oct 22 14:05:58 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 22 Oct 2018 14:05:58 -0500
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
Message-ID: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>

Hi, 

   I am exploring the nonlinear eigenvalue problem solver in Slepc. 

   From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 

   Is there a way to include support for complex eigenpairs in a library complied with real scalars? 

Regards,
Manav



From jroman at dsic.upv.es  Mon Oct 22 14:40:09 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Mon, 22 Oct 2018 21:40:09 +0200
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
Message-ID: <B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>



> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
> 
> Hi, 
> 
>   I am exploring the nonlinear eigenvalue problem solver in Slepc. 
> 
>   From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
> 
>   Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
> 
> Regards,
> Manav
> 
> 

Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.

Jose

From bhatiamanav at gmail.com  Mon Oct 22 14:40:24 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 22 Oct 2018 14:40:24 -0500
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
Message-ID: <EC8E208D-1DC3-448B-AD9F-FF6F10BC8C1F@gmail.com>

As a followup to this, if I am using a shell matrix for eigensolution (linear or nonlinear eigenproblems), what operations should be defined for the shell matrix? 

-Manav

> On Oct 22, 2018, at 2:05 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Hi, 
> 
>   I am exploring the nonlinear eigenvalue problem solver in Slepc. 
> 
>   From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
> 
>   Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
> 
> Regards,
> Manav
> 
> 


From jroman at dsic.upv.es  Mon Oct 22 14:56:28 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Mon, 22 Oct 2018 21:56:28 +0200
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <EC8E208D-1DC3-448B-AD9F-FF6F10BC8C1F@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<EC8E208D-1DC3-448B-AD9F-FF6F10BC8C1F@gmail.com>
Message-ID: <F7225F61-5F73-4317-B3B3-FB24E8A5AE9C@dsic.upv.es>

It depends on the solver. For instance, NEPRII builds the matrix T(lambda) and then uses it for matrix-vector multiplications and also for linear solves. So the required operations depend on which preconditioner you use for the linear solves. This example can use Jacobi preconditioner: http://slepc.upv.es/documentation/current/src/nep/examples/tutorials/ex21.c.html

Jose


> El 22 oct 2018, a las 21:40, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
> 
> As a followup to this, if I am using a shell matrix for eigensolution (linear or nonlinear eigenproblems), what operations should be defined for the shell matrix? 
> 
> -Manav
> 
>> On Oct 22, 2018, at 2:05 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Hi, 
>> 
>>  I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>> 
>>  From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>> 
>>  Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>> 
>> Regards,
>> Manav
>> 
>> 
> 


From bhatiamanav at gmail.com  Mon Oct 22 15:01:12 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 22 Oct 2018 15:01:12 -0500
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
Message-ID: <98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>

Thanks, Jose. 

How difficult would it be to add the support for the general case (if at all possible)? 

My eigenvalue problem is of the form shown in the attachment. Beta is the eigenvalue and X_s^\Delta is the eigenvector. While some of the matrices are known, others are defined only as matrix vector products. 

I am interested in eigenvalues with the largest real part. I expect to find complex eigenvalues, although for a small subset of cases these will be real. 

What is your recommendation for attacking this problem with the nonlinear eigenvalue support in Slepc? 

Would appreciate your guidance. 

Regards,
Manav



> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> 
> 
>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>> 
>> Hi, 
>> 
>>  I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>> 
>>  From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>> 
>>  Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>> 
>> Regards,
>> Manav
>> 
>> 
> 
> Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.
> 
> Jose

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/bf7b1bc6/attachment-0002.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: PastedGraphic-1.pdf
Type: application/pdf
Size: 226918 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/bf7b1bc6/attachment-0001.pdf>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/bf7b1bc6/attachment-0003.html>

From weizhuo2 at illinois.edu  Mon Oct 22 15:05:40 2018
From: weizhuo2 at illinois.edu (Weizhuo Wang)
Date: Mon, 22 Oct 2018 15:05:40 -0500
Subject: [petsc-users] Increasing norm with finer mesh
In-Reply-To: <CAJ98EDrh4VGwLNuZGOfX88HMubRDsNfSz2swHhL9PM0LjKVHrw@mail.gmail.com>
References: <CABvwWnLw46P5HH8BwCGifmnJnyeceZ7y-chpUt7s3D4vKji04A@mail.gmail.com>
	<CAMYG4Gm9Y9KXciid2TYFfv4Bzt3PK459R_zdTd24zoq=yVZRYw@mail.gmail.com>
	<CABvwWnL7-e6qLsv=LsSqyP9hR135Hk1ouSASajycmXXtwx899g@mail.gmail.com>
	<BEA681F7-9E74-45F2-8EDD-FF600D10A37C@anl.gov>
	<CABvwWnKzb3DLHRS1vO8oNGroFy0i3ks=i5reGCnEaLw8h594hw@mail.gmail.com>
	<CADOhEh5Osf1MFN42U6Nkd=vdkwq=viZpkV=+4fM5YuE+YEL5Kw@mail.gmail.com>
	<CABvwWnKpiuH7898w_Levax8DEo-432bguu9kKQHYM+LUGb=aiA@mail.gmail.com>
	<CADOhEh5qF4-Y4i_ROigY5_6fW-woGGM=b2As3u2khxn+u1cPyQ@mail.gmail.com>
	<CABvwWn+dTM3rqYV7qgStKtnf8_i=vZbJ-dD2FjEaS-dQYDc6kA@mail.gmail.com>
	<CAMYG4GmOwMMiwiwFQQh9U-pSvGyq-aj5kQeC9+yDMe-GHkT+fQ@mail.gmail.com>
	<CABvwWn+iki3sdb9SjQ3ym_x4w5ACvbm-SD2f9N_QMEDjnPn5=Q@mail.gmail.com>
	<CAMYG4G=ZR_HBs=xuc9HK7FwjLWAhV3Xu4Q8a+H63w=wPhMtDxg@mail.gmail.com>
	<CABvwWnKKwPWY8H06vVkd3TOyFLseJoCxA2PwzYFWOqqP=f15Kw@mail.gmail.com>
	<CADOhEh7NKSJdgxa51-OvUmp7gbV5hZixSU2Jh71oD7cNq5791Q@mail.gmail.com>
	<CABvwWnJj3SYxiCzYckBoC7srTOKeQL+vAzG72b1XhycBJ_OGNQ@mail.gmail.com>
	<CAMYG4GkVrYuE5V5vHcmqo-zqs8j4yJ6ShyfX791qUfrEouj9WQ@mail.gmail.com>
	<CADOhEh71pvnPskDyKzjidppyLbo3ib71sBKzygtw1_YT_ArkWA@mail.gmail.com>
	<CABvwWnKMfVNJ8RZj3+6FUmhH8T8s-Qdsb_1zn8MsV1yssuTF5g@mail.gmail.com>
	<CAMYG4GnOrQwM5qvDYti8DhwLKRxitRTQNVf+ii6g4EforGyYBg@mail.gmail.com>
	<CABvwWnJv_C6gSu3gLr-aLup7BRtEKVtQ+u0T_qx57a9K-ntqtw@mail.gmail.com>
	<CAJ98EDrh4VGwLNuZGOfX88HMubRDsNfSz2swHhL9PM0LjKVHrw@mail.gmail.com>
Message-ID: <CABvwWn+TevDAn6bDLRbSg+U-ZNLG7u5yY1ctS0_LO5RN5s1xNw@mail.gmail.com>

Amazing, right preconditioning fixes the problem. Thanks a lot!


On Tue, Oct 16, 2018 at 8:31 PM Dave May <dave.mayhem23 at gmail.com> wrote:

>
>
> On Wed, 17 Oct 2018 at 03:15, Weizhuo Wang <weizhuo2 at illinois.edu> wrote:
>
>> I just tried both, neither of them make a difference. I got exactly the
>> same curve with either combination.
>>
>
> Try using right preconditioning.
>
>
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/KSP/KSPSetPCSide.html
>
>
> Use the options:
>
> -ksp_type gmres -ksp_pc_side right -ksp_rtol 1e-12
>
> Or;
>
> -ksp_type fgmres  -ksp_rtol 1e-12
>
> Fgmres does right preconditioning my default
>
>
>
>> Thanks!
>>
>> Wang weizhuo
>>
>> On Tue, Oct 16, 2018 at 8:06 PM Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Oct 16, 2018 at 7:26 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>> wrote:
>>>
>>>> Hello again!
>>>>
>>>> After some tweaking the code is giving right answers now. However it
>>>> start to disagree with MATLAB results ('traditional' way using matrix
>>>> inverse) when the grid is larger than 100*100. My PhD advisor and I
>>>> suspects that the default dimension of the Krylov subspace is 100 in the
>>>> test case we are running. If so, is there a way to increase the size of the
>>>> subspace?
>>>>
>>>
>>> 1) The default subspace size is 30, not 100. You can increase the
>>> subspace size using
>>>
>>>        -ksp_gmres_restart n
>>>
>>> 2) The problem is likely your tolerance. The default solver tolerance is
>>> 1e-5. You can change it using
>>>
>>>        -ksp_rtol 1e-9
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>> [image: Disagrees.png]
>>>>
>>>> Thanks!
>>>>
>>>> Wang Weizhuo
>>>>
>>>> On Tue, Oct 9, 2018 at 2:50 AM Mark Adams <mfadams at lbl.gov> wrote:
>>>>
>>>>> To reiterate what Matt is saying, you seem to have the exact solution
>>>>> on a 10x10 grid. That makes no sense unless the solution can be represented
>>>>> exactly by your FE space (eg, u(x,y) = x + y).
>>>>>
>>>>> On Mon, Oct 8, 2018 at 9:33 PM Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Oct 8, 2018 at 9:28 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> The code is attached in case anyone wants to take a look, I will try
>>>>>>> the high frequency scenario later.
>>>>>>>
>>>>>>
>>>>>> That is not the error. It is superconvergence at the vertices. The
>>>>>> real solution is trigonometric, so your
>>>>>> linear interpolants or whatever you use is not going to get the right
>>>>>> value in between mesh points. You
>>>>>> need to do a real integral over the whole interval to get the L_2
>>>>>> error.
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> On Mon, Oct 8, 2018 at 7:58 PM Mark Adams <mfadams at lbl.gov> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Oct 8, 2018 at 6:58 PM Weizhuo Wang <weizhuo2 at illinois.edu>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> The first plot is the norm with the flag -pc_type lu with respect
>>>>>>>>> to number of grids in one axis (n), and the second plot is the norm without
>>>>>>>>> the flag -pc_type lu.
>>>>>>>>>
>>>>>>>>
>>>>>>>> So you are using the default PC w/o LU. The default is ILU. This
>>>>>>>> will reduce high frequency effectively but is not effective on the low
>>>>>>>> frequency error. Don't expect your algebraic error reduction to be at the
>>>>>>>> same scale as the residual reduction (what KSP measures).
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Wang Weizhuo
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cse.buffalo.edu_-7Eknepley_&d=DwMFaQ&c=OCIEmEwdEq_aNlsP4fF3gFqSN-E3mlr2t9JcDdfOZag&r=hsLktHsuxNfF6zyuWGCN8x-6ghPYxhx4cV62Hya47oo&m=EFM29ATgv4U8PjXEtfgMkuxKr5DGscMlH-j769W5W_4&s=grgSL2LaDCthvYvvFITmeOOWPCwgmNfYRPs94N8kmOs&e=>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Wang Weizhuo
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Wang Weizhuo
>>
>

-- 
Wang Weizhuo
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/7f0f655d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Disagrees.png
Type: image/png
Size: 81967 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/7f0f655d/attachment-0001.png>

From andrewh0 at uw.edu  Mon Oct 22 18:43:21 2018
From: andrewh0 at uw.edu (Andrew Ho)
Date: Mon, 22 Oct 2018 16:43:21 -0700
Subject: [petsc-users] Shell Matrix Operations required for KSP solvers?
Message-ID: <CADhXwguPc8-advDNkOA+VaQPr=EfM27=bg+p17e2tT_wz3VfxQ@mail.gmail.com>

I have a specialized matrix structure I'm trying to take advantage of for
solving large scale (non)linear systems. I think for this purpose using a
Shell matrix is sufficient for interfacing with PETSc's KSP linear solvers.

Looking at the examples which use shell matrices, it seems most only
require implementing MatMult, and sometimes MatMultTranspose. Is there a
list of what operations are required (or optional but good to have) for the
different KSP solver types? This is specifically for the KSP solve itself,
not constructing the actual matrix. I'd also be interested if any of the
required/optional operations changes if preconditioners (left and/or right)
are used.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/3dd8f97f/attachment.html>

From knepley at gmail.com  Mon Oct 22 19:22:42 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 22 Oct 2018 20:22:42 -0400
Subject: [petsc-users] Shell Matrix Operations required for KSP solvers?
In-Reply-To: <CADhXwguPc8-advDNkOA+VaQPr=EfM27=bg+p17e2tT_wz3VfxQ@mail.gmail.com>
References: <CADhXwguPc8-advDNkOA+VaQPr=EfM27=bg+p17e2tT_wz3VfxQ@mail.gmail.com>
Message-ID: <CAMYG4G=-sCkyjqGOiJPZa6SY8Pfv4NgTFpXD-ZP0rhzV3+JE4g@mail.gmail.com>

On Mon, Oct 22, 2018 at 7:44 PM Andrew Ho <andrewh0 at uw.edu> wrote:

> I have a specialized matrix structure I'm trying to take advantage of for
> solving large scale (non)linear systems. I think for this purpose using a
> Shell matrix is sufficient for interfacing with PETSc's KSP linear solvers.
>
> Looking at the examples which use shell matrices, it seems most only
> require implementing MatMult, and sometimes MatMultTranspose. Is there a
> list of what operations are required (or optional but good to have) for the
> different KSP solver types? This is specifically for the KSP solve itself,
> not constructing the actual matrix. I'd also be interested if any of the
> required/optional operations changes if preconditioners (left and/or right)
> are used.
>

There is no list, but its hard to think of another operation KSP would ask
for. Preconditioners are another story unfortunately. They often want
explicit access to matrix entries. its really unusual for KSPs to work
without a good preconditioner (the notable exception being well-conditioned
systems like some boundary integral operators).

  Thanks,

    Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181022/0ec506ff/attachment.html>

From Peterspy at outlook.com  Mon Oct 22 20:55:46 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Tue, 23 Oct 2018 01:55:46 +0000
Subject: [petsc-users] Searching for a suitable example case for parallel
 linear solver
Message-ID: <HK0PR04MB2547896236632B054A36C50ED0F50@HK0PR04MB2547.apcprd04.prod.outlook.com>

Hi all,
I am a new user of PETSC, and I am trying to use KSP to solve a 3D Poisson problem. The 3D Poisson problem is differenced by myself, which means that matrix A is given already.
I have found several examples in KSP tutorials but I want to know which one is suitable for me in ?ex2f; ex6f; ex15f; ex21f; ex44f; ex45f?
Thanks in advance.
Regards,
Pengyang

----------------------------------------------------------------
Pengyang Song?????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86) 15762264643

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/e9feb713/attachment.html>

From jed at jedbrown.org  Mon Oct 22 22:01:38 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 22 Oct 2018 21:01:38 -0600
Subject: [petsc-users] Searching for a suitable example case for
 parallel linear solver
In-Reply-To: <HK0PR04MB2547896236632B054A36C50ED0F50@HK0PR04MB2547.apcprd04.prod.outlook.com>
References: <HK0PR04MB2547896236632B054A36C50ED0F50@HK0PR04MB2547.apcprd04.prod.outlook.com>
Message-ID: <87woq9pdrh.fsf@jedbrown.org>

If you have a stored matrix (strongly discouraged as a workflow, but
handy for tests), you can run ksp/ksp/examples/tutorials/ex10.c (no need
to edit code).  If you have a structured grid and want a fast solver,
you should consider using geometric multigrid; see
src/ksp/ksp/examples/tutorials/ex22f.F90 which has that capability.

SONG Pengyang <Peterspy at outlook.com> writes:

> Hi all,
> I am a new user of PETSC, and I am trying to use KSP to solve a 3D Poisson problem. The 3D Poisson problem is differenced by myself, which means that matrix A is given already.
> I have found several examples in KSP tutorials but I want to know which one is suitable for me in ?ex2f; ex6f; ex15f; ex21f; ex44f; ex45f?
> Thanks in advance.
> Regards,
> Pengyang
>
> ----------------------------------------------------------------
> Pengyang Song?????
> MSc candidate in Physical Oceanography
> College of Oceanic and Atmospheric Sciences
> Ocean University of China
> No. 238 Songling Road, Qingdao 266100, Shandong, China
> TEL: (+86) 15762264643

From ysjosh.lo at gmail.com  Tue Oct 23 02:11:32 2018
From: ysjosh.lo at gmail.com (Josh L)
Date: Tue, 23 Oct 2018 02:11:32 -0500
Subject: [petsc-users] DMPlex, output value on gauss point to vertex
Message-ID: <CA+2ebbDr-arsQbY0=29gusK0O7QdEe=mjDv7=aDEu_iSd+z1KQ@mail.gmail.com>

Hi,

My FEM codes calculate the stress field on gauss point, but when I output
the stress field, I want it to on vertex. My old codes find the gauss
points that surround a vertex and calculate the average.
Is there any better way to do it with DMPlex?


The PetscSF given by DMPlexDistribute has the info how mesh points are
distributed.
Can I create a mapping from it or it can be used to map mesh point in
natural ordering to global ordering or the other way around?

Thanks,
Josh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/0c0d6a54/attachment.html>

From dave.mayhem23 at gmail.com  Tue Oct 23 02:27:38 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Tue, 23 Oct 2018 09:27:38 +0200
Subject: [petsc-users] Shell Matrix Operations required for KSP solvers?
In-Reply-To: <CAMYG4G=-sCkyjqGOiJPZa6SY8Pfv4NgTFpXD-ZP0rhzV3+JE4g@mail.gmail.com>
References: <CADhXwguPc8-advDNkOA+VaQPr=EfM27=bg+p17e2tT_wz3VfxQ@mail.gmail.com>
	<CAMYG4G=-sCkyjqGOiJPZa6SY8Pfv4NgTFpXD-ZP0rhzV3+JE4g@mail.gmail.com>
Message-ID: <CAJ98EDp6HX04HjfyhahupC2wHK3-TksnW1WsO9frBdzXf11e5A@mail.gmail.com>

On Tue, 23 Oct 2018 at 02:24, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 22, 2018 at 7:44 PM Andrew Ho <andrewh0 at uw.edu> wrote:
>
>> I have a specialized matrix structure I'm trying to take advantage of for
>> solving large scale (non)linear systems. I think for this purpose using a
>> Shell matrix is sufficient for interfacing with PETSc's KSP linear solvers.
>>
>> Looking at the examples which use shell matrices, it seems most only
>> require implementing MatMult, and sometimes MatMultTranspose. Is there a
>> list of what operations are required (or optional but good to have) for the
>> different KSP solver types? This is specifically for the KSP solve itself,
>> not constructing the actual matrix. I'd also be interested if any of the
>> required/optional operations changes if preconditioners (left and/or right)
>> are used.
>>
>
> There is no list, but its hard to think of another operation KSP would ask
> for. Preconditioners are another story unfortunately. They often want
> explicit access to matrix entries. its really unusual for KSPs to work
> without a good preconditioner (the notable exception being well-conditioned
> systems like some boundary integral operators).
>

The most basic PC is Jacobi - for that to work you need to implement
MatGetDiagonal.

Thanks,
  Dave



>   Thanks,
>
>     Matt
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/86c190aa/attachment-0001.html>

From yann.jobic at univ-amu.fr  Tue Oct 23 02:56:16 2018
From: yann.jobic at univ-amu.fr (Yann Jobic)
Date: Tue, 23 Oct 2018 09:56:16 +0200
Subject: [petsc-users] DMPlex, output value on gauss point to vertex
In-Reply-To: <CA+2ebbDr-arsQbY0=29gusK0O7QdEe=mjDv7=aDEu_iSd+z1KQ@mail.gmail.com>
References: <CA+2ebbDr-arsQbY0=29gusK0O7QdEe=mjDv7=aDEu_iSd+z1KQ@mail.gmail.com>
Message-ID: <cccc3dde-80c5-279f-27cd-d4a19938e8c7@univ-amu.fr>

Hi,

You may want to look at :
https://www.mcs.anl.gov/petsc/petsc-current/include/petsc/private/petscfeimpl.h.html#PetscFEInterpolate_Static
It only gives the structure of what you are looking for.

In order to use it, you have to use the FE implementation of Petsc, by 
looking at the examples.

I hope it helps,

Yann


Le 23/10/2018 ? 09:11, Josh L a ?crit?:
> Hi,
>
> My FEM codes calculate the stress field on gauss point, but when I 
> output the stress field, I want it to on vertex. My old codes find the 
> gauss points that surround a vertex and calculate the average.
> Is there any better way to do it with DMPlex?
>
>
> The PetscSF given by DMPlexDistribute has the info how mesh points are 
> distributed.
> Can I create a mapping from it or it can be used to map mesh point in 
> natural ordering to global ordering or the other way around?
>
> Thanks,
> Josh
>

-- 
___________________________

Yann JOBIC
HPC engineer
IUSTI-CNRS UMR 7343 - Polytech Marseille
Technop?le de Ch?teau Gombert
5 rue Enrico Fermi
13453 Marseille cedex 13
Tel : (33) 4 91 10 69 43
Fax : (33) 4 91 10 69 69


From jroman at dsic.upv.es  Tue Oct 23 03:43:32 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Oct 2018 10:43:32 +0200
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
	<98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
Message-ID: <701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>

If eigenvalues are complex then NLEIGS also needs to work in complex arithmetic because it needs a region of the complex plane containing the wanted eigenvalues. It seems that complex arithmetic is the only change in your problem.

Jose


> El 22 oct 2018, a las 22:01, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
> 
> Thanks, Jose. 
> 
> How difficult would it be to add the support for the general case (if at all possible)? 
> 
> My eigenvalue problem is of the form shown in the attachment. Beta is the eigenvalue and X_s^\Delta is the eigenvector. While some of the matrices are known, others are defined only as matrix vector products. 
> 
> I am interested in eigenvalues with the largest real part. I expect to find complex eigenvalues, although for a small subset of cases these will be real. 
> 
> What is your recommendation for attacking this problem with the nonlinear eigenvalue support in Slepc? 
> 
> Would appreciate your guidance. 
> 
> Regards,
> Manav
> 
> <PastedGraphic-1.pdf>
> 
> 
>> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>> 
>> 
>> 
>>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>> 
>>> Hi, 
>>> 
>>>  I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>>> 
>>>  From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>>> 
>>>  Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>>> 
>>> Regards,
>>> Manav
>>> 
>>> 
>> 
>> Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.
>> 
>> Jose
> 


From amfoggia at gmail.com  Tue Oct 23 05:13:36 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Tue, 23 Oct 2018 12:13:36 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
Message-ID: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>

Hello,

I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real
eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are
the only options I set for the solver. My aim is to be able to
predict/estimate the time-to-solution. To do so, I was doing a scaling of
the code for different sizes of matrices and for different number of MPI
processes. As I was not observing a good scaling I checked the number of
iterations of the solver (given by EPSGetIterationNumber). I've encounter
that for the **same size** of matrix (that meaning, the same problem), when
I change the number of MPI processes, the amount of iterations changes, and
the behaviour is not monotonic. This are the numbers I've got:

# procs   # iters
960          157
992          189
1024        338
1056        190
1120        174
2048        136

I've checked the mailing list for a similar situation and I've found
another person with the same problem but in another solver ("[SLEPc] GD is
not deterministic when using different number of cores", Nov 19 2015), but
I think the solution this person finds does not apply to my problem
(removing "-eps_harmonic" option).

Can you give me any hint on what is the reason for this behaviour? Is there
a way to prevent this? It's not possible to estimate/predict any time
consumption for bigger problems if the number of iterations varies this
much.

Ale
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/cffc6b84/attachment.html>

From jroman at dsic.upv.es  Tue Oct 23 05:59:29 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Oct 2018 12:59:29 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
Message-ID: <4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>

There is an undocumented option:

  -bv_reproducible_random

It will force the initial vector of the Krylov subspace to be the same irrespective of the number of MPI processes. This should be used for scaling analyses as the one you are trying to do.

An additional comment is that we strongly recommend to use the default solver (Krylov-Schur), which will do Lanczos with implicit restart. It is generally faster and more stable.

Jose



> El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> Hello, 
> 
> I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are the only options I set for the solver. My aim is to be able to predict/estimate the time-to-solution. To do so, I was doing a scaling of the code for different sizes of matrices and for different number of MPI processes. As I was not observing a good scaling I checked the number of iterations of the solver (given by EPSGetIterationNumber). I've encounter that for the **same size** of matrix (that meaning, the same problem), when I change the number of MPI processes, the amount of iterations changes, and the behaviour is not monotonic. This are the numbers I've got:
> 
> # procs   # iters
> 960          157
> 992          189
> 1024        338
> 1056        190
> 1120        174
> 2048        136
> 
> I've checked the mailing list for a similar situation and I've found another person with the same problem but in another solver ("[SLEPc] GD is not deterministic when using different number of cores", Nov 19 2015), but I think the solution this person finds does not apply to my problem (removing "-eps_harmonic" option).
> 
> Can you give me any hint on what is the reason for this behaviour? Is there a way to prevent this? It's not possible to estimate/predict any time consumption for bigger problems if the number of iterations varies this much.
> 
> Ale


From knepley at gmail.com  Tue Oct 23 06:53:27 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Oct 2018 07:53:27 -0400
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
Message-ID: <CAMYG4G=msJn9eQ=f_o8LLRwiU=JXPt8n+Xo43zUjxxDNtBh-kQ@mail.gmail.com>

On Tue, Oct 23, 2018 at 6:24 AM Ale Foggia <amfoggia at gmail.com> wrote:

> Hello,
>
> I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real
> eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are
> the only options I set for the solver. My aim is to be able to
> predict/estimate the time-to-solution. To do so, I was doing a scaling of
> the code for different sizes of matrices and for different number of MPI
> processes. As I was not observing a good scaling I checked the number of
> iterations of the solver (given by EPSGetIterationNumber). I've encounter
> that for the **same size** of matrix (that meaning, the same problem), when
> I change the number of MPI processes, the amount of iterations changes, and
> the behaviour is not monotonic. This are the numbers I've got:
>

I am sure you know this, but this test is strong scaling and will top out
when the individual problem sizes become too small (we see this at several
thousand unknowns).

  Thanks,

    Matt


>
> # procs   # iters
> 960          157
> 992          189
> 1024        338
> 1056        190
> 1120        174
> 2048        136
>
> I've checked the mailing list for a similar situation and I've found
> another person with the same problem but in another solver ("[SLEPc] GD is
> not deterministic when using different number of cores", Nov 19 2015), but
> I think the solution this person finds does not apply to my problem
> (removing "-eps_harmonic" option).
>
> Can you give me any hint on what is the reason for this behaviour? Is
> there a way to prevent this? It's not possible to estimate/predict any time
> consumption for bigger problems if the number of iterations varies this
> much.
>
> Ale
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/38a1c9d1/attachment.html>

From amfoggia at gmail.com  Tue Oct 23 08:17:38 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Tue, 23 Oct 2018 15:17:38 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
Message-ID: <CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>

Hello Jose, thanks for your answer.

El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

> There is an undocumented option:
>
>   -bv_reproducible_random
>
> It will force the initial vector of the Krylov subspace to be the same
> irrespective of the number of MPI processes. This should be used for
> scaling analyses as the one you are trying to do.
>

What about when I'm not doing the scaling? Now I would like to ask for
computing time for bigger size problems, should I also use this option in
that case? Because, what happens if I have a "bad" configuration? Meaning,
I ask for some time, enough if I take into account the "correct" scaling,
but when I run it takes double the time/iterations, like it happened before
when changing from 960 to 1024 processes?

>
> An additional comment is that we strongly recommend to use the default
> solver (Krylov-Schur), which will do Lanczos with implicit restart. It is
> generally faster and more stable.
>

I will be doing Dynamical Lanczos, that means that I'll need the "matrix
whose rows are the eigenvectors of the tridiagonal matrix" (so, according
to the Lanczos Technical Report notation, I need the "matrix whose rows are
the eigenvectors of T_m", which should be the same as the vectors y_i). I
checked the Technical Report for Krylov-Schur also and I think I can get
the same information also from that solver, but I'm not sure. Can you
confirm this please?
Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the
notation on the Report), my idea to get them was to retrieve the invariant
subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply
it with the eigenvectors that I get with EPSGetEigenvector. Is there
another easier (or with less computations) way to get this?


> Jose
>
>
>
> > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> > Hello,
> >
> > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real
> eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are
> the only options I set for the solver. My aim is to be able to
> predict/estimate the time-to-solution. To do so, I was doing a scaling of
> the code for different sizes of matrices and for different number of MPI
> processes. As I was not observing a good scaling I checked the number of
> iterations of the solver (given by EPSGetIterationNumber). I've encounter
> that for the **same size** of matrix (that meaning, the same problem), when
> I change the number of MPI processes, the amount of iterations changes, and
> the behaviour is not monotonic. This are the numbers I've got:
> >
> > # procs   # iters
> > 960          157
> > 992          189
> > 1024        338
> > 1056        190
> > 1120        174
> > 2048        136
> >
> > I've checked the mailing list for a similar situation and I've found
> another person with the same problem but in another solver ("[SLEPc] GD is
> not deterministic when using different number of cores", Nov 19 2015), but
> I think the solution this person finds does not apply to my problem
> (removing "-eps_harmonic" option).
> >
> > Can you give me any hint on what is the reason for this behaviour? Is
> there a way to prevent this? It's not possible to estimate/predict any time
> consumption for bigger problems if the number of iterations varies this
> much.
> >
> > Ale
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/0d5ee144/attachment-0001.html>

From jroman at dsic.upv.es  Tue Oct 23 08:33:39 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Oct 2018 15:33:39 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
Message-ID: <F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>



> El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> Hello Jose, thanks for your answer.
> 
> El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> There is an undocumented option:
> 
>   -bv_reproducible_random
> 
> It will force the initial vector of the Krylov subspace to be the same irrespective of the number of MPI processes. This should be used for scaling analyses as the one you are trying to do.
> 
> What about when I'm not doing the scaling? Now I would like to ask for computing time for bigger size problems, should I also use this option in that case? Because, what happens if I have a "bad" configuration? Meaning, I ask for some time, enough if I take into account the "correct" scaling, but when I run it takes double the time/iterations, like it happened before when changing from 960 to 1024 processes?

When you increase the matrix size the spectrum of the matrix changes and probably also the convergence, so the computation time is not easy to predict in advance.

> 
> An additional comment is that we strongly recommend to use the default solver (Krylov-Schur), which will do Lanczos with implicit restart. It is generally faster and more stable.
> 
> I will be doing Dynamical Lanczos, that means that I'll need the "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so, according to the Lanczos Technical Report notation, I need the "matrix whose rows are the eigenvectors of T_m", which should be the same as the vectors y_i). I checked the Technical Report for Krylov-Schur also and I think I can get the same information also from that solver, but I'm not sure. Can you confirm this please? 
> Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the notation on the Report), my idea to get them was to retrieve the invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply it with the eigenvectors that I get with EPSGetEigenvector. Is there another easier (or with less computations) way to get this?

In Krylov-Schur the tridiagonal matrix T_m becomes arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The relevant information can be obtained with EPSGetBV() and EPSGetDS(). But this is a "developer level" interface. We could help you get this running. Send a small problem matrix to slepc-maint together with a more detailed description of what you need to compute.

Jose

> 
> 
> Jose
> 
> 
> 
> > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> > 
> > Hello, 
> > 
> > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are the only options I set for the solver. My aim is to be able to predict/estimate the time-to-solution. To do so, I was doing a scaling of the code for different sizes of matrices and for different number of MPI processes. As I was not observing a good scaling I checked the number of iterations of the solver (given by EPSGetIterationNumber). I've encounter that for the **same size** of matrix (that meaning, the same problem), when I change the number of MPI processes, the amount of iterations changes, and the behaviour is not monotonic. This are the numbers I've got:
> > 
> > # procs   # iters
> > 960          157
> > 992          189
> > 1024        338
> > 1056        190
> > 1120        174
> > 2048        136
> > 
> > I've checked the mailing list for a similar situation and I've found another person with the same problem but in another solver ("[SLEPc] GD is not deterministic when using different number of cores", Nov 19 2015), but I think the solution this person finds does not apply to my problem (removing "-eps_harmonic" option).
> > 
> > Can you give me any hint on what is the reason for this behaviour? Is there a way to prevent this? It's not possible to estimate/predict any time consumption for bigger problems if the number of iterations varies this much.
> > 
> > Ale
> 


From amfoggia at gmail.com  Tue Oct 23 08:46:16 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Tue, 23 Oct 2018 15:46:16 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
Message-ID: <CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>

El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

>
>
> > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> > Hello Jose, thanks for your answer.
> >
> > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> > There is an undocumented option:
> >
> >   -bv_reproducible_random
> >
> > It will force the initial vector of the Krylov subspace to be the same
> irrespective of the number of MPI processes. This should be used for
> scaling analyses as the one you are trying to do.
> >
> > What about when I'm not doing the scaling? Now I would like to ask for
> computing time for bigger size problems, should I also use this option in
> that case? Because, what happens if I have a "bad" configuration? Meaning,
> I ask for some time, enough if I take into account the "correct" scaling,
> but when I run it takes double the time/iterations, like it happened before
> when changing from 960 to 1024 processes?
>
> When you increase the matrix size the spectrum of the matrix changes and
> probably also the convergence, so the computation time is not easy to
> predict in advance.
>

Okey, I'll keep that in mine. I thought that, even if the spectrum changes,
if I had a behaviour/tendency for 6 or 7 smaller cases I could predict more
or less the time. It was working this way until I found this "iterations
problem" which doubled the time of execution for the same size problem. To
be completely sure, do you suggest me or not to use this run-time option
when going in production? Can you elaborate a bit in the effect this
option? Is the (huge) difference I got in the number of iterations
something expected?


> >
> > An additional comment is that we strongly recommend to use the default
> solver (Krylov-Schur), which will do Lanczos with implicit restart. It is
> generally faster and more stable.
> >
> > I will be doing Dynamical Lanczos, that means that I'll need the "matrix
> whose rows are the eigenvectors of the tridiagonal matrix" (so, according
> to the Lanczos Technical Report notation, I need the "matrix whose rows are
> the eigenvectors of T_m", which should be the same as the vectors y_i). I
> checked the Technical Report for Krylov-Schur also and I think I can get
> the same information also from that solver, but I'm not sure. Can you
> confirm this please?
> > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the
> notation on the Report), my idea to get them was to retrieve the invariant
> subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply
> it with the eigenvectors that I get with EPSGetEigenvector. Is there
> another easier (or with less computations) way to get this?
>
> In Krylov-Schur the tridiagonal matrix T_m becomes
> arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The
> relevant information can be obtained with EPSGetBV() and EPSGetDS(). But
> this is a "developer level" interface. We could help you get this running.
> Send a small problem matrix to slepc-maint together with a more detailed
> description of what you need to compute.
>

Thanks! When I get to that part I'll write to slepc-maint for help.


> Jose
>
> >
> >
> > Jose
> >
> >
> >
> > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> > >
> > > Hello,
> > >
> > > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest
> real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those
> are the only options I set for the solver. My aim is to be able to
> predict/estimate the time-to-solution. To do so, I was doing a scaling of
> the code for different sizes of matrices and for different number of MPI
> processes. As I was not observing a good scaling I checked the number of
> iterations of the solver (given by EPSGetIterationNumber). I've encounter
> that for the **same size** of matrix (that meaning, the same problem), when
> I change the number of MPI processes, the amount of iterations changes, and
> the behaviour is not monotonic. This are the numbers I've got:
> > >
> > > # procs   # iters
> > > 960          157
> > > 992          189
> > > 1024        338
> > > 1056        190
> > > 1120        174
> > > 2048        136
> > >
> > > I've checked the mailing list for a similar situation and I've found
> another person with the same problem but in another solver ("[SLEPc] GD is
> not deterministic when using different number of cores", Nov 19 2015), but
> I think the solution this person finds does not apply to my problem
> (removing "-eps_harmonic" option).
> > >
> > > Can you give me any hint on what is the reason for this behaviour? Is
> there a way to prevent this? It's not possible to estimate/predict any time
> consumption for bigger problems if the number of iterations varies this
> much.
> > >
> > > Ale
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/3779f64f/attachment.html>

From bhatiamanav at gmail.com  Tue Oct 23 09:10:12 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Tue, 23 Oct 2018 09:10:12 -0500
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
	<98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
	<701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>
Message-ID: <F0F26ECE-2992-4C50-AFFF-523891712E4B@gmail.com>

Thanks for the clarification. 

Does this also apply to the standard non-hermitian eigenvalue problem? Do I need to compile with complex numbers if I want to capture the complex eigenvalues? Or does it work with real number support? 

Thanks
Manav

Sent from my iPhone

> On Oct 23, 2018, at 3:43 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> If eigenvalues are complex then NLEIGS also needs to work in complex arithmetic because it needs a region of the complex plane containing the wanted eigenvalues. It seems that complex arithmetic is the only change in your problem.
> 
> Jose
> 
> 
>> El 22 oct 2018, a las 22:01, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>> 
>> Thanks, Jose. 
>> 
>> How difficult would it be to add the support for the general case (if at all possible)? 
>> 
>> My eigenvalue problem is of the form shown in the attachment. Beta is the eigenvalue and X_s^\Delta is the eigenvector. While some of the matrices are known, others are defined only as matrix vector products. 
>> 
>> I am interested in eigenvalues with the largest real part. I expect to find complex eigenvalues, although for a small subset of cases these will be real. 
>> 
>> What is your recommendation for attacking this problem with the nonlinear eigenvalue support in Slepc? 
>> 
>> Would appreciate your guidance. 
>> 
>> Regards,
>> Manav
>> 
>> <PastedGraphic-1.pdf>
>> 
>> 
>>> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>> 
>>> 
>>> 
>>>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>>> 
>>>> Hi, 
>>>> 
>>>> I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>>>> 
>>>> From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>>>> 
>>>> Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>>>> 
>>>> Regards,
>>>> Manav
>>>> 
>>>> 
>>> 
>>> Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.
>>> 
>>> Jose
>> 
> 

From knepley at gmail.com  Tue Oct 23 09:40:27 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Oct 2018 10:40:27 -0400
Subject: [petsc-users] DMPlex, output value on gauss point to vertex
In-Reply-To: <CA+2ebbDr-arsQbY0=29gusK0O7QdEe=mjDv7=aDEu_iSd+z1KQ@mail.gmail.com>
References: <CA+2ebbDr-arsQbY0=29gusK0O7QdEe=mjDv7=aDEu_iSd+z1KQ@mail.gmail.com>
Message-ID: <CAMYG4GmxjnssjwD5bywhgt3icK5qt_k9hwnCAMrzYyEddGc1QA@mail.gmail.com>

On Tue, Oct 23, 2018 at 3:12 AM Josh L <ysjosh.lo at gmail.com> wrote:

> Hi,
>
> My FEM codes calculate the stress field on gauss point, but when I output
> the stress field, I want it to on vertex. My old codes find the gauss
> points that surround a vertex and calculate the average.
> Is there any better way to do it with DMPlex?
>

In general, I would project the answer into a finite element space for
output, unless you can think of a reason that this is unacceptable.
You can easily project functions of finite element fields using
DMPlexProjectField(). You would give a pointwise function that would
calculate your stress and everything else should be automatic.


> The PetscSF given by DMPlexDistribute has the info how mesh points are
> distributed.
> Can I create a mapping from it or it can be used to map mesh point in
> natural ordering to global ordering or the other way around?
>

I would really discourage this kind of thinking. This is what made
historical FEM code impossible to use or optimize.

  Thanks,

    Matt


> Thanks,
> Josh
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/71f6d869/attachment.html>

From jroman at dsic.upv.es  Tue Oct 23 09:42:40 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Oct 2018 16:42:40 +0200
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <F0F26ECE-2992-4C50-AFFF-523891712E4B@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
	<98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
	<701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>
	<F0F26ECE-2992-4C50-AFFF-523891712E4B@gmail.com>
Message-ID: <70A96A4A-6607-4CD6-8F3A-0D6071542ECC@dsic.upv.es>



> El 23 oct 2018, a las 16:10, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
> 
> Thanks for the clarification. 
> 
> Does this also apply to the standard non-hermitian eigenvalue problem? Do I need to compile with complex numbers if I want to capture the complex eigenvalues? Or does it work with real number support? 

No, linear eigenproblems (EPS) can be solved with real scalars for complex eigenvalues, but nonlinear eigenproblems (NEP) cannot.

Jose

> 
> Thanks
> Manav
> 
> Sent from my iPhone
> 
>> On Oct 23, 2018, at 3:43 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>> 
>> If eigenvalues are complex then NLEIGS also needs to work in complex arithmetic because it needs a region of the complex plane containing the wanted eigenvalues. It seems that complex arithmetic is the only change in your problem.
>> 
>> Jose
>> 
>> 
>>> El 22 oct 2018, a las 22:01, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>> 
>>> Thanks, Jose. 
>>> 
>>> How difficult would it be to add the support for the general case (if at all possible)? 
>>> 
>>> My eigenvalue problem is of the form shown in the attachment. Beta is the eigenvalue and X_s^\Delta is the eigenvector. While some of the matrices are known, others are defined only as matrix vector products. 
>>> 
>>> I am interested in eigenvalues with the largest real part. I expect to find complex eigenvalues, although for a small subset of cases these will be real. 
>>> 
>>> What is your recommendation for attacking this problem with the nonlinear eigenvalue support in Slepc? 
>>> 
>>> Would appreciate your guidance. 
>>> 
>>> Regards,
>>> Manav
>>> 
>>> <PastedGraphic-1.pdf>
>>> 
>>> 
>>>> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>>> 
>>>> 
>>>> 
>>>>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>>>> 
>>>>> Hi, 
>>>>> 
>>>>> I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>>>>> 
>>>>> From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>>>>> 
>>>>> Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>>>>> 
>>>>> Regards,
>>>>> Manav
>>>>> 
>>>>> 
>>>> 
>>>> Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.
>>>> 
>>>> Jose
>>> 
>> 


From jroman at dsic.upv.es  Tue Oct 23 09:48:43 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 23 Oct 2018 16:48:43 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
Message-ID: <8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>



> El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> 
> 
> El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> 
> 
> > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> > 
> > Hello Jose, thanks for your answer.
> > 
> > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > There is an undocumented option:
> > 
> >   -bv_reproducible_random
> > 
> > It will force the initial vector of the Krylov subspace to be the same irrespective of the number of MPI processes. This should be used for scaling analyses as the one you are trying to do.
> > 
> > What about when I'm not doing the scaling? Now I would like to ask for computing time for bigger size problems, should I also use this option in that case? Because, what happens if I have a "bad" configuration? Meaning, I ask for some time, enough if I take into account the "correct" scaling, but when I run it takes double the time/iterations, like it happened before when changing from 960 to 1024 processes?
> 
> When you increase the matrix size the spectrum of the matrix changes and probably also the convergence, so the computation time is not easy to predict in advance.
> 
> Okey, I'll keep that in mine. I thought that, even if the spectrum changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could predict more or less the time. It was working this way until I found this "iterations problem" which doubled the time of execution for the same size problem. To be completely sure, do you suggest me or not to use this run-time option when going in production? Can you elaborate a bit in the effect this option? Is the (huge) difference I got in the number of iterations something expected?

Ideally if you have a rough approximation of the eigenvector, you set it as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc generates a random initial vector, that is start the search blindly. The difference between using one random vector or another may be large, depending on the problem. Krylov-Schur is usually less sensitive to the initial vector. 

Jose

> 
> 
> > 
> > An additional comment is that we strongly recommend to use the default solver (Krylov-Schur), which will do Lanczos with implicit restart. It is generally faster and more stable.
> > 
> > I will be doing Dynamical Lanczos, that means that I'll need the "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so, according to the Lanczos Technical Report notation, I need the "matrix whose rows are the eigenvectors of T_m", which should be the same as the vectors y_i). I checked the Technical Report for Krylov-Schur also and I think I can get the same information also from that solver, but I'm not sure. Can you confirm this please? 
> > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the notation on the Report), my idea to get them was to retrieve the invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply it with the eigenvectors that I get with EPSGetEigenvector. Is there another easier (or with less computations) way to get this?
> 
> In Krylov-Schur the tridiagonal matrix T_m becomes arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The relevant information can be obtained with EPSGetBV() and EPSGetDS(). But this is a "developer level" interface. We could help you get this running. Send a small problem matrix to slepc-maint together with a more detailed description of what you need to compute.
> 
> Thanks! When I get to that part I'll write to slepc-maint for help.
> 
> 
> Jose
> 
> > 
> > 
> > Jose
> > 
> > 
> > 
> > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > 
> > > Hello, 
> > > 
> > > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are the only options I set for the solver. My aim is to be able to predict/estimate the time-to-solution. To do so, I was doing a scaling of the code for different sizes of matrices and for different number of MPI processes. As I was not observing a good scaling I checked the number of iterations of the solver (given by EPSGetIterationNumber). I've encounter that for the **same size** of matrix (that meaning, the same problem), when I change the number of MPI processes, the amount of iterations changes, and the behaviour is not monotonic. This are the numbers I've got:
> > > 
> > > # procs   # iters
> > > 960          157
> > > 992          189
> > > 1024        338
> > > 1056        190
> > > 1120        174
> > > 2048        136
> > > 
> > > I've checked the mailing list for a similar situation and I've found another person with the same problem but in another solver ("[SLEPc] GD is not deterministic when using different number of cores", Nov 19 2015), but I think the solution this person finds does not apply to my problem (removing "-eps_harmonic" option).
> > > 
> > > Can you give me any hint on what is the reason for this behaviour? Is there a way to prevent this? It's not possible to estimate/predict any time consumption for bigger problems if the number of iterations varies this much.
> > > 
> > > Ale
> > 


From bhatiamanav at gmail.com  Tue Oct 23 09:51:51 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Tue, 23 Oct 2018 09:51:51 -0500
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <70A96A4A-6607-4CD6-8F3A-0D6071542ECC@dsic.upv.es>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
	<98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
	<701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>
	<F0F26ECE-2992-4C50-AFFF-523891712E4B@gmail.com>
	<70A96A4A-6607-4CD6-8F3A-0D6071542ECC@dsic.upv.es>
Message-ID: <3814CB35-A7AF-48D1-8C5E-52BAA09E4B54@gmail.com>

Really interesting! 

So this is a limitation of the algorithm and not the implementation. 

The challenge is that the eigenvalue solution  in my workflow is a small component of a large computation done with real numbers in an optimization problem. I could do the whole thing with complex numbers, but would be wasting a lot of cpu cycles doing that. 

Do you know if there is anyway to link to both real and complex versions of the library or switch between them at runtime? 

-Manav

Sent from my iPhone

> On Oct 23, 2018, at 9:42 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> 
> 
>> El 23 oct 2018, a las 16:10, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>> 
>> Thanks for the clarification. 
>> 
>> Does this also apply to the standard non-hermitian eigenvalue problem? Do I need to compile with complex numbers if I want to capture the complex eigenvalues? Or does it work with real number support? 
> 
> No, linear eigenproblems (EPS) can be solved with real scalars for complex eigenvalues, but nonlinear eigenproblems (NEP) cannot.
> 
> Jose
> 
>> 
>> Thanks
>> Manav
>> 
>> Sent from my iPhone
>> 
>>> On Oct 23, 2018, at 3:43 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>> 
>>> If eigenvalues are complex then NLEIGS also needs to work in complex arithmetic because it needs a region of the complex plane containing the wanted eigenvalues. It seems that complex arithmetic is the only change in your problem.
>>> 
>>> Jose
>>> 
>>> 
>>>> El 22 oct 2018, a las 22:01, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>>> 
>>>> Thanks, Jose. 
>>>> 
>>>> How difficult would it be to add the support for the general case (if at all possible)? 
>>>> 
>>>> My eigenvalue problem is of the form shown in the attachment. Beta is the eigenvalue and X_s^\Delta is the eigenvector. While some of the matrices are known, others are defined only as matrix vector products. 
>>>> 
>>>> I am interested in eigenvalues with the largest real part. I expect to find complex eigenvalues, although for a small subset of cases these will be real. 
>>>> 
>>>> What is your recommendation for attacking this problem with the nonlinear eigenvalue support in Slepc? 
>>>> 
>>>> Would appreciate your guidance. 
>>>> 
>>>> Regards,
>>>> Manav
>>>> 
>>>> <PastedGraphic-1.pdf>
>>>> 
>>>> 
>>>>> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es> wrote:
>>>>> 
>>>>> 
>>>>> 
>>>>>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com> escribi?:
>>>>>> 
>>>>>> Hi, 
>>>>>> 
>>>>>> I am exploring the nonlinear eigenvalue problem solver in Slepc. 
>>>>>> 
>>>>>> From the notes in "Sec 6.4: Retrieving the Solution?, it appears that if I expect to find complex eigenpairs then I must compile the library (and Petsc) with complex scalars. Is that correct? 
>>>>>> 
>>>>>> Is there a way to include support for complex eigenpairs in a library complied with real scalars? 
>>>>>> 
>>>>>> Regards,
>>>>>> Manav
>>>>>> 
>>>>>> 
>>>>> 
>>>>> Currently, the only combination that supports complex eigenpairs with real scalars is the split form for the nonlinear function with the NLEIGS solver.
>>>>> 
>>>>> Jose
>>>> 
>>> 
> 

From knepley at gmail.com  Tue Oct 23 09:58:25 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Oct 2018 10:58:25 -0400
Subject: [petsc-users] Slepc: Nonlinear eigenvalue problem
In-Reply-To: <3814CB35-A7AF-48D1-8C5E-52BAA09E4B54@gmail.com>
References: <61B9AB6C-C7AE-4E6A-990D-9409AA3CA0A9@gmail.com>
	<B43BABC6-D7E5-41F1-8E64-74171C95386F@dsic.upv.es>
	<98D6E03D-2804-47EA-94CF-219B3A2CD9DE@gmail.com>
	<701859C7-BC17-4121-929B-579DE4285BB7@dsic.upv.es>
	<F0F26ECE-2992-4C50-AFFF-523891712E4B@gmail.com>
	<70A96A4A-6607-4CD6-8F3A-0D6071542ECC@dsic.upv.es>
	<3814CB35-A7AF-48D1-8C5E-52BAA09E4B54@gmail.com>
Message-ID: <CAMYG4G=SMJaOjas807mVwQOzg1KPjBadeyC7r2uRF4aPTn-jtw@mail.gmail.com>

On Tue, Oct 23, 2018 at 10:53 AM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Really interesting!
>
> So this is a limitation of the algorithm and not the implementation.
>
> The challenge is that the eigenvalue solution  in my workflow is a small
> component of a large computation done with real numbers in an optimization
> problem. I could do the whole thing with complex numbers, but would be
> wasting a lot of cpu cycles doing that.
>
> Do you know if there is anyway to link to both real and complex versions
> of the library or switch between them at runtime?
>

Unfortunately, that is a huge amount of work. One group has done it, but we
do not support it.

  Thanks,

    Matt


> -Manav
>
> Sent from my iPhone
>
> > On Oct 23, 2018, at 9:42 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> >
> >
> >
> >> El 23 oct 2018, a las 16:10, Manav Bhatia <bhatiamanav at gmail.com>
> escribi?:
> >>
> >> Thanks for the clarification.
> >>
> >> Does this also apply to the standard non-hermitian eigenvalue problem?
> Do I need to compile with complex numbers if I want to capture the complex
> eigenvalues? Or does it work with real number support?
> >
> > No, linear eigenproblems (EPS) can be solved with real scalars for
> complex eigenvalues, but nonlinear eigenproblems (NEP) cannot.
> >
> > Jose
> >
> >>
> >> Thanks
> >> Manav
> >>
> >> Sent from my iPhone
> >>
> >>> On Oct 23, 2018, at 3:43 AM, Jose E. Roman <jroman at dsic.upv.es> wrote:
> >>>
> >>> If eigenvalues are complex then NLEIGS also needs to work in complex
> arithmetic because it needs a region of the complex plane containing the
> wanted eigenvalues. It seems that complex arithmetic is the only change in
> your problem.
> >>>
> >>> Jose
> >>>
> >>>
> >>>> El 22 oct 2018, a las 22:01, Manav Bhatia <bhatiamanav at gmail.com>
> escribi?:
> >>>>
> >>>> Thanks, Jose.
> >>>>
> >>>> How difficult would it be to add the support for the general case (if
> at all possible)?
> >>>>
> >>>> My eigenvalue problem is of the form shown in the attachment. Beta is
> the eigenvalue and X_s^\Delta is the eigenvector. While some of the
> matrices are known, others are defined only as matrix vector products.
> >>>>
> >>>> I am interested in eigenvalues with the largest real part. I expect
> to find complex eigenvalues, although for a small subset of cases these
> will be real.
> >>>>
> >>>> What is your recommendation for attacking this problem with the
> nonlinear eigenvalue support in Slepc?
> >>>>
> >>>> Would appreciate your guidance.
> >>>>
> >>>> Regards,
> >>>> Manav
> >>>>
> >>>> <PastedGraphic-1.pdf>
> >>>>
> >>>>
> >>>>> On Oct 22, 2018, at 2:40 PM, Jose E. Roman <jroman at dsic.upv.es>
> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>>> El 22 oct 2018, a las 21:05, Manav Bhatia <bhatiamanav at gmail.com>
> escribi?:
> >>>>>>
> >>>>>> Hi,
> >>>>>>
> >>>>>> I am exploring the nonlinear eigenvalue problem solver in Slepc.
> >>>>>>
> >>>>>> From the notes in "Sec 6.4: Retrieving the Solution?, it appears
> that if I expect to find complex eigenpairs then I must compile the library
> (and Petsc) with complex scalars. Is that correct?
> >>>>>>
> >>>>>> Is there a way to include support for complex eigenpairs in a
> library complied with real scalars?
> >>>>>>
> >>>>>> Regards,
> >>>>>> Manav
> >>>>>>
> >>>>>>
> >>>>>
> >>>>> Currently, the only combination that supports complex eigenpairs
> with real scalars is the split form for the nonlinear function with the
> NLEIGS solver.
> >>>>>
> >>>>> Jose
> >>>>
> >>>
> >
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/b03da3ce/attachment.html>

From yjwu16 at gmail.com  Tue Oct 23 10:35:32 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Tue, 23 Oct 2018 23:35:32 +0800
Subject: [petsc-users] Problems about Compiling Multifile Program
Message-ID: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>

Dear Petsc developer:
Hi,
Thank you very much for your continuous help, I recently encountered some
difficulties in developing programs on Petsc.

1. I want to use my class definition (class1. h) and class functions
(class1. cpp) files in my Petsc program (myprogram. c) and compile my
program. I have written other program before:
g++ -c class1.cpp myprogram.c
g++ -o myprogram *.o
I also have some knowledge of Makefile, but I don't know how to modify it
to achieve compilation. I really want your help.

2.  In my class definition and function implementation program, I want to
use Petsc data structures, such as PetscInt, PetscReal. In order to
maintain consistency of the main program. What kind of header files should
I add to my code, or do I need to add any code?

Thanks,
Yingjie
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/76d536eb/attachment.html>

From knepley at gmail.com  Tue Oct 23 12:46:10 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 23 Oct 2018 13:46:10 -0400
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
Message-ID: <CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>

On Tue, Oct 23, 2018 at 11:37 AM Yingjie Wu <yjwu16 at gmail.com> wrote:

> Dear Petsc developer:
> Hi,
> Thank you very much for your continuous help, I recently encountered some
> difficulties in developing programs on Petsc.
>
> 1. I want to use my class definition (class1. h) and class functions
> (class1. cpp) files in my Petsc program (myprogram. c) and compile my
> program. I have written other program before:
> g++ -c class1.cpp myprogram.c
> g++ -o myprogram *.o
> I also have some knowledge of Makefile, but I don't know how to modify it
> to achieve compilation. I really want your help.
>

With PETSc makefiles, we have automatic rules, so you can use

myprogram: class1.o myprogram.o
       ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}


> 2.  In my class definition and function implementation program, I want to
> use Petsc data structures, such as PetscInt, PetscReal. In order to
> maintain consistency of the main program. What kind of header files should
> I add to my code, or do I need to add any code?
>

The easiest thing to do is just

#include <petsc.h>

  Thanks,

    Matt


> Thanks,
> Yingjie
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/b413c13f/attachment.html>

From overholt at capesim.com  Tue Oct 23 13:29:59 2018
From: overholt at capesim.com (Matthew Overholt)
Date: Tue, 23 Oct 2018 14:29:59 -0400
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
	<CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
Message-ID: <CAMSLWku9HJ+mkj1bdkoFH7bO34WjTh+2Y532vBCdfGhfxfZ2ag@mail.gmail.com>

Here is a sample makefile, like what I use with the Intel MPI compilers
(used during PETSc configuration) and MKL library.

Matt Overholt

---------------------------------------------
#
# makefile for Linux using the Intel C++ Compiler, MKL & MPI libraries +
OpenMP
#   usage: make
#   or: make clean
#
# Although explicit paths are used for all libraries and compilers below,
it is
# still necessary to first call the Intel scripts as follows:
#   if [ -z "$I_MPI_ROOT" ]; then
#     source
/opt/intel/compilers_and_libraries/linux/mpi/intel64/bin/mpivars.sh
#   fi
#   if [ -z "$MKLROOT" ]; then
#     source /opt/intel/parallel_studio_xe_2018/bin/psxevars.sh intel64
#   fi
# Use OMP_NUM_THREADS to set # of threads

### MPI Compilers for C++ and C ###
COMPILER_DIR = /opt/intel/compilers_and_libraries/linux/mpi/intel64/bin
COMPILER = $(COMPILER_DIR)/mpiicpc
CCOMPILER = $(COMPILER_DIR)/mpiicc

### Intel MKL & MPI libraries ###
MKL_DIR = /opt/intel/compilers_and_libraries/linux/mkl
MKL_INC = $(MKL_DIR)/include
MKL_LIB = $(MKL_DIR)/lib/intel64

### Compiling Flags (debug and optimized) ###
# CCFLAGS = -c -O0 -g -traceback -heap-arrays -check=stack,uninit
-I$(MKL_INC) -qopenmp
CCFLAGS = -c -I$(MKL_INC) -qopenmp

### Linking Flags (debug and optimized) ###
# CLFLAGS = -g -lstdc++ -qopenmp
CLFLAGS = -lstdc++ -qopenmp

### MKL Libraries ###
# Sequential threading layer
# MKLFLAGS = -Wl,--start-group \
# ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a \
# ${MKLROOT}/lib/intel64/libmkl_sequential.a \
# ${MKLROOT}/lib/intel64/libmkl_core.a \
# -Wl,--end-group -lpthread -lm -ldl
# OpenMP parallel threading layer
MKLFLAGS = -Wl,--start-group \
$(MKL_LIB)/libmkl_intel_lp64.a \
$(MKL_LIB)/libmkl_intel_thread.a \
$(MKL_LIB)/libmkl_core.a \
$(MKL_LIB)/libmkl_blacs_intelmpi_lp64.a \
-Wl,--end-group -liomp5 -lpthread -lm -ldl

TARGET = myprogram

### Object Files ###
OBJFILES = \
class1.o \
class2.o

all: $(TARGET)

#
# Build (link) it
$(TARGET) : $(OBJFILES)
$(COMPILER) -o $(TARGET) $(OBJFILES) $(CLFLAGS) $(MKLFLAGS)
echo $(TARGET) BUILT AS A 64-BIT LINUX APPLICATION!

#
# Compile the source files with C and CPP MPI compilers
# CPP
class1.o : class1.cpp
$(COMPILER) $(CCFLAGS) class1.cpp

# C
class2.o : class2.c
$(CCOMPILER) $(CCFLAGS) class2.c

# main
myprogram.o : myprogram.c
$(CCOMPILER) $(CCFLAGS) myprogram.c

# to clean up
.PHONY : clean

clean:
-rm -f $(OBJFILES)
-rm -f $(TARGET)
echo myprogram cleaned

---------------------------------------------
On Tue, Oct 23, 2018 at 1:46 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 23, 2018 at 11:37 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Dear Petsc developer:
>> Hi,
>> Thank you very much for your continuous help, I recently encountered some
>> difficulties in developing programs on Petsc.
>>
>> 1. I want to use my class definition (class1. h) and class functions
>> (class1. cpp) files in my Petsc program (myprogram. c) and compile my
>> program. I have written other program before:
>> g++ -c class1.cpp myprogram.c
>> g++ -o myprogram *.o
>> I also have some knowledge of Makefile, but I don't know how to modify it
>> to achieve compilation. I really want your help.
>>
>
> With PETSc makefiles, we have automatic rules, so you can use
>
> myprogram: class1.o myprogram.o
>        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
>
>
>> 2.  In my class definition and function implementation program, I want to
>> use Petsc data structures, such as PetscInt, PetscReal. In order to
>> maintain consistency of the main program. What kind of header files should
>> I add to my code, or do I need to add any code?
>>
>
> The easiest thing to do is just
>
> #include <petsc.h>
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Yingjie
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/9f088e5d/attachment.html>

From overholt at capesim.com  Tue Oct 23 13:33:43 2018
From: overholt at capesim.com (Matthew Overholt)
Date: Tue, 23 Oct 2018 14:33:43 -0400
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <CAMSLWku9HJ+mkj1bdkoFH7bO34WjTh+2Y532vBCdfGhfxfZ2ag@mail.gmail.com>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
	<CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
	<CAMSLWku9HJ+mkj1bdkoFH7bO34WjTh+2Y532vBCdfGhfxfZ2ag@mail.gmail.com>
Message-ID: <CAMSLWkv2bPko9LpjfsCFFKBPFcB45Zs1Rk_S+_Up82jPbGhvAQ@mail.gmail.com>

Correction:

OBJFILES = \
class1.o \
class2.o \
        myprogram.o

Matt Overholt

On Tue, Oct 23, 2018 at 2:29 PM Matthew Overholt <overholt at capesim.com>
wrote:

> Here is a sample makefile, like what I use with the Intel MPI compilers
> (used during PETSc configuration) and MKL library.
>
> Matt Overholt
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/0b0f48ab/attachment.html>

From jed at jedbrown.org  Tue Oct 23 17:12:13 2018
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 23 Oct 2018 16:12:13 -0600
Subject: [petsc-users] Assistant Professorship in Computational
 Earth-Surface Process Modeling at CU Boulder
Message-ID: <871s8g71oi.fsf@jedbrown.org>

The Institute of Arctic and Alpine Research (INSTAAR) and Community
Surface Dynamics Modeling System (CSDMS) at the University of Colorado
invite applications for a tenure-track assistant professor position in
Computational Earth-Surface Process Modeling, with an August 2019
start. CSDMS is an NSF-sponsored facility that supports modeling and
prediction of erosion, transport, and deposition of sediment in
landscapes and sedimentary basins over a broad range of environments and
time and space scales. The candidate will join a team at INSTAAR at the
University of Colorado, Boulder, responsible for promoting the growth of
CSDMS by developing multidisciplinary and international collaboration in
support of earth surface processes modeling. We seek a scientist who
takes advantage of new opportunities in computing and remote sensing to
advance understanding of landscape, coastal, and/or seascape
geomorphology and sediment dynamics. Disciplinary areas of particular
interest include, but are not limited to: modeling of terrestrial,
coastal, cryospheric, and/or marine processes; use of high-resolution
topographic data to test process models; earth surface interactions with
the climate system; modeling human impacts; high-performance computing;
and community scientific software development.

CSDMS is hosted at INSTAAR. The new assistant professor would be
rostered in the Research & Innovation Office (RIO) and also become a
tenure-track professor in one of INSTAAR?s affiliated academic
departments and programs that best suits their training and interests
(Geological Sciences; Civil, Environmental, and Architectural
Engineering; Atmospheric and Ocean Sciences; Computer Science).

A strong commitment to teaching, and mentorship at the undergraduate and
graduate levels, and to outreach activities, is expected. The candidate
is expected to engage in development of coursework in his or her area of
research expertise, and to teach at the undergraduate and graduate level
in the home department. Incorporation of computing skills in
undergraduate and graduate curricula is desirable.

Applicants should have a Ph.D. in geosciences, engineering, or related
field at the time of appointment. We seek a strong team player with
excellent communication and networking skills, who is focused on
achieving multidisciplinary research goals. We welcome candidates who
will bring diverse intellectual, geographical, gender, and ethnic
perspectives to CSDMS and the University of Colorado campus community.

For further information and to apply:

  https://jobs.colorado.edu/jobs/JobDetail/?jobId=13976

From abdullahasivas at gmail.com  Tue Oct 23 18:16:01 2018
From: abdullahasivas at gmail.com (Abdullah Ali Sivas)
Date: Tue, 23 Oct 2018 19:16:01 -0400
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
Message-ID: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>

Dear all,

I have a series of linear systems coming from a PDE for which BDDC is an
optimal preconditioner. These linear systems are assembled and I read them
from a file, then convert into MATIS as required (as in
https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
). I expect each of the systems converge to the solution in almost same
number of iterations but I don't observe it. I think it is because I do not
provide enough information to the preconditioner. I can get a list of inner
dofs and interface dofs. However, I do not know how to use them. Has anyone
have any insights about it or done something similar?

Best wishes,
Abdullah Ali Sivas
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181023/135a19b0/attachment-0001.html>

From jed at jedbrown.org  Tue Oct 23 22:16:22 2018
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 23 Oct 2018 21:16:22 -0600
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
Message-ID: <87sh0w5915.fsf@jedbrown.org>

Did you assemble "Neumann" problems that are compatible with your
definition of interior/interface degrees of freedom?

Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:

> Dear all,
>
> I have a series of linear systems coming from a PDE for which BDDC is an
> optimal preconditioner. These linear systems are assembled and I read them
> from a file, then convert into MATIS as required (as in
> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
> ). I expect each of the systems converge to the solution in almost same
> number of iterations but I don't observe it. I think it is because I do not
> provide enough information to the preconditioner. I can get a list of inner
> dofs and interface dofs. However, I do not know how to use them. Has anyone
> have any insights about it or done something similar?
>
> Best wishes,
> Abdullah Ali Sivas

From abdullahasivas at gmail.com  Wed Oct 24 00:12:35 2018
From: abdullahasivas at gmail.com (Abdullah Ali Sivas)
Date: Wed, 24 Oct 2018 01:12:35 -0400
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <87sh0w5915.fsf@jedbrown.org>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
Message-ID: <CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>

Hi Jed,

Thanks for your reply. The assembled matrix I have corresponds to the full
problem on the full mesh. There are no "Neumann" problems (or any sort of
domain decomposition) defined in the code generates the matrix. However, I
think assembling the full problem is equivalent to implicitly assembling
the "Neumann" problems, since the system can be partitioned as;

[A_{LL} | A_{LI}]  [u_L]     [F]
-----------|------------ -------- = -----
[A_{IL}  |A_{II} ]   [u_I]      [G]

and G should correspond to the Neumann problem. I might be thinking wrong
(or maybe I completely misunderstood the idea), if so please correct me.
But I think that the problem is that I am not explicitly telling PCBDDC
which dofs are interface dofs.

Regards,
Abdullah Ali Sivas

On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:

> Did you assemble "Neumann" problems that are compatible with your
> definition of interior/interface degrees of freedom?
>
> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>
> > Dear all,
> >
> > I have a series of linear systems coming from a PDE for which BDDC is an
> > optimal preconditioner. These linear systems are assembled and I read
> them
> > from a file, then convert into MATIS as required (as in
> >
> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
> > ). I expect each of the systems converge to the solution in almost same
> > number of iterations but I don't observe it. I think it is because I do
> not
> > provide enough information to the preconditioner. I can get a list of
> inner
> > dofs and interface dofs. However, I do not know how to use them. Has
> anyone
> > have any insights about it or done something similar?
> >
> > Best wishes,
> > Abdullah Ali Sivas
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/3281d8c2/attachment.html>

From amfoggia at gmail.com  Wed Oct 24 02:09:40 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Wed, 24 Oct 2018 09:09:40 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
	<8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
Message-ID: <CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>

I've tried the option that you gave me but I still get different number of
iterations when changing the number of MPI processes: I did 960 procs and
1024 procs and I got 435 and 176 iterations, respectively.

El mar., 23 oct. 2018 a las 16:48, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

>
>
> > El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> >
> >
> > El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> >
> >
> > > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> > >
> > > Hello Jose, thanks for your answer.
> > >
> > > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> > > There is an undocumented option:
> > >
> > >   -bv_reproducible_random
> > >
> > > It will force the initial vector of the Krylov subspace to be the same
> irrespective of the number of MPI processes. This should be used for
> scaling analyses as the one you are trying to do.
> > >
> > > What about when I'm not doing the scaling? Now I would like to ask for
> computing time for bigger size problems, should I also use this option in
> that case? Because, what happens if I have a "bad" configuration? Meaning,
> I ask for some time, enough if I take into account the "correct" scaling,
> but when I run it takes double the time/iterations, like it happened before
> when changing from 960 to 1024 processes?
> >
> > When you increase the matrix size the spectrum of the matrix changes and
> probably also the convergence, so the computation time is not easy to
> predict in advance.
> >
> > Okey, I'll keep that in mine. I thought that, even if the spectrum
> changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could
> predict more or less the time. It was working this way until I found this
> "iterations problem" which doubled the time of execution for the same size
> problem. To be completely sure, do you suggest me or not to use this
> run-time option when going in production? Can you elaborate a bit in the
> effect this option? Is the (huge) difference I got in the number of
> iterations something expected?
>
> Ideally if you have a rough approximation of the eigenvector, you set it
> as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc generates
> a random initial vector, that is start the search blindly. The difference
> between using one random vector or another may be large, depending on the
> problem. Krylov-Schur is usually less sensitive to the initial vector.
>
> Jose
>
> >
> >
> > >
> > > An additional comment is that we strongly recommend to use the default
> solver (Krylov-Schur), which will do Lanczos with implicit restart. It is
> generally faster and more stable.
> > >
> > > I will be doing Dynamical Lanczos, that means that I'll need the
> "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so,
> according to the Lanczos Technical Report notation, I need the "matrix
> whose rows are the eigenvectors of T_m", which should be the same as the
> vectors y_i). I checked the Technical Report for Krylov-Schur also and I
> think I can get the same information also from that solver, but I'm not
> sure. Can you confirm this please?
> > > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following
> the notation on the Report), my idea to get them was to retrieve the
> invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then
> multiply it with the eigenvectors that I get with EPSGetEigenvector. Is
> there another easier (or with less computations) way to get this?
> >
> > In Krylov-Schur the tridiagonal matrix T_m becomes
> arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The
> relevant information can be obtained with EPSGetBV() and EPSGetDS(). But
> this is a "developer level" interface. We could help you get this running.
> Send a small problem matrix to slepc-maint together with a more detailed
> description of what you need to compute.
> >
> > Thanks! When I get to that part I'll write to slepc-maint for help.
> >
> >
> > Jose
> >
> > >
> > >
> > > Jose
> > >
> > >
> > >
> > > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > >
> > > > Hello,
> > > >
> > > > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest
> real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those
> are the only options I set for the solver. My aim is to be able to
> predict/estimate the time-to-solution. To do so, I was doing a scaling of
> the code for different sizes of matrices and for different number of MPI
> processes. As I was not observing a good scaling I checked the number of
> iterations of the solver (given by EPSGetIterationNumber). I've encounter
> that for the **same size** of matrix (that meaning, the same problem), when
> I change the number of MPI processes, the amount of iterations changes, and
> the behaviour is not monotonic. This are the numbers I've got:
> > > >
> > > > # procs   # iters
> > > > 960          157
> > > > 992          189
> > > > 1024        338
> > > > 1056        190
> > > > 1120        174
> > > > 2048        136
> > > >
> > > > I've checked the mailing list for a similar situation and I've found
> another person with the same problem but in another solver ("[SLEPc] GD is
> not deterministic when using different number of cores", Nov 19 2015), but
> I think the solution this person finds does not apply to my problem
> (removing "-eps_harmonic" option).
> > > >
> > > > Can you give me any hint on what is the reason for this behaviour?
> Is there a way to prevent this? It's not possible to estimate/predict any
> time consumption for bigger problems if the number of iterations varies
> this much.
> > > >
> > > > Ale
> > >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/43159f99/attachment.html>

From jroman at dsic.upv.es  Wed Oct 24 03:52:20 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 24 Oct 2018 10:52:20 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
	<8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
	<CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>
Message-ID: <3F900479-3123-413E-B279-9DE624E825F4@dsic.upv.es>

This is very strange. Make sure you call EPSSetFromOptions in the code. Do iteration counts change also for two different runs with the same number of processes?
Maybe Lanczos with default options is too sensitive (by default it does not reorthogonalize). Suggest using Krylov-Schur or Lanczos with full reorthogonalization (EPSLanczosSetReorthog).
Also, send the output of -eps_view to see if there is anything abnormal.

Jose


> El 24 oct 2018, a las 9:09, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> I've tried the option that you gave me but I still get different number of iterations when changing the number of MPI processes: I did 960 procs and 1024 procs and I got 435 and 176 iterations, respectively.
> 
> El mar., 23 oct. 2018 a las 16:48, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> 
> 
> > El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com> escribi?:
> > 
> > 
> > 
> > El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > 
> > 
> > > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > 
> > > Hello Jose, thanks for your answer.
> > > 
> > > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > > There is an undocumented option:
> > > 
> > >   -bv_reproducible_random
> > > 
> > > It will force the initial vector of the Krylov subspace to be the same irrespective of the number of MPI processes. This should be used for scaling analyses as the one you are trying to do.
> > > 
> > > What about when I'm not doing the scaling? Now I would like to ask for computing time for bigger size problems, should I also use this option in that case? Because, what happens if I have a "bad" configuration? Meaning, I ask for some time, enough if I take into account the "correct" scaling, but when I run it takes double the time/iterations, like it happened before when changing from 960 to 1024 processes?
> > 
> > When you increase the matrix size the spectrum of the matrix changes and probably also the convergence, so the computation time is not easy to predict in advance.
> > 
> > Okey, I'll keep that in mine. I thought that, even if the spectrum changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could predict more or less the time. It was working this way until I found this "iterations problem" which doubled the time of execution for the same size problem. To be completely sure, do you suggest me or not to use this run-time option when going in production? Can you elaborate a bit in the effect this option? Is the (huge) difference I got in the number of iterations something expected?
> 
> Ideally if you have a rough approximation of the eigenvector, you set it as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc generates a random initial vector, that is start the search blindly. The difference between using one random vector or another may be large, depending on the problem. Krylov-Schur is usually less sensitive to the initial vector. 
> 
> Jose
> 
> > 
> > 
> > > 
> > > An additional comment is that we strongly recommend to use the default solver (Krylov-Schur), which will do Lanczos with implicit restart. It is generally faster and more stable.
> > > 
> > > I will be doing Dynamical Lanczos, that means that I'll need the "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so, according to the Lanczos Technical Report notation, I need the "matrix whose rows are the eigenvectors of T_m", which should be the same as the vectors y_i). I checked the Technical Report for Krylov-Schur also and I think I can get the same information also from that solver, but I'm not sure. Can you confirm this please? 
> > > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the notation on the Report), my idea to get them was to retrieve the invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply it with the eigenvectors that I get with EPSGetEigenvector. Is there another easier (or with less computations) way to get this?
> > 
> > In Krylov-Schur the tridiagonal matrix T_m becomes arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The relevant information can be obtained with EPSGetBV() and EPSGetDS(). But this is a "developer level" interface. We could help you get this running. Send a small problem matrix to slepc-maint together with a more detailed description of what you need to compute.
> > 
> > Thanks! When I get to that part I'll write to slepc-maint for help.
> > 
> > 
> > Jose
> > 
> > > 
> > > 
> > > Jose
> > > 
> > > 
> > > 
> > > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > > 
> > > > Hello, 
> > > > 
> > > > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are the only options I set for the solver. My aim is to be able to predict/estimate the time-to-solution. To do so, I was doing a scaling of the code for different sizes of matrices and for different number of MPI processes. As I was not observing a good scaling I checked the number of iterations of the solver (given by EPSGetIterationNumber). I've encounter that for the **same size** of matrix (that meaning, the same problem), when I change the number of MPI processes, the amount of iterations changes, and the behaviour is not monotonic. This are the numbers I've got:
> > > > 
> > > > # procs   # iters
> > > > 960          157
> > > > 992          189
> > > > 1024        338
> > > > 1056        190
> > > > 1120        174
> > > > 2048        136
> > > > 
> > > > I've checked the mailing list for a similar situation and I've found another person with the same problem but in another solver ("[SLEPc] GD is not deterministic when using different number of cores", Nov 19 2015), but I think the solution this person finds does not apply to my problem (removing "-eps_harmonic" option).
> > > > 
> > > > Can you give me any hint on what is the reason for this behaviour? Is there a way to prevent this? It's not possible to estimate/predict any time consumption for bigger problems if the number of iterations varies this much.
> > > > 
> > > > Ale
> > > 
> 


From stefano.zampini at gmail.com  Wed Oct 24 04:28:04 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 24 Oct 2018 12:28:04 +0300
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
	<CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
Message-ID: <CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>

Abdullah,

The "Neumann" problems Jed is referring to result from assembling your
problem on each subdomain ( = MPI process) separately.
Assuming you are using FEM, these problems have been historically  named
"Neumann" as they correspond to a problem with natural boundary conditions
(Neumann bc for Poisson).
Note that in PETSc the subdomain decomposition is associated with the mesh
decomposition.

When converting from an assembled AIJ matrix to a MATIS format, such
"Neumann" information is lost.
You can disassemble an AIJ matrix, in the sense that you can find local
matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
but you cannot guarantee (unless if you solve an optimization problem) that
the disassembling will produce subdomain Neumann problems that are
consistent with your FEM problem.

I have added such disassembling code a few months ago, just to have another
alternative for preconditioning AIJ matrices in PETSc; there are few tweaks
one can do to improve the quality of the disassembling, but I discourage
its usage unless you don't have access to the FEM assembly code.

With that said, what problem are you trying to solve? Are you using DMDA or
DMPlex? What are the results you obtained with using the automatic
disassembling?

Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
abdullahasivas at gmail.com> ha scritto:

> Hi Jed,
>
> Thanks for your reply. The assembled matrix I have corresponds to the full
> problem on the full mesh. There are no "Neumann" problems (or any sort of
> domain decomposition) defined in the code generates the matrix. However, I
> think assembling the full problem is equivalent to implicitly assembling
> the "Neumann" problems, since the system can be partitioned as;
>
> [A_{LL} | A_{LI}]  [u_L]     [F]
> -----------|------------ -------- = -----
> [A_{IL}  |A_{II} ]   [u_I]      [G]
>
> and G should correspond to the Neumann problem. I might be thinking wrong
> (or maybe I completely misunderstood the idea), if so please correct me.
> But I think that the problem is that I am not explicitly telling PCBDDC
> which dofs are interface dofs.
>
> Regards,
> Abdullah Ali Sivas
>
> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>
>> Did you assemble "Neumann" problems that are compatible with your
>> definition of interior/interface degrees of freedom?
>>
>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>
>> > Dear all,
>> >
>> > I have a series of linear systems coming from a PDE for which BDDC is an
>> > optimal preconditioner. These linear systems are assembled and I read
>> them
>> > from a file, then convert into MATIS as required (as in
>> >
>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>> > ). I expect each of the systems converge to the solution in almost same
>> > number of iterations but I don't observe it. I think it is because I do
>> not
>> > provide enough information to the preconditioner. I can get a list of
>> inner
>> > dofs and interface dofs. However, I do not know how to use them. Has
>> anyone
>> > have any insights about it or done something similar?
>> >
>> > Best wishes,
>> > Abdullah Ali Sivas
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/533509a2/attachment.html>

From wence at gmx.li  Wed Oct 24 05:29:28 2018
From: wence at gmx.li (Lawrence Mitchell)
Date: Wed, 24 Oct 2018 11:29:28 +0100
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
Message-ID: <2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>

Hi Max,

(I'm cc'ing in the petsc-users mailing list which may have more advice, if you are using PETSc you should definitely subscribe!

> On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com> wrote:
> 
> Hello Lawrence,
> 
> sorry to message you out of the blue. My name is Max and I found your post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246 ) on DMPlex being able to read periodic gmsh files. I am currently trying to do just that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD work. So far I haven? t found any documentation on the periodic BC?s with DMPlex and gmsh in the official petsc documentation. 
> I was wondering whether you?d be so kind as to point me in a general direction concerning how to achieve this. You seem experienced in using petsc and I would greatly appreciate your help. 


I think the answer is "it depends". If you're just using DMPlex directly and all the of the functionality with PetscDS, then I /think/ that reading periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh format [v2]) "just works".

From my side, the issue is to do with mapping that coordinate field into one that I understand (within Firedrake). You may not have this problem.

Thanks,

Lawrence

From knepley at gmail.com  Wed Oct 24 05:49:50 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Oct 2018 06:49:50 -0400
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
Message-ID: <CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>

On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:

> Hi Max,
>
> (I'm cc'ing in the petsc-users mailing list which may have more advice, if
> you are using PETSc you should definitely subscribe!
>
> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com>
> wrote:
> >
> > Hello Lawrence,
> >
> > sorry to message you out of the blue. My name is Max and I found your
> post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246
> ) on DMPlex being able to read periodic gmsh files. I am currently trying
> to do just that (creating a periodic DMPlex mesh with gmsh) in the context
> of my PhD work. So far I haven? t found any documentation on the periodic
> BC?s with DMPlex and gmsh in the official petsc documentation.
> > I was wondering whether you?d be so kind as to point me in a general
> direction concerning how to achieve this. You seem experienced in using
> petsc and I would greatly appreciate your help.
>
>
> I think the answer is "it depends". If you're just using DMPlex directly
> and all the of the functionality with PetscDS, then I /think/ that reading
> periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh
> format [v2]) "just works".
>

There are two phases here: topological and geometric. DMPlex represents the
periodic topological entity directly. For example,  a circle is just a
segment with one end hooked to the other. Vertices are not duplicated, or
mapped to each other. This makes topology simple and easy to implement.
However, then geometry is more complicated. What Plex does is allow
coordinates to be represented by a discontinuous field taking values on
cells, in addition to vertices. In our circle example, each cells near the
cut will have 2 coordinates, one for each vertex, but they will not agree
across the cut. If you define a periodic domain, then Plex can construct
this coordinate field automatically using DMPlexLocalize(). These DG
coordinates are then used by the integration routines.


> From my side, the issue is to do with mapping that coordinate field into
> one that I understand (within Firedrake). You may not have this problem.
>

Firedrake uses its own coordinate mapping and integration routines, so they
must manage the second part independently. I hope to get change this
slightly soon by making the Firedrake representation a DMField, so that it
looks the same to Plex.

  Thanks,

    Matt


> Thanks,
>
> Lawrence



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/f8c6ab4d/attachment.html>

From amfoggia at gmail.com  Wed Oct 24 07:59:02 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Wed, 24 Oct 2018 14:59:02 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <3F900479-3123-413E-B279-9DE624E825F4@dsic.upv.es>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
	<8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
	<CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>
	<3F900479-3123-413E-B279-9DE624E825F4@dsic.upv.es>
Message-ID: <CANCZWytd1gWi5DUmYN39fKq9G3Fwpq_e8B1o2Sn3vS+Bm7grqw@mail.gmail.com>

The functions called to set the solver are (in this order): EPSCreate();
EPSSetOperators(); EPSSetProblemType(EPS_HEP); EPSSetType(EPSLANCZOS);
EPSSetWhichEigenpairs(EPS_SMALLEST_REAL); EPSSetFromOptions();

The output of -eps_view for each run is:
=================================================================
EPS Object: 960 MPI processes
  type: lanczos
    LOCAL reorthogonalization
  problem type: symmetric eigenvalue problem
  selected portion of the spectrum: smallest real parts
  number of eigenvalues (nev): 1
  number of column vectors (ncv): 16
  maximum dimension of projected problem (mpd): 16
  maximum number of iterations: 291700777
  tolerance: 1e-08
  convergence test: relative to the eigenvalue
BV Object: 960 MPI processes
  type: svec
  17 columns of global length 2333606220
  vector orthogonalization method: modified Gram-Schmidt
  orthogonalization refinement: if needed (eta: 0.7071)
  block orthogonalization method: GS
  doing matmult as a single matrix-matrix product
  generating random vectors independent of the number of processes
DS Object: 960 MPI processes
  type: hep
  parallel operation mode: REDUNDANT
  solving the problem with: Implicit QR method (_steqr)
ST Object: 960 MPI processes
  type: shift
  shift: 0.
  number of matrices: 1
=================================================================
EPS Object: 1024 MPI processes
  type: lanczos
    LOCAL reorthogonalization
  problem type: symmetric eigenvalue problem
  selected portion of the spectrum: smallest real parts
  number of eigenvalues (nev): 1
  number of column vectors (ncv): 16
  maximum dimension of projected problem (mpd): 16
  maximum number of iterations: 291700777
  tolerance: 1e-08
  convergence test: relative to the eigenvalue
BV Object: 1024 MPI processes
  type: svec
  17 columns of global length 2333606220
  vector orthogonalization method: modified Gram-Schmidt
  orthogonalization refinement: if needed (eta: 0.7071)
  block orthogonalization method: GS
  doing matmult as a single matrix-matrix product
  generating random vectors independent of the number of processes
DS Object: 1024 MPI processes
  type: hep
  parallel operation mode: REDUNDANT
  solving the problem with: Implicit QR method (_steqr)
ST Object: 1024 MPI processes
  type: shift
  shift: 0.
  number of matrices: 1
=================================================================

I run again the same configurations and I got the same result in term of
the number of iterations.

I also tried the full reorthogonalization (always with the
-bv_reproducible_random option) but I still get different number of
iterations: for 960 procs I get 172 iters, and for 1024 I get 362 iters.
The -esp_view output for this case (only for 960 procs, the other one has
the same information -except the number of processes-) is:
=================================================================
EPS Object: 960 MPI processes
  type: lanczos
    FULL reorthogonalization
  problem type: symmetric eigenvalue problem
  selected portion of the spectrum: smallest real parts
  number of eigenvalues (nev): 1
  number of column vectors (ncv): 16
  maximum dimension of projected problem (mpd): 16
  maximum number of iterations: 291700777
  tolerance: 1e-08
  convergence test: relative to the eigenvalue
BV Object: 960 MPI processes
  type: svec
  17 columns of global length 2333606220
  vector orthogonalization method: classical Gram-Schmidt
  orthogonalization refinement: if needed (eta: 0.7071)
  block orthogonalization method: GS
  doing matmult as a single matrix-matrix product
  generating random vectors independent of the number of processes
DS Object: 960 MPI processes
  type: hep
  parallel operation mode: REDUNDANT
  solving the problem with: Implicit QR method (_steqr)
ST Object: 960 MPI processes
  type: shift
  shift: 0.
  number of matrices: 1
=================================================================

El mi?., 24 oct. 2018 a las 10:52, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

> This is very strange. Make sure you call EPSSetFromOptions in the code. Do
> iteration counts change also for two different runs with the same number of
> processes?
> Maybe Lanczos with default options is too sensitive (by default it does
> not reorthogonalize). Suggest using Krylov-Schur or Lanczos with full
> reorthogonalization (EPSLanczosSetReorthog).
> Also, send the output of -eps_view to see if there is anything abnormal.
>
> Jose
>
>
> > El 24 oct 2018, a las 9:09, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> > I've tried the option that you gave me but I still get different number
> of iterations when changing the number of MPI processes: I did 960 procs
> and 1024 procs and I got 435 and 176 iterations, respectively.
> >
> > El mar., 23 oct. 2018 a las 16:48, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> >
> >
> > > El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com> escribi?:
> > >
> > >
> > >
> > > El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> > >
> > >
> > > > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > >
> > > > Hello Jose, thanks for your answer.
> > > >
> > > > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<
> jroman at dsic.upv.es>) escribi?:
> > > > There is an undocumented option:
> > > >
> > > >   -bv_reproducible_random
> > > >
> > > > It will force the initial vector of the Krylov subspace to be the
> same irrespective of the number of MPI processes. This should be used for
> scaling analyses as the one you are trying to do.
> > > >
> > > > What about when I'm not doing the scaling? Now I would like to ask
> for computing time for bigger size problems, should I also use this option
> in that case? Because, what happens if I have a "bad" configuration?
> Meaning, I ask for some time, enough if I take into account the "correct"
> scaling, but when I run it takes double the time/iterations, like it
> happened before when changing from 960 to 1024 processes?
> > >
> > > When you increase the matrix size the spectrum of the matrix changes
> and probably also the convergence, so the computation time is not easy to
> predict in advance.
> > >
> > > Okey, I'll keep that in mine. I thought that, even if the spectrum
> changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could
> predict more or less the time. It was working this way until I found this
> "iterations problem" which doubled the time of execution for the same size
> problem. To be completely sure, do you suggest me or not to use this
> run-time option when going in production? Can you elaborate a bit in the
> effect this option? Is the (huge) difference I got in the number of
> iterations something expected?
> >
> > Ideally if you have a rough approximation of the eigenvector, you set it
> as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc generates
> a random initial vector, that is start the search blindly. The difference
> between using one random vector or another may be large, depending on the
> problem. Krylov-Schur is usually less sensitive to the initial vector.
> >
> > Jose
> >
> > >
> > >
> > > >
> > > > An additional comment is that we strongly recommend to use the
> default solver (Krylov-Schur), which will do Lanczos with implicit restart.
> It is generally faster and more stable.
> > > >
> > > > I will be doing Dynamical Lanczos, that means that I'll need the
> "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so,
> according to the Lanczos Technical Report notation, I need the "matrix
> whose rows are the eigenvectors of T_m", which should be the same as the
> vectors y_i). I checked the Technical Report for Krylov-Schur also and I
> think I can get the same information also from that solver, but I'm not
> sure. Can you confirm this please?
> > > > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following
> the notation on the Report), my idea to get them was to retrieve the
> invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then
> multiply it with the eigenvectors that I get with EPSGetEigenvector. Is
> there another easier (or with less computations) way to get this?
> > >
> > > In Krylov-Schur the tridiagonal matrix T_m becomes
> arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The
> relevant information can be obtained with EPSGetBV() and EPSGetDS(). But
> this is a "developer level" interface. We could help you get this running.
> Send a small problem matrix to slepc-maint together with a more detailed
> description of what you need to compute.
> > >
> > > Thanks! When I get to that part I'll write to slepc-maint for help.
> > >
> > >
> > > Jose
> > >
> > > >
> > > >
> > > > Jose
> > > >
> > > >
> > > >
> > > > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I'm currently using Lanczos solver (EPSLANCZOS) to get the
> smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem
> (EPS_HEP). Those are the only options I set for the solver. My aim is to be
> able to predict/estimate the time-to-solution. To do so, I was doing a
> scaling of the code for different sizes of matrices and for different
> number of MPI processes. As I was not observing a good scaling I checked
> the number of iterations of the solver (given by EPSGetIterationNumber).
> I've encounter that for the **same size** of matrix (that meaning, the same
> problem), when I change the number of MPI processes, the amount of
> iterations changes, and the behaviour is not monotonic. This are the
> numbers I've got:
> > > > >
> > > > > # procs   # iters
> > > > > 960          157
> > > > > 992          189
> > > > > 1024        338
> > > > > 1056        190
> > > > > 1120        174
> > > > > 2048        136
> > > > >
> > > > > I've checked the mailing list for a similar situation and I've
> found another person with the same problem but in another solver ("[SLEPc]
> GD is not deterministic when using different number of cores", Nov 19
> 2015), but I think the solution this person finds does not apply to my
> problem (removing "-eps_harmonic" option).
> > > > >
> > > > > Can you give me any hint on what is the reason for this behaviour?
> Is there a way to prevent this? It's not possible to estimate/predict any
> time consumption for bigger problems if the number of iterations varies
> this much.
> > > > >
> > > > > Ale
> > > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/41633380/attachment-0001.html>

From stefano.zampini at gmail.com  Wed Oct 24 08:11:53 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Wed, 24 Oct 2018 16:11:53 +0300
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
Message-ID: <CAGPUisgbxYuwP5QpP0nvV1S8j3mcjL7oPMy8fgvnqm4H=g3k+A@mail.gmail.com>

To read periodic meshes from GMSH, you need to use the option
-dm_plex_gmsh_periodic and DMPlexCreateFromFile
See  src/dm/impls/plex/examples/tests/ex1.c. An example runs

$ ./ex1 -filename
${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
-dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape

generating periodic meshes in gmsh may be tricky, Lisandro for sure may
advice.

Il giorno mer 24 ott 2018 alle ore 13:51 Matthew Knepley <knepley at gmail.com>
ha scritto:

> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:
>
>> Hi Max,
>>
>> (I'm cc'ing in the petsc-users mailing list which may have more advice,
>> if you are using PETSc you should definitely subscribe!
>>
>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com>
>> wrote:
>> >
>> > Hello Lawrence,
>> >
>> > sorry to message you out of the blue. My name is Max and I found your
>> post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246
>> ) on DMPlex being able to read periodic gmsh files. I am currently trying
>> to do just that (creating a periodic DMPlex mesh with gmsh) in the context
>> of my PhD work. So far I haven? t found any documentation on the periodic
>> BC?s with DMPlex and gmsh in the official petsc documentation.
>> > I was wondering whether you?d be so kind as to point me in a general
>> direction concerning how to achieve this. You seem experienced in using
>> petsc and I would greatly appreciate your help.
>>
>>
>> I think the answer is "it depends". If you're just using DMPlex directly
>> and all the of the functionality with PetscDS, then I /think/ that reading
>> periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh
>> format [v2]) "just works".
>>
>
> There are two phases here: topological and geometric. DMPlex represents
> the periodic topological entity directly. For example,  a circle is just a
> segment with one end hooked to the other. Vertices are not duplicated, or
> mapped to each other. This makes topology simple and easy to implement.
> However, then geometry is more complicated. What Plex does is allow
> coordinates to be represented by a discontinuous field taking values on
> cells, in addition to vertices. In our circle example, each cells near the
> cut will have 2 coordinates, one for each vertex, but they will not agree
> across the cut. If you define a periodic domain, then Plex can construct
> this coordinate field automatically using DMPlexLocalize(). These DG
> coordinates are then used by the integration routines.
>
>
>> From my side, the issue is to do with mapping that coordinate field into
>> one that I understand (within Firedrake). You may not have this problem.
>>
>
> Firedrake uses its own coordinate mapping and integration routines, so
> they must manage the second part independently. I hope to get change this
> slightly soon by making the Firedrake representation a DMField, so that it
> looks the same to Plex.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>>
>> Lawrence
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/13b6aa72/attachment.html>

From griesser.jan at googlemail.com  Wed Oct 24 08:36:40 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 24 Oct 2018 15:36:40 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
Message-ID: <CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>

Hey,
i tried to run my program as you said with -bv_type vecs and/or -bv_type
mat, but instead of the PETScInt overflow i now get an MPI Error 9 message,
which i assume (after googling a little bit around) should be a memory
problem. I tried to run it also on slightly bigger compute nodes on our
cluster with 20 cores with each 12 GB and 24 GB but the problem still
remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
Do you have maybe an idea what the error could be? I appended also the
python method i used to solve the problem. I also tried to solve the
problem with spectrum solving but the error message remains the same.

def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
# Create the solver
# E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
solver)
E = SLEPc.EPS().create()

# Set the preconditioner
pc = PETSc.PC().create()
pc.setType(pc.Type.BJACOBI)

# Set the linear solver
# Create the KSP object
ksp = PETSc.KSP().create()
# Create the solver, in this case GMRES
ksp.setType(ksp.Type.GMRES)
# Set the tolerances of the GMRES solver
        # Link it to the PC
ksp.setPC(pc)

# Set up the spectral transformations
st = SLEPc.ST().create()
st.setType("shift")
st.setKSP(ksp)
# MPD stands for "maximum projected dimension". It has to due with
computational cost, please read Chap. 2.6.5 of SLEPc docu for
# an explanation. At the moment mpd is only a guess
E.setDimensions(nev=NEV, mpd = MPD)
# Eigenvalues should be real, therefore we start to order them from the
smallest real value |l.real|
E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
# Since the dynamical matrix is symmetric and real it is hermitian
E.setProblemType(SLEPc.EPS.ProblemType.HEP)
# Use the Krylov Schur method to solve the eigenvalue problem
E.setType(E.Type.KRYLOVSCHUR)
# Set the convergence criterion to relative to the eigenvalue and the
maximal number of iterations
E.setConvergenceTest(E.Conv.REL)
E.setTolerances(tol = 1e-8, max_it = 5000)
# Set the matrix in order to solve
E.setOperators(DynMatrix_nn, None)
# Sets EPS options from the options database. This routine must be called
before `setUp()` if the user is to be allowed to set dthe solver type.
E.setFromOptions()
# Sets up all the internal data structures necessary for the execution of
the eigensolver.
E.setUp()

# Solve eigenvalue problem
E.solve()

Print = PETSc.Sys.Print

Print()
Print("****************************")
Print("***SLEPc Solution Results***")
Print("****************************")

its = E.getIterationNumber()
Print("Number of iterations of the method: ", its)
eps_type = E.getType()
Print("Solution method: ", eps_type)
nev, ncv, mpd = E.getDimensions()
Print("Number of requested eigenvalues: ", nev)
Print("Number of computeded eigenvectors: ", ncv)
tol, maxit = E.getTolerances()
Print("Stopping condition: (tol, maxit)", (tol, maxit))
# Get the type of convergence
conv_test = E.getConvergenceTest()
Print("Selected convergence test: ", conv_test)
# Get the used spectral transformation
get_st = E.getST()
Print("Selected spectral transformation: ", get_st)
# Get the applied direct solver
get_ksp = E.getDS()
Print("Selected direct solver: ", get_ksp)
nconv = E.getConverged()
Print("Number of converged eigenpairs: ", nconv)
.....



Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
bsmith at mcs.anl.gov>:

>
>
> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
> >
> >
> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <griesser.jan at googlemail.com>
> wrote:
> > With more than 1 MPI process you mean i should use spectrum slicing in
> divide the full problem in smaller subproblems?
> > The --with-64-bit-indices is not a possibility for me since i configured
> petsc with mumps, which does not allow to use the 64-bit version (At least
> this was the error message when i tried to configure PETSc )
> >
> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
> integer feature" and "full 64-bit integer version" as well.
>
>     They use to achieve this by compiling with special Fortran flags to
> promote integers to 64 bit; this is too fragile for our taste so we never
> hooked PETSc up wit it. If they have a dependable way of using 64 bit
> integers we should add that to our mumps.py and test it.
>
>    Barry
>
> >
> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
> jroman at dsic.upv.es>:
> > To use BVVECS just add the command-line option -bv_type vecs
> > This causes to use a separate Vec for each column, instead of a single
> long Vec of size n*m. But it is considerably slower than the default.
> >
> > Anyway, for such large problems you should consider using more than 1
> MPI process. In that case the error may disappear because the local size is
> smaller than 768000.
> >
> > Jose
> >
> >
> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
> escribi?:
> > >
> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
> griesser.jan at googlemail.com> wrote:
> > > Hi all,
> > > i am using slepc4py and petsc4py to solve for the smallest real
> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
> 30k x 30k solving for the smallest soutions works quite well, but when i
> increase the dimension of my system to around A = 768000 x 768000 or 3
> million x 3 million and ask for the smallest real 3000 (the number is
> increasing with increasing system size) eigenvalues and eigenvectors i get
> the output (for the 768000):
> > >  The product 4001 times 768000 overflows the size of PetscInt;
> consider reducing the number of columns, or use BVVECS instead
> > > i understand that the requested number of eigenvectors and eigenvalues
> is causing an overflow but i do not understand the solution of the problem
> which is stated in the error message. Can someone tell me what exactly
> BVVECS is and how i can use it? Or is there any other solution to my
> problem ?
> > >
> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > > Thank you very much in advance,
> > > Jan
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/1db75cad/attachment.html>

From jroman at dsic.upv.es  Wed Oct 24 08:48:15 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 24 Oct 2018 15:48:15 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CANCZWytd1gWi5DUmYN39fKq9G3Fwpq_e8B1o2Sn3vS+Bm7grqw@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
	<8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
	<CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>
	<3F900479-3123-413E-B279-9DE624E825F4@dsic.upv.es>
	<CANCZWytd1gWi5DUmYN39fKq9G3Fwpq_e8B1o2Sn3vS+Bm7grqw@mail.gmail.com>
Message-ID: <AA616809-7FDD-46AF-A60E-EB41C66FD0ED@dsic.upv.es>

Everything seems correct. I don't know, maybe your problem is very sensitive? Is the eigenvalue tiny?
I would still try with Krylov-Schur.
Jose


> El 24 oct 2018, a las 14:59, Ale Foggia <amfoggia at gmail.com> escribi?:
> 
> The functions called to set the solver are (in this order): EPSCreate(); EPSSetOperators(); EPSSetProblemType(EPS_HEP); EPSSetType(EPSLANCZOS); EPSSetWhichEigenpairs(EPS_SMALLEST_REAL); EPSSetFromOptions();
> 
> The output of -eps_view for each run is:
> =================================================================
> EPS Object: 960 MPI processes
>   type: lanczos
>     LOCAL reorthogonalization
>   problem type: symmetric eigenvalue problem
>   selected portion of the spectrum: smallest real parts
>   number of eigenvalues (nev): 1
>   number of column vectors (ncv): 16
>   maximum dimension of projected problem (mpd): 16
>   maximum number of iterations: 291700777
>   tolerance: 1e-08
>   convergence test: relative to the eigenvalue
> BV Object: 960 MPI processes
>   type: svec
>   17 columns of global length 2333606220
>   vector orthogonalization method: modified Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
>   generating random vectors independent of the number of processes
> DS Object: 960 MPI processes
>   type: hep
>   parallel operation mode: REDUNDANT
>   solving the problem with: Implicit QR method (_steqr)
> ST Object: 960 MPI processes
>   type: shift
>   shift: 0.
>   number of matrices: 1
> =================================================================
> EPS Object: 1024 MPI processes
>   type: lanczos
>     LOCAL reorthogonalization
>   problem type: symmetric eigenvalue problem
>   selected portion of the spectrum: smallest real parts
>   number of eigenvalues (nev): 1
>   number of column vectors (ncv): 16
>   maximum dimension of projected problem (mpd): 16
>   maximum number of iterations: 291700777
>   tolerance: 1e-08
>   convergence test: relative to the eigenvalue
> BV Object: 1024 MPI processes
>   type: svec
>   17 columns of global length 2333606220
>   vector orthogonalization method: modified Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
>   generating random vectors independent of the number of processes
> DS Object: 1024 MPI processes
>   type: hep
>   parallel operation mode: REDUNDANT
>   solving the problem with: Implicit QR method (_steqr)
> ST Object: 1024 MPI processes
>   type: shift
>   shift: 0.
>   number of matrices: 1
> =================================================================
> 
> I run again the same configurations and I got the same result in term of the number of iterations.
> 
> I also tried the full reorthogonalization (always with the -bv_reproducible_random option) but I still get different number of iterations: for 960 procs I get 172 iters, and for 1024 I get 362 iters. The -esp_view output for this case (only for 960 procs, the other one has the same information -except the number of processes-) is:
> =================================================================
> EPS Object: 960 MPI processes
>   type: lanczos
>     FULL reorthogonalization
>   problem type: symmetric eigenvalue problem
>   selected portion of the spectrum: smallest real parts
>   number of eigenvalues (nev): 1
>   number of column vectors (ncv): 16
>   maximum dimension of projected problem (mpd): 16
>   maximum number of iterations: 291700777
>   tolerance: 1e-08
>   convergence test: relative to the eigenvalue
> BV Object: 960 MPI processes
>   type: svec
>   17 columns of global length 2333606220
>   vector orthogonalization method: classical Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
>   generating random vectors independent of the number of processes
> DS Object: 960 MPI processes
>   type: hep
>   parallel operation mode: REDUNDANT
>   solving the problem with: Implicit QR method (_steqr)
> ST Object: 960 MPI processes
>   type: shift
>   shift: 0.
>   number of matrices: 1
> =================================================================
> 
> El mi?., 24 oct. 2018 a las 10:52, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> This is very strange. Make sure you call EPSSetFromOptions in the code. Do iteration counts change also for two different runs with the same number of processes?
> Maybe Lanczos with default options is too sensitive (by default it does not reorthogonalize). Suggest using Krylov-Schur or Lanczos with full reorthogonalization (EPSLanczosSetReorthog).
> Also, send the output of -eps_view to see if there is anything abnormal.
> 
> Jose
> 
> 
> > El 24 oct 2018, a las 9:09, Ale Foggia <amfoggia at gmail.com> escribi?:
> > 
> > I've tried the option that you gave me but I still get different number of iterations when changing the number of MPI processes: I did 960 procs and 1024 procs and I got 435 and 176 iterations, respectively.
> > 
> > El mar., 23 oct. 2018 a las 16:48, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > 
> > 
> > > El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > 
> > > 
> > > 
> > > El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > > 
> > > 
> > > > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > > 
> > > > Hello Jose, thanks for your answer.
> > > > 
> > > > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<jroman at dsic.upv.es>) escribi?:
> > > > There is an undocumented option:
> > > > 
> > > >   -bv_reproducible_random
> > > > 
> > > > It will force the initial vector of the Krylov subspace to be the same irrespective of the number of MPI processes. This should be used for scaling analyses as the one you are trying to do.
> > > > 
> > > > What about when I'm not doing the scaling? Now I would like to ask for computing time for bigger size problems, should I also use this option in that case? Because, what happens if I have a "bad" configuration? Meaning, I ask for some time, enough if I take into account the "correct" scaling, but when I run it takes double the time/iterations, like it happened before when changing from 960 to 1024 processes?
> > > 
> > > When you increase the matrix size the spectrum of the matrix changes and probably also the convergence, so the computation time is not easy to predict in advance.
> > > 
> > > Okey, I'll keep that in mine. I thought that, even if the spectrum changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could predict more or less the time. It was working this way until I found this "iterations problem" which doubled the time of execution for the same size problem. To be completely sure, do you suggest me or not to use this run-time option when going in production? Can you elaborate a bit in the effect this option? Is the (huge) difference I got in the number of iterations something expected?
> > 
> > Ideally if you have a rough approximation of the eigenvector, you set it as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc generates a random initial vector, that is start the search blindly. The difference between using one random vector or another may be large, depending on the problem. Krylov-Schur is usually less sensitive to the initial vector. 
> > 
> > Jose
> > 
> > > 
> > > 
> > > > 
> > > > An additional comment is that we strongly recommend to use the default solver (Krylov-Schur), which will do Lanczos with implicit restart. It is generally faster and more stable.
> > > > 
> > > > I will be doing Dynamical Lanczos, that means that I'll need the "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so, according to the Lanczos Technical Report notation, I need the "matrix whose rows are the eigenvectors of T_m", which should be the same as the vectors y_i). I checked the Technical Report for Krylov-Schur also and I think I can get the same information also from that solver, but I'm not sure. Can you confirm this please? 
> > > > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i (following the notation on the Report), my idea to get them was to retrieve the invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and then multiply it with the eigenvectors that I get with EPSGetEigenvector. Is there another easier (or with less computations) way to get this?
> > > 
> > > In Krylov-Schur the tridiagonal matrix T_m becomes arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The relevant information can be obtained with EPSGetBV() and EPSGetDS(). But this is a "developer level" interface. We could help you get this running. Send a small problem matrix to slepc-maint together with a more detailed description of what you need to compute.
> > > 
> > > Thanks! When I get to that part I'll write to slepc-maint for help.
> > > 
> > > 
> > > Jose
> > > 
> > > > 
> > > > 
> > > > Jose
> > > > 
> > > > 
> > > > 
> > > > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com> escribi?:
> > > > > 
> > > > > Hello, 
> > > > > 
> > > > > I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are the only options I set for the solver. My aim is to be able to predict/estimate the time-to-solution. To do so, I was doing a scaling of the code for different sizes of matrices and for different number of MPI processes. As I was not observing a good scaling I checked the number of iterations of the solver (given by EPSGetIterationNumber). I've encounter that for the **same size** of matrix (that meaning, the same problem), when I change the number of MPI processes, the amount of iterations changes, and the behaviour is not monotonic. This are the numbers I've got:
> > > > > 
> > > > > # procs   # iters
> > > > > 960          157
> > > > > 992          189
> > > > > 1024        338
> > > > > 1056        190
> > > > > 1120        174
> > > > > 2048        136
> > > > > 
> > > > > I've checked the mailing list for a similar situation and I've found another person with the same problem but in another solver ("[SLEPc] GD is not deterministic when using different number of cores", Nov 19 2015), but I think the solution this person finds does not apply to my problem (removing "-eps_harmonic" option).
> > > > > 
> > > > > Can you give me any hint on what is the reason for this behaviour? Is there a way to prevent this? It's not possible to estimate/predict any time consumption for bigger problems if the number of iterations varies this much.
> > > > > 
> > > > > Ale
> > > > 
> > 
> 


From knepley at gmail.com  Wed Oct 24 09:01:37 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Oct 2018 10:01:37 -0400
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
Message-ID: <CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>

On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <griesser.jan at googlemail.com>
wrote:

> Hey,
> i tried to run my program as you said with -bv_type vecs and/or -bv_type
> mat, but instead of the PETScInt overflow i now get an MPI Error 9
>

Send the actual error.

  Thanks,

    Matt


> message, which i assume (after googling a little bit around) should be a
> memory problem. I tried to run it also on slightly bigger compute nodes on
> our cluster with 20 cores with each 12 GB and 24 GB but the problem still
> remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
> i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
> Do you have maybe an idea what the error could be? I appended also the
> python method i used to solve the problem. I also tried to solve the
> problem with spectrum solving but the error message remains the same.
>
> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
> # Create the solver
> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
> solver)
> E = SLEPc.EPS().create()
>
> # Set the preconditioner
> pc = PETSc.PC().create()
> pc.setType(pc.Type.BJACOBI)
>
> # Set the linear solver
> # Create the KSP object
> ksp = PETSc.KSP().create()
> # Create the solver, in this case GMRES
> ksp.setType(ksp.Type.GMRES)
> # Set the tolerances of the GMRES solver
>         # Link it to the PC
> ksp.setPC(pc)
>
> # Set up the spectral transformations
> st = SLEPc.ST().create()
> st.setType("shift")
> st.setKSP(ksp)
> # MPD stands for "maximum projected dimension". It has to due with
> computational cost, please read Chap. 2.6.5 of SLEPc docu for
> # an explanation. At the moment mpd is only a guess
> E.setDimensions(nev=NEV, mpd = MPD)
> # Eigenvalues should be real, therefore we start to order them from the
> smallest real value |l.real|
> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
> # Since the dynamical matrix is symmetric and real it is hermitian
> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
> # Use the Krylov Schur method to solve the eigenvalue problem
> E.setType(E.Type.KRYLOVSCHUR)
> # Set the convergence criterion to relative to the eigenvalue and the
> maximal number of iterations
> E.setConvergenceTest(E.Conv.REL)
> E.setTolerances(tol = 1e-8, max_it = 5000)
> # Set the matrix in order to solve
> E.setOperators(DynMatrix_nn, None)
> # Sets EPS options from the options database. This routine must be called
> before `setUp()` if the user is to be allowed to set dthe solver type.
> E.setFromOptions()
> # Sets up all the internal data structures necessary for the execution of
> the eigensolver.
> E.setUp()
>
> # Solve eigenvalue problem
> E.solve()
>
> Print = PETSc.Sys.Print
>
> Print()
> Print("****************************")
> Print("***SLEPc Solution Results***")
> Print("****************************")
>
> its = E.getIterationNumber()
> Print("Number of iterations of the method: ", its)
> eps_type = E.getType()
> Print("Solution method: ", eps_type)
> nev, ncv, mpd = E.getDimensions()
> Print("Number of requested eigenvalues: ", nev)
> Print("Number of computeded eigenvectors: ", ncv)
> tol, maxit = E.getTolerances()
> Print("Stopping condition: (tol, maxit)", (tol, maxit))
> # Get the type of convergence
> conv_test = E.getConvergenceTest()
> Print("Selected convergence test: ", conv_test)
> # Get the used spectral transformation
> get_st = E.getST()
> Print("Selected spectral transformation: ", get_st)
> # Get the applied direct solver
> get_ksp = E.getDS()
> Print("Selected direct solver: ", get_ksp)
> nconv = E.getConverged()
> Print("Number of converged eigenpairs: ", nconv)
> .....
>
>
>
> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
> bsmith at mcs.anl.gov>:
>
>>
>>
>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>> wrote:
>> >
>> >
>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>> griesser.jan at googlemail.com> wrote:
>> > With more than 1 MPI process you mean i should use spectrum slicing in
>> divide the full problem in smaller subproblems?
>> > The --with-64-bit-indices is not a possibility for me since i
>> configured petsc with mumps, which does not allow to use the 64-bit version
>> (At least this was the error message when i tried to configure PETSc )
>> >
>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>> integer feature" and "full 64-bit integer version" as well.
>>
>>     They use to achieve this by compiling with special Fortran flags to
>> promote integers to 64 bit; this is too fragile for our taste so we never
>> hooked PETSc up wit it. If they have a dependable way of using 64 bit
>> integers we should add that to our mumps.py and test it.
>>
>>    Barry
>>
>> >
>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>> jroman at dsic.upv.es>:
>> > To use BVVECS just add the command-line option -bv_type vecs
>> > This causes to use a separate Vec for each column, instead of a single
>> long Vec of size n*m. But it is considerably slower than the default.
>> >
>> > Anyway, for such large problems you should consider using more than 1
>> MPI process. In that case the error may disappear because the local size is
>> smaller than 768000.
>> >
>> > Jose
>> >
>> >
>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>> escribi?:
>> > >
>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>> griesser.jan at googlemail.com> wrote:
>> > > Hi all,
>> > > i am using slepc4py and petsc4py to solve for the smallest real
>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>> 30k x 30k solving for the smallest soutions works quite well, but when i
>> increase the dimension of my system to around A = 768000 x 768000 or 3
>> million x 3 million and ask for the smallest real 3000 (the number is
>> increasing with increasing system size) eigenvalues and eigenvectors i get
>> the output (for the 768000):
>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>> consider reducing the number of columns, or use BVVECS instead
>> > > i understand that the requested number of eigenvectors and
>> eigenvalues is causing an overflow but i do not understand the solution of
>> the problem which is stated in the error message. Can someone tell me what
>> exactly BVVECS is and how i can use it? Or is there any other solution to
>> my problem ?
>> > >
>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>> > >
>> > >   Thanks,
>> > >
>> > >     Matt
>> > >
>> > > Thank you very much in advance,
>> > > Jan
>> > >
>> > >
>> > >
>> > > --
>> > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > -- Norbert Wiener
>> > >
>> > > https://www.cse.buffalo.edu/~knepley/
>> >
>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/af7ad114/attachment.html>

From griesser.jan at googlemail.com  Wed Oct 24 09:03:08 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 24 Oct 2018 16:03:08 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
Message-ID: <CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>

This is the error message i get from my program:
Shape of the dynamical matrix:  (1500000, 1500000)

===================================================================================
=   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
=   PID 122676 RUNNING AT n3512.nemo.privat
=   EXIT CODE: 9
=   CLEANING UP REMAINING PROCESSES
=   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
===================================================================================
   Intel(R) MPI Library troubleshooting guide:
      https://software.intel.com/node/561764
===================================================================================


Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
knepley at gmail.com>:

> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <griesser.jan at googlemail.com>
> wrote:
>
>> Hey,
>> i tried to run my program as you said with -bv_type vecs and/or -bv_type
>> mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>
>
> Send the actual error.
>
>   Thanks,
>
>     Matt
>
>
>> message, which i assume (after googling a little bit around) should be a
>> memory problem. I tried to run it also on slightly bigger compute nodes on
>> our cluster with 20 cores with each 12 GB and 24 GB but the problem still
>> remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
>> i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>> Do you have maybe an idea what the error could be? I appended also the
>> python method i used to solve the problem. I also tried to solve the
>> problem with spectrum solving but the error message remains the same.
>>
>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>> # Create the solver
>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
>> solver)
>> E = SLEPc.EPS().create()
>>
>> # Set the preconditioner
>> pc = PETSc.PC().create()
>> pc.setType(pc.Type.BJACOBI)
>>
>> # Set the linear solver
>> # Create the KSP object
>> ksp = PETSc.KSP().create()
>> # Create the solver, in this case GMRES
>> ksp.setType(ksp.Type.GMRES)
>> # Set the tolerances of the GMRES solver
>>         # Link it to the PC
>> ksp.setPC(pc)
>>
>> # Set up the spectral transformations
>> st = SLEPc.ST().create()
>> st.setType("shift")
>> st.setKSP(ksp)
>> # MPD stands for "maximum projected dimension". It has to due with
>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>> # an explanation. At the moment mpd is only a guess
>> E.setDimensions(nev=NEV, mpd = MPD)
>> # Eigenvalues should be real, therefore we start to order them from the
>> smallest real value |l.real|
>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>> # Since the dynamical matrix is symmetric and real it is hermitian
>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>> # Use the Krylov Schur method to solve the eigenvalue problem
>> E.setType(E.Type.KRYLOVSCHUR)
>> # Set the convergence criterion to relative to the eigenvalue and the
>> maximal number of iterations
>> E.setConvergenceTest(E.Conv.REL)
>> E.setTolerances(tol = 1e-8, max_it = 5000)
>> # Set the matrix in order to solve
>> E.setOperators(DynMatrix_nn, None)
>> # Sets EPS options from the options database. This routine must be called
>> before `setUp()` if the user is to be allowed to set dthe solver type.
>> E.setFromOptions()
>> # Sets up all the internal data structures necessary for the execution of
>> the eigensolver.
>> E.setUp()
>>
>> # Solve eigenvalue problem
>> E.solve()
>>
>> Print = PETSc.Sys.Print
>>
>> Print()
>> Print("****************************")
>> Print("***SLEPc Solution Results***")
>> Print("****************************")
>>
>> its = E.getIterationNumber()
>> Print("Number of iterations of the method: ", its)
>> eps_type = E.getType()
>> Print("Solution method: ", eps_type)
>> nev, ncv, mpd = E.getDimensions()
>> Print("Number of requested eigenvalues: ", nev)
>> Print("Number of computeded eigenvectors: ", ncv)
>> tol, maxit = E.getTolerances()
>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>> # Get the type of convergence
>> conv_test = E.getConvergenceTest()
>> Print("Selected convergence test: ", conv_test)
>> # Get the used spectral transformation
>> get_st = E.getST()
>> Print("Selected spectral transformation: ", get_st)
>> # Get the applied direct solver
>> get_ksp = E.getDS()
>> Print("Selected direct solver: ", get_ksp)
>> nconv = E.getConverged()
>> Print("Number of converged eigenpairs: ", nconv)
>> .....
>>
>>
>>
>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>> bsmith at mcs.anl.gov>:
>>
>>>
>>>
>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>> wrote:
>>> >
>>> >
>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>>> griesser.jan at googlemail.com> wrote:
>>> > With more than 1 MPI process you mean i should use spectrum slicing in
>>> divide the full problem in smaller subproblems?
>>> > The --with-64-bit-indices is not a possibility for me since i
>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>> (At least this was the error message when i tried to configure PETSc )
>>> >
>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>>> integer feature" and "full 64-bit integer version" as well.
>>>
>>>     They use to achieve this by compiling with special Fortran flags to
>>> promote integers to 64 bit; this is too fragile for our taste so we never
>>> hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>> integers we should add that to our mumps.py and test it.
>>>
>>>    Barry
>>>
>>> >
>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>> jroman at dsic.upv.es>:
>>> > To use BVVECS just add the command-line option -bv_type vecs
>>> > This causes to use a separate Vec for each column, instead of a single
>>> long Vec of size n*m. But it is considerably slower than the default.
>>> >
>>> > Anyway, for such large problems you should consider using more than 1
>>> MPI process. In that case the error may disappear because the local size is
>>> smaller than 768000.
>>> >
>>> > Jose
>>> >
>>> >
>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>> escribi?:
>>> > >
>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>>> griesser.jan at googlemail.com> wrote:
>>> > > Hi all,
>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>> million x 3 million and ask for the smallest real 3000 (the number is
>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>> the output (for the 768000):
>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>> consider reducing the number of columns, or use BVVECS instead
>>> > > i understand that the requested number of eigenvectors and
>>> eigenvalues is causing an overflow but i do not understand the solution of
>>> the problem which is stated in the error message. Can someone tell me what
>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>> my problem ?
>>> > >
>>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>>> > >
>>> > >   Thanks,
>>> > >
>>> > >     Matt
>>> > >
>>> > > Thank you very much in advance,
>>> > > Jan
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > > -- Norbert Wiener
>>> > >
>>> > > https://www.cse.buffalo.edu/~knepley/
>>> >
>>>
>>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/20c1fd46/attachment-0001.html>

From jroman at dsic.upv.es  Wed Oct 24 09:13:23 2018
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Wed, 24 Oct 2018 16:13:23 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
	<CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
Message-ID: <C93E4B02-6F29-4752-A0E1-8C6FBE2B3265@dsic.upv.es>

The program seems correct, although in your case you don't need any KSP or preconditioner, so you could remove all code related to PC, KSP and ST - anyway, this should not affect memory consumption.

Add the option -eps_view_pre and send the output.
Jose


> El 24 oct 2018, a las 16:03, Jan Grie?er <griesser.jan at googlemail.com> escribi?:
> 
> This is the error message i get from my program: 
> Shape of the dynamical matrix:  (1500000, 1500000)
> 
> ===================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   PID 122676 RUNNING AT n3512.nemo.privat
> =   EXIT CODE: 9
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
> ===================================================================================
>    Intel(R) MPI Library troubleshooting guide:
>       https://software.intel.com/node/561764
> ===================================================================================
> 
> 
> Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <knepley at gmail.com>:
> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> Hey,
> i tried to run my program as you said with -bv_type vecs and/or -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9
> 
> Send the actual error.
> 
>   Thanks,
> 
>     Matt
>  
> message, which i assume (after googling a little bit around) should be a memory problem. I tried to run it also on slightly bigger compute nodes on our cluster with 20 cores with each 12 GB and 24 GB but the problem still remains. The actual limit appears to be a 1.5 Million x 1.5 Million where i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues. 
> Do you have maybe an idea what the error could be? I appended also the python method i used to solve the problem. I also tried to solve the problem with spectrum solving but the error message remains the same. 
> 
> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
> 	# Create the solver 
> 	# E is used as an acronym for the EPS solver (EPS = Eigenvalue problem solver)
> 	E = SLEPc.EPS().create()
> 
> 	# Set the preconditioner 
> 	pc = PETSc.PC().create()
> 	pc.setType(pc.Type.BJACOBI)
> 
> 	# Set the linear solver 
> 	# Create the KSP object
> 	ksp = PETSc.KSP().create()
> 	# Create the solver, in this case GMRES
> 	ksp.setType(ksp.Type.GMRES)
> 	# Set the tolerances of the GMRES solver
>         # Link it to the PC
> 	ksp.setPC(pc)
> 
> 	# Set up the spectral transformations 
> 	st = SLEPc.ST().create()
> 	st.setType("shift")
> 	st.setKSP(ksp)
> 	
> 	# MPD stands for "maximum projected dimension". It has to due with computational cost, please read Chap. 2.6.5 of SLEPc docu for 
> 	# an explanation. At the moment mpd is only a guess
> 	E.setDimensions(nev=NEV, mpd = MPD)
> 	# Eigenvalues should be real, therefore we start to order them from the smallest real value |l.real|
> 	E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
> 	# Since the dynamical matrix is symmetric and real it is hermitian 
> 	E.setProblemType(SLEPc.EPS.ProblemType.HEP)
> 	# Use the Krylov Schur method to solve the eigenvalue problem
> 	E.setType(E.Type.KRYLOVSCHUR)
> 	# Set the convergence criterion to relative to the eigenvalue and the maximal number of iterations
> 	E.setConvergenceTest(E.Conv.REL)
> 	E.setTolerances(tol = 1e-8, max_it = 5000)
> 	# Set the matrix in order to solve 
> 	E.setOperators(DynMatrix_nn, None)
> 	# Sets EPS options from the options database. This routine must be called before `setUp()` if the user is to be allowed to set dthe solver type.
> 	E.setFromOptions()
> 	# Sets up all the internal data structures necessary for the execution of the eigensolver.
> 	E.setUp()
> 
> 	# Solve eigenvalue problem     
> 	E.solve()
> 
> 	Print = PETSc.Sys.Print
> 
> 	Print()
> 	Print("****************************")
> 	Print("***SLEPc Solution Results***")
> 	Print("****************************")
> 
> 	its = E.getIterationNumber()
> 	Print("Number of iterations of the method: ", its)
> 	eps_type = E.getType()
> 	Print("Solution method: ", eps_type)
> 	nev, ncv, mpd = E.getDimensions()
> 	Print("Number of requested eigenvalues: ", nev)
> 	Print("Number of computeded eigenvectors: ", ncv)
> 	tol, maxit = E.getTolerances()
> 	Print("Stopping condition: (tol, maxit)", (tol, maxit))
> 	# Get the type of convergence
> 	conv_test = E.getConvergenceTest()
> 	Print("Selected convergence test: ", conv_test)
> 	# Get the used spectral transformation
> 	get_st = E.getST()
> 	Print("Selected spectral transformation: ", get_st)
> 	# Get the applied direct solver
> 	get_ksp = E.getDS()
> 	Print("Selected direct solver: ", get_ksp)
> 	nconv = E.getConverged()
> 	Print("Number of converged eigenpairs: ", nconv)
> .....
> 
> 
> 
> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <bsmith at mcs.anl.gov>:
> 
> 
> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov> wrote:
> > 
> > 
> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> > With more than 1 MPI process you mean i should use spectrum slicing in divide the full problem in smaller subproblems? 
> > The --with-64-bit-indices is not a possibility for me since i configured petsc with mumps, which does not allow to use the 64-bit version (At least this was the error message when i tried to configure PETSc )
> >  
> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit integer feature" and "full 64-bit integer version" as well. 
> 
>     They use to achieve this by compiling with special Fortran flags to promote integers to 64 bit; this is too fragile for our taste so we never hooked PETSc up wit it. If they have a dependable way of using 64 bit integers we should add that to our mumps.py and test it.
> 
>    Barry
> 
> > 
> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <jroman at dsic.upv.es>:
> > To use BVVECS just add the command-line option -bv_type vecs
> > This causes to use a separate Vec for each column, instead of a single long Vec of size n*m. But it is considerably slower than the default.
> > 
> > Anyway, for such large problems you should consider using more than 1 MPI process. In that case the error may disappear because the local size is smaller than 768000.
> > 
> > Jose
> > 
> > 
> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com> escribi?:
> > > 
> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <griesser.jan at googlemail.com> wrote:
> > > Hi all,
> > > i am using slepc4py and petsc4py to solve for the smallest real eigenvalues and eigenvectors. For my test cases with a matrix A of the size 30k x 30k solving for the smallest soutions works quite well, but when i increase the dimension of my system to around A = 768000 x 768000 or 3 million x 3 million and ask for the smallest real 3000 (the number is increasing with increasing system size) eigenvalues and eigenvectors i get the output (for the 768000): 
> > >  The product 4001 times 768000 overflows the size of PetscInt; consider reducing the number of columns, or use BVVECS instead
> > > i understand that the requested number of eigenvectors and eigenvalues is causing an overflow but i do not understand the solution of the problem which is stated in the error message. Can someone tell me what exactly BVVECS is and how i can use it? Or is there any other solution to my problem ?
> > > 
> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
> > > 
> > >   Thanks,
> > > 
> > >     Matt
> > >  
> > > Thank you very much in advance,
> > > Jan 
> > > 
> > > 
> > > 
> > > -- 
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > > 
> > > https://www.cse.buffalo.edu/~knepley/
> > 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From knepley at gmail.com  Wed Oct 24 09:14:35 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Oct 2018 10:14:35 -0400
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
	<CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
Message-ID: <CAMYG4GkxiHzF0UrFKR4fkD4pzcj_URi4zEQdFQ4CjCZWRpz4uA@mail.gmail.com>

On Wed, Oct 24, 2018 at 10:03 AM Jan Grie?er <griesser.jan at googlemail.com>
wrote:

> This is the error message i get from my program:
> Shape of the dynamical matrix:  (1500000, 1500000)
>

Petsc installs a signal handler, so there should be a PETSc-specific
message before this one. Is something eating
your output?

  Thanks,

     Matt


>
> ===================================================================================
> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
> =   PID 122676 RUNNING AT n3512.nemo.privat
> =   EXIT CODE: 9
> =   CLEANING UP REMAINING PROCESSES
> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>
> ===================================================================================
>    Intel(R) MPI Library troubleshooting guide:
>       https://software.intel.com/node/561764
>
> ===================================================================================
>
>
> Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
> knepley at gmail.com>:
>
>> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <griesser.jan at googlemail.com>
>> wrote:
>>
>>> Hey,
>>> i tried to run my program as you said with -bv_type vecs and/or -bv_type
>>> mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>>
>>
>> Send the actual error.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> message, which i assume (after googling a little bit around) should be a
>>> memory problem. I tried to run it also on slightly bigger compute nodes on
>>> our cluster with 20 cores with each 12 GB and 24 GB but the problem still
>>> remains. The actual limit appears to be a 1.5 Million x 1.5 Million where
>>> i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>>> Do you have maybe an idea what the error could be? I appended also the
>>> python method i used to solve the problem. I also tried to solve the
>>> problem with spectrum solving but the error message remains the same.
>>>
>>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>>> # Create the solver
>>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
>>> solver)
>>> E = SLEPc.EPS().create()
>>>
>>> # Set the preconditioner
>>> pc = PETSc.PC().create()
>>> pc.setType(pc.Type.BJACOBI)
>>>
>>> # Set the linear solver
>>> # Create the KSP object
>>> ksp = PETSc.KSP().create()
>>> # Create the solver, in this case GMRES
>>> ksp.setType(ksp.Type.GMRES)
>>> # Set the tolerances of the GMRES solver
>>>         # Link it to the PC
>>> ksp.setPC(pc)
>>>
>>> # Set up the spectral transformations
>>> st = SLEPc.ST().create()
>>> st.setType("shift")
>>> st.setKSP(ksp)
>>> # MPD stands for "maximum projected dimension". It has to due with
>>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>>> # an explanation. At the moment mpd is only a guess
>>> E.setDimensions(nev=NEV, mpd = MPD)
>>> # Eigenvalues should be real, therefore we start to order them from the
>>> smallest real value |l.real|
>>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>>> # Since the dynamical matrix is symmetric and real it is hermitian
>>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>>> # Use the Krylov Schur method to solve the eigenvalue problem
>>> E.setType(E.Type.KRYLOVSCHUR)
>>> # Set the convergence criterion to relative to the eigenvalue and the
>>> maximal number of iterations
>>> E.setConvergenceTest(E.Conv.REL)
>>> E.setTolerances(tol = 1e-8, max_it = 5000)
>>> # Set the matrix in order to solve
>>> E.setOperators(DynMatrix_nn, None)
>>> # Sets EPS options from the options database. This routine must be
>>> called before `setUp()` if the user is to be allowed to set dthe solver
>>> type.
>>> E.setFromOptions()
>>> # Sets up all the internal data structures necessary for the execution
>>> of the eigensolver.
>>> E.setUp()
>>>
>>> # Solve eigenvalue problem
>>> E.solve()
>>>
>>> Print = PETSc.Sys.Print
>>>
>>> Print()
>>> Print("****************************")
>>> Print("***SLEPc Solution Results***")
>>> Print("****************************")
>>>
>>> its = E.getIterationNumber()
>>> Print("Number of iterations of the method: ", its)
>>> eps_type = E.getType()
>>> Print("Solution method: ", eps_type)
>>> nev, ncv, mpd = E.getDimensions()
>>> Print("Number of requested eigenvalues: ", nev)
>>> Print("Number of computeded eigenvectors: ", ncv)
>>> tol, maxit = E.getTolerances()
>>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>>> # Get the type of convergence
>>> conv_test = E.getConvergenceTest()
>>> Print("Selected convergence test: ", conv_test)
>>> # Get the used spectral transformation
>>> get_st = E.getST()
>>> Print("Selected spectral transformation: ", get_st)
>>> # Get the applied direct solver
>>> get_ksp = E.getDS()
>>> Print("Selected direct solver: ", get_ksp)
>>> nconv = E.getConverged()
>>> Print("Number of converged eigenpairs: ", nconv)
>>> .....
>>>
>>>
>>>
>>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>>> bsmith at mcs.anl.gov>:
>>>
>>>>
>>>>
>>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>>> wrote:
>>>> >
>>>> >
>>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>>>> griesser.jan at googlemail.com> wrote:
>>>> > With more than 1 MPI process you mean i should use spectrum slicing
>>>> in divide the full problem in smaller subproblems?
>>>> > The --with-64-bit-indices is not a possibility for me since i
>>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>>> (At least this was the error message when i tried to configure PETSc )
>>>> >
>>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>>>> integer feature" and "full 64-bit integer version" as well.
>>>>
>>>>     They use to achieve this by compiling with special Fortran flags to
>>>> promote integers to 64 bit; this is too fragile for our taste so we never
>>>> hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>>> integers we should add that to our mumps.py and test it.
>>>>
>>>>    Barry
>>>>
>>>> >
>>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>>> jroman at dsic.upv.es>:
>>>> > To use BVVECS just add the command-line option -bv_type vecs
>>>> > This causes to use a separate Vec for each column, instead of a
>>>> single long Vec of size n*m. But it is considerably slower than the default.
>>>> >
>>>> > Anyway, for such large problems you should consider using more than 1
>>>> MPI process. In that case the error may disappear because the local size is
>>>> smaller than 768000.
>>>> >
>>>> > Jose
>>>> >
>>>> >
>>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>>> escribi?:
>>>> > >
>>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>>>> griesser.jan at googlemail.com> wrote:
>>>> > > Hi all,
>>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>>> million x 3 million and ask for the smallest real 3000 (the number is
>>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>>> the output (for the 768000):
>>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>>> consider reducing the number of columns, or use BVVECS instead
>>>> > > i understand that the requested number of eigenvectors and
>>>> eigenvalues is causing an overflow but i do not understand the solution of
>>>> the problem which is stated in the error message. Can someone tell me what
>>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>>> my problem ?
>>>> > >
>>>> > > You can also reconfigure with 64-bit integers: --with-64-bit-indices
>>>> > >
>>>> > >   Thanks,
>>>> > >
>>>> > >     Matt
>>>> > >
>>>> > > Thank you very much in advance,
>>>> > > Jan
>>>> > >
>>>> > >
>>>> > >
>>>> > > --
>>>> > > What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> > > -- Norbert Wiener
>>>> > >
>>>> > > https://www.cse.buffalo.edu/~knepley/
>>>> >
>>>>
>>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/3f957cd0/attachment-0001.html>

From Peterspy at outlook.com  Wed Oct 24 10:06:40 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Wed, 24 Oct 2018 15:06:40 +0000
Subject: [petsc-users] Problems with a Laplacian problem with Neumann
 boundary condition
Message-ID: <HK0PR04MB254711040527C83435580484D0F60@HK0PR04MB2547.apcprd04.prod.outlook.com>

Hi all,
I am trying to change PETSC KSP tutorial ex34 to fortran code ex34f and I referred to ex22f, but it somehow seems wrong in my code.
Important error information points to MatSetValuesStencil() [argument out of range].
Can anyone look over my code and give any advice?
Many thanks.
----------------------------------------------------------------
SONG Pengyang ?????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86)15762264643

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/8666509c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex34f.F90
Type: application/octet-stream
Size: 9597 bytes
Desc: ex34f.F90
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/8666509c/attachment.obj>

From yjwu16 at gmail.com  Wed Oct 24 10:30:09 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Wed, 24 Oct 2018 23:30:09 +0800
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
	<CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
Message-ID: <CAAMEhBhB7yUmG_GzuL3QApUdEhm930jYeJ68aafU3HbN36CWnQ@mail.gmail.com>

Thank you very much for your reply.

Because the files I added are written in c++, including definitions of
classes and functions. I encountered a mistake when I used your method.
myprogram: class1.o myprogram.o
       ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
Error information is : unknown type name 'class'
It seems that I did not compile using the c++ compiler.

How should I set it up so that it can compile successfully?

Thanks,
Yingjie

Matthew Knepley <knepley at gmail.com> ?2018?10?24??? ??1:46???

> On Tue, Oct 23, 2018 at 11:37 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
>
>> Dear Petsc developer:
>> Hi,
>> Thank you very much for your continuous help, I recently encountered some
>> difficulties in developing programs on Petsc.
>>
>> 1. I want to use my class definition (class1. h) and class functions
>> (class1. cpp) files in my Petsc program (myprogram. c) and compile my
>> program. I have written other program before:
>> g++ -c class1.cpp myprogram.c
>> g++ -o myprogram *.o
>> I also have some knowledge of Makefile, but I don't know how to modify it
>> to achieve compilation. I really want your help.
>>
>
> With PETSc makefiles, we have automatic rules, so you can use
>
> myprogram: class1.o myprogram.o
>        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
>
>
>> 2.  In my class definition and function implementation program, I want to
>> use Petsc data structures, such as PetscInt, PetscReal. In order to
>> maintain consistency of the main program. What kind of header files should
>> I add to my code, or do I need to add any code?
>>
>
> The easiest thing to do is just
>
> #include <petsc.h>
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>> Yingjie
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/1f1056f0/attachment.html>

From balay at mcs.anl.gov  Wed Oct 24 10:35:28 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Wed, 24 Oct 2018 15:35:28 +0000
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <CAAMEhBhB7yUmG_GzuL3QApUdEhm930jYeJ68aafU3HbN36CWnQ@mail.gmail.com>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
	<CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
	<CAAMEhBhB7yUmG_GzuL3QApUdEhm930jYeJ68aafU3HbN36CWnQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.21.1810241033290.2457@asterix>

Please send complete logs. [i.e the complete compile output from your makefile. Also send your makefile]

If your c++ code uses .cxx - it should get compiled with a c++ compiler.

You might benefit by change CLINKER to CXXLINKER

Satish

On Wed, 24 Oct 2018, Yingjie Wu wrote:

> Thank you very much for your reply.
> 
> Because the files I added are written in c++, including definitions of
> classes and functions. I encountered a mistake when I used your method.
> myprogram: class1.o myprogram.o
>        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
> Error information is : unknown type name 'class'
> It seems that I did not compile using the c++ compiler.
> 
> How should I set it up so that it can compile successfully?
> 
> Thanks,
> Yingjie
> 
> Matthew Knepley <knepley at gmail.com> ?2018?10?24??? ??1:46???
> 
> > On Tue, Oct 23, 2018 at 11:37 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
> >
> >> Dear Petsc developer:
> >> Hi,
> >> Thank you very much for your continuous help, I recently encountered some
> >> difficulties in developing programs on Petsc.
> >>
> >> 1. I want to use my class definition (class1. h) and class functions
> >> (class1. cpp) files in my Petsc program (myprogram. c) and compile my
> >> program. I have written other program before:
> >> g++ -c class1.cpp myprogram.c
> >> g++ -o myprogram *.o
> >> I also have some knowledge of Makefile, but I don't know how to modify it
> >> to achieve compilation. I really want your help.
> >>
> >
> > With PETSc makefiles, we have automatic rules, so you can use
> >
> > myprogram: class1.o myprogram.o
> >        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
> >
> >
> >> 2.  In my class definition and function implementation program, I want to
> >> use Petsc data structures, such as PetscInt, PetscReal. In order to
> >> maintain consistency of the main program. What kind of header files should
> >> I add to my code, or do I need to add any code?
> >>
> >
> > The easiest thing to do is just
> >
> > #include <petsc.h>
> >
> >   Thanks,
> >
> >     Matt
> >
> >
> >> Thanks,
> >> Yingjie
> >>
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> > experiments is infinitely more interesting than any results to which their
> > experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
> > <http://www.cse.buffalo.edu/~knepley/>
> >
> 

From imilian.hartig at gmail.com  Wed Oct 24 10:36:11 2018
From: imilian.hartig at gmail.com (Maximilian Hartig)
Date: Wed, 24 Oct 2018 17:36:11 +0200
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
Message-ID: <6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>



> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li <mailto:wence at gmx.li>> wrote:
> Hi Max,
> 
> (I'm cc'ing in the petsc-users mailing list which may have more advice, if you are using PETSc you should definitely subscribe!
> 
> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com <mailto:imilian.hartig at gmail.com>> wrote:
> > 
> > Hello Lawrence,
> > 
> > sorry to message you out of the blue. My name is Max and I found your post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246 <https://github.com/firedrakeproject/firedrake/issues/1246> ) on DMPlex being able to read periodic gmsh files. I am currently trying to do just that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD work. So far I haven? t found any documentation on the periodic BC?s with DMPlex and gmsh in the official petsc documentation. 
> > I was wondering whether you?d be so kind as to point me in a general direction concerning how to achieve this. You seem experienced in using petsc and I would greatly appreciate your help. 
> 
> 
> I think the answer is "it depends". If you're just using DMPlex directly and all the of the functionality with PetscDS, then I /think/ that reading periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh format [v2]) "just works".
> 
> There are two phases here: topological and geometric. DMPlex represents the periodic topological entity directly. For example,  a circle is just a segment with one end hooked to the other. Vertices are not duplicated, or mapped to each other. This makes topology simple and easy to implement. However, then geometry is more complicated. What Plex does is allow coordinates to be represented by a discontinuous field taking values on cells, in addition to vertices. In our circle example, each cells near the cut will have 2 coordinates, one for each vertex, but they will not agree across the cut. If you define a periodic domain, then Plex can construct this coordinate field automatically using DMPlexLocalize(). These DG coordinates are then used by the integration routines.

Ok, I think I understand the concept. DMPlex reads information about both topology and coordinates from the .msh file. Creating a periodic mesh in gmsh then should allow DMPlex to identify the periodic boundaries as the ?cut? and build the mesh topology accordingly. Coordinate information is handled separately.
That means, as Lawrence suggested, building periodic meshes in gmsh and reading them in to petsc?s DMPlex should indeed ?just work?.  (From the user perspective). The only extra step is to call DMLocalizeCoordinates() after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make sense of this. 
>  
> From my side, the issue is to do with mapping that coordinate field into one that I understand (within Firedrake). You may not have this problem.
> 
> Firedrake uses its own coordinate mapping and integration routines, so they must manage the second part independently. I hope to get change this slightly soon by making the Firedrake representation a DMField, so that it looks the same to Plex.
> 
>   Thanks,
> 
>     Matt
>  
> Thanks,
> 
> Lawrence
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <https://www.cse.buffalo.edu/~knepley/>
> To read periodic meshes from GMSH, you need to use the option -dm_plex_gmsh_periodic and DMPlexCreateFromFile

Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But using this option I now generate a segmentation fault error when calling VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?

 
> 
> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
> 
> $ ./ex1 -filename ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
> 
> generating periodic meshes in gmsh may be tricky, Lisandro for sure may advice.

Thanks,
Max

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/b9423857/attachment.html>

From yjwu16 at gmail.com  Wed Oct 24 10:48:15 2018
From: yjwu16 at gmail.com (Yingjie Wu)
Date: Wed, 24 Oct 2018 23:48:15 +0800
Subject: [petsc-users] Problems about Compiling Multifile Program
In-Reply-To: <alpine.LFD.2.21.1810241033290.2457@asterix>
References: <CAAMEhBiOveGUJgMyPPHYe=tnoeZ69_LjQ7KjQC9p-c+HHQju3g@mail.gmail.com>
	<CAMYG4G=uTdcspskP=CuH2OsLyXtp1vme=Pd5KP=WK0C9RfymZw@mail.gmail.com>
	<CAAMEhBhB7yUmG_GzuL3QApUdEhm930jYeJ68aafU3HbN36CWnQ@mail.gmail.com>
	<alpine.LFD.2.21.1810241033290.2457@asterix>
Message-ID: <CAAMEhBhMnp1=HeM6DpUHHD5gJw74=K-HD2fu2P16GBT7tr9krw@mail.gmail.com>

Sorry, I made a mistake. I have a.C file with the same name in the current
directory. When I remove it, the compiler can be passed.
It seems that ${CLINKER} can automatically identify the extension of files
and compile them with different compilers.

Thanks for helping me.
Yingjie

Balay, Satish <balay at mcs.anl.gov> ?2018?10?24??? ??11:35???

> Please send complete logs. [i.e the complete compile output from your
> makefile. Also send your makefile]
>
> If your c++ code uses .cxx - it should get compiled with a c++ compiler.
>
> You might benefit by change CLINKER to CXXLINKER
>
> Satish
>
> On Wed, 24 Oct 2018, Yingjie Wu wrote:
>
> > Thank you very much for your reply.
> >
> > Because the files I added are written in c++, including definitions of
> > classes and functions. I encountered a mistake when I used your method.
> > myprogram: class1.o myprogram.o
> >        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
> > Error information is : unknown type name 'class'
> > It seems that I did not compile using the c++ compiler.
> >
> > How should I set it up so that it can compile successfully?
> >
> > Thanks,
> > Yingjie
> >
> > Matthew Knepley <knepley at gmail.com> ?2018?10?24??? ??1:46???
> >
> > > On Tue, Oct 23, 2018 at 11:37 AM Yingjie Wu <yjwu16 at gmail.com> wrote:
> > >
> > >> Dear Petsc developer:
> > >> Hi,
> > >> Thank you very much for your continuous help, I recently encountered
> some
> > >> difficulties in developing programs on Petsc.
> > >>
> > >> 1. I want to use my class definition (class1. h) and class functions
> > >> (class1. cpp) files in my Petsc program (myprogram. c) and compile my
> > >> program. I have written other program before:
> > >> g++ -c class1.cpp myprogram.c
> > >> g++ -o myprogram *.o
> > >> I also have some knowledge of Makefile, but I don't know how to
> modify it
> > >> to achieve compilation. I really want your help.
> > >>
> > >
> > > With PETSc makefiles, we have automatic rules, so you can use
> > >
> > > myprogram: class1.o myprogram.o
> > >        ${CLINKER} -o myprogram class1.o myprogram.o ${PETSC_LIB}
> > >
> > >
> > >> 2.  In my class definition and function implementation program, I
> want to
> > >> use Petsc data structures, such as PetscInt, PetscReal. In order to
> > >> maintain consistency of the main program. What kind of header files
> should
> > >> I add to my code, or do I need to add any code?
> > >>
> > >
> > > The easiest thing to do is just
> > >
> > > #include <petsc.h>
> > >
> > >   Thanks,
> > >
> > >     Matt
> > >
> > >
> > >> Thanks,
> > >> Yingjie
> > >>
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their
> > > experiments is infinitely more interesting than any results to which
> their
> > > experiments lead.
> > > -- Norbert Wiener
> > >
> > > https://www.cse.buffalo.edu/~knepley/
> > > <http://www.cse.buffalo.edu/~knepley/>
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/9f4b2f9d/attachment.html>

From griesser.jan at googlemail.com  Wed Oct 24 11:07:49 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 24 Oct 2018 18:07:49 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAMYG4GkxiHzF0UrFKR4fkD4pzcj_URi4zEQdFQ4CjCZWRpz4uA@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
	<CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
	<CAMYG4GkxiHzF0UrFKR4fkD4pzcj_URi4zEQdFQ4CjCZWRpz4uA@mail.gmail.com>
Message-ID: <CAB_-EyUfpLmcRHNxwWiAwdZibXz-K8oLkiYWvyaV43yPJ-=d6g@mail.gmail.com>

For some reason i get only this error message, also when is use the
-eps_view_pre. I started the program with nev=1, there the output is  (with
-bv_type vecs -bv_type mat -eps_view_pre)
EPS Object: 20 MPI processes
  type: krylovschur
    50% of basis vectors kept after restart
    using the locking variant
  problem type: symmetric eigenvalue problem
  selected portion of the spectrum: smallest real parts
  number of eigenvalues (nev): 1
  number of column vectors (ncv): 3
  maximum dimension of projected problem (mpd): 2
  maximum number of iterations: 1000
  tolerance: 1e-08
  convergence test: relative to the eigenvalue
BV Object: 20 MPI processes
  type: mat
  4 columns of global length 1500000
  vector orthogonalization method: classical Gram-Schmidt
  orthogonalization refinement: if needed (eta: 0.7071)
  block orthogonalization method: GS
  doing matmult as a single matrix-matrix product
DS Object: 20 MPI processes
  type: hep
  parallel operation mode: REDUNDANT
  solving the problem with: Implicit QR method (_steqr)
ST Object: 20 MPI processes
  type: shift
  shift: 0.
  number of matrices: 1




Am Mi., 24. Okt. 2018 um 16:14 Uhr schrieb Matthew Knepley <
knepley at gmail.com>:

> On Wed, Oct 24, 2018 at 10:03 AM Jan Grie?er <griesser.jan at googlemail.com>
> wrote:
>
>> This is the error message i get from my program:
>> Shape of the dynamical matrix:  (1500000, 1500000)
>>
>
> Petsc installs a signal handler, so there should be a PETSc-specific
> message before this one. Is something eating
> your output?
>
>   Thanks,
>
>      Matt
>
>
>>
>> ===================================================================================
>> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>> =   PID 122676 RUNNING AT n3512.nemo.privat
>> =   EXIT CODE: 9
>> =   CLEANING UP REMAINING PROCESSES
>> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>
>> ===================================================================================
>>    Intel(R) MPI Library troubleshooting guide:
>>       https://software.intel.com/node/561764
>>
>> ===================================================================================
>>
>>
>> Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
>> knepley at gmail.com>:
>>
>>> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <griesser.jan at googlemail.com>
>>> wrote:
>>>
>>>> Hey,
>>>> i tried to run my program as you said with -bv_type vecs and/or
>>>> -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>>>
>>>
>>> Send the actual error.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> message, which i assume (after googling a little bit around) should be
>>>> a memory problem. I tried to run it also on slightly bigger compute nodes
>>>> on our cluster with 20 cores with each 12 GB and 24 GB but the problem
>>>> still remains. The actual limit appears to be a 1.5 Million x 1.5 Million
>>>> where i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>>>> Do you have maybe an idea what the error could be? I appended also the
>>>> python method i used to solve the problem. I also tried to solve the
>>>> problem with spectrum solving but the error message remains the same.
>>>>
>>>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>>>> # Create the solver
>>>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
>>>> solver)
>>>> E = SLEPc.EPS().create()
>>>>
>>>> # Set the preconditioner
>>>> pc = PETSc.PC().create()
>>>> pc.setType(pc.Type.BJACOBI)
>>>>
>>>> # Set the linear solver
>>>> # Create the KSP object
>>>> ksp = PETSc.KSP().create()
>>>> # Create the solver, in this case GMRES
>>>> ksp.setType(ksp.Type.GMRES)
>>>> # Set the tolerances of the GMRES solver
>>>>         # Link it to the PC
>>>> ksp.setPC(pc)
>>>>
>>>> # Set up the spectral transformations
>>>> st = SLEPc.ST().create()
>>>> st.setType("shift")
>>>> st.setKSP(ksp)
>>>> # MPD stands for "maximum projected dimension". It has to due with
>>>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>>>> # an explanation. At the moment mpd is only a guess
>>>> E.setDimensions(nev=NEV, mpd = MPD)
>>>> # Eigenvalues should be real, therefore we start to order them from the
>>>> smallest real value |l.real|
>>>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>>>> # Since the dynamical matrix is symmetric and real it is hermitian
>>>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>>>> # Use the Krylov Schur method to solve the eigenvalue problem
>>>> E.setType(E.Type.KRYLOVSCHUR)
>>>> # Set the convergence criterion to relative to the eigenvalue and the
>>>> maximal number of iterations
>>>> E.setConvergenceTest(E.Conv.REL)
>>>> E.setTolerances(tol = 1e-8, max_it = 5000)
>>>> # Set the matrix in order to solve
>>>> E.setOperators(DynMatrix_nn, None)
>>>> # Sets EPS options from the options database. This routine must be
>>>> called before `setUp()` if the user is to be allowed to set dthe solver
>>>> type.
>>>> E.setFromOptions()
>>>> # Sets up all the internal data structures necessary for the execution
>>>> of the eigensolver.
>>>> E.setUp()
>>>>
>>>> # Solve eigenvalue problem
>>>> E.solve()
>>>>
>>>> Print = PETSc.Sys.Print
>>>>
>>>> Print()
>>>> Print("****************************")
>>>> Print("***SLEPc Solution Results***")
>>>> Print("****************************")
>>>>
>>>> its = E.getIterationNumber()
>>>> Print("Number of iterations of the method: ", its)
>>>> eps_type = E.getType()
>>>> Print("Solution method: ", eps_type)
>>>> nev, ncv, mpd = E.getDimensions()
>>>> Print("Number of requested eigenvalues: ", nev)
>>>> Print("Number of computeded eigenvectors: ", ncv)
>>>> tol, maxit = E.getTolerances()
>>>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>>>> # Get the type of convergence
>>>> conv_test = E.getConvergenceTest()
>>>> Print("Selected convergence test: ", conv_test)
>>>> # Get the used spectral transformation
>>>> get_st = E.getST()
>>>> Print("Selected spectral transformation: ", get_st)
>>>> # Get the applied direct solver
>>>> get_ksp = E.getDS()
>>>> Print("Selected direct solver: ", get_ksp)
>>>> nconv = E.getConverged()
>>>> Print("Number of converged eigenpairs: ", nconv)
>>>> .....
>>>>
>>>>
>>>>
>>>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>>>> bsmith at mcs.anl.gov>:
>>>>
>>>>>
>>>>>
>>>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>>>> wrote:
>>>>> >
>>>>> >
>>>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>>>>> griesser.jan at googlemail.com> wrote:
>>>>> > With more than 1 MPI process you mean i should use spectrum slicing
>>>>> in divide the full problem in smaller subproblems?
>>>>> > The --with-64-bit-indices is not a possibility for me since i
>>>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>>>> (At least this was the error message when i tried to configure PETSc )
>>>>> >
>>>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>>>>> integer feature" and "full 64-bit integer version" as well.
>>>>>
>>>>>     They use to achieve this by compiling with special Fortran flags
>>>>> to promote integers to 64 bit; this is too fragile for our taste so we
>>>>> never hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>>>> integers we should add that to our mumps.py and test it.
>>>>>
>>>>>    Barry
>>>>>
>>>>> >
>>>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>>>> jroman at dsic.upv.es>:
>>>>> > To use BVVECS just add the command-line option -bv_type vecs
>>>>> > This causes to use a separate Vec for each column, instead of a
>>>>> single long Vec of size n*m. But it is considerably slower than the default.
>>>>> >
>>>>> > Anyway, for such large problems you should consider using more than
>>>>> 1 MPI process. In that case the error may disappear because the local size
>>>>> is smaller than 768000.
>>>>> >
>>>>> > Jose
>>>>> >
>>>>> >
>>>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>>>> escribi?:
>>>>> > >
>>>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>>>>> griesser.jan at googlemail.com> wrote:
>>>>> > > Hi all,
>>>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>>>> million x 3 million and ask for the smallest real 3000 (the number is
>>>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>>>> the output (for the 768000):
>>>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>>>> consider reducing the number of columns, or use BVVECS instead
>>>>> > > i understand that the requested number of eigenvectors and
>>>>> eigenvalues is causing an overflow but i do not understand the solution of
>>>>> the problem which is stated in the error message. Can someone tell me what
>>>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>>>> my problem ?
>>>>> > >
>>>>> > > You can also reconfigure with 64-bit integers:
>>>>> --with-64-bit-indices
>>>>> > >
>>>>> > >   Thanks,
>>>>> > >
>>>>> > >     Matt
>>>>> > >
>>>>> > > Thank you very much in advance,
>>>>> > > Jan
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > > -- Norbert Wiener
>>>>> > >
>>>>> > > https://www.cse.buffalo.edu/~knepley/
>>>>> >
>>>>>
>>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/a5424641/attachment-0001.html>

From iancclin at umich.edu  Wed Oct 24 11:13:27 2018
From: iancclin at umich.edu (Ian Lin)
Date: Wed, 24 Oct 2018 12:13:27 -0400
Subject: [petsc-users] Usage of MATMPISBAIJ
Message-ID: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>

Hi PETSc team,

I am currently thinking of doing a Sparse-Dense matrix matrix multiplication, where the Sparse matrix is a larger symmetric matrix, and the dense matrix is a skinny matrix. The sparse matrix is used mainly because of the memory issue, and the sparsity is just around 10~20%. Thus I am thinking of using MPISBAIJ type to reduce the memory cost. Does MPISBAIJ support MatMatMult with MPIDense matrix? If so, how would the data be distributed over processors? Would it be the same as MPIAIJ (row are by default evenly distributed over processors, but that does not make sense to MPISBAIJ as I suppose the lower part of the matrix is not stored)? Also, just another quick question, in your experience, considering the overhead cost for sparse algorithm, how sparse should a matrix be for the sparse algorithm to outperform dense algorithm in terms of computation time?

Thanks a lot for your help.

Best regards,
Ian

From griesser.jan at googlemail.com  Wed Oct 24 11:29:49 2018
From: griesser.jan at googlemail.com (=?UTF-8?B?SmFuIEdyaWXDn2Vy?=)
Date: Wed, 24 Oct 2018 18:29:49 +0200
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyUfpLmcRHNxwWiAwdZibXz-K8oLkiYWvyaV43yPJ-=d6g@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
	<CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
	<CAMYG4GkxiHzF0UrFKR4fkD4pzcj_URi4zEQdFQ4CjCZWRpz4uA@mail.gmail.com>
	<CAB_-EyUfpLmcRHNxwWiAwdZibXz-K8oLkiYWvyaV43yPJ-=d6g@mail.gmail.com>
Message-ID: <CAB_-EyWBaStLuxmD_b5ziWAfucPZMmB5PYRnpZdOgFe0e4HzLg@mail.gmail.com>

I also run it with the -log_summary :
---------------------------------------------- PETSc Performance Summary:
----------------------------------------------



      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


/work/ws/nemo/fr_jg1080-FreeSurface_Glass-0/GlassSystems/PeriodicSystems/N500000T0.001/SolveEigenvalueProblem_par/Test/Eigensolver_petsc_slepc_no_argparse.py
on a arch-linux2-c-debug named int02.nemo.privat with 20 processors, by
fr_jg1080 Wed Oct 24 18:26:30 2018
Using Petsc Release Version 3.9.4, Sep, 11, 2018

                         Max       Max/Min        Avg      Total
Time (sec):           7.474e+02      1.00000   7.474e+02
Objects:              3.600e+01      1.00000   3.600e+01
Flop:                 1.090e+11      1.00346   1.089e+11  2.177e+12
Flop/sec:            1.459e+08      1.00346   1.457e+08  2.913e+09
Memory:               3.950e+08      1.00296              7.891e+09
MPI Messages:         3.808e+04      1.00000   3.808e+04  7.615e+05
MPI Message Lengths:  2.211e+10      1.00217   5.802e+05  4.419e+11
MPI Reductions:       1.023e+05      1.00000

Flop counting convention: 1 flop = 1 real number operation of type
(multiply/divide/add/subtract)
                            e.g., VecAXPY() for real vectors of length N
--> 2N flop
                            and VecAXPY() for complex vectors of length N
--> 8N flop

Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---
-- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts
 %Total     Avg         %Total   counts   %Total
 0:      Main Stage: 7.4739e+02 100.0%  2.1773e+12 100.0%  7.615e+05
100.0%  5.802e+05      100.0%  1.022e+05 100.0%

------------------------------------------------------------------------------------------------------------------------
See the 'Profiling' chapter of the users' manual for details on
interpreting output.
Phase summary info:
   Count: number of times phase was executed
   Time and Flop: Max - maximum over all processors
                   Ratio - ratio of maximum to minimum over all processors
   Mess: number of messages sent
   Avg. len: average message length (bytes)
   Reduct: number of global reductions
   Global: entire computation
   Stage: stages of a computation. Set stages with PetscLogStagePush() and
PetscLogStagePop().
      %T - percent time in this phase         %F - percent flop in this
phase
      %M - percent messages in this phase     %L - percent message lengths
in this phase
      %R - percent reductions in this phase
   Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over
all processors)
------------------------------------------------------------------------------------------------------------------------


      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Event                Count      Time (sec)     Flop
     --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len
Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSidedF         2 1.0 2.6670e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet                 2 1.0 6.8650e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecScatterBegin     2002 1.0 1.4380e+01 1.0 0.00e+00 0.0 7.6e+05 5.8e+05
0.0e+00  2  0100100  0   2  0100100  0     0
VecScatterEnd       2002 1.0 3.7604e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  4  0  0  0  0   4  0  0  0  0     0
VecSetRandom           1 1.0 1.6440e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMult             2002 1.0 6.0846e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05
0.0e+00 71 94100100  0  71 94100100  0  3376
MatAssemblyBegin       3 1.0 2.8129e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
6.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatAssemblyEnd         3 1.0 8.5094e+00 1.0 0.00e+00 0.0 7.6e+02 1.5e+05
3.6e+01  1  0  0  0  0   1  0  0  0  0     0
EPSSetUp               1 1.0 1.7351e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
7.6e+01  0  0  0  0  0   0  0  0  0  0     0
EPSSolve               1 1.0 6.7891e+02 1.0 1.09e+11 1.0 7.6e+05 5.8e+05
1.0e+05 91100100100100  91100100100100  3207
STSetUp                1 1.0 2.2221e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
6.0e+00  0  0  0  0  0   0  0  0  0  0     0
STApply             2002 1.0 6.0879e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05
0.0e+00 71 94100100  0  71 94100100  0  3374
BVCopy               999 1.0 2.7157e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BVMultVec           4004 1.0 2.2918e+00 1.0 2.10e+09 1.0 0.0e+00 0.0e+00
1.6e+04  0  2  0  0 16   0  2  0  0 16 18332
BVMultInPlace        999 1.0 4.8399e+01 1.0 1.20e+09 1.0 0.0e+00 0.0e+00
0.0e+00  6  1  0  0  0   6  1  0  0  0   495
BVDotVec            4004 1.0 1.0835e+01 1.0 2.70e+09 1.0 0.0e+00 0.0e+00
2.0e+04  1  2  0  0 20   1  2  0  0 20  4986
BVOrthogonalizeV    2003 1.0 1.3272e+01 1.0 4.80e+09 1.0 0.0e+00 0.0e+00
5.2e+04  2  4  0  0 51   2  4  0  0 51  7236
BVScale             2003 1.0 2.3521e-01 1.0 1.50e+08 1.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0 12773
BVSetRandom            1 1.0 1.6456e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
4.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSSolve             1000 1.0 3.3338e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSVectors           1000 1.0 6.0029e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
DSOther             2999 1.0 7.8770e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
0.0e+00  0  0  0  0  0   0  0  0  0  0     0
------------------------------------------------------------------------------------------------------------------------

Memory usage is given in bytes:

Object Type          Creations   Destructions     Memory  Descendants' Mem.
Reports information only for process 0.

--- Event Stage 0: Main Stage

              Viewer     2              1          840     0.
         PetscRandom     1              1          646     0.
           Index Set     4              4      5510472     0.
              Vector     9              9     11629608     0.
         Vec Scatter     2              2         1936     0.
              Matrix    10             10    331855732     0.
      Preconditioner     1              1         1000     0.
       Krylov Solver     1              1         1176     0.
          EPS Solver     1              1         1600     0.
  Spectral Transform     2              2         1624     0.
       Basis Vectors     1              1         2168     0.
       Direct Solver     1              1         2520     0.
              Region     1              1          672     0.
========================================================================================================================
Average time to get PetscTime(): 1.19209e-07
Average time for MPI_Barrier(): 2.67982e-05
Average time for zero size MPI_Send(): 1.08957e-05
#PETSc Option Table entries:
-bv_type mat
-eps_view_pre
-log_summary
#End of PETSc Option Table entries
Compiled without FORTRAN kernels
Compiled with full precision matrices (default)
sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
sizeof(PetscScalar) 8 sizeof(PetscInt) 4
Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
--download-mumps --with-shared-libraries=True --download-scalapack
-----------------------------------------
Libraries compiled on 2018-10-17 20:02:31 on login2.nemo.privat
Machine characteristics:
Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core
Using PETSc directory: /home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4
Using PETSc arch: arch-linux2-c-debug
-----------------------------------------

Using C compiler: mpicc  -fPIC  -wd1572 -g
Using Fortran compiler: mpif90  -fPIC -g
-----------------------------------------

Using include paths:
-I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/include
-I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/include
-----------------------------------------

Using C linker: mpicc
Using Fortran linker: mpif90
Using libraries:
-Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
-L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
-lpetsc
-Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
-L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
-Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt
-L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt
-Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib
-L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib
-Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64
-L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64
-Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin
-L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin
-Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
-L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
-Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt
-Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lcmumps -ldmumps
-lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lX11
-lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport
-lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl
-----------------------------------------



      ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################


Am Mi., 24. Okt. 2018 um 18:07 Uhr schrieb Jan Grie?er <
griesser.jan at googlemail.com>:

> For some reason i get only this error message, also when is use the
> -eps_view_pre. I started the program with nev=1, there the output is  (with
> -bv_type vecs -bv_type mat -eps_view_pre)
> EPS Object: 20 MPI processes
>   type: krylovschur
>     50% of basis vectors kept after restart
>     using the locking variant
>   problem type: symmetric eigenvalue problem
>   selected portion of the spectrum: smallest real parts
>   number of eigenvalues (nev): 1
>   number of column vectors (ncv): 3
>   maximum dimension of projected problem (mpd): 2
>   maximum number of iterations: 1000
>   tolerance: 1e-08
>   convergence test: relative to the eigenvalue
> BV Object: 20 MPI processes
>   type: mat
>   4 columns of global length 1500000
>   vector orthogonalization method: classical Gram-Schmidt
>   orthogonalization refinement: if needed (eta: 0.7071)
>   block orthogonalization method: GS
>   doing matmult as a single matrix-matrix product
> DS Object: 20 MPI processes
>   type: hep
>   parallel operation mode: REDUNDANT
>   solving the problem with: Implicit QR method (_steqr)
> ST Object: 20 MPI processes
>   type: shift
>   shift: 0.
>   number of matrices: 1
>
>
>
>
> Am Mi., 24. Okt. 2018 um 16:14 Uhr schrieb Matthew Knepley <
> knepley at gmail.com>:
>
>> On Wed, Oct 24, 2018 at 10:03 AM Jan Grie?er <griesser.jan at googlemail.com>
>> wrote:
>>
>>> This is the error message i get from my program:
>>> Shape of the dynamical matrix:  (1500000, 1500000)
>>>
>>
>> Petsc installs a signal handler, so there should be a PETSc-specific
>> message before this one. Is something eating
>> your output?
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>>
>>> ===================================================================================
>>> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>> =   PID 122676 RUNNING AT n3512.nemo.privat
>>> =   EXIT CODE: 9
>>> =   CLEANING UP REMAINING PROCESSES
>>> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>>
>>> ===================================================================================
>>>    Intel(R) MPI Library troubleshooting guide:
>>>       https://software.intel.com/node/561764
>>>
>>> ===================================================================================
>>>
>>>
>>> Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
>>> knepley at gmail.com>:
>>>
>>>> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <
>>>> griesser.jan at googlemail.com> wrote:
>>>>
>>>>> Hey,
>>>>> i tried to run my program as you said with -bv_type vecs and/or
>>>>> -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>>>>
>>>>
>>>> Send the actual error.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> message, which i assume (after googling a little bit around) should be
>>>>> a memory problem. I tried to run it also on slightly bigger compute nodes
>>>>> on our cluster with 20 cores with each 12 GB and 24 GB but the problem
>>>>> still remains. The actual limit appears to be a 1.5 Million x 1.5 Million
>>>>> where i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>>>>> Do you have maybe an idea what the error could be? I appended also the
>>>>> python method i used to solve the problem. I also tried to solve the
>>>>> problem with spectrum solving but the error message remains the same.
>>>>>
>>>>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>>>>> # Create the solver
>>>>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue problem
>>>>> solver)
>>>>> E = SLEPc.EPS().create()
>>>>>
>>>>> # Set the preconditioner
>>>>> pc = PETSc.PC().create()
>>>>> pc.setType(pc.Type.BJACOBI)
>>>>>
>>>>> # Set the linear solver
>>>>> # Create the KSP object
>>>>> ksp = PETSc.KSP().create()
>>>>> # Create the solver, in this case GMRES
>>>>> ksp.setType(ksp.Type.GMRES)
>>>>> # Set the tolerances of the GMRES solver
>>>>>         # Link it to the PC
>>>>> ksp.setPC(pc)
>>>>>
>>>>> # Set up the spectral transformations
>>>>> st = SLEPc.ST().create()
>>>>> st.setType("shift")
>>>>> st.setKSP(ksp)
>>>>> # MPD stands for "maximum projected dimension". It has to due with
>>>>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>>>>> # an explanation. At the moment mpd is only a guess
>>>>> E.setDimensions(nev=NEV, mpd = MPD)
>>>>> # Eigenvalues should be real, therefore we start to order them from
>>>>> the smallest real value |l.real|
>>>>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>>>>> # Since the dynamical matrix is symmetric and real it is hermitian
>>>>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>>>>> # Use the Krylov Schur method to solve the eigenvalue problem
>>>>> E.setType(E.Type.KRYLOVSCHUR)
>>>>> # Set the convergence criterion to relative to the eigenvalue and the
>>>>> maximal number of iterations
>>>>> E.setConvergenceTest(E.Conv.REL)
>>>>> E.setTolerances(tol = 1e-8, max_it = 5000)
>>>>> # Set the matrix in order to solve
>>>>> E.setOperators(DynMatrix_nn, None)
>>>>> # Sets EPS options from the options database. This routine must be
>>>>> called before `setUp()` if the user is to be allowed to set dthe solver
>>>>> type.
>>>>> E.setFromOptions()
>>>>> # Sets up all the internal data structures necessary for the execution
>>>>> of the eigensolver.
>>>>> E.setUp()
>>>>>
>>>>> # Solve eigenvalue problem
>>>>> E.solve()
>>>>>
>>>>> Print = PETSc.Sys.Print
>>>>>
>>>>> Print()
>>>>> Print("****************************")
>>>>> Print("***SLEPc Solution Results***")
>>>>> Print("****************************")
>>>>>
>>>>> its = E.getIterationNumber()
>>>>> Print("Number of iterations of the method: ", its)
>>>>> eps_type = E.getType()
>>>>> Print("Solution method: ", eps_type)
>>>>> nev, ncv, mpd = E.getDimensions()
>>>>> Print("Number of requested eigenvalues: ", nev)
>>>>> Print("Number of computeded eigenvectors: ", ncv)
>>>>> tol, maxit = E.getTolerances()
>>>>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>>>>> # Get the type of convergence
>>>>> conv_test = E.getConvergenceTest()
>>>>> Print("Selected convergence test: ", conv_test)
>>>>> # Get the used spectral transformation
>>>>> get_st = E.getST()
>>>>> Print("Selected spectral transformation: ", get_st)
>>>>> # Get the applied direct solver
>>>>> get_ksp = E.getDS()
>>>>> Print("Selected direct solver: ", get_ksp)
>>>>> nconv = E.getConverged()
>>>>> Print("Number of converged eigenpairs: ", nconv)
>>>>> .....
>>>>>
>>>>>
>>>>>
>>>>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>>>>> bsmith at mcs.anl.gov>:
>>>>>
>>>>>>
>>>>>>
>>>>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>>>>> wrote:
>>>>>> >
>>>>>> >
>>>>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>>>>>> griesser.jan at googlemail.com> wrote:
>>>>>> > With more than 1 MPI process you mean i should use spectrum slicing
>>>>>> in divide the full problem in smaller subproblems?
>>>>>> > The --with-64-bit-indices is not a possibility for me since i
>>>>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>>>>> (At least this was the error message when i tried to configure PETSc )
>>>>>> >
>>>>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective 64-bit
>>>>>> integer feature" and "full 64-bit integer version" as well.
>>>>>>
>>>>>>     They use to achieve this by compiling with special Fortran flags
>>>>>> to promote integers to 64 bit; this is too fragile for our taste so we
>>>>>> never hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>>>>> integers we should add that to our mumps.py and test it.
>>>>>>
>>>>>>    Barry
>>>>>>
>>>>>> >
>>>>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>>>>> jroman at dsic.upv.es>:
>>>>>> > To use BVVECS just add the command-line option -bv_type vecs
>>>>>> > This causes to use a separate Vec for each column, instead of a
>>>>>> single long Vec of size n*m. But it is considerably slower than the default.
>>>>>> >
>>>>>> > Anyway, for such large problems you should consider using more than
>>>>>> 1 MPI process. In that case the error may disappear because the local size
>>>>>> is smaller than 768000.
>>>>>> >
>>>>>> > Jose
>>>>>> >
>>>>>> >
>>>>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>>>>> escribi?:
>>>>>> > >
>>>>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>>>>>> griesser.jan at googlemail.com> wrote:
>>>>>> > > Hi all,
>>>>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>>>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>>>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>>>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>>>>> million x 3 million and ask for the smallest real 3000 (the number is
>>>>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>>>>> the output (for the 768000):
>>>>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>>>>> consider reducing the number of columns, or use BVVECS instead
>>>>>> > > i understand that the requested number of eigenvectors and
>>>>>> eigenvalues is causing an overflow but i do not understand the solution of
>>>>>> the problem which is stated in the error message. Can someone tell me what
>>>>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>>>>> my problem ?
>>>>>> > >
>>>>>> > > You can also reconfigure with 64-bit integers:
>>>>>> --with-64-bit-indices
>>>>>> > >
>>>>>> > >   Thanks,
>>>>>> > >
>>>>>> > >     Matt
>>>>>> > >
>>>>>> > > Thank you very much in advance,
>>>>>> > > Jan
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > > -- Norbert Wiener
>>>>>> > >
>>>>>> > > https://www.cse.buffalo.edu/~knepley/
>>>>>> >
>>>>>>
>>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/1be3fab0/attachment-0001.html>

From knepley at gmail.com  Wed Oct 24 11:46:34 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Oct 2018 12:46:34 -0400
Subject: [petsc-users] PetscInt overflow
In-Reply-To: <CAB_-EyWBaStLuxmD_b5ziWAfucPZMmB5PYRnpZdOgFe0e4HzLg@mail.gmail.com>
References: <CAB_-EyWXU3KqfVAjA7sByPUB8eQhTE_oOexqpnrZKCL+QtpZeQ@mail.gmail.com>
	<CAMYG4G=MpoT+neMSwk-02ZtNeWq-m0r+VkxPpd61y1BynGV9bQ@mail.gmail.com>
	<97941479-55EC-4D8B-B8D4-13675EBBF628@dsic.upv.es>
	<CAB_-EyU9z3bAO3OwcZhmwkTh0xMLifxVeso6s2V5epzV2tnyKw@mail.gmail.com>
	<CA+MQGp8gdPn_vwyvcT9wektbEBbt=eC4QfJ+Og=bmNgDObiYQQ@mail.gmail.com>
	<C2976750-9232-4F82-B6CA-FF9E06BAA8AF@anl.gov>
	<CAB_-EyU-_OoKWd7fWYX9bAhckHdO6K_YmzrSi8-6UR9P4-CAUw@mail.gmail.com>
	<CAMYG4G=gz1A6MBK-kg4NcvWz7gGOQAV37PB9NgCGQho7Z_Hs4Q@mail.gmail.com>
	<CAB_-EyXrJzmAE1H15MiUiTndkOHhC81BmxLwGQyUpaXfSqSLqg@mail.gmail.com>
	<CAMYG4GkxiHzF0UrFKR4fkD4pzcj_URi4zEQdFQ4CjCZWRpz4uA@mail.gmail.com>
	<CAB_-EyUfpLmcRHNxwWiAwdZibXz-K8oLkiYWvyaV43yPJ-=d6g@mail.gmail.com>
	<CAB_-EyWBaStLuxmD_b5ziWAfucPZMmB5PYRnpZdOgFe0e4HzLg@mail.gmail.com>
Message-ID: <CAMYG4G=tKtdgX9btHpLZiVFfs_wpzGPFhY69oXxESr-aRp3uxg@mail.gmail.com>

As it says, if you are looking at performance, you should configure using
--with-debugging=0.

It says SLEPc is using 8GB. Is this what you see?

  Matt

On Wed, Oct 24, 2018 at 12:30 PM Jan Grie?er <griesser.jan at googlemail.com>
wrote:

> I also run it with the -log_summary :
> ---------------------------------------------- PETSc Performance Summary:
> ----------------------------------------------
>
>
>
>       ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
>
>
> /work/ws/nemo/fr_jg1080-FreeSurface_Glass-0/GlassSystems/PeriodicSystems/N500000T0.001/SolveEigenvalueProblem_par/Test/Eigensolver_petsc_slepc_no_argparse.py
> on a arch-linux2-c-debug named int02.nemo.privat with 20 processors, by
> fr_jg1080 Wed Oct 24 18:26:30 2018
> Using Petsc Release Version 3.9.4, Sep, 11, 2018
>
>                          Max       Max/Min        Avg      Total
> Time (sec):           7.474e+02      1.00000   7.474e+02
> Objects:              3.600e+01      1.00000   3.600e+01
> Flop:                 1.090e+11      1.00346   1.089e+11  2.177e+12
> Flop/sec:            1.459e+08      1.00346   1.457e+08  2.913e+09
> Memory:               3.950e+08      1.00296              7.891e+09
> MPI Messages:         3.808e+04      1.00000   3.808e+04  7.615e+05
> MPI Message Lengths:  2.211e+10      1.00217   5.802e+05  4.419e+11
> MPI Reductions:       1.023e+05      1.00000
>
> Flop counting convention: 1 flop = 1 real number operation of type
> (multiply/divide/add/subtract)
>                             e.g., VecAXPY() for real vectors of length N
> --> 2N flop
>                             and VecAXPY() for complex vectors of length N
> --> 8N flop
>
> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages
> ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total   counts
>  %Total     Avg         %Total   counts   %Total
>  0:      Main Stage: 7.4739e+02 100.0%  2.1773e+12 100.0%  7.615e+05
> 100.0%  5.802e+05      100.0%  1.022e+05 100.0%
>
>
> ------------------------------------------------------------------------------------------------------------------------
> See the 'Profiling' chapter of the users' manual for details on
> interpreting output.
> Phase summary info:
>    Count: number of times phase was executed
>    Time and Flop: Max - maximum over all processors
>                    Ratio - ratio of maximum to minimum over all processors
>    Mess: number of messages sent
>    Avg. len: average message length (bytes)
>    Reduct: number of global reductions
>    Global: entire computation
>    Stage: stages of a computation. Set stages with PetscLogStagePush() and
> PetscLogStagePop().
>       %T - percent time in this phase         %F - percent flop in this
> phase
>       %M - percent messages in this phase     %L - percent message lengths
> in this phase
>       %R - percent reductions in this phase
>    Total Mflop/s: 10e-6 * (sum of flop over all processors)/(max time over
> all processors)
>
> ------------------------------------------------------------------------------------------------------------------------
>
>
>       ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
>
>
> Event                Count      Time (sec)     Flop
>      --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len
> Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>
> ------------------------------------------------------------------------------------------------------------------------
>
> --- Event Stage 0: Main Stage
>
> BuildTwoSidedF         2 1.0 2.6670e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet                 2 1.0 6.8650e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecScatterBegin     2002 1.0 1.4380e+01 1.0 0.00e+00 0.0 7.6e+05 5.8e+05
> 0.0e+00  2  0100100  0   2  0100100  0     0
> VecScatterEnd       2002 1.0 3.7604e+01 1.5 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  4  0  0  0  0   4  0  0  0  0     0
> VecSetRandom           1 1.0 1.6440e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMult             2002 1.0 6.0846e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05
> 0.0e+00 71 94100100  0  71 94100100  0  3376
> MatAssemblyBegin       3 1.0 2.8129e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 6.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatAssemblyEnd         3 1.0 8.5094e+00 1.0 0.00e+00 0.0 7.6e+02 1.5e+05
> 3.6e+01  1  0  0  0  0   1  0  0  0  0     0
> EPSSetUp               1 1.0 1.7351e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 7.6e+01  0  0  0  0  0   0  0  0  0  0     0
> EPSSolve               1 1.0 6.7891e+02 1.0 1.09e+11 1.0 7.6e+05 5.8e+05
> 1.0e+05 91100100100100  91100100100100  3207
> STSetUp                1 1.0 2.2221e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00
> 6.0e+00  0  0  0  0  0   0  0  0  0  0     0
> STApply             2002 1.0 6.0879e+02 1.2 1.03e+11 1.0 7.6e+05 5.8e+05
> 0.0e+00 71 94100100  0  71 94100100  0  3374
> BVCopy               999 1.0 2.7157e-01 1.2 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> BVMultVec           4004 1.0 2.2918e+00 1.0 2.10e+09 1.0 0.0e+00 0.0e+00
> 1.6e+04  0  2  0  0 16   0  2  0  0 16 18332
> BVMultInPlace        999 1.0 4.8399e+01 1.0 1.20e+09 1.0 0.0e+00 0.0e+00
> 0.0e+00  6  1  0  0  0   6  1  0  0  0   495
> BVDotVec            4004 1.0 1.0835e+01 1.0 2.70e+09 1.0 0.0e+00 0.0e+00
> 2.0e+04  1  2  0  0 20   1  2  0  0 20  4986
> BVOrthogonalizeV    2003 1.0 1.3272e+01 1.0 4.80e+09 1.0 0.0e+00 0.0e+00
> 5.2e+04  2  4  0  0 51   2  4  0  0 51  7236
> BVScale             2003 1.0 2.3521e-01 1.0 1.50e+08 1.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0 12773
> BVSetRandom            1 1.0 1.6456e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 4.0e+00  0  0  0  0  0   0  0  0  0  0     0
> DSSolve             1000 1.0 3.3338e+00 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> DSVectors           1000 1.0 6.0029e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> DSOther             2999 1.0 7.8770e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>
> ------------------------------------------------------------------------------------------------------------------------
>
> Memory usage is given in bytes:
>
> Object Type          Creations   Destructions     Memory  Descendants' Mem.
> Reports information only for process 0.
>
> --- Event Stage 0: Main Stage
>
>               Viewer     2              1          840     0.
>          PetscRandom     1              1          646     0.
>            Index Set     4              4      5510472     0.
>               Vector     9              9     11629608     0.
>          Vec Scatter     2              2         1936     0.
>               Matrix    10             10    331855732     0.
>       Preconditioner     1              1         1000     0.
>        Krylov Solver     1              1         1176     0.
>           EPS Solver     1              1         1600     0.
>   Spectral Transform     2              2         1624     0.
>        Basis Vectors     1              1         2168     0.
>        Direct Solver     1              1         2520     0.
>               Region     1              1          672     0.
>
> ========================================================================================================================
> Average time to get PetscTime(): 1.19209e-07
> Average time for MPI_Barrier(): 2.67982e-05
> Average time for zero size MPI_Send(): 1.08957e-05
> #PETSc Option Table entries:
> -bv_type mat
> -eps_view_pre
> -log_summary
> #End of PETSc Option Table entries
> Compiled without FORTRAN kernels
> Compiled with full precision matrices (default)
> sizeof(short) 2 sizeof(int) 4 sizeof(long) 8 sizeof(void*) 8
> sizeof(PetscScalar) 8 sizeof(PetscInt) 4
> Configure options: --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90
> --download-mumps --with-shared-libraries=True --download-scalapack
> -----------------------------------------
> Libraries compiled on 2018-10-17 20:02:31 on login2.nemo.privat
> Machine characteristics:
> Linux-3.10.0-693.21.1.el7.x86_64-x86_64-with-centos-7.4.1708-Core
> Using PETSc directory: /home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4
> Using PETSc arch: arch-linux2-c-debug
> -----------------------------------------
>
> Using C compiler: mpicc  -fPIC  -wd1572 -g
> Using Fortran compiler: mpif90  -fPIC -g
> -----------------------------------------
>
> Using include paths:
> -I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/include
> -I/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/include
> -----------------------------------------
>
> Using C linker: mpicc
> Using Fortran linker: mpif90
> Using libraries:
> -Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
> -L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
> -lpetsc
> -Wl,-rpath,/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
> -L/home/fr/fr_fr/fr_jg1080/Libaries/petsc-3.9.4/arch-linux2-c-debug/lib
> -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt
> -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib/debug_mt
> -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib
> -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/mpi/intel64/lib
> -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64
> -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries/linux/lib/intel64
> -Wl,-rpath,/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin
> -L/opt/bwhpc/common/compiler/intel/2018.3.222/compilers_and_libraries_2018.3.222/linux/compiler/lib/intel64_lin
> -Wl,-rpath,/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -L/usr/lib/gcc/x86_64-redhat-linux/4.8.5
> -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib/debug_mt
> -Wl,-rpath,/opt/intel/mpi-rt/2017.0.0/intel64/lib -lcmumps -ldmumps
> -lsmumps -lzmumps -lmumps_common -lpord -lscalapack -llapack -lblas -lX11
> -lstdc++ -ldl -lmpifort -lmpi -lmpigi -lrt -lpthread -lifport
> -lifcoremt_pic -limf -lsvml -lm -lipgo -lirc -lgcc_s -lirc_s -lstdc++ -ldl
> -----------------------------------------
>
>
>
>       ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
>
>
> Am Mi., 24. Okt. 2018 um 18:07 Uhr schrieb Jan Grie?er <
> griesser.jan at googlemail.com>:
>
>> For some reason i get only this error message, also when is use the
>> -eps_view_pre. I started the program with nev=1, there the output is  (with
>> -bv_type vecs -bv_type mat -eps_view_pre)
>> EPS Object: 20 MPI processes
>>   type: krylovschur
>>     50% of basis vectors kept after restart
>>     using the locking variant
>>   problem type: symmetric eigenvalue problem
>>   selected portion of the spectrum: smallest real parts
>>   number of eigenvalues (nev): 1
>>   number of column vectors (ncv): 3
>>   maximum dimension of projected problem (mpd): 2
>>   maximum number of iterations: 1000
>>   tolerance: 1e-08
>>   convergence test: relative to the eigenvalue
>> BV Object: 20 MPI processes
>>   type: mat
>>   4 columns of global length 1500000
>>   vector orthogonalization method: classical Gram-Schmidt
>>   orthogonalization refinement: if needed (eta: 0.7071)
>>   block orthogonalization method: GS
>>   doing matmult as a single matrix-matrix product
>> DS Object: 20 MPI processes
>>   type: hep
>>   parallel operation mode: REDUNDANT
>>   solving the problem with: Implicit QR method (_steqr)
>> ST Object: 20 MPI processes
>>   type: shift
>>   shift: 0.
>>   number of matrices: 1
>>
>>
>>
>>
>> Am Mi., 24. Okt. 2018 um 16:14 Uhr schrieb Matthew Knepley <
>> knepley at gmail.com>:
>>
>>> On Wed, Oct 24, 2018 at 10:03 AM Jan Grie?er <
>>> griesser.jan at googlemail.com> wrote:
>>>
>>>> This is the error message i get from my program:
>>>> Shape of the dynamical matrix:  (1500000, 1500000)
>>>>
>>>
>>> Petsc installs a signal handler, so there should be a PETSc-specific
>>> message before this one. Is something eating
>>> your output?
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>>
>>>> ===================================================================================
>>>> =   BAD TERMINATION OF ONE OF YOUR APPLICATION PROCESSES
>>>> =   PID 122676 RUNNING AT n3512.nemo.privat
>>>> =   EXIT CODE: 9
>>>> =   CLEANING UP REMAINING PROCESSES
>>>> =   YOU CAN IGNORE THE BELOW CLEANUP MESSAGES
>>>>
>>>> ===================================================================================
>>>>    Intel(R) MPI Library troubleshooting guide:
>>>>       https://software.intel.com/node/561764
>>>>
>>>> ===================================================================================
>>>>
>>>>
>>>> Am Mi., 24. Okt. 2018 um 16:01 Uhr schrieb Matthew Knepley <
>>>> knepley at gmail.com>:
>>>>
>>>>> On Wed, Oct 24, 2018 at 9:38 AM Jan Grie?er <
>>>>> griesser.jan at googlemail.com> wrote:
>>>>>
>>>>>> Hey,
>>>>>> i tried to run my program as you said with -bv_type vecs and/or
>>>>>> -bv_type mat, but instead of the PETScInt overflow i now get an MPI Error 9
>>>>>>
>>>>>
>>>>> Send the actual error.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> message, which i assume (after googling a little bit around) should
>>>>>> be a memory problem. I tried to run it also on slightly bigger compute
>>>>>> nodes on our cluster with 20 cores with each 12 GB and 24 GB but the
>>>>>> problem still remains. The actual limit appears to be a 1.5 Million x 1.5
>>>>>> Million where i  searched for NEV = 1500 and MPD = 500/ 200 eigenvalues.
>>>>>> Do you have maybe an idea what the error could be? I appended also
>>>>>> the python method i used to solve the problem. I also tried to solve the
>>>>>> problem with spectrum solving but the error message remains the same.
>>>>>>
>>>>>> def solve_eigensystem(DynMatrix_nn, NEV, MPD, Dimension):
>>>>>> # Create the solver
>>>>>> # E is used as an acronym for the EPS solver (EPS = Eigenvalue
>>>>>> problem solver)
>>>>>> E = SLEPc.EPS().create()
>>>>>>
>>>>>> # Set the preconditioner
>>>>>> pc = PETSc.PC().create()
>>>>>> pc.setType(pc.Type.BJACOBI)
>>>>>>
>>>>>> # Set the linear solver
>>>>>> # Create the KSP object
>>>>>> ksp = PETSc.KSP().create()
>>>>>> # Create the solver, in this case GMRES
>>>>>> ksp.setType(ksp.Type.GMRES)
>>>>>> # Set the tolerances of the GMRES solver
>>>>>>         # Link it to the PC
>>>>>> ksp.setPC(pc)
>>>>>>
>>>>>> # Set up the spectral transformations
>>>>>> st = SLEPc.ST().create()
>>>>>> st.setType("shift")
>>>>>> st.setKSP(ksp)
>>>>>> # MPD stands for "maximum projected dimension". It has to due with
>>>>>> computational cost, please read Chap. 2.6.5 of SLEPc docu for
>>>>>> # an explanation. At the moment mpd is only a guess
>>>>>> E.setDimensions(nev=NEV, mpd = MPD)
>>>>>> # Eigenvalues should be real, therefore we start to order them from
>>>>>> the smallest real value |l.real|
>>>>>> E.setWhichEigenpairs(E.Which.SMALLEST_REAL)
>>>>>> # Since the dynamical matrix is symmetric and real it is hermitian
>>>>>> E.setProblemType(SLEPc.EPS.ProblemType.HEP)
>>>>>> # Use the Krylov Schur method to solve the eigenvalue problem
>>>>>> E.setType(E.Type.KRYLOVSCHUR)
>>>>>> # Set the convergence criterion to relative to the eigenvalue and the
>>>>>> maximal number of iterations
>>>>>> E.setConvergenceTest(E.Conv.REL)
>>>>>> E.setTolerances(tol = 1e-8, max_it = 5000)
>>>>>> # Set the matrix in order to solve
>>>>>> E.setOperators(DynMatrix_nn, None)
>>>>>> # Sets EPS options from the options database. This routine must be
>>>>>> called before `setUp()` if the user is to be allowed to set dthe solver
>>>>>> type.
>>>>>> E.setFromOptions()
>>>>>> # Sets up all the internal data structures necessary for the
>>>>>> execution of the eigensolver.
>>>>>> E.setUp()
>>>>>>
>>>>>> # Solve eigenvalue problem
>>>>>> E.solve()
>>>>>>
>>>>>> Print = PETSc.Sys.Print
>>>>>>
>>>>>> Print()
>>>>>> Print("****************************")
>>>>>> Print("***SLEPc Solution Results***")
>>>>>> Print("****************************")
>>>>>>
>>>>>> its = E.getIterationNumber()
>>>>>> Print("Number of iterations of the method: ", its)
>>>>>> eps_type = E.getType()
>>>>>> Print("Solution method: ", eps_type)
>>>>>> nev, ncv, mpd = E.getDimensions()
>>>>>> Print("Number of requested eigenvalues: ", nev)
>>>>>> Print("Number of computeded eigenvectors: ", ncv)
>>>>>> tol, maxit = E.getTolerances()
>>>>>> Print("Stopping condition: (tol, maxit)", (tol, maxit))
>>>>>> # Get the type of convergence
>>>>>> conv_test = E.getConvergenceTest()
>>>>>> Print("Selected convergence test: ", conv_test)
>>>>>> # Get the used spectral transformation
>>>>>> get_st = E.getST()
>>>>>> Print("Selected spectral transformation: ", get_st)
>>>>>> # Get the applied direct solver
>>>>>> get_ksp = E.getDS()
>>>>>> Print("Selected direct solver: ", get_ksp)
>>>>>> nconv = E.getConverged()
>>>>>> Print("Number of converged eigenpairs: ", nconv)
>>>>>> .....
>>>>>>
>>>>>>
>>>>>>
>>>>>> Am Fr., 19. Okt. 2018 um 21:00 Uhr schrieb Smith, Barry F. <
>>>>>> bsmith at mcs.anl.gov>:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> > On Oct 19, 2018, at 7:56 AM, Zhang, Junchao <jczhang at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> >
>>>>>>> > On Fri, Oct 19, 2018 at 4:02 AM Jan Grie?er <
>>>>>>> griesser.jan at googlemail.com> wrote:
>>>>>>> > With more than 1 MPI process you mean i should use spectrum
>>>>>>> slicing in divide the full problem in smaller subproblems?
>>>>>>> > The --with-64-bit-indices is not a possibility for me since i
>>>>>>> configured petsc with mumps, which does not allow to use the 64-bit version
>>>>>>> (At least this was the error message when i tried to configure PETSc )
>>>>>>> >
>>>>>>> > MUMPS 5.1.2 manual chapter 2.4.2 says it supports "Selective
>>>>>>> 64-bit integer feature" and "full 64-bit integer version" as well.
>>>>>>>
>>>>>>>     They use to achieve this by compiling with special Fortran flags
>>>>>>> to promote integers to 64 bit; this is too fragile for our taste so we
>>>>>>> never hooked PETSc up wit it. If they have a dependable way of using 64 bit
>>>>>>> integers we should add that to our mumps.py and test it.
>>>>>>>
>>>>>>>    Barry
>>>>>>>
>>>>>>> >
>>>>>>> > Am Mi., 17. Okt. 2018 um 18:24 Uhr schrieb Jose E. Roman <
>>>>>>> jroman at dsic.upv.es>:
>>>>>>> > To use BVVECS just add the command-line option -bv_type vecs
>>>>>>> > This causes to use a separate Vec for each column, instead of a
>>>>>>> single long Vec of size n*m. But it is considerably slower than the default.
>>>>>>> >
>>>>>>> > Anyway, for such large problems you should consider using more
>>>>>>> than 1 MPI process. In that case the error may disappear because the local
>>>>>>> size is smaller than 768000.
>>>>>>> >
>>>>>>> > Jose
>>>>>>> >
>>>>>>> >
>>>>>>> > > El 17 oct 2018, a las 17:58, Matthew Knepley <knepley at gmail.com>
>>>>>>> escribi?:
>>>>>>> > >
>>>>>>> > > On Wed, Oct 17, 2018 at 11:54 AM Jan Grie?er <
>>>>>>> griesser.jan at googlemail.com> wrote:
>>>>>>> > > Hi all,
>>>>>>> > > i am using slepc4py and petsc4py to solve for the smallest real
>>>>>>> eigenvalues and eigenvectors. For my test cases with a matrix A of the size
>>>>>>> 30k x 30k solving for the smallest soutions works quite well, but when i
>>>>>>> increase the dimension of my system to around A = 768000 x 768000 or 3
>>>>>>> million x 3 million and ask for the smallest real 3000 (the number is
>>>>>>> increasing with increasing system size) eigenvalues and eigenvectors i get
>>>>>>> the output (for the 768000):
>>>>>>> > >  The product 4001 times 768000 overflows the size of PetscInt;
>>>>>>> consider reducing the number of columns, or use BVVECS instead
>>>>>>> > > i understand that the requested number of eigenvectors and
>>>>>>> eigenvalues is causing an overflow but i do not understand the solution of
>>>>>>> the problem which is stated in the error message. Can someone tell me what
>>>>>>> exactly BVVECS is and how i can use it? Or is there any other solution to
>>>>>>> my problem ?
>>>>>>> > >
>>>>>>> > > You can also reconfigure with 64-bit integers:
>>>>>>> --with-64-bit-indices
>>>>>>> > >
>>>>>>> > >   Thanks,
>>>>>>> > >
>>>>>>> > >     Matt
>>>>>>> > >
>>>>>>> > > Thank you very much in advance,
>>>>>>> > > Jan
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > --
>>>>>>> > > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > > -- Norbert Wiener
>>>>>>> > >
>>>>>>> > > https://www.cse.buffalo.edu/~knepley/
>>>>>>> >
>>>>>>>
>>>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/f10e063b/attachment-0001.html>

From hzhang at mcs.anl.gov  Wed Oct 24 11:49:34 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Wed, 24 Oct 2018 16:49:34 +0000
Subject: [petsc-users] Usage of MATMPISBAIJ
In-Reply-To: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
References: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
Message-ID: <CAGCphBuhSNMYGYgTBVW6GQ-owN+aTG7a=Bfbnqf8na2HNzcZQQ@mail.gmail.com>

Ian:
I am currently thinking of doing a Sparse-Dense matrix matrix multiplication, where the Sparse matrix is a larger symmetric matrix, and the dense matrix is a skinny matrix. The sparse matrix is used mainly because of the memory issue, and the sparsity is just around 10~20%. Thus I am thinking of using MPISBAIJ type to reduce the memory cost. Does MPISBAIJ support MatMatMult with MPIDense matrix? If so, how would the data be distributed over processors? Would it be the same as MPIAIJ (row are by default evenly distributed over processors, but that does not make sense to MPISBAIJ as I suppose the lower part of the matrix is not stored)?
We do not support MatMatMult with MPISBAIJ and MPIDense matrix, only MPIAIJ and MPIDense matrix.

Also, just another quick question, in your experience, considering the overhead cost for sparse algorithm, how sparse should a matrix be for the sparse algorithm to outperform dense algorithm in terms of computation time?
Sparse matrix normally is defined as the sparsity less than 10%, or less than 1%. I would consider 20% as a dense matrix.
The complexity of C=A*B is O(nnzA*nnzB), nnzA = num of nonzeros of A.
Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/c890a90e/attachment.html>

From abdullahasivas at gmail.com  Wed Oct 24 12:59:23 2018
From: abdullahasivas at gmail.com (Abdullah Ali Sivas)
Date: Wed, 24 Oct 2018 13:59:23 -0400
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
	<CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
	<CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>
Message-ID: <CAPyJGmCEOCN5-HLSFxvtEPbL2nnPmCkQUhdOC1q4hW2qFMfk2g@mail.gmail.com>

Hi Stefano,

I am trying to solve the div-div problem (or grad-div problem in strong
form) with a H(div)-conforming FEM. I am getting the matrices from an
external source (to be clear, from an ngsolve script) and I am not sure if
it is possible to get a MATIS matrix out of that. So I am just treating it
as if I am not able to access the assembly code. The results are 2, 31, 26,
27, 31 iterations, respectively, for matrix sizes 282, 1095, 4314, 17133,
67242, 267549. However, norm of the residual also grows significantly;
7.38369e-09 for 1095 and 5.63828e-07 for 267549. I can try larger sizes, or
maybe this is expected for this case.

As a side question, if we are dividing the domain into number of MPI
processes subdomains, does it mean that convergence is affected negatively
by the increasing number of processes? I know that alternating Schwarz
method and some other domain decomposition methods sometimes suffer from
the decreasing radius of the subdomains. It sounds like BDDC is pretty
similar to those by your description.

Best wishes,
Abdullah Ali Sivas

On Wed, 24 Oct 2018 at 05:28, Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> Abdullah,
>
> The "Neumann" problems Jed is referring to result from assembling your
> problem on each subdomain ( = MPI process) separately.
> Assuming you are using FEM, these problems have been historically  named
> "Neumann" as they correspond to a problem with natural boundary conditions
> (Neumann bc for Poisson).
> Note that in PETSc the subdomain decomposition is associated with the mesh
> decomposition.
>
> When converting from an assembled AIJ matrix to a MATIS format, such
> "Neumann" information is lost.
> You can disassemble an AIJ matrix, in the sense that you can find local
> matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
> but you cannot guarantee (unless if you solve an optimization problem) that
> the disassembling will produce subdomain Neumann problems that are
> consistent with your FEM problem.
>
> I have added such disassembling code a few months ago, just to have
> another alternative for preconditioning AIJ matrices in PETSc; there are
> few tweaks one can do to improve the quality of the disassembling, but I
> discourage its usage unless you don't have access to the FEM assembly code.
>
> With that said, what problem are you trying to solve? Are you using DMDA
> or DMPlex? What are the results you obtained with using the automatic
> disassembling?
>
> Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
> abdullahasivas at gmail.com> ha scritto:
>
>> Hi Jed,
>>
>> Thanks for your reply. The assembled matrix I have corresponds to the
>> full problem on the full mesh. There are no "Neumann" problems (or any sort
>> of domain decomposition) defined in the code generates the matrix. However,
>> I think assembling the full problem is equivalent to implicitly assembling
>> the "Neumann" problems, since the system can be partitioned as;
>>
>> [A_{LL} | A_{LI}]  [u_L]     [F]
>> -----------|------------ -------- = -----
>> [A_{IL}  |A_{II} ]   [u_I]      [G]
>>
>> and G should correspond to the Neumann problem. I might be thinking wrong
>> (or maybe I completely misunderstood the idea), if so please correct me.
>> But I think that the problem is that I am not explicitly telling PCBDDC
>> which dofs are interface dofs.
>>
>> Regards,
>> Abdullah Ali Sivas
>>
>> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>>
>>> Did you assemble "Neumann" problems that are compatible with your
>>> definition of interior/interface degrees of freedom?
>>>
>>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>>
>>> > Dear all,
>>> >
>>> > I have a series of linear systems coming from a PDE for which BDDC is
>>> an
>>> > optimal preconditioner. These linear systems are assembled and I read
>>> them
>>> > from a file, then convert into MATIS as required (as in
>>> >
>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>>> > ). I expect each of the systems converge to the solution in almost same
>>> > number of iterations but I don't observe it. I think it is because I
>>> do not
>>> > provide enough information to the preconditioner. I can get a list of
>>> inner
>>> > dofs and interface dofs. However, I do not know how to use them. Has
>>> anyone
>>> > have any insights about it or done something similar?
>>> >
>>> > Best wishes,
>>> > Abdullah Ali Sivas
>>>
>>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/64933a44/attachment.html>

From knepley at gmail.com  Wed Oct 24 13:04:01 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 24 Oct 2018 14:04:01 -0400
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
Message-ID: <CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>

On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <imilian.hartig at gmail.com>
wrote:

>
>
> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:
>
>> Hi Max,
>>
>> (I'm cc'ing in the petsc-users mailing list which may have more advice,
>> if you are using PETSc you should definitely subscribe!
>>
>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com>
>> wrote:
>> >
>> > Hello Lawrence,
>> >
>> > sorry to message you out of the blue. My name is Max and I found your
>> post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246
>>  ) on DMPlex being able to read periodic gmsh files. I am currently
>> trying to do just that (creating a periodic DMPlex mesh with gmsh) in the
>> context of my PhD work. So far I haven? t found any documentation on the
>> periodic BC?s with DMPlex and gmsh in the official petsc documentation.
>> > I was wondering whether you?d be so kind as to point me in a general
>> direction concerning how to achieve this. You seem experienced in using
>> petsc and I would greatly appreciate your help.
>>
>>
>> I think the answer is "it depends". If you're just using DMPlex directly
>> and all the of the functionality with PetscDS, then I /think/ that reading
>> periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh
>> format [v2]) "just works".
>>
>
> There are two phases here: topological and geometric. DMPlex represents
> the periodic topological entity directly. For example,  a circle is just a
> segment with one end hooked to the other. Vertices are not duplicated, or
> mapped to each other. This makes topology simple and easy to implement.
> However, then geometry is more complicated. What Plex does is allow
> coordinates to be represented by a discontinuous field taking values on
> cells, in addition to vertices. In our circle example, each cells near the
> cut will have 2 coordinates, one for each vertex, but they will not agree
> across the cut. If you define a periodic domain, then Plex can construct
> this coordinate field automatically using DMPlexLocalize(). These DG
> coordinates are then used by the integration routines.
>
>
> Ok, I think I understand the concept. DMPlex reads information about both
> topology and coordinates from the .msh file. Creating a periodic mesh in
> gmsh then should allow DMPlex to identify the periodic boundaries as the
> ?cut? and build the mesh topology accordingly. Coordinate information is
> handled separately.
> That means, as Lawrence suggested, building periodic meshes in gmsh and
> reading them in to petsc?s DMPlex should indeed ?just work?.  (From the
> user perspective). The only extra step is to call DMLocalizeCoordinates()
> after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make
> sense of this.
>
>
>
>> From my side, the issue is to do with mapping that coordinate field into
>> one that I understand (within Firedrake). You may not have this problem.
>>
>
> Firedrake uses its own coordinate mapping and integration routines, so
> they must manage the second part independently. I hope to get change this
> slightly soon by making the Firedrake representation a DMField, so that it
> looks the same to Plex.
>
>   Thanks,
>
>     Matt
>
>
>> Thanks,
>>
>> Lawrence
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
>
>
> To read periodic meshes from GMSH, you need to use the option
> -dm_plex_gmsh_periodic and DMPlexCreateFromFile
>
>
> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But
> using this option I now generate a segmentation fault error when calling
> VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>

 Small example? VTK is deprecated. HDF5 should work, although it will
require you to have proper coordinates I think. We have to
think about what you mean. If its for a checkpoint, there is no problem,
but for viz, those programs do not understand periodicity. Thus I embed it
in a higher dimensional space.

   Matt

> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>
> $ ./ex1 -filename
> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>
> generating periodic meshes in gmsh may be tricky, Lisandro for sure may
> advice.
>
>
> Thanks,
> Max
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181024/c1d3ef47/attachment.html>

From amfoggia at gmail.com  Thu Oct 25 06:48:09 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Thu, 25 Oct 2018 13:48:09 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <CAMYG4G=msJn9eQ=f_o8LLRwiU=JXPt8n+Xo43zUjxxDNtBh-kQ@mail.gmail.com>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<CAMYG4G=msJn9eQ=f_o8LLRwiU=JXPt8n+Xo43zUjxxDNtBh-kQ@mail.gmail.com>
Message-ID: <CANCZWyuD_Ab4FNAnYrWAWuO3iZFEWW0wymfq7PrzZ-UXPguLUw@mail.gmail.com>

El mar., 23 oct. 2018 a las 13:53, Matthew Knepley (<knepley at gmail.com>)
escribi?:

> On Tue, Oct 23, 2018 at 6:24 AM Ale Foggia <amfoggia at gmail.com> wrote:
>
>> Hello,
>>
>> I'm currently using Lanczos solver (EPSLANCZOS) to get the smallest real
>> eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem (EPS_HEP). Those are
>> the only options I set for the solver. My aim is to be able to
>> predict/estimate the time-to-solution. To do so, I was doing a scaling of
>> the code for different sizes of matrices and for different number of MPI
>> processes. As I was not observing a good scaling I checked the number of
>> iterations of the solver (given by EPSGetIterationNumber). I've encounter
>> that for the **same size** of matrix (that meaning, the same problem), when
>> I change the number of MPI processes, the amount of iterations changes, and
>> the behaviour is not monotonic. This are the numbers I've got:
>>
>
> I am sure you know this, but this test is strong scaling and will top out
> when the individual problem sizes become too small (we see this at several
> thousand unknowns).
>

Thanks for pointing this out, we are aware of that and I've been "playing"
around to try to see by myself this behaviour. Now, I think I'll go with
the Krylov-Schur method because is the only solution to the problem of the
number of iterations. With this I think I'll be able to see the individual
problem size effect in the scaling.


>   Thanks,
>
>     Matt
>
>
>>
>> # procs   # iters
>> 960          157
>> 992          189
>> 1024        338
>> 1056        190
>> 1120        174
>> 2048        136
>>
>> I've checked the mailing list for a similar situation and I've found
>> another person with the same problem but in another solver ("[SLEPc] GD is
>> not deterministic when using different number of cores", Nov 19 2015), but
>> I think the solution this person finds does not apply to my problem
>> (removing "-eps_harmonic" option).
>>
>> Can you give me any hint on what is the reason for this behaviour? Is
>> there a way to prevent this? It's not possible to estimate/predict any time
>> consumption for bigger problems if the number of iterations varies this
>> much.
>>
>> Ale
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/6c3d593c/attachment-0001.html>

From amfoggia at gmail.com  Thu Oct 25 06:54:04 2018
From: amfoggia at gmail.com (Ale Foggia)
Date: Thu, 25 Oct 2018 13:54:04 +0200
Subject: [petsc-users] [SLEPc] Number of iterations changes with MPI
 processes in Lanczos
In-Reply-To: <AA616809-7FDD-46AF-A60E-EB41C66FD0ED@dsic.upv.es>
References: <CANCZWytd=nkSJH4f4ufiKo1_o0EPCRLB1mWKCqg9b2GqNejaDA@mail.gmail.com>
	<4EE63176-0CF1-47D7-83BD-A95162946BFD@dsic.upv.es>
	<CANCZWys4Co4PHEwhoryNxk8aQTfFP4W9AAruS3biVy9RP8pA4A@mail.gmail.com>
	<F6E8C112-6546-4719-8C82-489766D78F2A@dsic.upv.es>
	<CANCZWyt7NNAsT4w6OvdgFQfmahtJHfw9_HqXZrLdDzTsy+ZNTw@mail.gmail.com>
	<8785019B-C52B-4022-B7A8-83F8466F9B71@dsic.upv.es>
	<CANCZWytqr76idf5XyD5QgDoOUSB3TXN_8jbw4T4ucpO+G8-+Bg@mail.gmail.com>
	<3F900479-3123-413E-B279-9DE624E825F4@dsic.upv.es>
	<CANCZWytd1gWi5DUmYN39fKq9G3Fwpq_e8B1o2Sn3vS+Bm7grqw@mail.gmail.com>
	<AA616809-7FDD-46AF-A60E-EB41C66FD0ED@dsic.upv.es>
Message-ID: <CANCZWyuGphFD01XOWcfZs0zO4i_F5L__LVT5tFmz77uXQGUJMw@mail.gmail.com>

No, the eigenvalue is around -15. I've tried KS and the number of
iterations differs in one when I change the number of MPI processes, which
seems fine for me. So, I'll see if this method is fine for my specific goal
or not, and I'll try to use. Thanks for the help.

El mi?., 24 oct. 2018 a las 15:48, Jose E. Roman (<jroman at dsic.upv.es>)
escribi?:

> Everything seems correct. I don't know, maybe your problem is very
> sensitive? Is the eigenvalue tiny?
> I would still try with Krylov-Schur.
> Jose
>
>
> > El 24 oct 2018, a las 14:59, Ale Foggia <amfoggia at gmail.com> escribi?:
> >
> > The functions called to set the solver are (in this order): EPSCreate();
> EPSSetOperators(); EPSSetProblemType(EPS_HEP); EPSSetType(EPSLANCZOS);
> EPSSetWhichEigenpairs(EPS_SMALLEST_REAL); EPSSetFromOptions();
> >
> > The output of -eps_view for each run is:
> > =================================================================
> > EPS Object: 960 MPI processes
> >   type: lanczos
> >     LOCAL reorthogonalization
> >   problem type: symmetric eigenvalue problem
> >   selected portion of the spectrum: smallest real parts
> >   number of eigenvalues (nev): 1
> >   number of column vectors (ncv): 16
> >   maximum dimension of projected problem (mpd): 16
> >   maximum number of iterations: 291700777
> >   tolerance: 1e-08
> >   convergence test: relative to the eigenvalue
> > BV Object: 960 MPI processes
> >   type: svec
> >   17 columns of global length 2333606220
> >   vector orthogonalization method: modified Gram-Schmidt
> >   orthogonalization refinement: if needed (eta: 0.7071)
> >   block orthogonalization method: GS
> >   doing matmult as a single matrix-matrix product
> >   generating random vectors independent of the number of processes
> > DS Object: 960 MPI processes
> >   type: hep
> >   parallel operation mode: REDUNDANT
> >   solving the problem with: Implicit QR method (_steqr)
> > ST Object: 960 MPI processes
> >   type: shift
> >   shift: 0.
> >   number of matrices: 1
> > =================================================================
> > EPS Object: 1024 MPI processes
> >   type: lanczos
> >     LOCAL reorthogonalization
> >   problem type: symmetric eigenvalue problem
> >   selected portion of the spectrum: smallest real parts
> >   number of eigenvalues (nev): 1
> >   number of column vectors (ncv): 16
> >   maximum dimension of projected problem (mpd): 16
> >   maximum number of iterations: 291700777
> >   tolerance: 1e-08
> >   convergence test: relative to the eigenvalue
> > BV Object: 1024 MPI processes
> >   type: svec
> >   17 columns of global length 2333606220
> >   vector orthogonalization method: modified Gram-Schmidt
> >   orthogonalization refinement: if needed (eta: 0.7071)
> >   block orthogonalization method: GS
> >   doing matmult as a single matrix-matrix product
> >   generating random vectors independent of the number of processes
> > DS Object: 1024 MPI processes
> >   type: hep
> >   parallel operation mode: REDUNDANT
> >   solving the problem with: Implicit QR method (_steqr)
> > ST Object: 1024 MPI processes
> >   type: shift
> >   shift: 0.
> >   number of matrices: 1
> > =================================================================
> >
> > I run again the same configurations and I got the same result in term of
> the number of iterations.
> >
> > I also tried the full reorthogonalization (always with the
> -bv_reproducible_random option) but I still get different number of
> iterations: for 960 procs I get 172 iters, and for 1024 I get 362 iters.
> The -esp_view output for this case (only for 960 procs, the other one has
> the same information -except the number of processes-) is:
> > =================================================================
> > EPS Object: 960 MPI processes
> >   type: lanczos
> >     FULL reorthogonalization
> >   problem type: symmetric eigenvalue problem
> >   selected portion of the spectrum: smallest real parts
> >   number of eigenvalues (nev): 1
> >   number of column vectors (ncv): 16
> >   maximum dimension of projected problem (mpd): 16
> >   maximum number of iterations: 291700777
> >   tolerance: 1e-08
> >   convergence test: relative to the eigenvalue
> > BV Object: 960 MPI processes
> >   type: svec
> >   17 columns of global length 2333606220
> >   vector orthogonalization method: classical Gram-Schmidt
> >   orthogonalization refinement: if needed (eta: 0.7071)
> >   block orthogonalization method: GS
> >   doing matmult as a single matrix-matrix product
> >   generating random vectors independent of the number of processes
> > DS Object: 960 MPI processes
> >   type: hep
> >   parallel operation mode: REDUNDANT
> >   solving the problem with: Implicit QR method (_steqr)
> > ST Object: 960 MPI processes
> >   type: shift
> >   shift: 0.
> >   number of matrices: 1
> > =================================================================
> >
> > El mi?., 24 oct. 2018 a las 10:52, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> > This is very strange. Make sure you call EPSSetFromOptions in the code.
> Do iteration counts change also for two different runs with the same number
> of processes?
> > Maybe Lanczos with default options is too sensitive (by default it does
> not reorthogonalize). Suggest using Krylov-Schur or Lanczos with full
> reorthogonalization (EPSLanczosSetReorthog).
> > Also, send the output of -eps_view to see if there is anything abnormal.
> >
> > Jose
> >
> >
> > > El 24 oct 2018, a las 9:09, Ale Foggia <amfoggia at gmail.com> escribi?:
> > >
> > > I've tried the option that you gave me but I still get different
> number of iterations when changing the number of MPI processes: I did 960
> procs and 1024 procs and I got 435 and 176 iterations, respectively.
> > >
> > > El mar., 23 oct. 2018 a las 16:48, Jose E. Roman (<jroman at dsic.upv.es>)
> escribi?:
> > >
> > >
> > > > El 23 oct 2018, a las 15:46, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > >
> > > >
> > > >
> > > > El mar., 23 oct. 2018 a las 15:33, Jose E. Roman (<
> jroman at dsic.upv.es>) escribi?:
> > > >
> > > >
> > > > > El 23 oct 2018, a las 15:17, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > > >
> > > > > Hello Jose, thanks for your answer.
> > > > >
> > > > > El mar., 23 oct. 2018 a las 12:59, Jose E. Roman (<
> jroman at dsic.upv.es>) escribi?:
> > > > > There is an undocumented option:
> > > > >
> > > > >   -bv_reproducible_random
> > > > >
> > > > > It will force the initial vector of the Krylov subspace to be the
> same irrespective of the number of MPI processes. This should be used for
> scaling analyses as the one you are trying to do.
> > > > >
> > > > > What about when I'm not doing the scaling? Now I would like to ask
> for computing time for bigger size problems, should I also use this option
> in that case? Because, what happens if I have a "bad" configuration?
> Meaning, I ask for some time, enough if I take into account the "correct"
> scaling, but when I run it takes double the time/iterations, like it
> happened before when changing from 960 to 1024 processes?
> > > >
> > > > When you increase the matrix size the spectrum of the matrix changes
> and probably also the convergence, so the computation time is not easy to
> predict in advance.
> > > >
> > > > Okey, I'll keep that in mine. I thought that, even if the spectrum
> changes, if I had a behaviour/tendency for 6 or 7 smaller cases I could
> predict more or less the time. It was working this way until I found this
> "iterations problem" which doubled the time of execution for the same size
> problem. To be completely sure, do you suggest me or not to use this
> run-time option when going in production? Can you elaborate a bit in the
> effect this option? Is the (huge) difference I got in the number of
> iterations something expected?
> > >
> > > Ideally if you have a rough approximation of the eigenvector, you set
> it as the initial vector with EPSSetInitialSpace(). Otherwise, SLEPc
> generates a random initial vector, that is start the search blindly. The
> difference between using one random vector or another may be large,
> depending on the problem. Krylov-Schur is usually less sensitive to the
> initial vector.
> > >
> > > Jose
> > >
> > > >
> > > >
> > > > >
> > > > > An additional comment is that we strongly recommend to use the
> default solver (Krylov-Schur), which will do Lanczos with implicit restart.
> It is generally faster and more stable.
> > > > >
> > > > > I will be doing Dynamical Lanczos, that means that I'll need the
> "matrix whose rows are the eigenvectors of the tridiagonal matrix" (so,
> according to the Lanczos Technical Report notation, I need the "matrix
> whose rows are the eigenvectors of T_m", which should be the same as the
> vectors y_i). I checked the Technical Report for Krylov-Schur also and I
> think I can get the same information also from that solver, but I'm not
> sure. Can you confirm this please?
> > > > > Also, as the vectors I want are given by V_m^(-1)*x_i=y_i
> (following the notation on the Report), my idea to get them was to retrieve
> the invariant subspace V_m (with EPSGetInvariantSubspace), invert it, and
> then multiply it with the eigenvectors that I get with EPSGetEigenvector.
> Is there another easier (or with less computations) way to get this?
> > > >
> > > > In Krylov-Schur the tridiagonal matrix T_m becomes
> arrowhead-plus-tridiagonal. Apart from this, it should be equivalent. The
> relevant information can be obtained with EPSGetBV() and EPSGetDS(). But
> this is a "developer level" interface. We could help you get this running.
> Send a small problem matrix to slepc-maint together with a more detailed
> description of what you need to compute.
> > > >
> > > > Thanks! When I get to that part I'll write to slepc-maint for help.
> > > >
> > > >
> > > > Jose
> > > >
> > > > >
> > > > >
> > > > > Jose
> > > > >
> > > > >
> > > > >
> > > > > > El 23 oct 2018, a las 12:13, Ale Foggia <amfoggia at gmail.com>
> escribi?:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I'm currently using Lanczos solver (EPSLANCZOS) to get the
> smallest real eigenvalue (EPS_SMALLEST_REAL) of a Hermitian problem
> (EPS_HEP). Those are the only options I set for the solver. My aim is to be
> able to predict/estimate the time-to-solution. To do so, I was doing a
> scaling of the code for different sizes of matrices and for different
> number of MPI processes. As I was not observing a good scaling I checked
> the number of iterations of the solver (given by EPSGetIterationNumber).
> I've encounter that for the **same size** of matrix (that meaning, the same
> problem), when I change the number of MPI processes, the amount of
> iterations changes, and the behaviour is not monotonic. This are the
> numbers I've got:
> > > > > >
> > > > > > # procs   # iters
> > > > > > 960          157
> > > > > > 992          189
> > > > > > 1024        338
> > > > > > 1056        190
> > > > > > 1120        174
> > > > > > 2048        136
> > > > > >
> > > > > > I've checked the mailing list for a similar situation and I've
> found another person with the same problem but in another solver ("[SLEPc]
> GD is not deterministic when using different number of cores", Nov 19
> 2015), but I think the solution this person finds does not apply to my
> problem (removing "-eps_harmonic" option).
> > > > > >
> > > > > > Can you give me any hint on what is the reason for this
> behaviour? Is there a way to prevent this? It's not possible to
> estimate/predict any time consumption for bigger problems if the number of
> iterations varies this much.
> > > > > >
> > > > > > Ale
> > > > >
> > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/0317d5a2/attachment.html>

From stefano.zampini at gmail.com  Thu Oct 25 07:11:07 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 25 Oct 2018 15:11:07 +0300
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
Message-ID: <CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>

Matt,

you can reproduce it via

$ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
-petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
-dm_plex_gmsh_periodic

Long time ago I added support for viewing meshes with periodic vertices in
the VTK_VTU viewer, but I did not fix the part that writes fields


Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley <knepley at gmail.com>
ha scritto:

> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <
> imilian.hartig at gmail.com> wrote:
>
>>
>>
>> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:
>>
>>> Hi Max,
>>>
>>> (I'm cc'ing in the petsc-users mailing list which may have more advice,
>>> if you are using PETSc you should definitely subscribe!
>>>
>>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com>
>>> wrote:
>>> >
>>> > Hello Lawrence,
>>> >
>>> > sorry to message you out of the blue. My name is Max and I found your
>>> post on GitHub (
>>> https://github.com/firedrakeproject/firedrake/issues/1246 ) on DMPlex
>>> being able to read periodic gmsh files. I am currently trying to do just
>>> that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD
>>> work. So far I haven? t found any documentation on the periodic BC?s with
>>> DMPlex and gmsh in the official petsc documentation.
>>> > I was wondering whether you?d be so kind as to point me in a general
>>> direction concerning how to achieve this. You seem experienced in using
>>> petsc and I would greatly appreciate your help.
>>>
>>>
>>> I think the answer is "it depends". If you're just using DMPlex directly
>>> and all the of the functionality with PetscDS, then I /think/ that reading
>>> periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh
>>> format [v2]) "just works".
>>>
>>
>> There are two phases here: topological and geometric. DMPlex represents
>> the periodic topological entity directly. For example,  a circle is just a
>> segment with one end hooked to the other. Vertices are not duplicated, or
>> mapped to each other. This makes topology simple and easy to implement.
>> However, then geometry is more complicated. What Plex does is allow
>> coordinates to be represented by a discontinuous field taking values on
>> cells, in addition to vertices. In our circle example, each cells near the
>> cut will have 2 coordinates, one for each vertex, but they will not agree
>> across the cut. If you define a periodic domain, then Plex can construct
>> this coordinate field automatically using DMPlexLocalize(). These DG
>> coordinates are then used by the integration routines.
>>
>>
>> Ok, I think I understand the concept. DMPlex reads information about both
>> topology and coordinates from the .msh file. Creating a periodic mesh in
>> gmsh then should allow DMPlex to identify the periodic boundaries as the
>> ?cut? and build the mesh topology accordingly. Coordinate information is
>> handled separately.
>> That means, as Lawrence suggested, building periodic meshes in gmsh and
>> reading them in to petsc?s DMPlex should indeed ?just work?.  (From the
>> user perspective). The only extra step is to call DMLocalizeCoordinates()
>> after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make
>> sense of this.
>>
>>
>>
>>> From my side, the issue is to do with mapping that coordinate field into
>>> one that I understand (within Firedrake). You may not have this problem.
>>>
>>
>> Firedrake uses its own coordinate mapping and integration routines, so
>> they must manage the second part independently. I hope to get change this
>> slightly soon by making the Firedrake representation a DMField, so that it
>> looks the same to Plex.
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Thanks,
>>>
>>> Lawrence
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>>
>>
>> To read periodic meshes from GMSH, you need to use the option
>> -dm_plex_gmsh_periodic and DMPlexCreateFromFile
>>
>>
>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But
>> using this option I now generate a segmentation fault error when calling
>> VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>>
>
>  Small example? VTK is deprecated. HDF5 should work, although it will
> require you to have proper coordinates I think. We have to
> think about what you mean. If its for a checkpoint, there is no problem,
> but for viz, those programs do not understand periodicity. Thus I embed it
> in a higher dimensional space.
>
>    Matt
>
>> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>>
>> $ ./ex1 -filename
>> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
>> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>>
>> generating periodic meshes in gmsh may be tricky, Lisandro for sure may
>> advice.
>>
>>
>> Thanks,
>> Max
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/604bac94/attachment-0001.html>

From stefano.zampini at gmail.com  Thu Oct 25 08:36:15 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 25 Oct 2018 16:36:15 +0300
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
	<CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
Message-ID: <CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>

Maybe this is a fix

diff --git a/src/dm/impls/plex/plexvtu.c b/src/dm/impls/plex/plexvtu.c
index acdea12c2f..1a8bbada6a 100644
--- a/src/dm/impls/plex/plexvtu.c
+++ b/src/dm/impls/plex/plexvtu.c
@@ -465,10 +465,11 @@ PetscErrorCode DMPlexVTKWriteAll_VTU(DM
dm,PetscViewer viewer)
                 if ((closure[v] >= vStart) && (closure[v] < vEnd)) {
                   PetscScalar *xpoint;

-                  ierr =
DMPlexPointLocalRead(dm,v,x,&xpoint);CHKERRQ(ierr);
+                  ierr =
DMPlexPointLocalRead(dm,closure[v],x,&xpoint);CHKERRQ(ierr);
                   y[cnt + off++] = xpoint[i];
                 }
               }
+              cnt += off;
               ierr = DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE,
&closureSize, &closure);CHKERRQ(ierr);
             }
           }

Max, does this fix your problem? If you confirm, I'll fix this in the maint
branch

If I run the below command line with the patch and with snes tutorials ex12
I get the nice picture attached
$ ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
-petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
-dm_plex_gmsh_periodic -x_periodicity periodic -y_periodicity periodic
-dm_refine 4

Il giorno gio 25 ott 2018 alle ore 15:11 Stefano Zampini <
stefano.zampini at gmail.com> ha scritto:

> Matt,
>
> you can reproduce it via
>
> $ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
> -dm_plex_gmsh_periodic
>
> Long time ago I added support for viewing meshes with periodic vertices in
> the VTK_VTU viewer, but I did not fix the part that writes fields
>
>
> Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley <
> knepley at gmail.com> ha scritto:
>
>> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <
>> imilian.hartig at gmail.com> wrote:
>>
>>>
>>>
>>> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:
>>>
>>>> Hi Max,
>>>>
>>>> (I'm cc'ing in the petsc-users mailing list which may have more advice,
>>>> if you are using PETSc you should definitely subscribe!
>>>>
>>>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com>
>>>> wrote:
>>>> >
>>>> > Hello Lawrence,
>>>> >
>>>> > sorry to message you out of the blue. My name is Max and I found your
>>>> post on GitHub (
>>>> https://github.com/firedrakeproject/firedrake/issues/1246 ) on DMPlex
>>>> being able to read periodic gmsh files. I am currently trying to do just
>>>> that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD
>>>> work. So far I haven? t found any documentation on the periodic BC?s with
>>>> DMPlex and gmsh in the official petsc documentation.
>>>> > I was wondering whether you?d be so kind as to point me in a general
>>>> direction concerning how to achieve this. You seem experienced in using
>>>> petsc and I would greatly appreciate your help.
>>>>
>>>>
>>>> I think the answer is "it depends". If you're just using DMPlex
>>>> directly and all the of the functionality with PetscDS, then I /think/ that
>>>> reading periodic meshes via gmsh (assuming you're using the appropriate
>>>> gmsh mesh format [v2]) "just works".
>>>>
>>>
>>> There are two phases here: topological and geometric. DMPlex represents
>>> the periodic topological entity directly. For example,  a circle is just a
>>> segment with one end hooked to the other. Vertices are not duplicated, or
>>> mapped to each other. This makes topology simple and easy to implement.
>>> However, then geometry is more complicated. What Plex does is allow
>>> coordinates to be represented by a discontinuous field taking values on
>>> cells, in addition to vertices. In our circle example, each cells near the
>>> cut will have 2 coordinates, one for each vertex, but they will not agree
>>> across the cut. If you define a periodic domain, then Plex can construct
>>> this coordinate field automatically using DMPlexLocalize(). These DG
>>> coordinates are then used by the integration routines.
>>>
>>>
>>> Ok, I think I understand the concept. DMPlex reads information about
>>> both topology and coordinates from the .msh file. Creating a periodic mesh
>>> in gmsh then should allow DMPlex to identify the periodic boundaries as the
>>> ?cut? and build the mesh topology accordingly. Coordinate information is
>>> handled separately.
>>> That means, as Lawrence suggested, building periodic meshes in gmsh and
>>> reading them in to petsc?s DMPlex should indeed ?just work?.  (From the
>>> user perspective). The only extra step is to call DMLocalizeCoordinates()
>>> after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make
>>> sense of this.
>>>
>>>
>>>
>>>> From my side, the issue is to do with mapping that coordinate field
>>>> into one that I understand (within Firedrake). You may not have this
>>>> problem.
>>>>
>>>
>>> Firedrake uses its own coordinate mapping and integration routines, so
>>> they must manage the second part independently. I hope to get change this
>>> slightly soon by making the Firedrake representation a DMField, so that it
>>> looks the same to Plex.
>>>
>>>   Thanks,
>>>
>>>     Matt
>>>
>>>
>>>> Thanks,
>>>>
>>>> Lawrence
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>>
>>>
>>> To read periodic meshes from GMSH, you need to use the option
>>> -dm_plex_gmsh_periodic and DMPlexCreateFromFile
>>>
>>>
>>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But
>>> using this option I now generate a segmentation fault error when calling
>>> VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>>>
>>
>>  Small example? VTK is deprecated. HDF5 should work, although it will
>> require you to have proper coordinates I think. We have to
>> think about what you mean. If its for a checkpoint, there is no problem,
>> but for viz, those programs do not understand periodicity. Thus I embed it
>> in a higher dimensional space.
>>
>>    Matt
>>
>>> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>>>
>>> $ ./ex1 -filename
>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
>>> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>>>
>>> generating periodic meshes in gmsh may be tricky, Lisandro for sure may
>>> advice.
>>>
>>>
>>> Thanks,
>>> Max
>>>
>>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>> https://www.cse.buffalo.edu/~knepley/
>> <http://www.cse.buffalo.edu/~knepley/>
>>
>
>
> --
> Stefano
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/e997dff5/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: periodic.png
Type: image/png
Size: 255676 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/e997dff5/attachment-0001.png>

From stefano.zampini at gmail.com  Thu Oct 25 08:53:03 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 25 Oct 2018 16:53:03 +0300
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAPyJGmCEOCN5-HLSFxvtEPbL2nnPmCkQUhdOC1q4hW2qFMfk2g@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
	<CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
	<CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>
	<CAPyJGmCEOCN5-HLSFxvtEPbL2nnPmCkQUhdOC1q4hW2qFMfk2g@mail.gmail.com>
Message-ID: <CAGPUisg-yqAAExLZ2kY0B0cR-8Pu+E_vm=kU7FKnCihM=eNS6g@mail.gmail.com>

How many processes (subdomains) are you using?
I would not say the number of iterations is bad, and it also seems to
plateau.
The grad-div problem is quite hard to be solved (unless you use
hybridization), you can really benefit from the "Neumann" assembly.
I believe you are using GMRES, as the preconditioned operator (i.e
M_BDDC^-1 A) is not guaranteed to be positive definite when you use the
automatic disassembling.
You may slightly improve the quality of the disassembling by using
-mat_is_disassemble_l2g_type nd -mat_partitioning_type parmetis if you have
PETSc compiled with ParMETIS support.


Il giorno mer 24 ott 2018 alle ore 20:59 Abdullah Ali Sivas <
abdullahasivas at gmail.com> ha scritto:

> Hi Stefano,
>
> I am trying to solve the div-div problem (or grad-div problem in strong
> form) with a H(div)-conforming FEM. I am getting the matrices from an
> external source (to be clear, from an ngsolve script) and I am not sure if
> it is possible to get a MATIS matrix out of that. So I am just treating it
> as if I am not able to access the assembly code. The results are 2, 31, 26,
> 27, 31 iterations, respectively, for matrix sizes 282, 1095, 4314, 17133,
> 67242, 267549. However, norm of the residual also grows significantly;
> 7.38369e-09 for 1095 and 5.63828e-07 for 267549. I can try larger sizes, or
> maybe this is expected for this case.
>
> As a side question, if we are dividing the domain into number of MPI
> processes subdomains, does it mean that convergence is affected negatively
> by the increasing number of processes? I know that alternating Schwarz
> method and some other domain decomposition methods sometimes suffer from
> the decreasing radius of the subdomains. It sounds like BDDC is pretty
> similar to those by your description.
>
> Best wishes,
> Abdullah Ali Sivas
>
> On Wed, 24 Oct 2018 at 05:28, Stefano Zampini <stefano.zampini at gmail.com>
> wrote:
>
>> Abdullah,
>>
>> The "Neumann" problems Jed is referring to result from assembling your
>> problem on each subdomain ( = MPI process) separately.
>> Assuming you are using FEM, these problems have been historically  named
>> "Neumann" as they correspond to a problem with natural boundary conditions
>> (Neumann bc for Poisson).
>> Note that in PETSc the subdomain decomposition is associated with the
>> mesh decomposition.
>>
>> When converting from an assembled AIJ matrix to a MATIS format, such
>> "Neumann" information is lost.
>> You can disassemble an AIJ matrix, in the sense that you can find local
>> matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
>> but you cannot guarantee (unless if you solve an optimization problem) that
>> the disassembling will produce subdomain Neumann problems that are
>> consistent with your FEM problem.
>>
>> I have added such disassembling code a few months ago, just to have
>> another alternative for preconditioning AIJ matrices in PETSc; there are
>> few tweaks one can do to improve the quality of the disassembling, but I
>> discourage its usage unless you don't have access to the FEM assembly code.
>>
>> With that said, what problem are you trying to solve? Are you using DMDA
>> or DMPlex? What are the results you obtained with using the automatic
>> disassembling?
>>
>> Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
>> abdullahasivas at gmail.com> ha scritto:
>>
>>> Hi Jed,
>>>
>>> Thanks for your reply. The assembled matrix I have corresponds to the
>>> full problem on the full mesh. There are no "Neumann" problems (or any sort
>>> of domain decomposition) defined in the code generates the matrix. However,
>>> I think assembling the full problem is equivalent to implicitly assembling
>>> the "Neumann" problems, since the system can be partitioned as;
>>>
>>> [A_{LL} | A_{LI}]  [u_L]     [F]
>>> -----------|------------ -------- = -----
>>> [A_{IL}  |A_{II} ]   [u_I]      [G]
>>>
>>> and G should correspond to the Neumann problem. I might be thinking
>>> wrong (or maybe I completely misunderstood the idea), if so please correct
>>> me. But I think that the problem is that I am not explicitly telling PCBDDC
>>> which dofs are interface dofs.
>>>
>>> Regards,
>>> Abdullah Ali Sivas
>>>
>>> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>>> Did you assemble "Neumann" problems that are compatible with your
>>>> definition of interior/interface degrees of freedom?
>>>>
>>>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>>>
>>>> > Dear all,
>>>> >
>>>> > I have a series of linear systems coming from a PDE for which BDDC is
>>>> an
>>>> > optimal preconditioner. These linear systems are assembled and I read
>>>> them
>>>> > from a file, then convert into MATIS as required (as in
>>>> >
>>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>>>> > ). I expect each of the systems converge to the solution in almost
>>>> same
>>>> > number of iterations but I don't observe it. I think it is because I
>>>> do not
>>>> > provide enough information to the preconditioner. I can get a list of
>>>> inner
>>>> > dofs and interface dofs. However, I do not know how to use them. Has
>>>> anyone
>>>> > have any insights about it or done something similar?
>>>> >
>>>> > Best wishes,
>>>> > Abdullah Ali Sivas
>>>>
>>>
>>
>> --
>> Stefano
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/820fa653/attachment.html>

From abdullahasivas at gmail.com  Thu Oct 25 09:19:53 2018
From: abdullahasivas at gmail.com (Abdullah Ali Sivas)
Date: Thu, 25 Oct 2018 10:19:53 -0400
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAGPUisg-yqAAExLZ2kY0B0cR-8Pu+E_vm=kU7FKnCihM=eNS6g@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
	<CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
	<CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>
	<CAPyJGmCEOCN5-HLSFxvtEPbL2nnPmCkQUhdOC1q4hW2qFMfk2g@mail.gmail.com>
	<CAGPUisg-yqAAExLZ2kY0B0cR-8Pu+E_vm=kU7FKnCihM=eNS6g@mail.gmail.com>
Message-ID: <CAPyJGmCCw7Z2dRs40w+cdzhPM3mHtwAh3y6-BckvH-VDSAbuBA@mail.gmail.com>

Right now, one to four. I am just running some tests with small matrices.
Later on, I am planning to do large scale tests hopefully up to 1024
processes. I was worried that iteration numbers may get worse.

I actually use hybridization and I was reading the preprint "Algebraic
Hybridization and Static Condensation with Application to Scalable H(div)
Preconditioning" by Dobrev et al. ( https://arxiv.org/abs/1801.08914 ) and
they show that multigrid is optimal for the grad-div problem discretized
with H(div) conforming FEMs when hybridized. That is actually why I think
that BDDC also would be optimal. I will look into ngsolve to see if I can
have such a domain decomposition. Maybe I can do it manually just as proof
of concept.

I am using GMRES. I was wondering if the application of BDDC is a linear
operator, if it is not maybe I should use FGMRES. But I could not find any
comments about that.

I will recompile PETSc with ParMETIS and try your suggestions. Thank you! I
will update you soon.

Best wishes,
Abdullah Ali Sivas

On Thu, 25 Oct 2018 at 09:53, Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> How many processes (subdomains) are you using?
> I would not say the number of iterations is bad, and it also seems to
> plateau.
> The grad-div problem is quite hard to be solved (unless you use
> hybridization), you can really benefit from the "Neumann" assembly.
> I believe you are using GMRES, as the preconditioned operator (i.e
> M_BDDC^-1 A) is not guaranteed to be positive definite when you use the
> automatic disassembling.
> You may slightly improve the quality of the disassembling by using
> -mat_is_disassemble_l2g_type nd -mat_partitioning_type parmetis if you have
> PETSc compiled with ParMETIS support.
>
>
> Il giorno mer 24 ott 2018 alle ore 20:59 Abdullah Ali Sivas <
> abdullahasivas at gmail.com> ha scritto:
>
>> Hi Stefano,
>>
>> I am trying to solve the div-div problem (or grad-div problem in strong
>> form) with a H(div)-conforming FEM. I am getting the matrices from an
>> external source (to be clear, from an ngsolve script) and I am not sure if
>> it is possible to get a MATIS matrix out of that. So I am just treating it
>> as if I am not able to access the assembly code. The results are 2, 31, 26,
>> 27, 31 iterations, respectively, for matrix sizes 282, 1095, 4314, 17133,
>> 67242, 267549. However, norm of the residual also grows significantly;
>> 7.38369e-09 for 1095 and 5.63828e-07 for 267549. I can try larger sizes, or
>> maybe this is expected for this case.
>>
>> As a side question, if we are dividing the domain into number of MPI
>> processes subdomains, does it mean that convergence is affected negatively
>> by the increasing number of processes? I know that alternating Schwarz
>> method and some other domain decomposition methods sometimes suffer from
>> the decreasing radius of the subdomains. It sounds like BDDC is pretty
>> similar to those by your description.
>>
>> Best wishes,
>> Abdullah Ali Sivas
>>
>> On Wed, 24 Oct 2018 at 05:28, Stefano Zampini <stefano.zampini at gmail.com>
>> wrote:
>>
>>> Abdullah,
>>>
>>> The "Neumann" problems Jed is referring to result from assembling your
>>> problem on each subdomain ( = MPI process) separately.
>>> Assuming you are using FEM, these problems have been historically  named
>>> "Neumann" as they correspond to a problem with natural boundary conditions
>>> (Neumann bc for Poisson).
>>> Note that in PETSc the subdomain decomposition is associated with the
>>> mesh decomposition.
>>>
>>> When converting from an assembled AIJ matrix to a MATIS format, such
>>> "Neumann" information is lost.
>>> You can disassemble an AIJ matrix, in the sense that you can find local
>>> matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
>>> but you cannot guarantee (unless if you solve an optimization problem) that
>>> the disassembling will produce subdomain Neumann problems that are
>>> consistent with your FEM problem.
>>>
>>> I have added such disassembling code a few months ago, just to have
>>> another alternative for preconditioning AIJ matrices in PETSc; there are
>>> few tweaks one can do to improve the quality of the disassembling, but I
>>> discourage its usage unless you don't have access to the FEM assembly code.
>>>
>>> With that said, what problem are you trying to solve? Are you using DMDA
>>> or DMPlex? What are the results you obtained with using the automatic
>>> disassembling?
>>>
>>> Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
>>> abdullahasivas at gmail.com> ha scritto:
>>>
>>>> Hi Jed,
>>>>
>>>> Thanks for your reply. The assembled matrix I have corresponds to the
>>>> full problem on the full mesh. There are no "Neumann" problems (or any sort
>>>> of domain decomposition) defined in the code generates the matrix. However,
>>>> I think assembling the full problem is equivalent to implicitly assembling
>>>> the "Neumann" problems, since the system can be partitioned as;
>>>>
>>>> [A_{LL} | A_{LI}]  [u_L]     [F]
>>>> -----------|------------ -------- = -----
>>>> [A_{IL}  |A_{II} ]   [u_I]      [G]
>>>>
>>>> and G should correspond to the Neumann problem. I might be thinking
>>>> wrong (or maybe I completely misunderstood the idea), if so please correct
>>>> me. But I think that the problem is that I am not explicitly telling PCBDDC
>>>> which dofs are interface dofs.
>>>>
>>>> Regards,
>>>> Abdullah Ali Sivas
>>>>
>>>> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>>> Did you assemble "Neumann" problems that are compatible with your
>>>>> definition of interior/interface degrees of freedom?
>>>>>
>>>>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>>>>
>>>>> > Dear all,
>>>>> >
>>>>> > I have a series of linear systems coming from a PDE for which BDDC
>>>>> is an
>>>>> > optimal preconditioner. These linear systems are assembled and I
>>>>> read them
>>>>> > from a file, then convert into MATIS as required (as in
>>>>> >
>>>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>>>>> > ). I expect each of the systems converge to the solution in almost
>>>>> same
>>>>> > number of iterations but I don't observe it. I think it is because I
>>>>> do not
>>>>> > provide enough information to the preconditioner. I can get a list
>>>>> of inner
>>>>> > dofs and interface dofs. However, I do not know how to use them. Has
>>>>> anyone
>>>>> > have any insights about it or done something similar?
>>>>> >
>>>>> > Best wishes,
>>>>> > Abdullah Ali Sivas
>>>>>
>>>>
>>>
>>> --
>>> Stefano
>>>
>>
>
> --
> Stefano
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/56c4a119/attachment.html>

From stefano.zampini at gmail.com  Thu Oct 25 09:26:36 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 25 Oct 2018 17:26:36 +0300
Subject: [petsc-users] Using BDDC preconditioner for assembled matrices
In-Reply-To: <CAPyJGmCCw7Z2dRs40w+cdzhPM3mHtwAh3y6-BckvH-VDSAbuBA@mail.gmail.com>
References: <CAPyJGmCNCEniwwS4BOFhui-VTk+Qw=0ji+5fTiQh9nGPzB2Org@mail.gmail.com>
	<87sh0w5915.fsf@jedbrown.org>
	<CAPyJGmATc=SnFhyKc9cxc=6SE6a9SeM_Qqo2Hs8fZzUjBfpu-w@mail.gmail.com>
	<CAGPUish8V4YTOSznH__EAc53qMJhqmoLc-tMvBLs7oKRVSd3vw@mail.gmail.com>
	<CAPyJGmCEOCN5-HLSFxvtEPbL2nnPmCkQUhdOC1q4hW2qFMfk2g@mail.gmail.com>
	<CAGPUisg-yqAAExLZ2kY0B0cR-8Pu+E_vm=kU7FKnCihM=eNS6g@mail.gmail.com>
	<CAPyJGmCCw7Z2dRs40w+cdzhPM3mHtwAh3y6-BckvH-VDSAbuBA@mail.gmail.com>
Message-ID: <CAGPUish2Jn6GxPv=QGCTMCLDNDVxxuMuh1inveEg+vbk4dhP+g@mail.gmail.com>

> I actually use hybridization and I was reading the preprint "Algebraic
> Hybridization and Static Condensation with Application to Scalable H(div)
> Preconditioning" by Dobrev et al. ( https://arxiv.org/abs/1801.08914 )
> and they show that multigrid is optimal for the grad-div problem
> discretized with H(div) conforming FEMs when hybridized. That is actually
> why I think that BDDC also would be optimal. I will look into ngsolve to
> see if I can have such a domain decomposition. Maybe I can do it manually
> just as proof of concept.
>
>
If you are using hybridization, you can use PCGAMG (i.e. -pc_type gamg)


> I am using GMRES. I was wondering if the application of BDDC is a linear
> operator, if it is not maybe I should use FGMRES. But I could not find any
> comments about that.
>
>
BDDC is linear. The problem is that when you disassemble an already
assembled matrix, the operator of the preconditioner is not guaranteed to
stay positive definite for positive definite assembled problems,



> I will recompile PETSc with ParMETIS and try your suggestions. Thank you!
> I will update you soon.
>
> Best wishes,
> Abdullah Ali Sivas
>
> On Thu, 25 Oct 2018 at 09:53, Stefano Zampini <stefano.zampini at gmail.com>
> wrote:
>
>> How many processes (subdomains) are you using?
>> I would not say the number of iterations is bad, and it also seems to
>> plateau.
>> The grad-div problem is quite hard to be solved (unless you use
>> hybridization), you can really benefit from the "Neumann" assembly.
>> I believe you are using GMRES, as the preconditioned operator (i.e
>> M_BDDC^-1 A) is not guaranteed to be positive definite when you use the
>> automatic disassembling.
>> You may slightly improve the quality of the disassembling by using
>> -mat_is_disassemble_l2g_type nd -mat_partitioning_type parmetis if you have
>> PETSc compiled with ParMETIS support.
>>
>>
>> Il giorno mer 24 ott 2018 alle ore 20:59 Abdullah Ali Sivas <
>> abdullahasivas at gmail.com> ha scritto:
>>
>>> Hi Stefano,
>>>
>>> I am trying to solve the div-div problem (or grad-div problem in strong
>>> form) with a H(div)-conforming FEM. I am getting the matrices from an
>>> external source (to be clear, from an ngsolve script) and I am not sure if
>>> it is possible to get a MATIS matrix out of that. So I am just treating it
>>> as if I am not able to access the assembly code. The results are 2, 31, 26,
>>> 27, 31 iterations, respectively, for matrix sizes 282, 1095, 4314, 17133,
>>> 67242, 267549. However, norm of the residual also grows significantly;
>>> 7.38369e-09 for 1095 and 5.63828e-07 for 267549. I can try larger sizes, or
>>> maybe this is expected for this case.
>>>
>>> As a side question, if we are dividing the domain into number of MPI
>>> processes subdomains, does it mean that convergence is affected negatively
>>> by the increasing number of processes? I know that alternating Schwarz
>>> method and some other domain decomposition methods sometimes suffer from
>>> the decreasing radius of the subdomains. It sounds like BDDC is pretty
>>> similar to those by your description.
>>>
>>> Best wishes,
>>> Abdullah Ali Sivas
>>>
>>> On Wed, 24 Oct 2018 at 05:28, Stefano Zampini <stefano.zampini at gmail.com>
>>> wrote:
>>>
>>>> Abdullah,
>>>>
>>>> The "Neumann" problems Jed is referring to result from assembling your
>>>> problem on each subdomain ( = MPI process) separately.
>>>> Assuming you are using FEM, these problems have been historically
>>>> named "Neumann" as they correspond to a problem with natural boundary
>>>> conditions (Neumann bc for Poisson).
>>>> Note that in PETSc the subdomain decomposition is associated with the
>>>> mesh decomposition.
>>>>
>>>> When converting from an assembled AIJ matrix to a MATIS format, such
>>>> "Neumann" information is lost.
>>>> You can disassemble an AIJ matrix, in the sense that you can find local
>>>> matrices A_j such that A = \sum_j R^T_j A_j R_j (as it is done in ex72.c),
>>>> but you cannot guarantee (unless if you solve an optimization problem) that
>>>> the disassembling will produce subdomain Neumann problems that are
>>>> consistent with your FEM problem.
>>>>
>>>> I have added such disassembling code a few months ago, just to have
>>>> another alternative for preconditioning AIJ matrices in PETSc; there are
>>>> few tweaks one can do to improve the quality of the disassembling, but I
>>>> discourage its usage unless you don't have access to the FEM assembly code.
>>>>
>>>> With that said, what problem are you trying to solve? Are you using
>>>> DMDA or DMPlex? What are the results you obtained with using the automatic
>>>> disassembling?
>>>>
>>>> Il giorno mer 24 ott 2018 alle ore 08:14 Abdullah Ali Sivas <
>>>> abdullahasivas at gmail.com> ha scritto:
>>>>
>>>>> Hi Jed,
>>>>>
>>>>> Thanks for your reply. The assembled matrix I have corresponds to the
>>>>> full problem on the full mesh. There are no "Neumann" problems (or any sort
>>>>> of domain decomposition) defined in the code generates the matrix. However,
>>>>> I think assembling the full problem is equivalent to implicitly assembling
>>>>> the "Neumann" problems, since the system can be partitioned as;
>>>>>
>>>>> [A_{LL} | A_{LI}]  [u_L]     [F]
>>>>> -----------|------------ -------- = -----
>>>>> [A_{IL}  |A_{II} ]   [u_I]      [G]
>>>>>
>>>>> and G should correspond to the Neumann problem. I might be thinking
>>>>> wrong (or maybe I completely misunderstood the idea), if so please correct
>>>>> me. But I think that the problem is that I am not explicitly telling PCBDDC
>>>>> which dofs are interface dofs.
>>>>>
>>>>> Regards,
>>>>> Abdullah Ali Sivas
>>>>>
>>>>> On Tue, 23 Oct 2018 at 23:16, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>>> Did you assemble "Neumann" problems that are compatible with your
>>>>>> definition of interior/interface degrees of freedom?
>>>>>>
>>>>>> Abdullah Ali Sivas <abdullahasivas at gmail.com> writes:
>>>>>>
>>>>>> > Dear all,
>>>>>> >
>>>>>> > I have a series of linear systems coming from a PDE for which BDDC
>>>>>> is an
>>>>>> > optimal preconditioner. These linear systems are assembled and I
>>>>>> read them
>>>>>> > from a file, then convert into MATIS as required (as in
>>>>>> >
>>>>>> https://www.mcs.anl.gov/petsc/petsc-current/src/ksp/ksp/examples/tutorials/ex72.c.html
>>>>>> > ). I expect each of the systems converge to the solution in almost
>>>>>> same
>>>>>> > number of iterations but I don't observe it. I think it is because
>>>>>> I do not
>>>>>> > provide enough information to the preconditioner. I can get a list
>>>>>> of inner
>>>>>> > dofs and interface dofs. However, I do not know how to use them.
>>>>>> Has anyone
>>>>>> > have any insights about it or done something similar?
>>>>>> >
>>>>>> > Best wishes,
>>>>>> > Abdullah Ali Sivas
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Stefano
>>>>
>>>
>>
>> --
>> Stefano
>>
>

-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/7b106241/attachment-0001.html>

From knepley at gmail.com  Thu Oct 25 09:43:57 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 25 Oct 2018 10:43:57 -0400
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
	<CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
	<CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
Message-ID: <CAMYG4GmzsHgvZ8RPQd8+O2jtVB9jsDP7+mgEjSmM+w6OnQ1t3Q@mail.gmail.com>

Good catch Stefano.

  Matt

On Thu, Oct 25, 2018 at 9:36 AM Stefano Zampini <stefano.zampini at gmail.com>
wrote:

> Maybe this is a fix
>
> diff --git a/src/dm/impls/plex/plexvtu.c b/src/dm/impls/plex/plexvtu.c
> index acdea12c2f..1a8bbada6a 100644
> --- a/src/dm/impls/plex/plexvtu.c
> +++ b/src/dm/impls/plex/plexvtu.c
> @@ -465,10 +465,11 @@ PetscErrorCode DMPlexVTKWriteAll_VTU(DM
> dm,PetscViewer viewer)
>                  if ((closure[v] >= vStart) && (closure[v] < vEnd)) {
>                    PetscScalar *xpoint;
>
> -                  ierr =
> DMPlexPointLocalRead(dm,v,x,&xpoint);CHKERRQ(ierr);
> +                  ierr =
> DMPlexPointLocalRead(dm,closure[v],x,&xpoint);CHKERRQ(ierr);
>                    y[cnt + off++] = xpoint[i];
>                  }
>                }
> +              cnt += off;
>                ierr = DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE,
> &closureSize, &closure);CHKERRQ(ierr);
>              }
>            }
>
> Max, does this fix your problem? If you confirm, I'll fix this in the
> maint branch
>
> If I run the below command line with the patch and with snes tutorials
> ex12 I get the nice picture attached
> $ ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
> -dm_plex_gmsh_periodic -x_periodicity periodic -y_periodicity periodic
> -dm_refine 4
>
> Il giorno gio 25 ott 2018 alle ore 15:11 Stefano Zampini <
> stefano.zampini at gmail.com> ha scritto:
>
>> Matt,
>>
>> you can reproduce it via
>>
>> $ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
>> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
>> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
>> -dm_plex_gmsh_periodic
>>
>> Long time ago I added support for viewing meshes with periodic vertices
>> in the VTK_VTU viewer, but I did not fix the part that writes fields
>>
>>
>> Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley <
>> knepley at gmail.com> ha scritto:
>>
>>> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <
>>> imilian.hartig at gmail.com> wrote:
>>>
>>>>
>>>>
>>>> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
>>>>
>>>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li> wrote:
>>>>
>>>>> Hi Max,
>>>>>
>>>>> (I'm cc'ing in the petsc-users mailing list which may have more
>>>>> advice, if you are using PETSc you should definitely subscribe!
>>>>>
>>>>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <
>>>>> imilian.hartig at gmail.com> wrote:
>>>>> >
>>>>> > Hello Lawrence,
>>>>> >
>>>>> > sorry to message you out of the blue. My name is Max and I found
>>>>> your post on GitHub (
>>>>> https://github.com/firedrakeproject/firedrake/issues/1246 ) on DMPlex
>>>>> being able to read periodic gmsh files. I am currently trying to do just
>>>>> that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD
>>>>> work. So far I haven? t found any documentation on the periodic BC?s with
>>>>> DMPlex and gmsh in the official petsc documentation.
>>>>> > I was wondering whether you?d be so kind as to point me in a general
>>>>> direction concerning how to achieve this. You seem experienced in using
>>>>> petsc and I would greatly appreciate your help.
>>>>>
>>>>>
>>>>> I think the answer is "it depends". If you're just using DMPlex
>>>>> directly and all the of the functionality with PetscDS, then I /think/ that
>>>>> reading periodic meshes via gmsh (assuming you're using the appropriate
>>>>> gmsh mesh format [v2]) "just works".
>>>>>
>>>>
>>>> There are two phases here: topological and geometric. DMPlex represents
>>>> the periodic topological entity directly. For example,  a circle is just a
>>>> segment with one end hooked to the other. Vertices are not duplicated, or
>>>> mapped to each other. This makes topology simple and easy to implement.
>>>> However, then geometry is more complicated. What Plex does is allow
>>>> coordinates to be represented by a discontinuous field taking values on
>>>> cells, in addition to vertices. In our circle example, each cells near the
>>>> cut will have 2 coordinates, one for each vertex, but they will not agree
>>>> across the cut. If you define a periodic domain, then Plex can construct
>>>> this coordinate field automatically using DMPlexLocalize(). These DG
>>>> coordinates are then used by the integration routines.
>>>>
>>>>
>>>> Ok, I think I understand the concept. DMPlex reads information about
>>>> both topology and coordinates from the .msh file. Creating a periodic mesh
>>>> in gmsh then should allow DMPlex to identify the periodic boundaries as the
>>>> ?cut? and build the mesh topology accordingly. Coordinate information is
>>>> handled separately.
>>>> That means, as Lawrence suggested, building periodic meshes in gmsh and
>>>> reading them in to petsc?s DMPlex should indeed ?just work?.  (From the
>>>> user perspective). The only extra step is to call DMLocalizeCoordinates()
>>>> after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make
>>>> sense of this.
>>>>
>>>>
>>>>
>>>>> From my side, the issue is to do with mapping that coordinate field
>>>>> into one that I understand (within Firedrake). You may not have this
>>>>> problem.
>>>>>
>>>>
>>>> Firedrake uses its own coordinate mapping and integration routines, so
>>>> they must manage the second part independently. I hope to get change this
>>>> slightly soon by making the Firedrake representation a DMField, so that it
>>>> looks the same to Plex.
>>>>
>>>>   Thanks,
>>>>
>>>>     Matt
>>>>
>>>>
>>>>> Thanks,
>>>>>
>>>>> Lawrence
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>>
>>>>
>>>> To read periodic meshes from GMSH, you need to use the option
>>>> -dm_plex_gmsh_periodic and DMPlexCreateFromFile
>>>>
>>>>
>>>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But
>>>> using this option I now generate a segmentation fault error when calling
>>>> VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>>>>
>>>
>>>  Small example? VTK is deprecated. HDF5 should work, although it will
>>> require you to have proper coordinates I think. We have to
>>> think about what you mean. If its for a checkpoint, there is no problem,
>>> but for viz, those programs do not understand periodicity. Thus I embed it
>>> in a higher dimensional space.
>>>
>>>    Matt
>>>
>>>> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>>>>
>>>> $ ./ex1 -filename
>>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
>>>> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>>>>
>>>> generating periodic meshes in gmsh may be tricky, Lisandro for sure may
>>>> advice.
>>>>
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>> https://www.cse.buffalo.edu/~knepley/
>>> <http://www.cse.buffalo.edu/~knepley/>
>>>
>>
>>
>> --
>> Stefano
>>
>
>
> --
> Stefano
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/8fb15752/attachment.html>

From stefano.zampini at gmail.com  Thu Oct 25 09:47:43 2018
From: stefano.zampini at gmail.com (Stefano Zampini)
Date: Thu, 25 Oct 2018 17:47:43 +0300
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAMYG4GmzsHgvZ8RPQd8+O2jtVB9jsDP7+mgEjSmM+w6OnQ1t3Q@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
	<CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
	<CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
	<CAMYG4GmzsHgvZ8RPQd8+O2jtVB9jsDP7+mgEjSmM+w6OnQ1t3Q@mail.gmail.com>
Message-ID: <CAGPUisgVL6r_s8aPXZa9E3_2SsVX4ZCqiW7e5OtbgEJUsHjXfg@mail.gmail.com>

Opened the PR
https://bitbucket.org/petsc/petsc/pull-requests/1203/fix-dump-vtk-field-with-periodic-meshes/diff

Il giorno gio 25 ott 2018 alle ore 17:44 Matthew Knepley <knepley at gmail.com>
ha scritto:

> Good catch Stefano.
>
>   Matt
>
> On Thu, Oct 25, 2018 at 9:36 AM Stefano Zampini <stefano.zampini at gmail.com>
> wrote:
>
>> Maybe this is a fix
>>
>> diff --git a/src/dm/impls/plex/plexvtu.c b/src/dm/impls/plex/plexvtu.c
>> index acdea12c2f..1a8bbada6a 100644
>> --- a/src/dm/impls/plex/plexvtu.c
>> +++ b/src/dm/impls/plex/plexvtu.c
>> @@ -465,10 +465,11 @@ PetscErrorCode DMPlexVTKWriteAll_VTU(DM
>> dm,PetscViewer viewer)
>>                  if ((closure[v] >= vStart) && (closure[v] < vEnd)) {
>>                    PetscScalar *xpoint;
>>
>> -                  ierr =
>> DMPlexPointLocalRead(dm,v,x,&xpoint);CHKERRQ(ierr);
>> +                  ierr =
>> DMPlexPointLocalRead(dm,closure[v],x,&xpoint);CHKERRQ(ierr);
>>                    y[cnt + off++] = xpoint[i];
>>                  }
>>                }
>> +              cnt += off;
>>                ierr = DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE,
>> &closureSize, &closure);CHKERRQ(ierr);
>>              }
>>            }
>>
>> Max, does this fix your problem? If you confirm, I'll fix this in the
>> maint branch
>>
>> If I run the below command line with the patch and with snes tutorials
>> ex12 I get the nice picture attached
>> $ ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
>> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
>> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
>> -dm_plex_gmsh_periodic -x_periodicity periodic -y_periodicity periodic
>> -dm_refine 4
>>
>> Il giorno gio 25 ott 2018 alle ore 15:11 Stefano Zampini <
>> stefano.zampini at gmail.com> ha scritto:
>>
>>> Matt,
>>>
>>> you can reproduce it via
>>>
>>> $ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type
>>> dirichlet -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
>>> -dm_plex_gmsh_periodic
>>>
>>> Long time ago I added support for viewing meshes with periodic vertices
>>> in the VTK_VTU viewer, but I did not fix the part that writes fields
>>>
>>>
>>> Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley <
>>> knepley at gmail.com> ha scritto:
>>>
>>>> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <
>>>> imilian.hartig at gmail.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com> wrote:
>>>>>
>>>>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li>
>>>>> wrote:
>>>>>
>>>>>> Hi Max,
>>>>>>
>>>>>> (I'm cc'ing in the petsc-users mailing list which may have more
>>>>>> advice, if you are using PETSc you should definitely subscribe!
>>>>>>
>>>>>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <
>>>>>> imilian.hartig at gmail.com> wrote:
>>>>>> >
>>>>>> > Hello Lawrence,
>>>>>> >
>>>>>> > sorry to message you out of the blue. My name is Max and I found
>>>>>> your post on GitHub (
>>>>>> https://github.com/firedrakeproject/firedrake/issues/1246 ) on
>>>>>> DMPlex being able to read periodic gmsh files. I am currently trying to do
>>>>>> just that (creating a periodic DMPlex mesh with gmsh) in the context of my
>>>>>> PhD work. So far I haven? t found any documentation on the periodic BC?s
>>>>>> with DMPlex and gmsh in the official petsc documentation.
>>>>>> > I was wondering whether you?d be so kind as to point me in a
>>>>>> general direction concerning how to achieve this. You seem experienced in
>>>>>> using petsc and I would greatly appreciate your help.
>>>>>>
>>>>>>
>>>>>> I think the answer is "it depends". If you're just using DMPlex
>>>>>> directly and all the of the functionality with PetscDS, then I /think/ that
>>>>>> reading periodic meshes via gmsh (assuming you're using the appropriate
>>>>>> gmsh mesh format [v2]) "just works".
>>>>>>
>>>>>
>>>>> There are two phases here: topological and geometric. DMPlex
>>>>> represents the periodic topological entity directly. For example,  a circle
>>>>> is just a segment with one end hooked to the other. Vertices are not
>>>>> duplicated, or mapped to each other. This makes topology simple and easy to
>>>>> implement. However, then geometry is more complicated. What Plex does is
>>>>> allow coordinates to be represented by a discontinuous field taking values
>>>>> on cells, in addition to vertices. In our circle example, each cells near
>>>>> the cut will have 2 coordinates, one for each vertex, but they will not
>>>>> agree across the cut. If you define a periodic domain, then Plex can
>>>>> construct this coordinate field automatically using DMPlexLocalize(). These
>>>>> DG coordinates are then used by the integration routines.
>>>>>
>>>>>
>>>>> Ok, I think I understand the concept. DMPlex reads information about
>>>>> both topology and coordinates from the .msh file. Creating a periodic mesh
>>>>> in gmsh then should allow DMPlex to identify the periodic boundaries as the
>>>>> ?cut? and build the mesh topology accordingly. Coordinate information is
>>>>> handled separately.
>>>>> That means, as Lawrence suggested, building periodic meshes in gmsh
>>>>> and reading them in to petsc?s DMPlex should indeed ?just work?.  (From the
>>>>> user perspective). The only extra step is to call DMLocalizeCoordinates()
>>>>> after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make
>>>>> sense of this.
>>>>>
>>>>>
>>>>>
>>>>>> From my side, the issue is to do with mapping that coordinate field
>>>>>> into one that I understand (within Firedrake). You may not have this
>>>>>> problem.
>>>>>>
>>>>>
>>>>> Firedrake uses its own coordinate mapping and integration routines, so
>>>>> they must manage the second part independently. I hope to get change this
>>>>> slightly soon by making the Firedrake representation a DMField, so that it
>>>>> looks the same to Plex.
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>     Matt
>>>>>
>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Lawrence
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>> https://www.cse.buffalo.edu/~knepley/
>>>>>
>>>>>
>>>>> To read periodic meshes from GMSH, you need to use the option
>>>>> -dm_plex_gmsh_periodic and DMPlexCreateFromFile
>>>>>
>>>>>
>>>>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But
>>>>> using this option I now generate a segmentation fault error when calling
>>>>> VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>>>>>
>>>>
>>>>  Small example? VTK is deprecated. HDF5 should work, although it will
>>>> require you to have proper coordinates I think. We have to
>>>> think about what you mean. If its for a checkpoint, there is no
>>>> problem, but for viz, those programs do not understand periodicity. Thus I
>>>> embed it in a higher dimensional space.
>>>>
>>>>    Matt
>>>>
>>>>> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>>>>>
>>>>> $ ./ex1 -filename
>>>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
>>>>> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>>>>>
>>>>> generating periodic meshes in gmsh may be tricky, Lisandro for sure
>>>>> may advice.
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Max
>>>>>
>>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>> https://www.cse.buffalo.edu/~knepley/
>>>> <http://www.cse.buffalo.edu/~knepley/>
>>>>
>>>
>>>
>>> --
>>> Stefano
>>>
>>
>>
>> --
>> Stefano
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/1c48678d/attachment-0001.html>

From stephen.wornom at inria.fr  Thu Oct 25 09:50:53 2018
From: stephen.wornom at inria.fr (Stephen Wornom)
Date: Thu, 25 Oct 2018 16:50:53 +0200 (CEST)
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAGPUisgVL6r_s8aPXZa9E3_2SsVX4ZCqiW7e5OtbgEJUsHjXfg@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
	<CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
	<CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
	<CAMYG4GmzsHgvZ8RPQd8+O2jtVB9jsDP7+mgEjSmM+w6OnQ1t3Q@mail.gmail.com>
	<CAGPUisgVL6r_s8aPXZa9E3_2SsVX4ZCqiW7e5OtbgEJUsHjXfg@mail.gmail.com>
Message-ID: <216197717.27572033.1540479053626.JavaMail.zimbra@inria.fr>

Did you forget to attach "The nice picture attached" 
Stephen 

> From: "Stefano Zampini" <stefano.zampini at gmail.com>
> To: "petsc-maint" <knepley at gmail.com>
> Cc: "imilian hartig" <imilian.hartig at gmail.com>, "PETSc users list"
> <petsc-users at mcs.anl.gov>
> Sent: Thursday, October 25, 2018 4:47:43 PM
> Subject: Re: [petsc-users] Periodic BC in petsc DMPlex / firedrake

> Opened the PR [
> https://bitbucket.org/petsc/petsc/pull-requests/1203/fix-dump-vtk-field-with-periodic-meshes/diff
> |
> https://bitbucket.org/petsc/petsc/pull-requests/1203/fix-dump-vtk-field-with-periodic-meshes/diff
> ]

> Il giorno gio 25 ott 2018 alle ore 17:44 Matthew Knepley < [
> mailto:knepley at gmail.com | knepley at gmail.com ] > ha scritto:

>> Good catch Stefano.
>> Matt

>> On Thu, Oct 25, 2018 at 9:36 AM Stefano Zampini < [
>> mailto:stefano.zampini at gmail.com | stefano.zampini at gmail.com ] > wrote:

>>> Maybe this is a fix
>>> diff --git a/src/dm/impls/plex/plexvtu.c b/src/dm/impls/plex/plexvtu.c
>>> index acdea12c2f..1a8bbada6a 100644
>>> --- a/src/dm/impls/plex/plexvtu.c
>>> +++ b/src/dm/impls/plex/plexvtu.c
>>> @@ -465,10 +465,11 @@ PetscErrorCode DMPlexVTKWriteAll_VTU(DM dm,PetscViewer
>>> viewer)
>>> if ((closure[v] >= vStart) && (closure[v] < vEnd)) {
>>> PetscScalar *xpoint;
>>> - ierr = DMPlexPointLocalRead(dm,v,x,&xpoint);CHKERRQ(ierr);
>>> + ierr = DMPlexPointLocalRead(dm,closure[v],x,&xpoint);CHKERRQ(ierr);
>>> y[cnt + off++] = xpoint[i];
>>> }
>>> }
>>> + cnt += off;
>>> ierr = DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE, &closureSize,
>>> &closure);CHKERRQ(ierr);
>>> }
>>> }

>>> Max, does this fix your problem? If you confirm, I'll fix this in the maint
>>> branch

>>> If I run the below command line with the patch and with snes tutorials ex12 I
>>> get the nice picture attached
>>> $ ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
>>> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
>>> -dm_plex_gmsh_periodic -x_periodicity periodic -y_periodicity periodic
>>> -dm_refine 4

>>> Il giorno gio 25 ott 2018 alle ore 15:11 Stefano Zampini < [
>>> mailto:stefano.zampini at gmail.com | stefano.zampini at gmail.com ] > ha scritto:

>>>> Matt,

>>>> you can reproduce it via
>>>> $ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet
>>>> -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f
>>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh
>>>> -dm_plex_gmsh_periodic

>>>> Long time ago I added support for viewing meshes with periodic vertices in the
>>>> VTK_VTU viewer, but I did not fix the part that writes fields

>>>> Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley < [
>>>> mailto:knepley at gmail.com | knepley at gmail.com ] > ha scritto:

>>>>> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig < [
>>>>> mailto:imilian.hartig at gmail.com | imilian.hartig at gmail.com ] > wrote:

>>>>>>> On 24. Oct 2018, at 12:49, Matthew Knepley < [ mailto:knepley at gmail.com |
>>>>>>> knepley at gmail.com ] > wrote:

>>>>>>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell < [ mailto:wence at gmx.li |
>>>>>>> wence at gmx.li ] > wrote:

>>>>>>>> Hi Max,

>>>>>>>> (I'm cc'ing in the petsc-users mailing list which may have more advice, if you
>>>>>>>> are using PETSc you should definitely subscribe!

>>>>>>>>> On 24 Oct 2018, at 09:27, Maximilian Hartig < [ mailto:imilian.hartig at gmail.com
>>>>>>>> > | imilian.hartig at gmail.com ] > wrote:

>>>>>>>> > Hello Lawrence,

>>>>>>>>> sorry to message you out of the blue. My name is Max and I found your post on
>>>>>>>>> GitHub ( [ https://github.com/firedrakeproject/firedrake/issues/1246 |
>>>>>>>>> https://github.com/firedrakeproject/firedrake/issues/1246 ] ) on DMPlex being
>>>>>>>>> able to read periodic gmsh files. I am currently trying to do just that
>>>>>>>>> (creating a periodic DMPlex mesh with gmsh) in the context of my PhD work. So
>>>>>>>>> far I haven? t found any documentation on the periodic BC?s with DMPlex and
>>>>>>>> > gmsh in the official petsc documentation.
>>>>>>>>> I was wondering whether you?d be so kind as to point me in a general direction
>>>>>>>>> concerning how to achieve this. You seem experienced in using petsc and I would
>>>>>>>> > greatly appreciate your help.

>>>>>>>> I think the answer is "it depends". If you're just using DMPlex directly and all
>>>>>>>> the of the functionality with PetscDS, then I /think/ that reading periodic
>>>>>>>> meshes via gmsh (assuming you're using the appropriate gmsh mesh format [v2])
>>>>>>>> "just works".

>>>>>>> There are two phases here: topological and geometric. DMPlex represents the
>>>>>>> periodic topological entity directly. For example, a circle is just a segment
>>>>>>> with one end hooked to the other. Vertices are not duplicated, or mapped to
>>>>>>> each other. This makes topology simple and easy to implement. However, then
>>>>>>> geometry is more complicated. What Plex does is allow coordinates to be
>>>>>>> represented by a discontinuous field taking values on cells, in addition to
>>>>>>> vertices. In our circle example, each cells near the cut will have 2
>>>>>>> coordinates, one for each vertex, but they will not agree across the cut. If
>>>>>>> you define a periodic domain, then Plex can construct this coordinate field
>>>>>>> automatically using DMPlexLocalize(). These DG coordinates are then used by the
>>>>>>> integration routines.

>>>>>> Ok, I think I understand the concept. DMPlex reads information about both
>>>>>> topology and coordinates from the .msh file. Creating a periodic mesh in gmsh
>>>>>> then should allow DMPlex to identify the periodic boundaries as the ?cut? and
>>>>>> build the mesh topology accordingly. Coordinate information is handled
>>>>>> separately.
>>>>>> That means, as Lawrence suggested, building periodic meshes in gmsh and reading
>>>>>> them in to petsc?s DMPlex should indeed ?just work?. (From the user
>>>>>> perspective). The only extra step is to call DMLocalizeCoordinates() after
>>>>>> DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make sense of
>>>>>> this.

>>>>>>>> From my side, the issue is to do with mapping that coordinate field into one
>>>>>>>> that I understand (within Firedrake). You may not have this problem.

>>>>>>> Firedrake uses its own coordinate mapping and integration routines, so they must
>>>>>>> manage the second part independently. I hope to get change this slightly soon
>>>>>>> by making the Firedrake representation a DMField, so that it looks the same to
>>>>>>> Plex.

>>>>>>> Thanks,

>>>>>>> Matt

>>>>>>>> Thanks,

>>>>>>>> Lawrence
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their experiments is
>>>>>>> infinitely more interesting than any results to which their experiments lead.
>>>>>>> -- Norbert Wiener

>>>>>>> [ https://www.cse.buffalo.edu/~knepley/ | https://www.cse.buffalo.edu/~knepley/
>>>>>>> ]

>>>>>>> To read periodic meshes from GMSH, you need to use the option
>>>>>>> -dm_plex_gmsh_periodic and DMPlexCreateFromFile

>>>>>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But using this
>>>>>> option I now generate a segmentation fault error when calling VecView() on the
>>>>>> solution vector with vtk and hdf5 viewers. Any suggestions?

>>>>> Small example? VTK is deprecated. HDF5 should work, although it will require you
>>>>> to have proper coordinates I think. We have to
>>>>> think about what you mean. If its for a checkpoint, there is no problem, but for
>>>>> viz, those programs do not understand periodicity. Thus I embed it in a higher
>>>>> dimensional space.

>>>>> Matt

>>>>>>> See src/dm/impls/plex/examples/tests/ex1.c. An example runs

>>>>>>> $ ./ex1 -filename
>>>>>>> ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh
>>>>>>> -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape

>>>>>>> generating periodic meshes in gmsh may be tricky, Lisandro for sure may advice.

>>>>>> Thanks,
>>>>>> Max

>>>>> --
>>>>> What most experimenters take for granted before they begin their experiments is
>>>>> infinitely more interesting than any results to which their experiments lead.
>>>>> -- Norbert Wiener

>>>>> [ http://www.cse.buffalo.edu/~knepley/ | https://www.cse.buffalo.edu/~knepley/ ]

>>>> --
>>>> Stefano

>>> --
>>> Stefano

>> --
>> What most experimenters take for granted before they begin their experiments is
>> infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener

>> [ http://www.cse.buffalo.edu/~knepley/ | https://www.cse.buffalo.edu/~knepley/ ]

> --
> Stefano
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/f682ac3c/attachment.html>

From imilian.hartig at gmail.com  Thu Oct 25 09:52:15 2018
From: imilian.hartig at gmail.com (Maximilian Hartig)
Date: Thu, 25 Oct 2018 16:52:15 +0200
Subject: [petsc-users] Periodic BC in petsc DMPlex / firedrake
In-Reply-To: <CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
References: <2D9A9125-F4A2-4A50-A5F9-4D7C9B86B540@gmail.com>
	<2852F36F-0CB0-4737-A489-38A476FDAEF2@gmx.li>
	<CAMYG4GmTZaQHjv=3ovMWWTYgXjZVcUWR5AjhZjYYNsGb90A4nA@mail.gmail.com>
	<6E65EDD0-AB80-4FE5-AF9F-49B9D4D8B9E3@gmail.com>
	<CAMYG4GksK0KtmmoQ7VpwcUqkocPDt3vtD=fOukUmLdce02m7Pw@mail.gmail.com>
	<CAGPUish0i-jhczq+y-MtECfaU8A7b9zFw=RKsb9csnuboveSWg@mail.gmail.com>
	<CAGPUisiQYqZkicWksN6rypEHXUdh5EeGR37QxOsoKBWqpUpiHg@mail.gmail.com>
Message-ID: <3D5F0B86-0E9F-4A00-A717-633AE1BEFC9D@gmail.com>

>> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But using this option I now generate a segmentation fault error when calling VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
>>  

> Small example? VTK is deprecated. HDF5 should work, although it will require you to have proper coordinates I think. We have to
> think about what you mean. If its for a checkpoint, there is no problem, but for viz, those programs do not understand periodicity. Thus I embed it in a higher dimensional space.
> 
>    Matt

Building the small example I realised that hdf5 wasn?t working altogether. I?m trying to fix this and see if I can run VecView with the periodic DMPlex then.
I planned using this for visualisation. 


> On 25. Oct 2018, at 15:36, Stefano Zampini <stefano.zampini at gmail.com> wrote:
> 
> Maybe this is a fix
> 
> diff --git a/src/dm/impls/plex/plexvtu.c b/src/dm/impls/plex/plexvtu.c
> index acdea12c2f..1a8bbada6a 100644
> --- a/src/dm/impls/plex/plexvtu.c
> +++ b/src/dm/impls/plex/plexvtu.c
> @@ -465,10 +465,11 @@ PetscErrorCode DMPlexVTKWriteAll_VTU(DM dm,PetscViewer viewer)
>                  if ((closure[v] >= vStart) && (closure[v] < vEnd)) {
>                    PetscScalar *xpoint;
>  
> -                  ierr = DMPlexPointLocalRead(dm,v,x,&xpoint);CHKERRQ(ierr);
> +                  ierr = DMPlexPointLocalRead(dm,closure[v],x,&xpoint);CHKERRQ(ierr);
>                    y[cnt + off++] = xpoint[i];
>                  }
>                }
> +              cnt += off;
>                ierr = DMPlexRestoreTransitiveClosure(dm, c, PETSC_TRUE, &closureSize, &closure);CHKERRQ(ierr);
>              }
>            }
> 
> Max, does this fix your problem? If you confirm, I'll fix this in the maint branch

It did indeed fix the issue! Thank you. I?ve had problems with hdf5 before and switched to vtk. It?s good news I can continue to use it.
> 
> If I run the below command line with the patch and with snes tutorials ex12 I get the nice picture attached
> $ ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh -dm_plex_gmsh_periodic -x_periodicity periodic -y_periodicity periodic -dm_refine 4
> 
> Il giorno gio 25 ott 2018 alle ore 15:11 Stefano Zampini <stefano.zampini at gmail.com <mailto:stefano.zampini at gmail.com>> ha scritto:
> Matt,
> 
> you can reproduce it via
> 
> $ valgrind ./ex12 -quiet -run_type test -interpolate 1 -bc_type dirichlet -petscspace_degree 1 -vec_view vtk:test.vtu:vtk_vtu -f ${PETSC_DIR}/share/petsc/datafiles/meshes/square_periodic.msh -dm_plex_gmsh_periodic
> 
> Long time ago I added support for viewing meshes with periodic vertices in the VTK_VTU viewer, but I did not fix the part that writes fields
> 
> 
> Il giorno mer 24 ott 2018 alle ore 21:04 Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> ha scritto:
> On Wed, Oct 24, 2018 at 11:36 AM Maximilian Hartig <imilian.hartig at gmail.com <mailto:imilian.hartig at gmail.com>> wrote:
> 
> 
>> On 24. Oct 2018, at 12:49, Matthew Knepley <knepley at gmail.com <mailto:knepley at gmail.com>> wrote:
>> 
>> On Wed, Oct 24, 2018 at 6:29 AM Lawrence Mitchell <wence at gmx.li <mailto:wence at gmx.li>> wrote:
>> Hi Max,
>> 
>> (I'm cc'ing in the petsc-users mailing list which may have more advice, if you are using PETSc you should definitely subscribe!
>> 
>> > On 24 Oct 2018, at 09:27, Maximilian Hartig <imilian.hartig at gmail.com <mailto:imilian.hartig at gmail.com>> wrote:
>> > 
>> > Hello Lawrence,
>> > 
>> > sorry to message you out of the blue. My name is Max and I found your post on GitHub (https://github.com/firedrakeproject/firedrake/issues/1246 <https://github.com/firedrakeproject/firedrake/issues/1246> ) on DMPlex being able to read periodic gmsh files. I am currently trying to do just that (creating a periodic DMPlex mesh with gmsh) in the context of my PhD work. So far I haven? t found any documentation on the periodic BC?s with DMPlex and gmsh in the official petsc documentation. 
>> > I was wondering whether you?d be so kind as to point me in a general direction concerning how to achieve this. You seem experienced in using petsc and I would greatly appreciate your help. 
>> 
>> 
>> I think the answer is "it depends". If you're just using DMPlex directly and all the of the functionality with PetscDS, then I /think/ that reading periodic meshes via gmsh (assuming you're using the appropriate gmsh mesh format [v2]) "just works".
>> 
>> There are two phases here: topological and geometric. DMPlex represents the periodic topological entity directly. For example,  a circle is just a segment with one end hooked to the other. Vertices are not duplicated, or mapped to each other. This makes topology simple and easy to implement. However, then geometry is more complicated. What Plex does is allow coordinates to be represented by a discontinuous field taking values on cells, in addition to vertices. In our circle example, each cells near the cut will have 2 coordinates, one for each vertex, but they will not agree across the cut. If you define a periodic domain, then Plex can construct this coordinate field automatically using DMPlexLocalize(). These DG coordinates are then used by the integration routines.
> 
> Ok, I think I understand the concept. DMPlex reads information about both topology and coordinates from the .msh file. Creating a periodic mesh in gmsh then should allow DMPlex to identify the periodic boundaries as the ?cut? and build the mesh topology accordingly. Coordinate information is handled separately.
> That means, as Lawrence suggested, building periodic meshes in gmsh and reading them in to petsc?s DMPlex should indeed ?just work?.  (From the user perspective). The only extra step is to call DMLocalizeCoordinates() after DMPlexReadFromFile(). Sorry to reiterate, I am just trying to make sense of this. 
>>  
>> From my side, the issue is to do with mapping that coordinate field into one that I understand (within Firedrake). You may not have this problem.
>> 
>> Firedrake uses its own coordinate mapping and integration routines, so they must manage the second part independently. I hope to get change this slightly soon by making the Firedrake representation a DMField, so that it looks the same to Plex.
>> 
>>   Thanks,
>> 
>>     Matt
>>  
>> Thanks,
>> 
>> Lawrence
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/ <https://www.cse.buffalo.edu/~knepley/>
>> To read periodic meshes from GMSH, you need to use the option -dm_plex_gmsh_periodic and DMPlexCreateFromFile
> 
> Ahh, thanks. I was missing the option " -dm_plex_gmsh_periodic ?. But using this option I now generate a segmentation fault error when calling VecView() on the solution vector with vtk and hdf5 viewers. Any suggestions?
> 
>  Small example? VTK is deprecated. HDF5 should work, although it will require you to have proper coordinates I think. We have to
> think about what you mean. If its for a checkpoint, there is no problem, but for viz, those programs do not understand periodicity. Thus I embed it in a higher dimensional space.
> 
>    Matt
>> See  src/dm/impls/plex/examples/tests/ex1.c. An example runs
>> 
>> $ ./ex1 -filename ${PETSC_DIR}/share/petsc/datafiles/meshes/cube_periodic_bin.msh -dm_plex_gmsh_periodic -dm_view ::ascii_info_detail -interpolate -test_shape
>> 
>> generating periodic meshes in gmsh may be tricky, Lisandro for sure may advice.
> 
> Thanks,
> Max
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
> 
> 
> -- 
> Stefano
> 
> 
> -- 
> Stefano
> <periodic.png>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181025/51a02d9c/attachment-0001.html>

From 648934229 at qq.com  Thu Oct 25 17:18:07 2018
From: 648934229 at qq.com (=?gb18030?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 06:18:07 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use --with-ar to
 specify an archiver"
Message-ID: <tencent_66005EB97387AAF422486D42@qq.com>

Hi,

When I try to configure PETSc to connect Matlab with the following command
./petsc-3.8.3/configure ?with-matlab
, it prompts
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
TESTING: checkArchiver from config.setCompilers(config/BuildSystem/config/setCompilers.py:1184)                  *******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Could not find a suitable archiver.  Use --with-ar to specify an archiver.
*******************************************************************************



I am a newbie on Petsc. Could you please tell me how I should specify an archiver?  I am on MacOX system. I have tried many ways but it still does not work.


Thank you very much for your help.


Best,
Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/5dfbb3c4/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 17:23:59 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 22:23:59 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-ar to specify an archiver"
In-Reply-To: <tencent_66005EB97387AAF422486D42@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
Message-ID: <alpine.LFD.2.21.1810251723260.7009@asterix>

Its best to stick with one list - so will follow up on petsc-maint.

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Hi,
> 
> When I try to configure PETSc to connect Matlab with the following command
> ./petsc-3.8.3/configure ??with-matlab
> , it prompts
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: checkArchiver from config.setCompilers(config/BuildSystem/config/setCompilers.py:1184)                  *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Could not find a suitable archiver.  Use --with-ar to specify an archiver.
> *******************************************************************************
> 
> 
> 
> I am a newbie on Petsc. Could you please tell me how I should specify an archiver?  I am on MacOX system. I have tried many ways but it still does not work.
> 
> 
> Thank you very much for your help.
> 
> 
> Best,
> Joe


From 648934229 at qq.com  Thu Oct 25 17:27:32 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 06:27:32 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251723260.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
Message-ID: <tencent_014E19233CF8B25F19D46D24@qq.com>

Hi Satish,


Thank you very much for your quick response.


The log file is as follow:


                              Pushing language C
                              Popping language C
                              Pushing language CUDA
                              Popping language CUDA
                              Pushing language Cxx
                              Popping language Cxx
                              Pushing language FC
                              Popping language FC


================================================================================
================================================================================
Starting Configure Run at Thu Oct 25 16:25:30 2018
Configure Options: --configModules=PETSc.Configure --optionsModule=config.compilerOptions --with-matlab
Working directory: /Users/zhihui/igx/deps/srcs/petsc-3.8.3
Machine platform:
('Darwin', 'Scott-Grad-MBP', '16.7.0', 'Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64', 'x86_64', 'i386')
Python version:
2.7.10 (default, Feb  7 2017, 00:08:15) 
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.34)]
================================================================================
          Pushing language C
          Popping language C
          Pushing language CUDA
          Popping language CUDA
          Pushing language Cxx
          Popping language Cxx
          Pushing language FC
          Popping language FC
================================================================================
TEST configureExternalPackagesDir from config.framework(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py:841)
TESTING: configureExternalPackagesDir from config.framework(config/BuildSystem/config/framework.py:841)
================================================================================
TEST configureDebuggers from config.utilities.debuggers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/utilities/debuggers.py:22)
TESTING: configureDebuggers from config.utilities.debuggers(config/BuildSystem/config/utilities/debuggers.py:22)
  Find a default debugger and determine its arguments
Checking for program /usr/local/bin/gdb...not found
Checking for program /usr/bin/gdb...not found
Checking for program /bin/gdb...not found
Checking for program /usr/sbin/gdb...not found
Checking for program /sbin/gdb...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/gdb...not found
Checking for program /Library/TeX/texbin/gdb...not found
Checking for program /Users/zhihui/gdb...not found
Checking for program /usr/local/bin/dbx...not found
Checking for program /usr/bin/dbx...not found
Checking for program /bin/dbx...not found
Checking for program /usr/sbin/dbx...not found
Checking for program /sbin/dbx...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/dbx...not found
Checking for program /Library/TeX/texbin/dbx...not found
Checking for program /Users/zhihui/dbx...not found
Checking for program /usr/local/bin/xdb...not found
Checking for program /usr/bin/xdb...not found
Checking for program /bin/xdb...not found
Checking for program /usr/sbin/xdb...not found
Checking for program /sbin/xdb...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/xdb...not found
Checking for program /Library/TeX/texbin/xdb...not found
Checking for program /Users/zhihui/xdb...not found
Checking for program /usr/local/bin/dsymutil...not found
Checking for program /usr/bin/dsymutil...found
                Defined make macro "DSYMUTIL" to "/usr/bin/dsymutil"
              Defined make macro "DSYMUTIL" to "/usr/bin/dsymutil"
================================================================================
TEST configureGit from config.sourceControl(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/sourceControl.py:24)
TESTING: configureGit from config.sourceControl(config/BuildSystem/config/sourceControl.py:24)
  Find the Git executable
Checking for program /usr/local/bin/git...not found
Checking for program /usr/bin/git...found
                Defined make macro "GIT" to "git"
Executing: git --version
stdout: git version 2.14.3 (Apple Git-98)
================================================================================
TEST configureMercurial from config.sourceControl(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/sourceControl.py:35)
TESTING: configureMercurial from config.sourceControl(config/BuildSystem/config/sourceControl.py:35)
  Find the Mercurial executable
Checking for program /usr/local/bin/hg...not found
Checking for program /usr/bin/hg...not found
Checking for program /bin/hg...not found
Checking for program /usr/sbin/hg...not found
Checking for program /sbin/hg...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/hg...not found
Checking for program /Library/TeX/texbin/hg...not found
Checking for program /Users/zhihui/hg...not found
================================================================================
TEST configureCLanguage from PETSc.options.languages(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/languages.py:27)
TESTING: configureCLanguage from PETSc.options.languages(config/PETSc/options/languages.py:27)
  Choose whether to compile the PETSc library using a C or C++ compiler
            C language is C
              Defined "CLANGUAGE_C" to "1"
================================================================================
TEST configureDirectories from PETSc.options.petscdir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/petscdir.py:23)
TESTING: configureDirectories from PETSc.options.petscdir(config/PETSc/options/petscdir.py:23)
  Checks PETSC_DIR and sets if not set
            Version Information:
            #define PETSC_VERSION_RELEASE    1
            #define PETSC_VERSION_MAJOR      3
            #define PETSC_VERSION_MINOR      8
            #define PETSC_VERSION_SUBMINOR   3
            #define PETSC_VERSION_PATCH      0
            #define PETSC_VERSION_DATE       "Dec, 09, 2017"
            #define PETSC_VERSION_GIT        "v3.8.3"
            #define PETSC_VERSION_DATE_GIT   "2017-12-09 22:33:04 -0600"
            #define PETSC_VERSION_EQ(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_ PETSC_VERSION_EQ
            #define PETSC_VERSION_LT(MAJOR,MINOR,SUBMINOR)          \
            #define PETSC_VERSION_LE(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_GT(MAJOR,MINOR,SUBMINOR) \
            #define PETSC_VERSION_GE(MAJOR,MINOR,SUBMINOR) \
              Defined make macro "DIR" to "/Users/zhihui/igx/deps/srcs/petsc-3.8.3"
================================================================================
TEST getDatafilespath from PETSc.options.dataFilesPath(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/dataFilesPath.py:29)
TESTING: getDatafilespath from PETSc.options.dataFilesPath(config/PETSc/options/dataFilesPath.py:29)
  Checks what DATAFILESPATH should be
================================================================================
TEST configureInstallationMethod from PETSc.options.petscclone(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/petscclone.py:20)
TESTING: configureInstallationMethod from PETSc.options.petscclone(config/PETSc/options/petscclone.py:20)
            This is a tarball installation
================================================================================
TEST configureArchitecture from PETSc.options.arch(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/arch.py:36)
TESTING: configureArchitecture from PETSc.options.arch(config/PETSc/options/arch.py:36)
  Checks PETSC_ARCH and sets if not set
              Defined "ARCH" to ""arch-darwin-c-debug""
================================================================================
TEST setInstallDir from PETSc.options.installDir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/installDir.py:35)
TESTING: setInstallDir from PETSc.options.installDir(config/PETSc/options/installDir.py:35)
   setup installDir to either prefix or if that is not set to PETSC_DIR/PETSC_ARCH
================================================================================
TEST saveReconfigure from PETSc.options.installDir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/installDir.py:75)
TESTING: saveReconfigure from PETSc.options.installDir(config/PETSc/options/installDir.py:75)
================================================================================
TEST cleanInstallDir from PETSc.options.installDir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/installDir.py:68)
TESTING: cleanInstallDir from PETSc.options.installDir(config/PETSc/options/installDir.py:68)
================================================================================
TEST configureInstallDir from PETSc.options.installDir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/installDir.py:52)
TESTING: configureInstallDir from PETSc.options.installDir(config/PETSc/options/installDir.py:52)
  Makes  installDir subdirectories if it does not exist for both prefix install location and PETSc work install location
            Changed persistence directory to /Users/zhihui/igx/deps/srcs/petsc-3.8.3/arch-darwin-c-debug/lib/petsc/conf
================================================================================
TEST restoreReconfigure from PETSc.options.installDir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/installDir.py:88)
TESTING: restoreReconfigure from PETSc.options.installDir(config/PETSc/options/installDir.py:88)
================================================================================
TEST setExternalPackagesDir from PETSc.options.externalpackagesdir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/externalpackagesdir.py:15)
TESTING: setExternalPackagesDir from PETSc.options.externalpackagesdir(config/PETSc/options/externalpackagesdir.py:15)
================================================================================
TEST cleanExternalpackagesDir from PETSc.options.externalpackagesdir(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/options/externalpackagesdir.py:22)
TESTING: cleanExternalpackagesDir from PETSc.options.externalpackagesdir(config/PETSc/options/externalpackagesdir.py:22)
================================================================================
TEST printEnvVariables from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1612)
TESTING: printEnvVariables from config.setCompilers(config/BuildSystem/config/setCompilers.py:1612)
            **** printenv ****
VERSIONER_PYTHON_PREFER_32_BIT=no
TERM_PROGRAM_VERSION=3.2.4
LOGNAME=zhihui
USER=zhihui
PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/Library/TeX/texbin
HOME=/Users/zhihui
TERM_PROGRAM=iTerm.app
LANG=en_US.UTF-8
TERM=xterm-256color
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.ybiDqNFoh0/Render
COLORFGBG=7;0
VERSIONER_PYTHON_VERSION=2.7
SHLVL=1
XPC_FLAGS=0x0
ITERM_SESSION_ID=w0t0p0:52215AF3-F8D7-4DDE-BC06-683584748154
_=./configure
TERM_SESSION_ID=w0t0p0:52215AF3-F8D7-4DDE-BC06-683584748154
XPC_SERVICE_NAME=0
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.XYegXE4MrG/Listeners
SHELL=/bin/bash
ITERM_PROFILE=Default
TMPDIR=tmp
OLDPWD=/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config
__CF_USER_TEXT_ENCODING=0x1F6:0x0:0x0
PWD=/Users/zhihui/igx/deps/srcs/petsc-3.8.3
COLORTERM=truecolor
================================================================================
TEST resetEnvCompilers from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1619)
TESTING: resetEnvCompilers from config.setCompilers(config/BuildSystem/config/setCompilers.py:1619)
================================================================================
TEST checkEnvCompilers from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1649)
TESTING: checkEnvCompilers from config.setCompilers(config/BuildSystem/config/setCompilers.py:1649)
================================================================================
TEST checkMPICompilerOverride from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1584)
TESTING: checkMPICompilerOverride from config.setCompilers(config/BuildSystem/config/setCompilers.py:1584)
  Check if --with-mpi-dir is used along with CC CXX or FC compiler options.
    This usually prevents mpi compilers from being used - so issue a warning
================================================================================
TEST requireMpiLdPath from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1603)
TESTING: requireMpiLdPath from config.setCompilers(config/BuildSystem/config/setCompilers.py:1603)
  OpenMPI wrappers require LD_LIBRARY_PATH set
================================================================================
TEST checkVendor from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:426)
TESTING: checkVendor from config.setCompilers(config/BuildSystem/config/setCompilers.py:426)
  Determine the compiler vendor
            Compiler vendor is ""
================================================================================
TEST checkInitialFlags from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:436)
TESTING: checkInitialFlags from config.setCompilers(config/BuildSystem/config/setCompilers.py:436)
  Initialize the compiler and linker flags
              Pushing language C
            Initialized CFLAGS to 
            Initialized CFLAGS to 
            Initialized LDFLAGS to 
              Popping language C
              Pushing language CUDA
            Initialized CUDAFLAGS to 
            Initialized CUDAFLAGS to 
            Initialized LDFLAGS to 
              Popping language CUDA
              Pushing language Cxx
            Initialized CXXFLAGS to 
            Initialized CXX_CXXFLAGS to 
            Initialized LDFLAGS to 
              Popping language Cxx
              Pushing language FC
            Initialized FFLAGS to 
            Initialized FFLAGS to 
            Initialized LDFLAGS to 
              Popping language FC
            Initialized CPPFLAGS to 
            Initialized CUDAPPFLAGS to 
            Initialized CXXCPPFLAGS to 
            Initialized CC_LINKER_FLAGS to []
            Initialized CXX_LINKER_FLAGS to []
            Initialized FC_LINKER_FLAGS to []
            Initialized CUDAC_LINKER_FLAGS to []
            Initialized sharedLibraryFlags to []
            Initialized dynamicLibraryFlags to []
================================================================================
TEST checkCCompiler from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:566)
TESTING: checkCCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:566)
  Locate a functional C compiler
Executing: mpicc --help
stdout:
OVERVIEW: clang LLVM compiler
USAGE: clang-6.0 [options] <inputs>
OPTIONS:
  -###                    Print (but do not run) the commands to run for this compilation
  --analyzer-output <value>
                          Static analyzer report output format (html|plist|plist-multi-file|plist-html|text).
  --analyze               Run the static analyzer
  -arcmt-migrate-emit-errors
                          Emit ARC errors even if the migrator can fix them
  -arcmt-migrate-report-output <value>
                          Output path for the plist report
  -B <dir>                Add <dir> to search path for binaries and object files used implicitly
  -CC                     Include comments from within macros in preprocessed output
  -cl-denorms-are-zero    OpenCL only. Allow denormals to be flushed to zero.
  -cl-fast-relaxed-math   OpenCL only. Sets -cl-finite-math-only and -cl-unsafe-math-optimizations, and defines __FAST_RELAXED_MATH__.
  -cl-finite-math-only    OpenCL only. Allow floating-point optimizations that assume arguments and results are not NaNs or +-Inf.
  -cl-fp32-correctly-rounded-divide-sqrt
                          OpenCL only. Specify that single precision floating-point divide and sqrt used in the program source are correctly rounded.
  -cl-kernel-arg-info     OpenCL only. Generate kernel argument metadata.
  -cl-mad-enable          OpenCL only. Allow use of less precise MAD computations in the generated binary.
  -cl-no-signed-zeros     OpenCL only. Allow use of less precise no signed zeros computations in the generated binary.
  -cl-opt-disable         OpenCL only. This option disables all optimizations. By default optimizations are enabled.
  -cl-single-precision-constant
                          OpenCL only. Treat double precision floating-point constant as single precision constant.
  -cl-std=<value>         OpenCL language standard to compile for.
  -cl-strict-aliasing     OpenCL only. This option is added for compatibility with OpenCL 1.0.
  -cl-unsafe-math-optimizations
                          OpenCL only. Allow unsafe floating-point optimizations.  Also implies -cl-no-signed-zeros and -cl-mad-enable.
  --config <value>        Specifies configuration file
  --cuda-compile-host-device
                          Compile CUDA code for both host and device (default).  Has no effect on non-CUDA compilations.
  --cuda-device-only      Compile CUDA code for device only
  --cuda-gpu-arch=<value> CUDA GPU architecture (e.g. sm_35).  May be specified more than once.
  --cuda-host-only        Compile CUDA code for host only.  Has no effect on non-CUDA compilations.
  --cuda-noopt-device-debug
                          Enable device-side debug info generation. Disables ptxas optimizations.
  --cuda-path=<value>     CUDA installation path
  -cxx-isystem <directory>
                          Add directory to the C++ SYSTEM include search path
  -C                      Include comments in preprocessed output
  -c                      Only run preprocess, compile, and assemble steps
  -dD                     Print macro definitions in -E mode in addition to normal output
  -dependency-dot <value> Filename to write DOT-formatted header dependencies to
  -dependency-file <value>
                          Filename (or -) to write dependency output to
  -dI                     Print include directives in -E mode in addition to normal output
  -dM                     Print macro definitions in -E mode instead of normal output
  -D <macro>=<value>      Define <macro> to <value> (or 1 if <value> omitted)
  -emit-ast               Emit Clang AST files for source inputs
  -emit-llvm              Use the LLVM representation for assembler and object files
  -E                      Only run the preprocessor
  -faligned-allocation    Enable C++17 aligned allocation functions
  -fallow-editor-placeholders
                          Treat editor placeholders as valid source code
  -fansi-escape-codes     Use ANSI escape codes for diagnostics
  -fapple-kext            Use Apple's kernel extensions ABI
  -fapple-pragma-pack     Enable Apple gcc-compatible #pragma pack handling
  -fapplication-extension Restrict code to those available for App Extensions
  -fblocks                Enable the 'blocks' language feature
  -fborland-extensions    Accept non-standard constructs supported by the Borland compiler
  -fbuild-session-file=<file>
                          Use the last modification time of <file> as the build session timestamp
  -fbuild-session-timestamp=<time since Epoch in seconds>
                          Time when the current build session started
  -fbuiltin-module-map    Load the clang builtins module map file.
  -fclang-abi-compat=<version>
                          Attempt to match the ABI of Clang <version>
  -fcolor-diagnostics     Use colors in diagnostics
  -fcomment-block-commands=<arg>
                          Treat each comma separated argument in <arg> as a documentation comment block command
  -fcoroutines-ts         Enable support for the C++ Coroutines TS
  -fcoverage-mapping      Generate coverage mapping to enable code coverage analysis
  -fcuda-approx-transcendentals
                          Use approximate transcendental functions
  -fcuda-flush-denormals-to-zero
                          Flush denormal floating point values to zero in CUDA device mode.
  -fcxx-exceptions        Enable C++ exceptions
  -fdata-sections         Place each data in its own section (ELF Only)
  -fdebug-info-for-profiling
                          Emit extra debug info to make sample profile more accurate.
  -fdebug-macro           Emit macro debug information
  -fdebug-prefix-map=<value>
                          remap file source paths in debug info
  -fdebug-types-section   Place debug types in their own section (ELF Only)
  -fdeclspec              Allow __declspec as a keyword
  -fdelayed-template-parsing
                          Parse templated function definitions at the end of the translation unit
  -fdiagnostics-absolute-paths
                          Print absolute paths in diagnostics
  -fdiagnostics-hotness-threshold=<number>
                          Prevent optimization remarks from being output if they do not have at least this profile count
  -fdiagnostics-parseable-fixits
                          Print fix-its in machine parseable form
  -fdiagnostics-print-source-range-info
                          Print source range spans in numeric form
  -fdiagnostics-show-hotness
                          Enable profile hotness information in diagnostic line
  -fdiagnostics-show-note-include-stack
                          Display include stacks for diagnostic notes
  -fdiagnostics-show-option
                          Print option name with mappable diagnostics
  -fdiagnostics-show-template-tree
                          Print a template comparison tree for differing templates
  -fdollars-in-identifiers
                          Allow '$' in identifiers
  -fdouble-square-bracket-attributes
                          Enable '[[]]' attributes in all C and C++ language modes
  -fdwarf-exceptions      Use DWARF style exceptions
  -fembed-bitcode-marker  Embed placeholder LLVM IR data as a marker
  -fembed-bitcode=<option>
                          Embed LLVM bitcode (option: off, all, bitcode, marker)
  -fembed-bitcode         Embed LLVM IR bitcode as data
  -femit-all-decls        Emit all declarations, even if unused
  -femulated-tls          Use emutls functions to access thread_local variables
  -fexceptions            Enable support for exception handling
  -fexperimental-isel     Enables the experimental global instruction selector
  -fexperimental-new-pass-manager
                          Enables an experimental new pass manager in LLVM.
  -ffast-math             Allow aggressive, lossy floating-point optimizations
  -ffine-grained-bitfield-accesses
                          Use separate accesses for bitfields with legal widths and alignments.
  -ffixed-r9              Reserve the r9 register (ARM only)
  -ffixed-x18             Reserve the x18 register (AArch64 only)
  -ffp-contract=<value>   Form fused FP ops (e.g. FMAs): fast (everywhere) | on (according to FP_CONTRACT pragma, default) | off (never fuse)
  -ffreestanding          Assert that the compilation takes place in a freestanding environment
  -ffunction-sections     Place each function in its own section (ELF Only)
  -fgnu-keywords          Allow GNU-extension keywords regardless of language standard
  -fgnu-runtime           Generate output compatible with the standard GNU Objective-C runtime
  -fgnu89-inline          Use the gnu89 inline semantics
  -fimplicit-module-maps  Implicitly search the file system for module map files.
  -finline-functions      Inline suitable functions
  -finline-hint-functions Inline functions which are (explicitly or implicitly) marked inline
  -finstrument-function-entry-bare
                          Instrument function entry only, after inlining, without arguments to the instrumentation call
  -finstrument-functions-after-inlining
                          Like -finstrument-functions, but insert the calls after inlining
  -finstrument-functions  Generate calls to instrument function entry and exit
  -fintegrated-as         Enable the integrated assembler
  -flto-jobs=<value>      Controls the backend parallelism of -flto=thin (default of 0 means the number of threads will be derived from the number of CPUs detected)
  -flto=<value>           Set LTO mode to either 'full' or 'thin'
  -flto                   Enable LTO in 'full' mode
  -fmath-errno            Require math functions to indicate errors by setting errno
  -fmax-type-align=<value>
                          Specify the maximum alignment to enforce on pointers lacking an explicit alignment
  -fmodule-file=[<name>=]<file>
                          Specify the mapping of module name to precompiled module file, or load a module file if name is omitted.
  -fmodule-map-file=<file>
                          Load this module map file
  -fmodule-name=<name>    Specify the name of the module to build
  -fmodules-cache-path=<directory>
                          Specify the module cache path
  -fmodules-decluse       Require declaration of modules used within a module
  -fmodules-disable-diagnostic-validation
                          Disable validation of the diagnostic options when loading the module
  -fmodules-ignore-macro=<value>
                          Ignore the definition of the given macro when building and loading modules
  -fmodules-prune-after=<seconds>
                          Specify the interval (in seconds) after which a module file will be considered unused
  -fmodules-prune-interval=<seconds>
                          Specify the interval (in seconds) between attempts to prune the module cache
  -fmodules-search-all    Search even non-imported modules to resolve references
  -fmodules-strict-decluse
                          Like -fmodules-decluse but requires all headers to be in modules
  -fmodules-ts            Enable support for the C++ Modules TS
  -fmodules-user-build-path <directory>
                          Specify the module user build path
  -fmodules-validate-once-per-build-session
                          Don't verify input files for the modules if the module has been successfully validated or loaded during this build session
  -fmodules-validate-system-headers
                          Validate the system headers that a module depends on when loading the module
  -fmodules               Enable the 'modules' language feature
  -fms-compatibility-version=<value>
                          Dot-separated value representing the Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default))
  -fms-compatibility      Enable full Microsoft Visual C++ compatibility
  -fms-extensions         Accept some non-standard constructs supported by the Microsoft compiler
  -fmsc-version=<value>   Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default))
  -fnew-alignment=<align> Specifies the largest alignment guaranteed by '::operator new(size_t)'
  -fno-access-control     Disable C++ access control
  -fno-assume-sane-operator-new
                          Don't assume that C++'s global operator new can't alias any pointer
  -fno-autolink           Disable generation of linker directives for automatic library linking
  -fno-builtin-<value>    Disable implicit builtin knowledge of a specific function
  -fno-builtin            Disable implicit builtin knowledge of functions
  -fno-common             Compile common globals like normal definitions
  -fno-constant-cfstrings Disable creation of CodeFoundation-type constant strings
  -fno-coverage-mapping   Disable code coverage analysis
  -fno-crash-diagnostics  Disable auto-generation of preprocessed source files and a script for reproduction during a clang crash
  -fno-debug-info-for-profiling
                          Do not emit extra debug info for sample profiler.
  -fno-debug-macro        Do not emit macro debug information
  -fno-declspec           Disallow __declspec as a keyword
  -fno-delayed-template-parsing
                          Disable delayed template parsing
  -fno-diagnostics-fixit-info
                          Do not include fixit information in diagnostics
  -fno-dollars-in-identifiers
                          Disallow '$' in identifiers
  -fno-double-square-bracket-attributes
                          Disable '[[]]' attributes in all C and C++ language modes
  -fno-elide-constructors Disable C++ copy constructor elision
  -fno-elide-type         Do not elide types when printing diagnostics
  -fno-experimental-isel  Disables the experimental global instruction selector
  -fno-experimental-new-pass-manager
                          Disables an experimental new pass manager in LLVM.
  -fno-fine-grained-bitfield-accesses
                          Use large-integer access for consecutive bitfield runs.
  -fno-gnu-inline-asm     Disable GNU style inline asm
  -fno-integrated-as      Disable the integrated assembler
  -fno-jump-tables        Do not use jump tables for lowering switches
  -fno-lax-vector-conversions
                          Disallow implicit conversions between vectors with a different number of elements or different element types
  -fno-lto                Disable LTO mode (default)
  -fno-merge-all-constants
                          Disallow merging of constants
  -fno-objc-infer-related-result-type
                          do not infer Objective-C related result type based on method family
  -fno-openmp-simd        Disable OpenMP code for SIMD-based constructs.
  -fno-operator-names     Do not treat C++ operator name keywords as synonyms for operators
  -fno-plt                Do not use the PLT to make function calls
  -fno-preserve-as-comments
                          Do not preserve comments in inline assembly
  -fno-profile-generate   Disable generation of profile instrumentation.
  -fno-profile-instr-generate
                          Disable generation of profile instrumentation.
  -fno-profile-instr-use  Disable using instrumentation data for profile-guided optimization
  -fno-reroll-loops       Turn off loop reroller
  -fno-rtlib-add-rpath    Do not add -rpath with architecture-specific resource directory to the linker flags
  -fno-rtti               Disable generation of rtti information
  -fno-sanitize-address-use-after-scope
                          Disable use-after-scope detection in AddressSanitizer
  -fno-sanitize-blacklist Don't use blacklist file for sanitizers
  -fno-sanitize-cfi-cross-dso
                          Disable control flow integrity (CFI) checks for cross-DSO calls.
  -fno-sanitize-coverage=<value>
                          Disable specified features of coverage instrumentation for Sanitizers
  -fno-sanitize-memory-track-origins
                          Disable origins tracking in MemorySanitizer
  -fno-sanitize-memory-use-after-dtor
                          Disable use-after-destroy detection in MemorySanitizer
  -fno-sanitize-recover=<value>
                          Disable recovery for specified sanitizers
  -fno-sanitize-stats     Disable sanitizer statistics gathering.
  -fno-sanitize-thread-atomics
                          Disable atomic operations instrumentation in ThreadSanitizer
  -fno-sanitize-thread-func-entry-exit
                          Disable function entry/exit instrumentation in ThreadSanitizer
  -fno-sanitize-thread-memory-access
                          Disable memory access instrumentation in ThreadSanitizer
  -fno-sanitize-trap=<value>
                          Disable trapping for specified sanitizers
  -fno-short-wchar        Force wchar_t to be an unsigned int
  -fno-show-column        Do not include column number on diagnostics
  -fno-show-source-location
                          Do not include source location information with diagnostics
  -fno-signed-char        Char is unsigned
  -fno-signed-zeros       Allow optimizations that ignore the sign of floating point zeros
  -fno-spell-checking     Disable spell-checking
  -fno-stack-protector    Disable the use of stack protectors
  -fno-standalone-debug   Limit debug information produced to reduce size of debug binary
  -fno-threadsafe-statics Do not emit code to make initialization of local statics thread safe
  -fno-trigraphs          Do not process trigraph sequences
  -fno-unroll-loops       Turn off loop unroller
  -fno-use-cxa-atexit     Don't use __cxa_atexit for calling destructors
  -fno-use-init-array     Don't use .init_array instead of .ctors
  -fnoopenmp-relocatable-target
                          Do not compile OpenMP target code as relocatable.
  -fobjc-arc-exceptions   Use EH-safe code when synthesizing retains and releases in -fobjc-arc
  -fobjc-arc              Synthesize retain and release calls for Objective-C pointers
  -fobjc-exceptions       Enable Objective-C exceptions
  -fobjc-runtime=<value>  Specify the target Objective-C runtime kind and version
  -fobjc-weak             Enable ARC-style weak references in Objective-C
  -fopenmp-relocatable-target
                          OpenMP target code is compiled as relocatable using the -c flag. For OpenMP targets the code is relocatable by default.
  -fopenmp-simd           Emit OpenMP code only for SIMD-based constructs.
  -fopenmp-targets=<value>
                          Specify comma-separated list of triples OpenMP offloading targets to be supported
  -foptimization-record-file=<value>
                          Specify the file name of any generated YAML optimization record
  -fpack-struct=<value>   Specify the default maximum struct packing alignment
  -fpascal-strings        Recognize and construct Pascal-style string literals
  -fpcc-struct-return     Override the default ABI to return all structs on the stack
  -fplt                   Use the PLT to make function calls
  -fplugin=<dsopath>      Load the named plugin (dynamic shared object)
  -fprebuilt-module-path=<directory>
                          Specify the prebuilt module path
  -fprofile-generate=<directory>
                          Generate instrumented code to collect execution counts into <directory>/default.profraw (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-generate      Generate instrumented code to collect execution counts into default.profraw (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-instr-generate=<file>
                          Generate instrumented code to collect execution counts into <file> (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-instr-generate
                          Generate instrumented code to collect execution counts into default.profraw file (overridden by '=' form of option or LLVM_PROFILE_FILE env var)
  -fprofile-instr-use=<value>
                          Use instrumentation data for profile-guided optimization
  -fprofile-sample-accurate
                          Specifies that the sample profile is accurate
  -fprofile-sample-use=<value>
                          Enable sample-based profile guided optimizations
  -fprofile-use=<pathname>
                          Use instrumentation data for profile-guided optimization. If pathname is a directory, it reads from <pathname>/default.profdata. Otherwise, it reads from file <pathname>.
  -freciprocal-math       Allow division operations to be reassociated
  -freg-struct-return     Override the default ABI to return small structs in registers
  -frelaxed-template-template-args
                          Enable C++17 relaxed template template argument matching
  -freroll-loops          Turn on loop reroller
  -frtlib-add-rpath       Add -rpath with architecture-specific resource directory to the linker flags
  -fsanitize-address-field-padding=<value>
                          Level of field padding for AddressSanitizer
  -fsanitize-address-globals-dead-stripping
                          Enable linker dead stripping of globals in AddressSanitizer
  -fsanitize-address-use-after-scope
                          Enable use-after-scope detection in AddressSanitizer
  -fsanitize-blacklist=<value>
                          Path to blacklist file for sanitizers
  -fsanitize-cfi-cross-dso
                          Enable control flow integrity (CFI) checks for cross-DSO calls.
  -fsanitize-cfi-icall-generalize-pointers
                          Generalize pointers in CFI indirect call type signature checks
  -fsanitize-coverage=<value>
                          Specify the type of coverage instrumentation for Sanitizers
  -fsanitize-memory-track-origins=<value>
                          Enable origins tracking in MemorySanitizer
  -fsanitize-memory-track-origins
                          Enable origins tracking in MemorySanitizer
  -fsanitize-memory-use-after-dtor
                          Enable use-after-destroy detection in MemorySanitizer
  -fsanitize-recover=<value>
                          Enable recovery for specified sanitizers
  -fsanitize-stats        Enable sanitizer statistics gathering.
  -fsanitize-thread-atomics
                          Enable atomic operations instrumentation in ThreadSanitizer (default)
  -fsanitize-thread-func-entry-exit
                          Enable function entry/exit instrumentation in ThreadSanitizer (default)
  -fsanitize-thread-memory-access
                          Enable memory access instrumentation in ThreadSanitizer (default)
  -fsanitize-trap=<value> Enable trapping for specified sanitizers
  -fsanitize-undefined-strip-path-components=<number>
                          Strip (or keep only, if negative) a given number of path components when emitting check metadata.
  -fsanitize=<check>      Turn on runtime checks for various forms of undefined or suspicious behavior. See user manual for available checks
  -fsave-optimization-record
                          Generate a YAML optimization record file
  -fseh-exceptions        Use SEH style exceptions
  -fshort-enums           Allocate to an enum type only as many bytes as it needs for the declared range of possible values
  -fshort-wchar           Force wchar_t to be a short unsigned int
  -fshow-overloads=<value>
                          Which overload candidates to show when overload resolution fails: best|all; defaults to all
  -fsized-deallocation    Enable C++14 sized global deallocation functions
  -fsjlj-exceptions       Use SjLj style exceptions
  -fslp-vectorize         Enable the superword-level parallelism vectorization passes
  -fsplit-dwarf-inlining  Place debug types in their own section (ELF Only)
  -fstack-protector-all   Force the usage of stack protectors for all functions
  -fstack-protector-strong
                          Use a strong heuristic to apply stack protectors to functions
  -fstack-protector       Enable stack protectors for functions potentially vulnerable to stack smashing
  -fstandalone-debug      Emit full debug info for all types used by the program
  -fstrict-enums          Enable optimizations based on the strict definition of an enum's value range
  -fstrict-return         Always treat control flow paths that fall off the end of a non-void function as unreachable
  -fstrict-vtable-pointers
                          Enable optimizations based on the strict rules for overwriting polymorphic C++ objects
  -fthinlto-index=<value> Perform ThinLTO importing using provided function summary index
  -ftrap-function=<value> Issue call to specified function rather than a trap instruction
  -ftrapv-handler=<function name>
                          Specify the function to be called on overflow
  -ftrapv                 Trap on integer overflow
  -ftrigraphs             Process trigraph sequences
  -funique-section-names  Use unique names for text and data sections (ELF Only)
  -funroll-loops          Turn on loop unroller
  -fuse-init-array        Use .init_array instead of .ctors
  -fveclib=<value>        Use the given vector functions library
  -fvectorize             Enable the loop vectorization passes
  -fvisibility-inlines-hidden
                          Give inline C++ member functions default visibility by default
  -fvisibility-ms-compat  Give global types 'default' visibility and global functions and variables 'hidden' visibility by default
  -fvisibility=<value>    Set the default symbol visibility for all global declarations
  -fwhole-program-vtables Enables whole-program vtable optimization. Requires -flto
  -fwrapv                 Treat signed integer overflow as two's complement
  -fwritable-strings      Store string literals as writable data
  -fxray-always-emit-customevents
                          Determine whether to always emit __xray_customevent(...) calls even if the function it appears in is not always instrumented.
  -fxray-always-instrument= <value>
                          Filename defining the whitelist for imbuing the 'always instrument' XRay attribute.
  -fxray-instruction-threshold= <value>
                          Sets the minimum function size to instrument with XRay
  -fxray-instrument       Generate XRay instrumentation sleds on function entry and exit
  -fxray-never-instrument= <value>
                          Filename defining the whitelist for imbuing the 'never instrument' XRay attribute.
  -fzvector               Enable System z vector language extension
  -F <value>              Add directory to framework include search path
  --gcc-toolchain=<value> Use the gcc toolchain at the given directory
  -gcodeview              Generate CodeView debug information
  -gdwarf-2               Generate source-level debug information with dwarf version 2
  -gdwarf-3               Generate source-level debug information with dwarf version 3
  -gdwarf-4               Generate source-level debug information with dwarf version 4
  -gdwarf-5               Generate source-level debug information with dwarf version 5
  -gline-tables-only      Emit debug line number tables only
  -gmodules               Generate debug info with external references to clang modules or precompiled headers
  -gz=<value>             DWARF debug sections compression type
  -gz                     DWARF debug sections compression type
  -G <size>               Put objects of at most <size> bytes into small data section (MIPS / Hexagon)
  -g                      Generate source-level debug information
  -help                   Display available options
  -H                      Show header includes and nesting depth
  -I-                     Restrict all prior -I flags to double-quoted inclusion and remove current directory from include path
  -idirafter <value>      Add directory to AFTER include search path
  -iframeworkwithsysroot <directory>
                          Add directory to SYSTEM framework search path, absolute paths are relative to -isysroot
  -iframework <value>     Add directory to SYSTEM framework search path
  -imacros <file>         Include macros from file before parsing
  -include-pch <file>     Include precompiled header file
  -include <file>         Include file before parsing
  -index-header-map       Make the next included directory (-I or -F) an indexer header map
  -iprefix <dir>          Set the -iwithprefix/-iwithprefixbefore prefix
  -iquote <directory>     Add directory to QUOTE include search path
  -isysroot <dir>         Set the system root directory (usually /)
  -isystem-after <directory>
                          Add directory to end of the SYSTEM include search path
  -isystem <directory>    Add directory to SYSTEM include search path
  -ivfsoverlay <value>    Overlay the virtual filesystem described by file over the real file system
  -iwithprefixbefore <dir>
                          Set directory to include search path with prefix
  -iwithprefix <dir>      Set directory to SYSTEM include search path with prefix
  -iwithsysroot <directory>
                          Add directory to SYSTEM include search path, absolute paths are relative to -isysroot
  -I <dir>                Add directory to include search path
  -L <dir>                Add directory to library search path
  -mabicalls              Enable SVR4-style position-independent code (Mips only)
  -malign-double          Align doubles to two words in structs (x86 only)
  -mbackchain             Link stack frames through backchain on System Z
  -mcrc                   Allow use of CRC instructions (ARM only)
  -MD                     Write a depfile containing user and system headers
  -meabi <value>          Set EABI type, e.g. 4, 5 or gnu (default depends on triple)
  -membedded-data         Place constants in the .rodata section instead of the .sdata section even if they meet the -G <size> threshold (MIPS)
  -mexecute-only          Disallow generation of data access to code sections (ARM only)
  -mextern-sdata          Assume that externally defined data is in the small data if it meets the -G <size> threshold (MIPS)
  -mfentry                Insert calls to fentry at function entry (x86 only)
  -mfix-cortex-a53-835769 Workaround Cortex-A53 erratum 835769 (AArch64 only)
  -mfp32                  Use 32-bit floating point registers (MIPS only)
  -mfp64                  Use 64-bit floating point registers (MIPS only)
  -MF <file>              Write depfile output from -MMD, -MD, -MM, or -M to <file>
  -mgeneral-regs-only     Generate code which only uses the general purpose registers (AArch64 only)
  -mglobal-merge          Enable merging of globals
  -mgpopt                 Use GP relative accesses for symbols known to be in a small data section (MIPS)
  -MG                     Add missing headers to depfile
  -mhvx-double            Enable Hexagon Double Vector eXtensions
  -mhvx-length=<value>    Set Hexagon Vector Length
  -mhvx=<value>           Enable Hexagon Vector eXtensions
  -mhvx                   Enable Hexagon Vector eXtensions
  -miamcu                 Use Intel MCU ABI
  --migrate               Run the migrator
  -mincremental-linker-compatible
                          (integrated-as) Emit an object file which can be used with an incremental linker
  -mios-version-min=<value>
                          Set iOS deployment target
  -MJ <value>             Write a compilation database entry per input
  -mllvm <value>          Additional arguments to forward to LLVM's option processing
  -mlocal-sdata           Extend the -G behaviour to object local data (MIPS)
  -mlong-calls            Generate branches with extended addressability, usually via indirect jumps.
  -mmacosx-version-min=<value>
                          Set Mac OS X deployment target
  -mmadd4                 Enable the generation of 4-operand madd.s, madd.d and related instructions.
  -MMD                    Write a depfile containing user headers
  -mms-bitfields          Set the default structure layout to be compatible with the Microsoft compiler standard
  -mmsa                   Enable MSA ASE (MIPS only)
  -mmt                    Enable MT ASE (MIPS only)
  -MM                     Like -MMD, but also implies -E and writes to stdout by default
  -mno-abicalls           Disable SVR4-style position-independent code (Mips only)
  -mno-embedded-data      Do not place constants in the .rodata section instead of the .sdata if they meet the -G <size> threshold (MIPS)
  -mno-execute-only       Allow generation of data access to code sections (ARM only)
  -mno-extern-sdata       Do not assume that externally defined data is in the small data if it meets the -G <size> threshold (MIPS)
  -mno-fix-cortex-a53-835769
                          Don't workaround Cortex-A53 erratum 835769 (AArch64 only)
  -mno-global-merge       Disable merging of globals
  -mno-gpopt              Do not use GP relative accesses for symbols known to be in a small data section (MIPS)
  -mno-hvx-double         Disable Hexagon Double Vector eXtensions
  -mno-hvx                Disable Hexagon Vector eXtensions
  -mno-implicit-float     Don't generate implicit floating point instructions
  -mno-incremental-linker-compatible
                          (integrated-as) Emit an object file which cannot be used with an incremental linker
  -mno-local-sdata        Do not extend the -G behaviour to object local data (MIPS)
  -mno-long-calls         Restore the default behaviour of not generating long calls
  -mno-madd4              Disable the generation of 4-operand madd.s, madd.d and related instructions.
  -mno-movt               Disallow use of movt/movw pairs (ARM only)
  -mno-ms-bitfields       Do not set the default structure layout to be compatible with the Microsoft compiler standard
  -mno-msa                Disable MSA ASE (MIPS only)
  -mno-mt                 Disable MT ASE (MIPS only)
  -mno-neg-immediates     Disallow converting instructions with negative immediates to their negation or inversion.
  -mno-restrict-it        Allow generation of deprecated IT blocks for ARMv8. It is off by default for ARMv8 Thumb mode
  -mno-unaligned-access   Force all memory accesses to be aligned (AArch32/AArch64 only)
  -mno-xnack              Disable XNACK (AMDGPU only)
  -mnocrc                 Disallow use of CRC instructions (ARM only)
  -module-dependency-dir <value>
                          Directory to dump module dependencies to
  -module-file-info       Provide information about a particular module file
  -momit-leaf-frame-pointer
                          Omit frame pointer setup for leaf functions
  -mpie-copy-relocations  Use copy relocations support for PIE builds
  -mprefer-vector-width=<value>
                          Specifies preferred vector width for auto-vectorization. Defaults to 'none' which allows target specific decisions.
  -MP                     Create phony target for each dependency (other than main file)
  -mqdsp6-compat          Enable hexagon-qdsp6 backward compatibility
  -MQ <value>             Specify name of main file output to quote in depfile
  -mrelax-all             (integrated-as) Relax all machine instructions
  -mrestrict-it           Disallow generation of deprecated IT blocks for ARMv8. It is on by default for ARMv8 Thumb mode.
  -mrtd                   Make StdCall calling convention the default
  -msoft-float            Use software floating point
  -mstack-alignment=<value>
                          Set the stack alignment
  -mstack-probe-size=<value>
                          Set the stack probe size
  -mstackrealign          Force realign the stack at entry to every function
  -mthread-model <value>  The thread model to use, e.g. posix, single (posix by default)
  -mtp=<value>            Read thread pointer from coprocessor register (ARM only)
  -MT <value>             Specify name of main file output in depfile
  -munaligned-access      Allow memory accesses to be unaligned (AArch32/AArch64 only)
  -MV                     Use NMake/Jom format for the depfile
  -mxnack                 Enable XNACK (AMDGPU only)
  -M                      Like -MD, but also implies -E and writes to stdout by default
  --no-cuda-gpu-arch=<value>
                          Remove GPU architecture (e.g. sm_35) from the list of GPUs to compile for. 'all' resets the list to its default value.
  --no-cuda-version-check Don't error out if the detected version of the CUDA install is too low for the requested CUDA gpu architecture.
  --no-system-header-prefix=<prefix>
                          Treat all #include paths starting with <prefix> as not including a system header.
  -nobuiltininc           Disable builtin #include directories
  -nostdinc++             Disable standard #include directories for the C++ standard library
  -ObjC++                 Treat source input files as Objective-C++ inputs
  -objcmt-atomic-property Make migration to 'atomic' properties
  -objcmt-migrate-all     Enable migration to modern ObjC
  -objcmt-migrate-annotation
                          Enable migration to property and method annotations
  -objcmt-migrate-designated-init
                          Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods
  -objcmt-migrate-instancetype
                          Enable migration to infer instancetype for method result type
  -objcmt-migrate-literals
                          Enable migration to modern ObjC literals
  -objcmt-migrate-ns-macros
                          Enable migration to NS_ENUM/NS_OPTIONS macros
  -objcmt-migrate-property-dot-syntax
                          Enable migration of setter/getter messages to property-dot syntax
  -objcmt-migrate-property
                          Enable migration to modern ObjC property
  -objcmt-migrate-protocol-conformance
                          Enable migration to add protocol conformance on classes
  -objcmt-migrate-readonly-property
                          Enable migration to modern ObjC readonly property
  -objcmt-migrate-readwrite-property
                          Enable migration to modern ObjC readwrite property
  -objcmt-migrate-subscripting
                          Enable migration to modern ObjC subscripting
  -objcmt-ns-nonatomic-iosonly
                          Enable migration to use NS_NONATOMIC_IOSONLY macro for setting property's 'atomic' attribute
  -objcmt-returns-innerpointer-property
                          Enable migration to annotate property with NS_RETURNS_INNER_POINTER
  -objcmt-whitelist-dir-path=<value>
                          Only modify files with a filename contained in the provided directory path
  -ObjC                   Treat source input files as Objective-C inputs
  -o <file>               Write output to <file>
  -pg                     Enable mcount instrumentation
  -pipe                   Use pipes between commands, when possible
  --precompile            Only precompile the input
  -print-file-name=<file> Print the full library path of <file>
  -print-ivar-layout      Enable Objective-C Ivar layout bitmap print trace
  -print-libgcc-file-name Print the library path for the currently used compiler runtime library ("libgcc.a" or "libclang_rt.builtins.*.a")
  -print-prog-name=<name> Print the full program path of <name>
  -print-resource-dir     Print the resource directory pathname
  -print-search-dirs      Print the paths used for finding libraries and programs
  -pthread                Support POSIX threads in generated code
  --ptxas-path=<value>    Path to ptxas (used for compiling CUDA code)
  -P                      Disable linemarker output in -E mode
  -Qunused-arguments      Don't emit warning for unused driver arguments
  -relocatable-pch        Whether to build a relocatable precompiled header
  -rewrite-legacy-objc    Rewrite Legacy Objective-C source to C++
  -rewrite-objc           Rewrite Objective-C source to C++
  -Rpass-analysis=<value> Report transformation analysis from optimization passes whose name matches the given POSIX regular expression
  -Rpass-missed=<value>   Report missed transformations by optimization passes whose name matches the given POSIX regular expression
  -Rpass=<value>          Report transformations performed by optimization passes whose name matches the given POSIX regular expression
  -rtlib=<value>          Compiler runtime library to use
  -R<remark>              Enable the specified remark
  -save-stats=<value>     Save llvm statistics.
  -save-stats             Save llvm statistics.
  -save-temps=<value>     Save intermediate compilation results.
  -save-temps             Save intermediate compilation results
  -serialize-diagnostics <value>
                          Serialize compiler diagnostics to a file
  -std=<value>            Language standard to compile for
  -stdlib=<value>         C++ standard library to use
  --system-header-prefix=<prefix>
                          Treat all #include paths starting with <prefix> as including a system header.
  -S                      Only run preprocess and compilation steps
  --target=<value>        Generate code for the given target
  -Tbss <addr>            Set starting address of BSS to <addr>
  -Tdata <addr>           Set starting address of BSS to <addr>
  -time                   Time individual commands
  -traditional-cpp        Enable some traditional CPP emulation
  -trigraphs              Process trigraph sequences
  -Ttext <addr>           Set starting address of BSS to <addr>
  -T <script>             Specify <script> as linker script
  -undef                  undef all system defines
  -U <macro>              Undefine macro <macro>
  --verify-debug-info     Verify the binary representation of debug output
  -verify-pch             Load and verify that a pre-compiled header file is not stale
  --version               Print version information
  -v                      Show commands to run and use verbose output
  -Wa,<arg>               Pass the comma separated arguments in <arg> to the assembler
  -Wdeprecated            Enable warnings for deprecated constructs and define __DEPRECATED
  -Wl,<arg>               Pass the comma separated arguments in <arg> to the linker
  -working-directory <value>
                          Resolve file paths relative to the specified directory
  -Wp,<arg>               Pass the comma separated arguments in <arg> to the preprocessor
  -W<warning>             Enable the specified warning
  -w                      Suppress all warnings
  -Xanalyzer <arg>        Pass <arg> to the static analyzer
  -Xassembler <arg>       Pass <arg> to the assembler
  -Xclang <arg>           Pass <arg> to the clang compiler
  -Xcuda-fatbinary <arg>  Pass <arg> to fatbinary invocation
  -Xcuda-ptxas <arg>      Pass <arg> to the ptxas assembler
  -Xlinker <arg>          Pass <arg> to the linker
  -Xopenmp-target=<arg>   Pass <arg> to the specified target offloading toolchain. The triple that identifies the toolchain must be provided after the equals sign.
  -Xopenmp-target <arg>   Pass <arg> to the target offloading toolchain.
  -Xpreprocessor <arg>    Pass <arg> to the preprocessor
  -x <language>           Treat subsequent input files as having type <language>
  -z <arg>                Pass -z <arg> to the linker
Checking for program /usr/local/bin/mpicc...found
                Defined make macro "CC" to "mpicc"
                Pushing language C
                            All intermediate test results are stored in /tmp/petsc-mjVUVK
                          All intermediate test results are stored in /tmp/petsc-mjVUVK/config.setCompilers
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                        Pushing language C
                        Popping language C
Executing: mpicc  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
ld: library not found for -lpetsc-ufod4vtr9mqHvKIQiVAm
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                            Pushing language C
                            Popping language C
Executing: mpicc  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-mjVUVK/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
                Popping language C
================================================================================
TEST checkCPreprocessor from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:599)
TESTING: checkCPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:599)
  Locate a functional C preprocessor
Checking for program /usr/local/bin/mpicc...found
                Defined make macro "CPP" to "mpicc -E"
              Pushing language C
Executing: mpicc -E  -I/tmp/petsc-mjVUVK/config.setCompilers  /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
stdout:
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.c"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 360 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.c" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/confdefs.h" 1
# 2 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.c" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conffix.h" 1
# 3 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.c" 2
# 1 "/usr/include/stdlib.h" 1 3 4
# 61 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/Availability.h" 1 3 4
# 190 "/usr/include/Availability.h" 3 4
# 1 "/usr/include/AvailabilityInternal.h" 1 3 4
# 191 "/usr/include/Availability.h" 2 3 4
# 62 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/_types.h" 1 3 4
# 27 "/usr/include/_types.h" 3 4
# 1 "/usr/include/sys/_types.h" 1 3 4
# 32 "/usr/include/sys/_types.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 587 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/sys/_symbol_aliasing.h" 1 3 4
# 588 "/usr/include/sys/cdefs.h" 2 3 4
# 653 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/sys/_posix_availability.h" 1 3 4
# 654 "/usr/include/sys/cdefs.h" 2 3 4
# 33 "/usr/include/sys/_types.h" 2 3 4
# 1 "/usr/include/machine/_types.h" 1 3 4
# 32 "/usr/include/machine/_types.h" 3 4
# 1 "/usr/include/i386/_types.h" 1 3 4
# 37 "/usr/include/i386/_types.h" 3 4
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef short __int16_t;
typedef unsigned short __uint16_t;
typedef int __int32_t;
typedef unsigned int __uint32_t;
typedef long long __int64_t;
typedef unsigned long long __uint64_t;
typedef long __darwin_intptr_t;
typedef unsigned int __darwin_natural_t;
# 70 "/usr/include/i386/_types.h" 3 4
typedef int __darwin_ct_rune_t;
typedef union {
 char __mbstate8[128];
 long long _mbstateL;
} __mbstate_t;
typedef __mbstate_t __darwin_mbstate_t;
typedef long int __darwin_ptrdiff_t;
typedef long unsigned int __darwin_size_t;
typedef __builtin_va_list __darwin_va_list;
typedef int __darwin_wchar_t;
typedef __darwin_wchar_t __darwin_rune_t;
typedef int __darwin_wint_t;
typedef unsigned long __darwin_clock_t;
typedef __uint32_t __darwin_socklen_t;
typedef long __darwin_ssize_t;
typedef long __darwin_time_t;
# 33 "/usr/include/machine/_types.h" 2 3 4
# 34 "/usr/include/sys/_types.h" 2 3 4
# 55 "/usr/include/sys/_types.h" 3 4
typedef __int64_t __darwin_blkcnt_t;
typedef __int32_t __darwin_blksize_t;
typedef __int32_t __darwin_dev_t;
typedef unsigned int __darwin_fsblkcnt_t;
typedef unsigned int __darwin_fsfilcnt_t;
typedef __uint32_t __darwin_gid_t;
typedef __uint32_t __darwin_id_t;
typedef __uint64_t __darwin_ino64_t;
typedef __darwin_ino64_t __darwin_ino_t;
typedef __darwin_natural_t __darwin_mach_port_name_t;
typedef __darwin_mach_port_name_t __darwin_mach_port_t;
typedef __uint16_t __darwin_mode_t;
typedef __int64_t __darwin_off_t;
typedef __int32_t __darwin_pid_t;
typedef __uint32_t __darwin_sigset_t;
typedef __int32_t __darwin_suseconds_t;
typedef __uint32_t __darwin_uid_t;
typedef __uint32_t __darwin_useconds_t;
typedef unsigned char __darwin_uuid_t[16];
typedef char __darwin_uuid_string_t[37];
# 1 "/usr/include/sys/_pthread/_pthread_types.h" 1 3 4
# 57 "/usr/include/sys/_pthread/_pthread_types.h" 3 4
struct __darwin_pthread_handler_rec {
 void (*__routine)(void *);
 void *__arg;
 struct __darwin_pthread_handler_rec *__next;
};
struct _opaque_pthread_attr_t {
 long __sig;
 char __opaque[56];
};
struct _opaque_pthread_cond_t {
 long __sig;
 char __opaque[40];
};
struct _opaque_pthread_condattr_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_mutex_t {
 long __sig;
 char __opaque[56];
};
struct _opaque_pthread_mutexattr_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_once_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_rwlock_t {
 long __sig;
 char __opaque[192];
};
struct _opaque_pthread_rwlockattr_t {
 long __sig;
 char __opaque[16];
};
struct _opaque_pthread_t {
 long __sig;
 struct __darwin_pthread_handler_rec *__cleanup_stack;
 char __opaque[8176];
};
typedef struct _opaque_pthread_attr_t __darwin_pthread_attr_t;
typedef struct _opaque_pthread_cond_t __darwin_pthread_cond_t;
typedef struct _opaque_pthread_condattr_t __darwin_pthread_condattr_t;
typedef unsigned long __darwin_pthread_key_t;
typedef struct _opaque_pthread_mutex_t __darwin_pthread_mutex_t;
typedef struct _opaque_pthread_mutexattr_t __darwin_pthread_mutexattr_t;
typedef struct _opaque_pthread_once_t __darwin_pthread_once_t;
typedef struct _opaque_pthread_rwlock_t __darwin_pthread_rwlock_t;
typedef struct _opaque_pthread_rwlockattr_t __darwin_pthread_rwlockattr_t;
typedef struct _opaque_pthread_t *__darwin_pthread_t;
# 81 "/usr/include/sys/_types.h" 2 3 4
# 28 "/usr/include/_types.h" 2 3 4
# 39 "/usr/include/_types.h" 3 4
typedef int __darwin_nl_item;
typedef int __darwin_wctrans_t;
typedef __uint32_t __darwin_wctype_t;
# 64 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/wait.h" 1 3 4
# 79 "/usr/include/sys/wait.h" 3 4
typedef enum {
 P_ALL,
 P_PID,
 P_PGID
} idtype_t;
# 1 "/usr/include/sys/_types/_pid_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_pid_t.h" 3 4
typedef __darwin_pid_t pid_t;
# 90 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/sys/_types/_id_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_id_t.h" 3 4
typedef __darwin_id_t id_t;
# 91 "/usr/include/sys/wait.h" 2 3 4
# 109 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/sys/signal.h" 1 3 4
# 73 "/usr/include/sys/signal.h" 3 4
# 1 "/usr/include/sys/appleapiopts.h" 1 3 4
# 74 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/machine/signal.h" 1 3 4
# 32 "/usr/include/machine/signal.h" 3 4
# 1 "/usr/include/i386/signal.h" 1 3 4
# 39 "/usr/include/i386/signal.h" 3 4
typedef int sig_atomic_t;
# 33 "/usr/include/machine/signal.h" 2 3 4
# 83 "/usr/include/sys/signal.h" 2 3 4
# 146 "/usr/include/sys/signal.h" 3 4
# 1 "/usr/include/machine/_mcontext.h" 1 3 4
# 29 "/usr/include/machine/_mcontext.h" 3 4
# 1 "/usr/include/i386/_mcontext.h" 1 3 4
# 33 "/usr/include/i386/_mcontext.h" 3 4
# 1 "/usr/include/mach/i386/_structs.h" 1 3 4
# 43 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_thread_state
{
    unsigned int __eax;
    unsigned int __ebx;
    unsigned int __ecx;
    unsigned int __edx;
    unsigned int __edi;
    unsigned int __esi;
    unsigned int __ebp;
    unsigned int __esp;
    unsigned int __ss;
    unsigned int __eflags;
    unsigned int __eip;
    unsigned int __cs;
    unsigned int __ds;
    unsigned int __es;
    unsigned int __fs;
    unsigned int __gs;
};
# 89 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_fp_control
{
    unsigned short __invalid :1,
        __denorm :1,
    __zdiv :1,
    __ovrfl :1,
    __undfl :1,
    __precis :1,
      :2,
    __pc :2,
    __rc :2,
             :1,
      :3;
};
typedef struct __darwin_fp_control __darwin_fp_control_t;
# 147 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_fp_status
{
    unsigned short __invalid :1,
        __denorm :1,
    __zdiv :1,
    __ovrfl :1,
    __undfl :1,
    __precis :1,
    __stkflt :1,
    __errsumm :1,
    __c0 :1,
    __c1 :1,
    __c2 :1,
    __tos :3,
    __c3 :1,
    __busy :1;
};
typedef struct __darwin_fp_status __darwin_fp_status_t;
# 191 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_mmst_reg
{
 char __mmst_reg[10];
 char __mmst_rsrv[6];
};
# 210 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_xmm_reg
{
 char __xmm_reg[16];
};
# 232 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_float_state
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 char __fpu_rsrv4[14*16];
 int __fpu_reserved1;
};
struct __darwin_i386_avx_state
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 char __fpu_rsrv4[14*16];
 int __fpu_reserved1;
 char __avx_reserved1[64];
 struct __darwin_xmm_reg __fpu_ymmh0;
 struct __darwin_xmm_reg __fpu_ymmh1;
 struct __darwin_xmm_reg __fpu_ymmh2;
 struct __darwin_xmm_reg __fpu_ymmh3;
 struct __darwin_xmm_reg __fpu_ymmh4;
 struct __darwin_xmm_reg __fpu_ymmh5;
 struct __darwin_xmm_reg __fpu_ymmh6;
 struct __darwin_xmm_reg __fpu_ymmh7;
};
# 402 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_exception_state
{
 __uint16_t __trapno;
 __uint16_t __cpu;
 __uint32_t __err;
 __uint32_t __faultvaddr;
};
# 422 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_debug_state32
{
 unsigned int __dr0;
 unsigned int __dr1;
 unsigned int __dr2;
 unsigned int __dr3;
 unsigned int __dr4;
 unsigned int __dr5;
 unsigned int __dr6;
 unsigned int __dr7;
};
# 454 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_thread_state64
{
 __uint64_t __rax;
 __uint64_t __rbx;
 __uint64_t __rcx;
 __uint64_t __rdx;
 __uint64_t __rdi;
 __uint64_t __rsi;
 __uint64_t __rbp;
 __uint64_t __rsp;
 __uint64_t __r8;
 __uint64_t __r9;
 __uint64_t __r10;
 __uint64_t __r11;
 __uint64_t __r12;
 __uint64_t __r13;
 __uint64_t __r14;
 __uint64_t __r15;
 __uint64_t __rip;
 __uint64_t __rflags;
 __uint64_t __cs;
 __uint64_t __fs;
 __uint64_t __gs;
};
# 509 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_float_state64
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 struct __darwin_xmm_reg __fpu_xmm8;
 struct __darwin_xmm_reg __fpu_xmm9;
 struct __darwin_xmm_reg __fpu_xmm10;
 struct __darwin_xmm_reg __fpu_xmm11;
 struct __darwin_xmm_reg __fpu_xmm12;
 struct __darwin_xmm_reg __fpu_xmm13;
 struct __darwin_xmm_reg __fpu_xmm14;
 struct __darwin_xmm_reg __fpu_xmm15;
 char __fpu_rsrv4[6*16];
 int __fpu_reserved1;
};
struct __darwin_x86_avx_state64
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 struct __darwin_xmm_reg __fpu_xmm8;
 struct __darwin_xmm_reg __fpu_xmm9;
 struct __darwin_xmm_reg __fpu_xmm10;
 struct __darwin_xmm_reg __fpu_xmm11;
 struct __darwin_xmm_reg __fpu_xmm12;
 struct __darwin_xmm_reg __fpu_xmm13;
 struct __darwin_xmm_reg __fpu_xmm14;
 struct __darwin_xmm_reg __fpu_xmm15;
 char __fpu_rsrv4[6*16];
 int __fpu_reserved1;
 char __avx_reserved1[64];
 struct __darwin_xmm_reg __fpu_ymmh0;
 struct __darwin_xmm_reg __fpu_ymmh1;
 struct __darwin_xmm_reg __fpu_ymmh2;
 struct __darwin_xmm_reg __fpu_ymmh3;
 struct __darwin_xmm_reg __fpu_ymmh4;
 struct __darwin_xmm_reg __fpu_ymmh5;
 struct __darwin_xmm_reg __fpu_ymmh6;
 struct __darwin_xmm_reg __fpu_ymmh7;
 struct __darwin_xmm_reg __fpu_ymmh8;
 struct __darwin_xmm_reg __fpu_ymmh9;
 struct __darwin_xmm_reg __fpu_ymmh10;
 struct __darwin_xmm_reg __fpu_ymmh11;
 struct __darwin_xmm_reg __fpu_ymmh12;
 struct __darwin_xmm_reg __fpu_ymmh13;
 struct __darwin_xmm_reg __fpu_ymmh14;
 struct __darwin_xmm_reg __fpu_ymmh15;
};
# 751 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_exception_state64
{
    __uint16_t __trapno;
    __uint16_t __cpu;
    __uint32_t __err;
    __uint64_t __faultvaddr;
};
# 771 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_debug_state64
{
 __uint64_t __dr0;
 __uint64_t __dr1;
 __uint64_t __dr2;
 __uint64_t __dr3;
 __uint64_t __dr4;
 __uint64_t __dr5;
 __uint64_t __dr6;
 __uint64_t __dr7;
};
# 34 "/usr/include/i386/_mcontext.h" 2 3 4
struct __darwin_mcontext32
{
 struct __darwin_i386_exception_state __es;
 struct __darwin_i386_thread_state __ss;
 struct __darwin_i386_float_state __fs;
};
struct __darwin_mcontext_avx32
{
 struct __darwin_i386_exception_state __es;
 struct __darwin_i386_thread_state __ss;
 struct __darwin_i386_avx_state __fs;
};
# 76 "/usr/include/i386/_mcontext.h" 3 4
struct __darwin_mcontext64
{
 struct __darwin_x86_exception_state64 __es;
 struct __darwin_x86_thread_state64 __ss;
 struct __darwin_x86_float_state64 __fs;
};
struct __darwin_mcontext_avx64
{
 struct __darwin_x86_exception_state64 __es;
 struct __darwin_x86_thread_state64 __ss;
 struct __darwin_x86_avx_state64 __fs;
};
# 115 "/usr/include/i386/_mcontext.h" 3 4
typedef struct __darwin_mcontext64 *mcontext_t;
# 30 "/usr/include/machine/_mcontext.h" 2 3 4
# 147 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_pthread/_pthread_attr_t.h" 1 3 4
# 30 "/usr/include/sys/_pthread/_pthread_attr_t.h" 3 4
typedef __darwin_pthread_attr_t pthread_attr_t;
# 149 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_sigaltstack.h" 1 3 4
# 36 "/usr/include/sys/_types/_sigaltstack.h" 3 4
struct __darwin_sigaltstack
{
 void *ss_sp;
 __darwin_size_t ss_size;
 int ss_flags;
};
typedef struct __darwin_sigaltstack stack_t;
# 151 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_ucontext.h" 1 3 4
# 34 "/usr/include/sys/_types/_ucontext.h" 3 4
struct __darwin_ucontext
{
 int uc_onstack;
 __darwin_sigset_t uc_sigmask;
 struct __darwin_sigaltstack uc_stack;
 struct __darwin_ucontext *uc_link;
 __darwin_size_t uc_mcsize;
 struct __darwin_mcontext64 *uc_mcontext;
};
typedef struct __darwin_ucontext ucontext_t;
# 152 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_sigset_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_sigset_t.h" 3 4
typedef __darwin_sigset_t sigset_t;
# 155 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_size_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_size_t.h" 3 4
typedef __darwin_size_t size_t;
# 156 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_uid_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_uid_t.h" 3 4
typedef __darwin_uid_t uid_t;
# 157 "/usr/include/sys/signal.h" 2 3 4
union sigval {
 int sival_int;
 void *sival_ptr;
};
struct sigevent {
 int sigev_notify;
 int sigev_signo;
 union sigval sigev_value;
 void (*sigev_notify_function)(union sigval);
 pthread_attr_t *sigev_notify_attributes;
};
typedef struct __siginfo {
 int si_signo;
 int si_errno;
 int si_code;
 pid_t si_pid;
 uid_t si_uid;
 int si_status;
 void *si_addr;
 union sigval si_value;
 long si_band;
 unsigned long __pad[7];
} siginfo_t;
# 269 "/usr/include/sys/signal.h" 3 4
union __sigaction_u {
 void (*__sa_handler)(int);
 void (*__sa_sigaction)(int, struct __siginfo *,
         void *);
};
struct __sigaction {
 union __sigaction_u __sigaction_u;
 void (*sa_tramp)(void *, int, int, siginfo_t *, void *);
 sigset_t sa_mask;
 int sa_flags;
};
struct sigaction {
 union __sigaction_u __sigaction_u;
 sigset_t sa_mask;
 int sa_flags;
};
# 331 "/usr/include/sys/signal.h" 3 4
typedef void (*sig_t)(int);
# 348 "/usr/include/sys/signal.h" 3 4
struct sigvec {
 void (*sv_handler)(int);
 int sv_mask;
 int sv_flags;
};
# 367 "/usr/include/sys/signal.h" 3 4
struct sigstack {
 char *ss_sp;
 int ss_onstack;
};
# 390 "/usr/include/sys/signal.h" 3 4
void (*signal(int, void (*)(int)))(int);
# 110 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/sys/resource.h" 1 3 4
# 72 "/usr/include/sys/resource.h" 3 4
# 1 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 1 3 4
# 63 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 18 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/sys/_types/_int8_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int8_t.h" 3 4
typedef signed char int8_t;
# 19 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int16_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int16_t.h" 3 4
typedef short int16_t;
# 20 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int32_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int32_t.h" 3 4
typedef int int32_t;
# 21 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int64_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int64_t.h" 3 4
typedef long long int64_t;
# 22 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint8_t.h" 1 3 4
# 31 "/usr/include/_types/_uint8_t.h" 3 4
typedef unsigned char uint8_t;
# 24 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint16_t.h" 1 3 4
# 31 "/usr/include/_types/_uint16_t.h" 3 4
typedef unsigned short uint16_t;
# 25 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint32_t.h" 1 3 4
# 31 "/usr/include/_types/_uint32_t.h" 3 4
typedef unsigned int uint32_t;
# 26 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint64_t.h" 1 3 4
# 31 "/usr/include/_types/_uint64_t.h" 3 4
typedef unsigned long long uint64_t;
# 27 "/usr/include/stdint.h" 2 3 4
typedef int8_t int_least8_t;
typedef int16_t int_least16_t;
typedef int32_t int_least32_t;
typedef int64_t int_least64_t;
typedef uint8_t uint_least8_t;
typedef uint16_t uint_least16_t;
typedef uint32_t uint_least32_t;
typedef uint64_t uint_least64_t;
typedef int8_t int_fast8_t;
typedef int16_t int_fast16_t;
typedef int32_t int_fast32_t;
typedef int64_t int_fast64_t;
typedef uint8_t uint_fast8_t;
typedef uint16_t uint_fast16_t;
typedef uint32_t uint_fast32_t;
typedef uint64_t uint_fast64_t;
# 1 "/usr/include/sys/_types/_intptr_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_intptr_t.h" 3 4
typedef __darwin_intptr_t intptr_t;
# 54 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_uintptr_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_uintptr_t.h" 3 4
typedef unsigned long uintptr_t;
# 55 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_intmax_t.h" 1 3 4
# 32 "/usr/include/_types/_intmax_t.h" 3 4
typedef long int intmax_t;
# 59 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uintmax_t.h" 1 3 4
# 32 "/usr/include/_types/_uintmax_t.h" 3 4
typedef long unsigned int uintmax_t;
# 60 "/usr/include/stdint.h" 2 3 4
# 64 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 2 3 4
# 73 "/usr/include/sys/resource.h" 2 3 4
# 1 "/usr/include/sys/_types/_timeval.h" 1 3 4
# 30 "/usr/include/sys/_types/_timeval.h" 3 4
struct timeval
{
 __darwin_time_t tv_sec;
 __darwin_suseconds_t tv_usec;
};
# 81 "/usr/include/sys/resource.h" 2 3 4
typedef __uint64_t rlim_t;
# 152 "/usr/include/sys/resource.h" 3 4
struct rusage {
 struct timeval ru_utime;
 struct timeval ru_stime;
# 163 "/usr/include/sys/resource.h" 3 4
 long ru_maxrss;
 long ru_ixrss;
 long ru_idrss;
 long ru_isrss;
 long ru_minflt;
 long ru_majflt;
 long ru_nswap;
 long ru_inblock;
 long ru_oublock;
 long ru_msgsnd;
 long ru_msgrcv;
 long ru_nsignals;
 long ru_nvcsw;
 long ru_nivcsw;
};
# 192 "/usr/include/sys/resource.h" 3 4
typedef void *rusage_info_t;
struct rusage_info_v0 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
};
struct rusage_info_v1 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
};
struct rusage_info_v2 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
 uint64_t ri_diskio_bytesread;
 uint64_t ri_diskio_byteswritten;
};
struct rusage_info_v3 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
 uint64_t ri_diskio_bytesread;
 uint64_t ri_diskio_byteswritten;
 uint64_t ri_cpu_time_qos_default;
 uint64_t ri_cpu_time_qos_maintenance;
 uint64_t ri_cpu_time_qos_background;
 uint64_t ri_cpu_time_qos_utility;
 uint64_t ri_cpu_time_qos_legacy;
 uint64_t ri_cpu_time_qos_user_initiated;
 uint64_t ri_cpu_time_qos_user_interactive;
 uint64_t ri_billed_system_time;
 uint64_t ri_serviced_system_time;
};
typedef struct rusage_info_v3 rusage_info_current;
# 325 "/usr/include/sys/resource.h" 3 4
struct rlimit {
 rlim_t rlim_cur;
 rlim_t rlim_max;
};
# 353 "/usr/include/sys/resource.h" 3 4
struct proc_rlimit_control_wakeupmon {
 uint32_t wm_flags;
 int32_t wm_rate;
};
# 385 "/usr/include/sys/resource.h" 3 4
int getpriority(int, id_t);
int getiopolicy_np(int, int) __attribute__((availability(macosx,introduced=10.5)));
int getrlimit(int, struct rlimit *) __asm("_" "getrlimit" );
int getrusage(int, struct rusage *);
int setpriority(int, id_t, int);
int setiopolicy_np(int, int, int) __attribute__((availability(macosx,introduced=10.5)));
int setrlimit(int, const struct rlimit *) __asm("_" "setrlimit" );
# 111 "/usr/include/sys/wait.h" 2 3 4
# 186 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/machine/endian.h" 1 3 4
# 35 "/usr/include/machine/endian.h" 3 4
# 1 "/usr/include/i386/endian.h" 1 3 4
# 99 "/usr/include/i386/endian.h" 3 4
# 1 "/usr/include/sys/_endian.h" 1 3 4
# 130 "/usr/include/sys/_endian.h" 3 4
# 1 "/usr/include/libkern/_OSByteOrder.h" 1 3 4
# 66 "/usr/include/libkern/_OSByteOrder.h" 3 4
# 1 "/usr/include/libkern/i386/_OSByteOrder.h" 1 3 4
# 44 "/usr/include/libkern/i386/_OSByteOrder.h" 3 4
static inline
__uint16_t
_OSSwapInt16(
    __uint16_t _data
)
{
    return ((__uint16_t)((_data << 8) | (_data >> 8)));
}
static inline
__uint32_t
_OSSwapInt32(
    __uint32_t _data
)
{
    return __builtin_bswap32(_data);
}
static inline
__uint64_t
_OSSwapInt64(
    __uint64_t _data
)
{
    return __builtin_bswap64(_data);
}
# 67 "/usr/include/libkern/_OSByteOrder.h" 2 3 4
# 131 "/usr/include/sys/_endian.h" 2 3 4
# 100 "/usr/include/i386/endian.h" 2 3 4
# 36 "/usr/include/machine/endian.h" 2 3 4
# 187 "/usr/include/sys/wait.h" 2 3 4
union wait {
 int w_status;
 struct {
  unsigned int w_Termsig:7,
    w_Coredump:1,
    w_Retcode:8,
    w_Filler:16;
 } w_T;
 struct {
  unsigned int w_Stopval:8,
    w_Stopsig:8,
    w_Filler:16;
 } w_S;
};
# 248 "/usr/include/sys/wait.h" 3 4
pid_t wait(int *) __asm("_" "wait" );
pid_t waitpid(pid_t, int *, int) __asm("_" "waitpid" );
int waitid(idtype_t, id_t, siginfo_t *, int) __asm("_" "waitid" );
pid_t wait3(int *, int, struct rusage *);
pid_t wait4(pid_t, int *, int, struct rusage *);
# 66 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/alloca.h" 1 3 4
# 32 "/usr/include/alloca.h" 3 4
void *alloca(size_t);
# 68 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_ct_rune_t.h" 1 3 4
# 31 "/usr/include/sys/_types/_ct_rune_t.h" 3 4
typedef __darwin_ct_rune_t ct_rune_t;
# 77 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_rune_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_rune_t.h" 3 4
typedef __darwin_rune_t rune_t;
# 78 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_wchar_t.h" 1 3 4
# 33 "/usr/include/sys/_types/_wchar_t.h" 3 4
typedef __darwin_wchar_t wchar_t;
# 81 "/usr/include/stdlib.h" 2 3 4
typedef struct {
 int quot;
 int rem;
} div_t;
typedef struct {
 long quot;
 long rem;
} ldiv_t;
typedef struct {
 long long quot;
 long long rem;
} lldiv_t;
# 1 "/usr/include/sys/_types/_null.h" 1 3 4
# 100 "/usr/include/stdlib.h" 2 3 4
# 117 "/usr/include/stdlib.h" 3 4
extern int __mb_cur_max;
# 128 "/usr/include/stdlib.h" 3 4
void abort(void) __attribute__((noreturn));
int abs(int) __attribute__((const));
int atexit(void (* _Nonnull)(void));
double atof(const char *);
int atoi(const char *);
long atol(const char *);
long long
  atoll(const char *);
void *bsearch(const void *__key, const void *__base, size_t __nel,
     size_t __width, int (* _Nonnull __compar)(const void *, const void *));
void *calloc(size_t __count, size_t __size) __attribute__((__warn_unused_result__));
div_t div(int, int) __attribute__((const));
void exit(int) __attribute__((noreturn));
void free(void *);
char *getenv(const char *);
long labs(long) __attribute__((const));
ldiv_t ldiv(long, long) __attribute__((const));
long long
  llabs(long long);
lldiv_t lldiv(long long, long long);
void *malloc(size_t __size) __attribute__((__warn_unused_result__));
int mblen(const char *__s, size_t __n);
size_t mbstowcs(wchar_t * restrict , const char * restrict, size_t);
int mbtowc(wchar_t * restrict, const char * restrict, size_t);
int posix_memalign(void **__memptr, size_t __alignment, size_t __size) __attribute__((availability(macosx,introduced=10.6)));
void qsort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int rand(void) ;
void *realloc(void *__ptr, size_t __size) __attribute__((__warn_unused_result__));
void srand(unsigned) ;
double strtod(const char *, char **) __asm("_" "strtod" );
float strtof(const char *, char **) __asm("_" "strtof" );
long strtol(const char *__str, char **__endptr, int __base);
long double
  strtold(const char *, char **);
long long
  strtoll(const char *__str, char **__endptr, int __base);
unsigned long
  strtoul(const char *__str, char **__endptr, int __base);
unsigned long long
  strtoull(const char *__str, char **__endptr, int __base);
# 185 "/usr/include/stdlib.h" 3 4
__attribute__((availability(macosx,introduced=10.0)))
__attribute__((availability(watchos,unavailable))) __attribute__((availability(tvos,unavailable)))
int system(const char *) __asm("_" "system" );
size_t wcstombs(char * restrict, const wchar_t * restrict, size_t);
int wctomb(char *, wchar_t);
void _Exit(int) __attribute__((noreturn));
long a64l(const char *);
double drand48(void);
char *ecvt(double, int, int *restrict, int *restrict);
double erand48(unsigned short[3]);
char *fcvt(double, int, int *restrict, int *restrict);
char *gcvt(double, int, char *);
int getsubopt(char **, char * const *, char **);
int grantpt(int);
char *initstate(unsigned, char *, size_t);
long jrand48(unsigned short[3]) ;
char *l64a(long);
void lcong48(unsigned short[7]);
long lrand48(void) ;
char *mktemp(char *);
int mkstemp(char *);
long mrand48(void) ;
long nrand48(unsigned short[3]) ;
int posix_openpt(int);
char *ptsname(int);
int putenv(char *) __asm("_" "putenv" );
long random(void) ;
int rand_r(unsigned *) ;
char *realpath(const char * restrict, char * restrict) __asm("_" "realpath" "$DARWIN_EXTSN");
unsigned short
 *seed48(unsigned short[3]);
int setenv(const char * __name, const char * __value, int __overwrite) __asm("_" "setenv" );
void setkey(const char *) __asm("_" "setkey" );
char *setstate(const char *);
void srand48(long);
void srandom(unsigned);
int unlockpt(int);
int unsetenv(const char *) __asm("_" "unsetenv" );
# 1 "/usr/include/machine/types.h" 1 3 4
# 35 "/usr/include/machine/types.h" 3 4
# 1 "/usr/include/i386/types.h" 1 3 4
# 81 "/usr/include/i386/types.h" 3 4
# 1 "/usr/include/sys/_types/_u_int8_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int8_t.h" 3 4
typedef unsigned char u_int8_t;
# 82 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int16_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int16_t.h" 3 4
typedef unsigned short u_int16_t;
# 83 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int32_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int32_t.h" 3 4
typedef unsigned int u_int32_t;
# 84 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int64_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int64_t.h" 3 4
typedef unsigned long long u_int64_t;
# 85 "/usr/include/i386/types.h" 2 3 4
typedef int64_t register_t;
# 97 "/usr/include/i386/types.h" 3 4
typedef u_int64_t user_addr_t;
typedef u_int64_t user_size_t;
typedef int64_t user_ssize_t;
typedef int64_t user_long_t;
typedef u_int64_t user_ulong_t;
typedef int64_t user_time_t;
typedef int64_t user_off_t;
typedef u_int64_t syscall_arg_t;
# 36 "/usr/include/machine/types.h" 2 3 4
# 252 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_dev_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_dev_t.h" 3 4
typedef __darwin_dev_t dev_t;
# 254 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_mode_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_mode_t.h" 3 4
typedef __darwin_mode_t mode_t;
# 255 "/usr/include/stdlib.h" 2 3 4
uint32_t arc4random(void);
void arc4random_addrandom(unsigned char * , int )
    __attribute__((availability(macosx,introduced=10.0))) __attribute__((availability(macosx,deprecated=10.12,message="use arc4random_stir")))
    __attribute__((availability(ios,introduced=2.0))) __attribute__((availability(ios,deprecated=10.0,message="use arc4random_stir")))
    __attribute__((availability(tvos,introduced=2.0))) __attribute__((availability(tvos,deprecated=10.0,message="use arc4random_stir")))
    __attribute__((availability(watchos,introduced=1.0))) __attribute__((availability(watchos,deprecated=3.0,message="use arc4random_stir")));
void arc4random_buf(void * __buf, size_t __nbytes) __attribute__((availability(macosx,introduced=10.7)));
void arc4random_stir(void);
uint32_t
  arc4random_uniform(uint32_t __upper_bound) __attribute__((availability(macosx,introduced=10.7)));
int atexit_b(void (^ _Nonnull)(void)) __attribute__((availability(macosx,introduced=10.6)));
void *bsearch_b(const void *__key, const void *__base, size_t __nel,
     size_t __width, int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
char *cgetcap(char *, const char *, int);
int cgetclose(void);
int cgetent(char **, char **, const char *);
int cgetfirst(char **, char **);
int cgetmatch(const char *, const char *);
int cgetnext(char **, char **);
int cgetnum(char *, const char *, long *);
int cgetset(const char *);
int cgetstr(char *, const char *, char **);
int cgetustr(char *, const char *, char **);
int daemon(int, int) __asm("_" "daemon" "$1050") __attribute__((availability(macosx,introduced=10.0,deprecated=10.5,message="Use posix_spawn APIs instead."))) __attribute__((availability(watchos,unavailable))) __attribute__((availability(tvos,unavailable)));
char *devname(dev_t, mode_t);
char *devname_r(dev_t, mode_t, char *buf, int len);
char *getbsize(int *, long *);
int getloadavg(double [], int);
const char
 *getprogname(void);
int heapsort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int heapsort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
int mergesort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int mergesort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort_r(void *__base, size_t __nel, size_t __width, void *,
     int (* _Nonnull __compar)(void *, const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void qsort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void qsort_r(void *__base, size_t __nel, size_t __width, void *,
     int (* _Nonnull __compar)(void *, const void *, const void *));
int radixsort(const unsigned char **__base, int __nel, const unsigned char *__table,
     unsigned __endbyte);
void setprogname(const char *);
int sradixsort(const unsigned char **__base, int __nel, const unsigned char *__table,
     unsigned __endbyte);
void sranddev(void);
void srandomdev(void);
void *reallocf(void *__ptr, size_t __size);
long long
  strtoq(const char *__str, char **__endptr, int __base);
unsigned long long
  strtouq(const char *__str, char **__endptr, int __base);
extern char *suboptarg;
void *valloc(size_t);
# 4 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.c" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
              Popping language C
================================================================================
TEST checkCUDACompiler from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:640)
TESTING: checkCUDACompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:640)
  Locate a functional CUDA compiler
================================================================================
TEST checkCUDAPreprocessor from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:680)
TESTING: checkCUDAPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:680)
  Locate a functional CUDA preprocessor
================================================================================
TEST checkCxxCompiler from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:792)
TESTING: checkCxxCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:792)
  Locate a functional Cxx compiler
Executing: mpicxx --help
stdout:
OVERVIEW: clang LLVM compiler
USAGE: clang-6.0 [options] <inputs>
OPTIONS:
  -###                    Print (but do not run) the commands to run for this compilation
  --analyzer-output <value>
                          Static analyzer report output format (html|plist|plist-multi-file|plist-html|text).
  --analyze               Run the static analyzer
  -arcmt-migrate-emit-errors
                          Emit ARC errors even if the migrator can fix them
  -arcmt-migrate-report-output <value>
                          Output path for the plist report
  -B <dir>                Add <dir> to search path for binaries and object files used implicitly
  -CC                     Include comments from within macros in preprocessed output
  -cl-denorms-are-zero    OpenCL only. Allow denormals to be flushed to zero.
  -cl-fast-relaxed-math   OpenCL only. Sets -cl-finite-math-only and -cl-unsafe-math-optimizations, and defines __FAST_RELAXED_MATH__.
  -cl-finite-math-only    OpenCL only. Allow floating-point optimizations that assume arguments and results are not NaNs or +-Inf.
  -cl-fp32-correctly-rounded-divide-sqrt
                          OpenCL only. Specify that single precision floating-point divide and sqrt used in the program source are correctly rounded.
  -cl-kernel-arg-info     OpenCL only. Generate kernel argument metadata.
  -cl-mad-enable          OpenCL only. Allow use of less precise MAD computations in the generated binary.
  -cl-no-signed-zeros     OpenCL only. Allow use of less precise no signed zeros computations in the generated binary.
  -cl-opt-disable         OpenCL only. This option disables all optimizations. By default optimizations are enabled.
  -cl-single-precision-constant
                          OpenCL only. Treat double precision floating-point constant as single precision constant.
  -cl-std=<value>         OpenCL language standard to compile for.
  -cl-strict-aliasing     OpenCL only. This option is added for compatibility with OpenCL 1.0.
  -cl-unsafe-math-optimizations
                          OpenCL only. Allow unsafe floating-point optimizations.  Also implies -cl-no-signed-zeros and -cl-mad-enable.
  --config <value>        Specifies configuration file
  --cuda-compile-host-device
                          Compile CUDA code for both host and device (default).  Has no effect on non-CUDA compilations.
  --cuda-device-only      Compile CUDA code for device only
  --cuda-gpu-arch=<value> CUDA GPU architecture (e.g. sm_35).  May be specified more than once.
  --cuda-host-only        Compile CUDA code for host only.  Has no effect on non-CUDA compilations.
  --cuda-noopt-device-debug
                          Enable device-side debug info generation. Disables ptxas optimizations.
  --cuda-path=<value>     CUDA installation path
  -cxx-isystem <directory>
                          Add directory to the C++ SYSTEM include search path
  -C                      Include comments in preprocessed output
  -c                      Only run preprocess, compile, and assemble steps
  -dD                     Print macro definitions in -E mode in addition to normal output
  -dependency-dot <value> Filename to write DOT-formatted header dependencies to
  -dependency-file <value>
                          Filename (or -) to write dependency output to
  -dI                     Print include directives in -E mode in addition to normal output
  -dM                     Print macro definitions in -E mode instead of normal output
  -D <macro>=<value>      Define <macro> to <value> (or 1 if <value> omitted)
  -emit-ast               Emit Clang AST files for source inputs
  -emit-llvm              Use the LLVM representation for assembler and object files
  -E                      Only run the preprocessor
  -faligned-allocation    Enable C++17 aligned allocation functions
  -fallow-editor-placeholders
                          Treat editor placeholders as valid source code
  -fansi-escape-codes     Use ANSI escape codes for diagnostics
  -fapple-kext            Use Apple's kernel extensions ABI
  -fapple-pragma-pack     Enable Apple gcc-compatible #pragma pack handling
  -fapplication-extension Restrict code to those available for App Extensions
  -fblocks                Enable the 'blocks' language feature
  -fborland-extensions    Accept non-standard constructs supported by the Borland compiler
  -fbuild-session-file=<file>
                          Use the last modification time of <file> as the build session timestamp
  -fbuild-session-timestamp=<time since Epoch in seconds>
                          Time when the current build session started
  -fbuiltin-module-map    Load the clang builtins module map file.
  -fclang-abi-compat=<version>
                          Attempt to match the ABI of Clang <version>
  -fcolor-diagnostics     Use colors in diagnostics
  -fcomment-block-commands=<arg>
                          Treat each comma separated argument in <arg> as a documentation comment block command
  -fcoroutines-ts         Enable support for the C++ Coroutines TS
  -fcoverage-mapping      Generate coverage mapping to enable code coverage analysis
  -fcuda-approx-transcendentals
                          Use approximate transcendental functions
  -fcuda-flush-denormals-to-zero
                          Flush denormal floating point values to zero in CUDA device mode.
  -fcxx-exceptions        Enable C++ exceptions
  -fdata-sections         Place each data in its own section (ELF Only)
  -fdebug-info-for-profiling
                          Emit extra debug info to make sample profile more accurate.
  -fdebug-macro           Emit macro debug information
  -fdebug-prefix-map=<value>
                          remap file source paths in debug info
  -fdebug-types-section   Place debug types in their own section (ELF Only)
  -fdeclspec              Allow __declspec as a keyword
  -fdelayed-template-parsing
                          Parse templated function definitions at the end of the translation unit
  -fdiagnostics-absolute-paths
                          Print absolute paths in diagnostics
  -fdiagnostics-hotness-threshold=<number>
                          Prevent optimization remarks from being output if they do not have at least this profile count
  -fdiagnostics-parseable-fixits
                          Print fix-its in machine parseable form
  -fdiagnostics-print-source-range-info
                          Print source range spans in numeric form
  -fdiagnostics-show-hotness
                          Enable profile hotness information in diagnostic line
  -fdiagnostics-show-note-include-stack
                          Display include stacks for diagnostic notes
  -fdiagnostics-show-option
                          Print option name with mappable diagnostics
  -fdiagnostics-show-template-tree
                          Print a template comparison tree for differing templates
  -fdollars-in-identifiers
                          Allow '$' in identifiers
  -fdouble-square-bracket-attributes
                          Enable '[[]]' attributes in all C and C++ language modes
  -fdwarf-exceptions      Use DWARF style exceptions
  -fembed-bitcode-marker  Embed placeholder LLVM IR data as a marker
  -fembed-bitcode=<option>
                          Embed LLVM bitcode (option: off, all, bitcode, marker)
  -fembed-bitcode         Embed LLVM IR bitcode as data
  -femit-all-decls        Emit all declarations, even if unused
  -femulated-tls          Use emutls functions to access thread_local variables
  -fexceptions            Enable support for exception handling
  -fexperimental-isel     Enables the experimental global instruction selector
  -fexperimental-new-pass-manager
                          Enables an experimental new pass manager in LLVM.
  -ffast-math             Allow aggressive, lossy floating-point optimizations
  -ffine-grained-bitfield-accesses
                          Use separate accesses for bitfields with legal widths and alignments.
  -ffixed-r9              Reserve the r9 register (ARM only)
  -ffixed-x18             Reserve the x18 register (AArch64 only)
  -ffp-contract=<value>   Form fused FP ops (e.g. FMAs): fast (everywhere) | on (according to FP_CONTRACT pragma, default) | off (never fuse)
  -ffreestanding          Assert that the compilation takes place in a freestanding environment
  -ffunction-sections     Place each function in its own section (ELF Only)
  -fgnu-keywords          Allow GNU-extension keywords regardless of language standard
  -fgnu-runtime           Generate output compatible with the standard GNU Objective-C runtime
  -fgnu89-inline          Use the gnu89 inline semantics
  -fimplicit-module-maps  Implicitly search the file system for module map files.
  -finline-functions      Inline suitable functions
  -finline-hint-functions Inline functions which are (explicitly or implicitly) marked inline
  -finstrument-function-entry-bare
                          Instrument function entry only, after inlining, without arguments to the instrumentation call
  -finstrument-functions-after-inlining
                          Like -finstrument-functions, but insert the calls after inlining
  -finstrument-functions  Generate calls to instrument function entry and exit
  -fintegrated-as         Enable the integrated assembler
  -flto-jobs=<value>      Controls the backend parallelism of -flto=thin (default of 0 means the number of threads will be derived from the number of CPUs detected)
  -flto=<value>           Set LTO mode to either 'full' or 'thin'
  -flto                   Enable LTO in 'full' mode
  -fmath-errno            Require math functions to indicate errors by setting errno
  -fmax-type-align=<value>
                          Specify the maximum alignment to enforce on pointers lacking an explicit alignment
  -fmodule-file=[<name>=]<file>
                          Specify the mapping of module name to precompiled module file, or load a module file if name is omitted.
  -fmodule-map-file=<file>
                          Load this module map file
  -fmodule-name=<name>    Specify the name of the module to build
  -fmodules-cache-path=<directory>
                          Specify the module cache path
  -fmodules-decluse       Require declaration of modules used within a module
  -fmodules-disable-diagnostic-validation
                          Disable validation of the diagnostic options when loading the module
  -fmodules-ignore-macro=<value>
                          Ignore the definition of the given macro when building and loading modules
  -fmodules-prune-after=<seconds>
                          Specify the interval (in seconds) after which a module file will be considered unused
  -fmodules-prune-interval=<seconds>
                          Specify the interval (in seconds) between attempts to prune the module cache
  -fmodules-search-all    Search even non-imported modules to resolve references
  -fmodules-strict-decluse
                          Like -fmodules-decluse but requires all headers to be in modules
  -fmodules-ts            Enable support for the C++ Modules TS
  -fmodules-user-build-path <directory>
                          Specify the module user build path
  -fmodules-validate-once-per-build-session
                          Don't verify input files for the modules if the module has been successfully validated or loaded during this build session
  -fmodules-validate-system-headers
                          Validate the system headers that a module depends on when loading the module
  -fmodules               Enable the 'modules' language feature
  -fms-compatibility-version=<value>
                          Dot-separated value representing the Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default))
  -fms-compatibility      Enable full Microsoft Visual C++ compatibility
  -fms-extensions         Accept some non-standard constructs supported by the Microsoft compiler
  -fmsc-version=<value>   Microsoft compiler version number to report in _MSC_VER (0 = don't define it (default))
  -fnew-alignment=<align> Specifies the largest alignment guaranteed by '::operator new(size_t)'
  -fno-access-control     Disable C++ access control
  -fno-assume-sane-operator-new
                          Don't assume that C++'s global operator new can't alias any pointer
  -fno-autolink           Disable generation of linker directives for automatic library linking
  -fno-builtin-<value>    Disable implicit builtin knowledge of a specific function
  -fno-builtin            Disable implicit builtin knowledge of functions
  -fno-common             Compile common globals like normal definitions
  -fno-constant-cfstrings Disable creation of CodeFoundation-type constant strings
  -fno-coverage-mapping   Disable code coverage analysis
  -fno-crash-diagnostics  Disable auto-generation of preprocessed source files and a script for reproduction during a clang crash
  -fno-debug-info-for-profiling
                          Do not emit extra debug info for sample profiler.
  -fno-debug-macro        Do not emit macro debug information
  -fno-declspec           Disallow __declspec as a keyword
  -fno-delayed-template-parsing
                          Disable delayed template parsing
  -fno-diagnostics-fixit-info
                          Do not include fixit information in diagnostics
  -fno-dollars-in-identifiers
                          Disallow '$' in identifiers
  -fno-double-square-bracket-attributes
                          Disable '[[]]' attributes in all C and C++ language modes
  -fno-elide-constructors Disable C++ copy constructor elision
  -fno-elide-type         Do not elide types when printing diagnostics
  -fno-experimental-isel  Disables the experimental global instruction selector
  -fno-experimental-new-pass-manager
                          Disables an experimental new pass manager in LLVM.
  -fno-fine-grained-bitfield-accesses
                          Use large-integer access for consecutive bitfield runs.
  -fno-gnu-inline-asm     Disable GNU style inline asm
  -fno-integrated-as      Disable the integrated assembler
  -fno-jump-tables        Do not use jump tables for lowering switches
  -fno-lax-vector-conversions
                          Disallow implicit conversions between vectors with a different number of elements or different element types
  -fno-lto                Disable LTO mode (default)
  -fno-merge-all-constants
                          Disallow merging of constants
  -fno-objc-infer-related-result-type
                          do not infer Objective-C related result type based on method family
  -fno-openmp-simd        Disable OpenMP code for SIMD-based constructs.
  -fno-operator-names     Do not treat C++ operator name keywords as synonyms for operators
  -fno-plt                Do not use the PLT to make function calls
  -fno-preserve-as-comments
                          Do not preserve comments in inline assembly
  -fno-profile-generate   Disable generation of profile instrumentation.
  -fno-profile-instr-generate
                          Disable generation of profile instrumentation.
  -fno-profile-instr-use  Disable using instrumentation data for profile-guided optimization
  -fno-reroll-loops       Turn off loop reroller
  -fno-rtlib-add-rpath    Do not add -rpath with architecture-specific resource directory to the linker flags
  -fno-rtti               Disable generation of rtti information
  -fno-sanitize-address-use-after-scope
                          Disable use-after-scope detection in AddressSanitizer
  -fno-sanitize-blacklist Don't use blacklist file for sanitizers
  -fno-sanitize-cfi-cross-dso
                          Disable control flow integrity (CFI) checks for cross-DSO calls.
  -fno-sanitize-coverage=<value>
                          Disable specified features of coverage instrumentation for Sanitizers
  -fno-sanitize-memory-track-origins
                          Disable origins tracking in MemorySanitizer
  -fno-sanitize-memory-use-after-dtor
                          Disable use-after-destroy detection in MemorySanitizer
  -fno-sanitize-recover=<value>
                          Disable recovery for specified sanitizers
  -fno-sanitize-stats     Disable sanitizer statistics gathering.
  -fno-sanitize-thread-atomics
                          Disable atomic operations instrumentation in ThreadSanitizer
  -fno-sanitize-thread-func-entry-exit
                          Disable function entry/exit instrumentation in ThreadSanitizer
  -fno-sanitize-thread-memory-access
                          Disable memory access instrumentation in ThreadSanitizer
  -fno-sanitize-trap=<value>
                          Disable trapping for specified sanitizers
  -fno-short-wchar        Force wchar_t to be an unsigned int
  -fno-show-column        Do not include column number on diagnostics
  -fno-show-source-location
                          Do not include source location information with diagnostics
  -fno-signed-char        Char is unsigned
  -fno-signed-zeros       Allow optimizations that ignore the sign of floating point zeros
  -fno-spell-checking     Disable spell-checking
  -fno-stack-protector    Disable the use of stack protectors
  -fno-standalone-debug   Limit debug information produced to reduce size of debug binary
  -fno-threadsafe-statics Do not emit code to make initialization of local statics thread safe
  -fno-trigraphs          Do not process trigraph sequences
  -fno-unroll-loops       Turn off loop unroller
  -fno-use-cxa-atexit     Don't use __cxa_atexit for calling destructors
  -fno-use-init-array     Don't use .init_array instead of .ctors
  -fnoopenmp-relocatable-target
                          Do not compile OpenMP target code as relocatable.
  -fobjc-arc-exceptions   Use EH-safe code when synthesizing retains and releases in -fobjc-arc
  -fobjc-arc              Synthesize retain and release calls for Objective-C pointers
  -fobjc-exceptions       Enable Objective-C exceptions
  -fobjc-runtime=<value>  Specify the target Objective-C runtime kind and version
  -fobjc-weak             Enable ARC-style weak references in Objective-C
  -fopenmp-relocatable-target
                          OpenMP target code is compiled as relocatable using the -c flag. For OpenMP targets the code is relocatable by default.
  -fopenmp-simd           Emit OpenMP code only for SIMD-based constructs.
  -fopenmp-targets=<value>
                          Specify comma-separated list of triples OpenMP offloading targets to be supported
  -foptimization-record-file=<value>
                          Specify the file name of any generated YAML optimization record
  -fpack-struct=<value>   Specify the default maximum struct packing alignment
  -fpascal-strings        Recognize and construct Pascal-style string literals
  -fpcc-struct-return     Override the default ABI to return all structs on the stack
  -fplt                   Use the PLT to make function calls
  -fplugin=<dsopath>      Load the named plugin (dynamic shared object)
  -fprebuilt-module-path=<directory>
                          Specify the prebuilt module path
  -fprofile-generate=<directory>
                          Generate instrumented code to collect execution counts into <directory>/default.profraw (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-generate      Generate instrumented code to collect execution counts into default.profraw (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-instr-generate=<file>
                          Generate instrumented code to collect execution counts into <file> (overridden by LLVM_PROFILE_FILE env var)
  -fprofile-instr-generate
                          Generate instrumented code to collect execution counts into default.profraw file (overridden by '=' form of option or LLVM_PROFILE_FILE env var)
  -fprofile-instr-use=<value>
                          Use instrumentation data for profile-guided optimization
  -fprofile-sample-accurate
                          Specifies that the sample profile is accurate
  -fprofile-sample-use=<value>
                          Enable sample-based profile guided optimizations
  -fprofile-use=<pathname>
                          Use instrumentation data for profile-guided optimization. If pathname is a directory, it reads from <pathname>/default.profdata. Otherwise, it reads from file <pathname>.
  -freciprocal-math       Allow division operations to be reassociated
  -freg-struct-return     Override the default ABI to return small structs in registers
  -frelaxed-template-template-args
                          Enable C++17 relaxed template template argument matching
  -freroll-loops          Turn on loop reroller
  -frtlib-add-rpath       Add -rpath with architecture-specific resource directory to the linker flags
  -fsanitize-address-field-padding=<value>
                          Level of field padding for AddressSanitizer
  -fsanitize-address-globals-dead-stripping
                          Enable linker dead stripping of globals in AddressSanitizer
  -fsanitize-address-use-after-scope
                          Enable use-after-scope detection in AddressSanitizer
  -fsanitize-blacklist=<value>
                          Path to blacklist file for sanitizers
  -fsanitize-cfi-cross-dso
                          Enable control flow integrity (CFI) checks for cross-DSO calls.
  -fsanitize-cfi-icall-generalize-pointers
                          Generalize pointers in CFI indirect call type signature checks
  -fsanitize-coverage=<value>
                          Specify the type of coverage instrumentation for Sanitizers
  -fsanitize-memory-track-origins=<value>
                          Enable origins tracking in MemorySanitizer
  -fsanitize-memory-track-origins
                          Enable origins tracking in MemorySanitizer
  -fsanitize-memory-use-after-dtor
                          Enable use-after-destroy detection in MemorySanitizer
  -fsanitize-recover=<value>
                          Enable recovery for specified sanitizers
  -fsanitize-stats        Enable sanitizer statistics gathering.
  -fsanitize-thread-atomics
                          Enable atomic operations instrumentation in ThreadSanitizer (default)
  -fsanitize-thread-func-entry-exit
                          Enable function entry/exit instrumentation in ThreadSanitizer (default)
  -fsanitize-thread-memory-access
                          Enable memory access instrumentation in ThreadSanitizer (default)
  -fsanitize-trap=<value> Enable trapping for specified sanitizers
  -fsanitize-undefined-strip-path-components=<number>
                          Strip (or keep only, if negative) a given number of path components when emitting check metadata.
  -fsanitize=<check>      Turn on runtime checks for various forms of undefined or suspicious behavior. See user manual for available checks
  -fsave-optimization-record
                          Generate a YAML optimization record file
  -fseh-exceptions        Use SEH style exceptions
  -fshort-enums           Allocate to an enum type only as many bytes as it needs for the declared range of possible values
  -fshort-wchar           Force wchar_t to be a short unsigned int
  -fshow-overloads=<value>
                          Which overload candidates to show when overload resolution fails: best|all; defaults to all
  -fsized-deallocation    Enable C++14 sized global deallocation functions
  -fsjlj-exceptions       Use SjLj style exceptions
  -fslp-vectorize         Enable the superword-level parallelism vectorization passes
  -fsplit-dwarf-inlining  Place debug types in their own section (ELF Only)
  -fstack-protector-all   Force the usage of stack protectors for all functions
  -fstack-protector-strong
                          Use a strong heuristic to apply stack protectors to functions
  -fstack-protector       Enable stack protectors for functions potentially vulnerable to stack smashing
  -fstandalone-debug      Emit full debug info for all types used by the program
  -fstrict-enums          Enable optimizations based on the strict definition of an enum's value range
  -fstrict-return         Always treat control flow paths that fall off the end of a non-void function as unreachable
  -fstrict-vtable-pointers
                          Enable optimizations based on the strict rules for overwriting polymorphic C++ objects
  -fthinlto-index=<value> Perform ThinLTO importing using provided function summary index
  -ftrap-function=<value> Issue call to specified function rather than a trap instruction
  -ftrapv-handler=<function name>
                          Specify the function to be called on overflow
  -ftrapv                 Trap on integer overflow
  -ftrigraphs             Process trigraph sequences
  -funique-section-names  Use unique names for text and data sections (ELF Only)
  -funroll-loops          Turn on loop unroller
  -fuse-init-array        Use .init_array instead of .ctors
  -fveclib=<value>        Use the given vector functions library
  -fvectorize             Enable the loop vectorization passes
  -fvisibility-inlines-hidden
                          Give inline C++ member functions default visibility by default
  -fvisibility-ms-compat  Give global types 'default' visibility and global functions and variables 'hidden' visibility by default
  -fvisibility=<value>    Set the default symbol visibility for all global declarations
  -fwhole-program-vtables Enables whole-program vtable optimization. Requires -flto
  -fwrapv                 Treat signed integer overflow as two's complement
  -fwritable-strings      Store string literals as writable data
  -fxray-always-emit-customevents
                          Determine whether to always emit __xray_customevent(...) calls even if the function it appears in is not always instrumented.
  -fxray-always-instrument= <value>
                          Filename defining the whitelist for imbuing the 'always instrument' XRay attribute.
  -fxray-instruction-threshold= <value>
                          Sets the minimum function size to instrument with XRay
  -fxray-instrument       Generate XRay instrumentation sleds on function entry and exit
  -fxray-never-instrument= <value>
                          Filename defining the whitelist for imbuing the 'never instrument' XRay attribute.
  -fzvector               Enable System z vector language extension
  -F <value>              Add directory to framework include search path
  --gcc-toolchain=<value> Use the gcc toolchain at the given directory
  -gcodeview              Generate CodeView debug information
  -gdwarf-2               Generate source-level debug information with dwarf version 2
  -gdwarf-3               Generate source-level debug information with dwarf version 3
  -gdwarf-4               Generate source-level debug information with dwarf version 4
  -gdwarf-5               Generate source-level debug information with dwarf version 5
  -gline-tables-only      Emit debug line number tables only
  -gmodules               Generate debug info with external references to clang modules or precompiled headers
  -gz=<value>             DWARF debug sections compression type
  -gz                     DWARF debug sections compression type
  -G <size>               Put objects of at most <size> bytes into small data section (MIPS / Hexagon)
  -g                      Generate source-level debug information
  -help                   Display available options
  -H                      Show header includes and nesting depth
  -I-                     Restrict all prior -I flags to double-quoted inclusion and remove current directory from include path
  -idirafter <value>      Add directory to AFTER include search path
  -iframeworkwithsysroot <directory>
                          Add directory to SYSTEM framework search path, absolute paths are relative to -isysroot
  -iframework <value>     Add directory to SYSTEM framework search path
  -imacros <file>         Include macros from file before parsing
  -include-pch <file>     Include precompiled header file
  -include <file>         Include file before parsing
  -index-header-map       Make the next included directory (-I or -F) an indexer header map
  -iprefix <dir>          Set the -iwithprefix/-iwithprefixbefore prefix
  -iquote <directory>     Add directory to QUOTE include search path
  -isysroot <dir>         Set the system root directory (usually /)
  -isystem-after <directory>
                          Add directory to end of the SYSTEM include search path
  -isystem <directory>    Add directory to SYSTEM include search path
  -ivfsoverlay <value>    Overlay the virtual filesystem described by file over the real file system
  -iwithprefixbefore <dir>
                          Set directory to include search path with prefix
  -iwithprefix <dir>      Set directory to SYSTEM include search path with prefix
  -iwithsysroot <directory>
                          Add directory to SYSTEM include search path, absolute paths are relative to -isysroot
  -I <dir>                Add directory to include search path
  -L <dir>                Add directory to library search path
  -mabicalls              Enable SVR4-style position-independent code (Mips only)
  -malign-double          Align doubles to two words in structs (x86 only)
  -mbackchain             Link stack frames through backchain on System Z
  -mcrc                   Allow use of CRC instructions (ARM only)
  -MD                     Write a depfile containing user and system headers
  -meabi <value>          Set EABI type, e.g. 4, 5 or gnu (default depends on triple)
  -membedded-data         Place constants in the .rodata section instead of the .sdata section even if they meet the -G <size> threshold (MIPS)
  -mexecute-only          Disallow generation of data access to code sections (ARM only)
  -mextern-sdata          Assume that externally defined data is in the small data if it meets the -G <size> threshold (MIPS)
  -mfentry                Insert calls to fentry at function entry (x86 only)
  -mfix-cortex-a53-835769 Workaround Cortex-A53 erratum 835769 (AArch64 only)
  -mfp32                  Use 32-bit floating point registers (MIPS only)
  -mfp64                  Use 64-bit floating point registers (MIPS only)
  -MF <file>              Write depfile output from -MMD, -MD, -MM, or -M to <file>
  -mgeneral-regs-only     Generate code which only uses the general purpose registers (AArch64 only)
  -mglobal-merge          Enable merging of globals
  -mgpopt                 Use GP relative accesses for symbols known to be in a small data section (MIPS)
  -MG                     Add missing headers to depfile
  -mhvx-double            Enable Hexagon Double Vector eXtensions
  -mhvx-length=<value>    Set Hexagon Vector Length
  -mhvx=<value>           Enable Hexagon Vector eXtensions
  -mhvx                   Enable Hexagon Vector eXtensions
  -miamcu                 Use Intel MCU ABI
  --migrate               Run the migrator
  -mincremental-linker-compatible
                          (integrated-as) Emit an object file which can be used with an incremental linker
  -mios-version-min=<value>
                          Set iOS deployment target
  -MJ <value>             Write a compilation database entry per input
  -mllvm <value>          Additional arguments to forward to LLVM's option processing
  -mlocal-sdata           Extend the -G behaviour to object local data (MIPS)
  -mlong-calls            Generate branches with extended addressability, usually via indirect jumps.
  -mmacosx-version-min=<value>
                          Set Mac OS X deployment target
  -mmadd4                 Enable the generation of 4-operand madd.s, madd.d and related instructions.
  -MMD                    Write a depfile containing user headers
  -mms-bitfields          Set the default structure layout to be compatible with the Microsoft compiler standard
  -mmsa                   Enable MSA ASE (MIPS only)
  -mmt                    Enable MT ASE (MIPS only)
  -MM                     Like -MMD, but also implies -E and writes to stdout by default
  -mno-abicalls           Disable SVR4-style position-independent code (Mips only)
  -mno-embedded-data      Do not place constants in the .rodata section instead of the .sdata if they meet the -G <size> threshold (MIPS)
  -mno-execute-only       Allow generation of data access to code sections (ARM only)
  -mno-extern-sdata       Do not assume that externally defined data is in the small data if it meets the -G <size> threshold (MIPS)
  -mno-fix-cortex-a53-835769
                          Don't workaround Cortex-A53 erratum 835769 (AArch64 only)
  -mno-global-merge       Disable merging of globals
  -mno-gpopt              Do not use GP relative accesses for symbols known to be in a small data section (MIPS)
  -mno-hvx-double         Disable Hexagon Double Vector eXtensions
  -mno-hvx                Disable Hexagon Vector eXtensions
  -mno-implicit-float     Don't generate implicit floating point instructions
  -mno-incremental-linker-compatible
                          (integrated-as) Emit an object file which cannot be used with an incremental linker
  -mno-local-sdata        Do not extend the -G behaviour to object local data (MIPS)
  -mno-long-calls         Restore the default behaviour of not generating long calls
  -mno-madd4              Disable the generation of 4-operand madd.s, madd.d and related instructions.
  -mno-movt               Disallow use of movt/movw pairs (ARM only)
  -mno-ms-bitfields       Do not set the default structure layout to be compatible with the Microsoft compiler standard
  -mno-msa                Disable MSA ASE (MIPS only)
  -mno-mt                 Disable MT ASE (MIPS only)
  -mno-neg-immediates     Disallow converting instructions with negative immediates to their negation or inversion.
  -mno-restrict-it        Allow generation of deprecated IT blocks for ARMv8. It is off by default for ARMv8 Thumb mode
  -mno-unaligned-access   Force all memory accesses to be aligned (AArch32/AArch64 only)
  -mno-xnack              Disable XNACK (AMDGPU only)
  -mnocrc                 Disallow use of CRC instructions (ARM only)
  -module-dependency-dir <value>
                          Directory to dump module dependencies to
  -module-file-info       Provide information about a particular module file
  -momit-leaf-frame-pointer
                          Omit frame pointer setup for leaf functions
  -mpie-copy-relocations  Use copy relocations support for PIE builds
  -mprefer-vector-width=<value>
                          Specifies preferred vector width for auto-vectorization. Defaults to 'none' which allows target specific decisions.
  -MP                     Create phony target for each dependency (other than main file)
  -mqdsp6-compat          Enable hexagon-qdsp6 backward compatibility
  -MQ <value>             Specify name of main file output to quote in depfile
  -mrelax-all             (integrated-as) Relax all machine instructions
  -mrestrict-it           Disallow generation of deprecated IT blocks for ARMv8. It is on by default for ARMv8 Thumb mode.
  -mrtd                   Make StdCall calling convention the default
  -msoft-float            Use software floating point
  -mstack-alignment=<value>
                          Set the stack alignment
  -mstack-probe-size=<value>
                          Set the stack probe size
  -mstackrealign          Force realign the stack at entry to every function
  -mthread-model <value>  The thread model to use, e.g. posix, single (posix by default)
  -mtp=<value>            Read thread pointer from coprocessor register (ARM only)
  -MT <value>             Specify name of main file output in depfile
  -munaligned-access      Allow memory accesses to be unaligned (AArch32/AArch64 only)
  -MV                     Use NMake/Jom format for the depfile
  -mxnack                 Enable XNACK (AMDGPU only)
  -M                      Like -MD, but also implies -E and writes to stdout by default
  --no-cuda-gpu-arch=<value>
                          Remove GPU architecture (e.g. sm_35) from the list of GPUs to compile for. 'all' resets the list to its default value.
  --no-cuda-version-check Don't error out if the detected version of the CUDA install is too low for the requested CUDA gpu architecture.
  --no-system-header-prefix=<prefix>
                          Treat all #include paths starting with <prefix> as not including a system header.
  -nobuiltininc           Disable builtin #include directories
  -nostdinc++             Disable standard #include directories for the C++ standard library
  -ObjC++                 Treat source input files as Objective-C++ inputs
  -objcmt-atomic-property Make migration to 'atomic' properties
  -objcmt-migrate-all     Enable migration to modern ObjC
  -objcmt-migrate-annotation
                          Enable migration to property and method annotations
  -objcmt-migrate-designated-init
                          Enable migration to infer NS_DESIGNATED_INITIALIZER for initializer methods
  -objcmt-migrate-instancetype
                          Enable migration to infer instancetype for method result type
  -objcmt-migrate-literals
                          Enable migration to modern ObjC literals
  -objcmt-migrate-ns-macros
                          Enable migration to NS_ENUM/NS_OPTIONS macros
  -objcmt-migrate-property-dot-syntax
                          Enable migration of setter/getter messages to property-dot syntax
  -objcmt-migrate-property
                          Enable migration to modern ObjC property
  -objcmt-migrate-protocol-conformance
                          Enable migration to add protocol conformance on classes
  -objcmt-migrate-readonly-property
                          Enable migration to modern ObjC readonly property
  -objcmt-migrate-readwrite-property
                          Enable migration to modern ObjC readwrite property
  -objcmt-migrate-subscripting
                          Enable migration to modern ObjC subscripting
  -objcmt-ns-nonatomic-iosonly
                          Enable migration to use NS_NONATOMIC_IOSONLY macro for setting property's 'atomic' attribute
  -objcmt-returns-innerpointer-property
                          Enable migration to annotate property with NS_RETURNS_INNER_POINTER
  -objcmt-whitelist-dir-path=<value>
                          Only modify files with a filename contained in the provided directory path
  -ObjC                   Treat source input files as Objective-C inputs
  -o <file>               Write output to <file>
  -pg                     Enable mcount instrumentation
  -pipe                   Use pipes between commands, when possible
  --precompile            Only precompile the input
  -print-file-name=<file> Print the full library path of <file>
  -print-ivar-layout      Enable Objective-C Ivar layout bitmap print trace
  -print-libgcc-file-name Print the library path for the currently used compiler runtime library ("libgcc.a" or "libclang_rt.builtins.*.a")
  -print-prog-name=<name> Print the full program path of <name>
  -print-resource-dir     Print the resource directory pathname
  -print-search-dirs      Print the paths used for finding libraries and programs
  -pthread                Support POSIX threads in generated code
  --ptxas-path=<value>    Path to ptxas (used for compiling CUDA code)
  -P                      Disable linemarker output in -E mode
  -Qunused-arguments      Don't emit warning for unused driver arguments
  -relocatable-pch        Whether to build a relocatable precompiled header
  -rewrite-legacy-objc    Rewrite Legacy Objective-C source to C++
  -rewrite-objc           Rewrite Objective-C source to C++
  -Rpass-analysis=<value> Report transformation analysis from optimization passes whose name matches the given POSIX regular expression
  -Rpass-missed=<value>   Report missed transformations by optimization passes whose name matches the given POSIX regular expression
  -Rpass=<value>          Report transformations performed by optimization passes whose name matches the given POSIX regular expression
  -rtlib=<value>          Compiler runtime library to use
  -R<remark>              Enable the specified remark
  -save-stats=<value>     Save llvm statistics.
  -save-stats             Save llvm statistics.
  -save-temps=<value>     Save intermediate compilation results.
  -save-temps             Save intermediate compilation results
  -serialize-diagnostics <value>
                          Serialize compiler diagnostics to a file
  -std=<value>            Language standard to compile for
  -stdlib=<value>         C++ standard library to use
  --system-header-prefix=<prefix>
                          Treat all #include paths starting with <prefix> as including a system header.
  -S                      Only run preprocess and compilation steps
  --target=<value>        Generate code for the given target
  -Tbss <addr>            Set starting address of BSS to <addr>
  -Tdata <addr>           Set starting address of BSS to <addr>
  -time                   Time individual commands
  -traditional-cpp        Enable some traditional CPP emulation
  -trigraphs              Process trigraph sequences
  -Ttext <addr>           Set starting address of BSS to <addr>
  -T <script>             Specify <script> as linker script
  -undef                  undef all system defines
  -U <macro>              Undefine macro <macro>
  --verify-debug-info     Verify the binary representation of debug output
  -verify-pch             Load and verify that a pre-compiled header file is not stale
  --version               Print version information
  -v                      Show commands to run and use verbose output
  -Wa,<arg>               Pass the comma separated arguments in <arg> to the assembler
  -Wdeprecated            Enable warnings for deprecated constructs and define __DEPRECATED
  -Wl,<arg>               Pass the comma separated arguments in <arg> to the linker
  -working-directory <value>
                          Resolve file paths relative to the specified directory
  -Wp,<arg>               Pass the comma separated arguments in <arg> to the preprocessor
  -W<warning>             Enable the specified warning
  -w                      Suppress all warnings
  -Xanalyzer <arg>        Pass <arg> to the static analyzer
  -Xassembler <arg>       Pass <arg> to the assembler
  -Xclang <arg>           Pass <arg> to the clang compiler
  -Xcuda-fatbinary <arg>  Pass <arg> to fatbinary invocation
  -Xcuda-ptxas <arg>      Pass <arg> to the ptxas assembler
  -Xlinker <arg>          Pass <arg> to the linker
  -Xopenmp-target=<arg>   Pass <arg> to the specified target offloading toolchain. The triple that identifies the toolchain must be provided after the equals sign.
  -Xopenmp-target <arg>   Pass <arg> to the target offloading toolchain.
  -Xpreprocessor <arg>    Pass <arg> to the preprocessor
  -x <language>           Treat subsequent input files as having type <language>
  -z <arg>                Pass -z <arg> to the linker
Checking for program /usr/local/bin/mpicxx...found
                Defined make macro "CXX" to "mpicxx"
                Pushing language Cxx
Executing: mpicxx -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers     /tmp/petsc-mjVUVK/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
Executing: mpicxx -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers     /tmp/petsc-mjVUVK/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Executing: mpicxx -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers     /tmp/petsc-mjVUVK/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                        Pushing language CXX
                        Popping language CXX
Executing: mpicxx  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
ld: library not found for -lpetsc-ufod4vtr9mqHvKIQiVAm
clang-6.0: error: linker command failed with exit code 1 (use -v to see invocation)
Executing: mpicxx -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers     /tmp/petsc-mjVUVK/config.setCompilers/conftest.cc 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"


int main() {
;
  return 0;
}
                            Pushing language CXX
                            Popping language CXX
Executing: mpicxx  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-mjVUVK/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
                Popping language Cxx
================================================================================
TEST checkCxxPreprocessor from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:830)
TESTING: checkCxxPreprocessor from config.setCompilers(config/BuildSystem/config/setCompilers.py:830)
  Locate a functional Cxx preprocessor
Checking for program /usr/local/bin/mpicxx...found
                Defined make macro "CXXCPP" to "mpicxx -E"
              Pushing language Cxx
Executing: mpicxx -E  -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.cc 
stdout:
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.cc"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 399 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.cc" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/confdefs.h" 1
# 2 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.cc" 2
# 1 "/tmp/petsc-mjVUVK/config.setCompilers/conffix.h" 1
extern "C" {
}
# 3 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.cc" 2
# 1 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 1 3
# 85 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 3
# 1 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/__config" 1 3
# 22 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/__config" 3
# 449 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/__config" 3
namespace std {
  inline namespace __1 {
  }
}
# 86 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 2 3
# 1 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdlib.h" 1 3
# 92 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdlib.h" 3
# 1 "/usr/include/stdlib.h" 1 3 4
# 61 "/usr/include/stdlib.h" 3 4
# 1 "/usr/include/Availability.h" 1 3 4
# 190 "/usr/include/Availability.h" 3 4
# 1 "/usr/include/AvailabilityInternal.h" 1 3 4
# 191 "/usr/include/Availability.h" 2 3 4
# 62 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/_types.h" 1 3 4
# 27 "/usr/include/_types.h" 3 4
# 1 "/usr/include/sys/_types.h" 1 3 4
# 32 "/usr/include/sys/_types.h" 3 4
# 1 "/usr/include/sys/cdefs.h" 1 3 4
# 587 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/sys/_symbol_aliasing.h" 1 3 4
# 588 "/usr/include/sys/cdefs.h" 2 3 4
# 653 "/usr/include/sys/cdefs.h" 3 4
# 1 "/usr/include/sys/_posix_availability.h" 1 3 4
# 654 "/usr/include/sys/cdefs.h" 2 3 4
# 33 "/usr/include/sys/_types.h" 2 3 4
# 1 "/usr/include/machine/_types.h" 1 3 4
# 32 "/usr/include/machine/_types.h" 3 4
# 1 "/usr/include/i386/_types.h" 1 3 4
# 37 "/usr/include/i386/_types.h" 3 4
typedef signed char __int8_t;
typedef unsigned char __uint8_t;
typedef short __int16_t;
typedef unsigned short __uint16_t;
typedef int __int32_t;
typedef unsigned int __uint32_t;
typedef long long __int64_t;
typedef unsigned long long __uint64_t;
typedef long __darwin_intptr_t;
typedef unsigned int __darwin_natural_t;
# 70 "/usr/include/i386/_types.h" 3 4
typedef int __darwin_ct_rune_t;
typedef union {
 char __mbstate8[128];
 long long _mbstateL;
} __mbstate_t;
typedef __mbstate_t __darwin_mbstate_t;
typedef long int __darwin_ptrdiff_t;
typedef long unsigned int __darwin_size_t;
typedef __builtin_va_list __darwin_va_list;
typedef int __darwin_wchar_t;
typedef __darwin_wchar_t __darwin_rune_t;
typedef int __darwin_wint_t;
typedef unsigned long __darwin_clock_t;
typedef __uint32_t __darwin_socklen_t;
typedef long __darwin_ssize_t;
typedef long __darwin_time_t;
# 33 "/usr/include/machine/_types.h" 2 3 4
# 34 "/usr/include/sys/_types.h" 2 3 4
# 55 "/usr/include/sys/_types.h" 3 4
typedef __int64_t __darwin_blkcnt_t;
typedef __int32_t __darwin_blksize_t;
typedef __int32_t __darwin_dev_t;
typedef unsigned int __darwin_fsblkcnt_t;
typedef unsigned int __darwin_fsfilcnt_t;
typedef __uint32_t __darwin_gid_t;
typedef __uint32_t __darwin_id_t;
typedef __uint64_t __darwin_ino64_t;
typedef __darwin_ino64_t __darwin_ino_t;
typedef __darwin_natural_t __darwin_mach_port_name_t;
typedef __darwin_mach_port_name_t __darwin_mach_port_t;
typedef __uint16_t __darwin_mode_t;
typedef __int64_t __darwin_off_t;
typedef __int32_t __darwin_pid_t;
typedef __uint32_t __darwin_sigset_t;
typedef __int32_t __darwin_suseconds_t;
typedef __uint32_t __darwin_uid_t;
typedef __uint32_t __darwin_useconds_t;
typedef unsigned char __darwin_uuid_t[16];
typedef char __darwin_uuid_string_t[37];
# 1 "/usr/include/sys/_pthread/_pthread_types.h" 1 3 4
# 57 "/usr/include/sys/_pthread/_pthread_types.h" 3 4
struct __darwin_pthread_handler_rec {
 void (*__routine)(void *);
 void *__arg;
 struct __darwin_pthread_handler_rec *__next;
};
struct _opaque_pthread_attr_t {
 long __sig;
 char __opaque[56];
};
struct _opaque_pthread_cond_t {
 long __sig;
 char __opaque[40];
};
struct _opaque_pthread_condattr_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_mutex_t {
 long __sig;
 char __opaque[56];
};
struct _opaque_pthread_mutexattr_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_once_t {
 long __sig;
 char __opaque[8];
};
struct _opaque_pthread_rwlock_t {
 long __sig;
 char __opaque[192];
};
struct _opaque_pthread_rwlockattr_t {
 long __sig;
 char __opaque[16];
};
struct _opaque_pthread_t {
 long __sig;
 struct __darwin_pthread_handler_rec *__cleanup_stack;
 char __opaque[8176];
};
typedef struct _opaque_pthread_attr_t __darwin_pthread_attr_t;
typedef struct _opaque_pthread_cond_t __darwin_pthread_cond_t;
typedef struct _opaque_pthread_condattr_t __darwin_pthread_condattr_t;
typedef unsigned long __darwin_pthread_key_t;
typedef struct _opaque_pthread_mutex_t __darwin_pthread_mutex_t;
typedef struct _opaque_pthread_mutexattr_t __darwin_pthread_mutexattr_t;
typedef struct _opaque_pthread_once_t __darwin_pthread_once_t;
typedef struct _opaque_pthread_rwlock_t __darwin_pthread_rwlock_t;
typedef struct _opaque_pthread_rwlockattr_t __darwin_pthread_rwlockattr_t;
typedef struct _opaque_pthread_t *__darwin_pthread_t;
# 81 "/usr/include/sys/_types.h" 2 3 4
# 28 "/usr/include/_types.h" 2 3 4
# 39 "/usr/include/_types.h" 3 4
typedef int __darwin_nl_item;
typedef int __darwin_wctrans_t;
typedef __uint32_t __darwin_wctype_t;
# 64 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/wait.h" 1 3 4
# 79 "/usr/include/sys/wait.h" 3 4
typedef enum {
 P_ALL,
 P_PID,
 P_PGID
} idtype_t;
# 1 "/usr/include/sys/_types/_pid_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_pid_t.h" 3 4
typedef __darwin_pid_t pid_t;
# 90 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/sys/_types/_id_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_id_t.h" 3 4
typedef __darwin_id_t id_t;
# 91 "/usr/include/sys/wait.h" 2 3 4
# 109 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/sys/signal.h" 1 3 4
# 73 "/usr/include/sys/signal.h" 3 4
# 1 "/usr/include/sys/appleapiopts.h" 1 3 4
# 74 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/machine/signal.h" 1 3 4
# 32 "/usr/include/machine/signal.h" 3 4
# 1 "/usr/include/i386/signal.h" 1 3 4
# 39 "/usr/include/i386/signal.h" 3 4
typedef int sig_atomic_t;
# 33 "/usr/include/machine/signal.h" 2 3 4
# 83 "/usr/include/sys/signal.h" 2 3 4
# 146 "/usr/include/sys/signal.h" 3 4
# 1 "/usr/include/machine/_mcontext.h" 1 3 4
# 29 "/usr/include/machine/_mcontext.h" 3 4
# 1 "/usr/include/i386/_mcontext.h" 1 3 4
# 33 "/usr/include/i386/_mcontext.h" 3 4
# 1 "/usr/include/mach/i386/_structs.h" 1 3 4
# 43 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_thread_state
{
    unsigned int __eax;
    unsigned int __ebx;
    unsigned int __ecx;
    unsigned int __edx;
    unsigned int __edi;
    unsigned int __esi;
    unsigned int __ebp;
    unsigned int __esp;
    unsigned int __ss;
    unsigned int __eflags;
    unsigned int __eip;
    unsigned int __cs;
    unsigned int __ds;
    unsigned int __es;
    unsigned int __fs;
    unsigned int __gs;
};
# 89 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_fp_control
{
    unsigned short __invalid :1,
        __denorm :1,
    __zdiv :1,
    __ovrfl :1,
    __undfl :1,
    __precis :1,
      :2,
    __pc :2,
    __rc :2,
             :1,
      :3;
};
typedef struct __darwin_fp_control __darwin_fp_control_t;
# 147 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_fp_status
{
    unsigned short __invalid :1,
        __denorm :1,
    __zdiv :1,
    __ovrfl :1,
    __undfl :1,
    __precis :1,
    __stkflt :1,
    __errsumm :1,
    __c0 :1,
    __c1 :1,
    __c2 :1,
    __tos :3,
    __c3 :1,
    __busy :1;
};
typedef struct __darwin_fp_status __darwin_fp_status_t;
# 191 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_mmst_reg
{
 char __mmst_reg[10];
 char __mmst_rsrv[6];
};
# 210 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_xmm_reg
{
 char __xmm_reg[16];
};
# 232 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_float_state
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 char __fpu_rsrv4[14*16];
 int __fpu_reserved1;
};
struct __darwin_i386_avx_state
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 char __fpu_rsrv4[14*16];
 int __fpu_reserved1;
 char __avx_reserved1[64];
 struct __darwin_xmm_reg __fpu_ymmh0;
 struct __darwin_xmm_reg __fpu_ymmh1;
 struct __darwin_xmm_reg __fpu_ymmh2;
 struct __darwin_xmm_reg __fpu_ymmh3;
 struct __darwin_xmm_reg __fpu_ymmh4;
 struct __darwin_xmm_reg __fpu_ymmh5;
 struct __darwin_xmm_reg __fpu_ymmh6;
 struct __darwin_xmm_reg __fpu_ymmh7;
};
# 402 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_i386_exception_state
{
 __uint16_t __trapno;
 __uint16_t __cpu;
 __uint32_t __err;
 __uint32_t __faultvaddr;
};
# 422 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_debug_state32
{
 unsigned int __dr0;
 unsigned int __dr1;
 unsigned int __dr2;
 unsigned int __dr3;
 unsigned int __dr4;
 unsigned int __dr5;
 unsigned int __dr6;
 unsigned int __dr7;
};
# 454 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_thread_state64
{
 __uint64_t __rax;
 __uint64_t __rbx;
 __uint64_t __rcx;
 __uint64_t __rdx;
 __uint64_t __rdi;
 __uint64_t __rsi;
 __uint64_t __rbp;
 __uint64_t __rsp;
 __uint64_t __r8;
 __uint64_t __r9;
 __uint64_t __r10;
 __uint64_t __r11;
 __uint64_t __r12;
 __uint64_t __r13;
 __uint64_t __r14;
 __uint64_t __r15;
 __uint64_t __rip;
 __uint64_t __rflags;
 __uint64_t __cs;
 __uint64_t __fs;
 __uint64_t __gs;
};
# 509 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_float_state64
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 struct __darwin_xmm_reg __fpu_xmm8;
 struct __darwin_xmm_reg __fpu_xmm9;
 struct __darwin_xmm_reg __fpu_xmm10;
 struct __darwin_xmm_reg __fpu_xmm11;
 struct __darwin_xmm_reg __fpu_xmm12;
 struct __darwin_xmm_reg __fpu_xmm13;
 struct __darwin_xmm_reg __fpu_xmm14;
 struct __darwin_xmm_reg __fpu_xmm15;
 char __fpu_rsrv4[6*16];
 int __fpu_reserved1;
};
struct __darwin_x86_avx_state64
{
 int __fpu_reserved[2];
 struct __darwin_fp_control __fpu_fcw;
 struct __darwin_fp_status __fpu_fsw;
 __uint8_t __fpu_ftw;
 __uint8_t __fpu_rsrv1;
 __uint16_t __fpu_fop;
 __uint32_t __fpu_ip;
 __uint16_t __fpu_cs;
 __uint16_t __fpu_rsrv2;
 __uint32_t __fpu_dp;
 __uint16_t __fpu_ds;
 __uint16_t __fpu_rsrv3;
 __uint32_t __fpu_mxcsr;
 __uint32_t __fpu_mxcsrmask;
 struct __darwin_mmst_reg __fpu_stmm0;
 struct __darwin_mmst_reg __fpu_stmm1;
 struct __darwin_mmst_reg __fpu_stmm2;
 struct __darwin_mmst_reg __fpu_stmm3;
 struct __darwin_mmst_reg __fpu_stmm4;
 struct __darwin_mmst_reg __fpu_stmm5;
 struct __darwin_mmst_reg __fpu_stmm6;
 struct __darwin_mmst_reg __fpu_stmm7;
 struct __darwin_xmm_reg __fpu_xmm0;
 struct __darwin_xmm_reg __fpu_xmm1;
 struct __darwin_xmm_reg __fpu_xmm2;
 struct __darwin_xmm_reg __fpu_xmm3;
 struct __darwin_xmm_reg __fpu_xmm4;
 struct __darwin_xmm_reg __fpu_xmm5;
 struct __darwin_xmm_reg __fpu_xmm6;
 struct __darwin_xmm_reg __fpu_xmm7;
 struct __darwin_xmm_reg __fpu_xmm8;
 struct __darwin_xmm_reg __fpu_xmm9;
 struct __darwin_xmm_reg __fpu_xmm10;
 struct __darwin_xmm_reg __fpu_xmm11;
 struct __darwin_xmm_reg __fpu_xmm12;
 struct __darwin_xmm_reg __fpu_xmm13;
 struct __darwin_xmm_reg __fpu_xmm14;
 struct __darwin_xmm_reg __fpu_xmm15;
 char __fpu_rsrv4[6*16];
 int __fpu_reserved1;
 char __avx_reserved1[64];
 struct __darwin_xmm_reg __fpu_ymmh0;
 struct __darwin_xmm_reg __fpu_ymmh1;
 struct __darwin_xmm_reg __fpu_ymmh2;
 struct __darwin_xmm_reg __fpu_ymmh3;
 struct __darwin_xmm_reg __fpu_ymmh4;
 struct __darwin_xmm_reg __fpu_ymmh5;
 struct __darwin_xmm_reg __fpu_ymmh6;
 struct __darwin_xmm_reg __fpu_ymmh7;
 struct __darwin_xmm_reg __fpu_ymmh8;
 struct __darwin_xmm_reg __fpu_ymmh9;
 struct __darwin_xmm_reg __fpu_ymmh10;
 struct __darwin_xmm_reg __fpu_ymmh11;
 struct __darwin_xmm_reg __fpu_ymmh12;
 struct __darwin_xmm_reg __fpu_ymmh13;
 struct __darwin_xmm_reg __fpu_ymmh14;
 struct __darwin_xmm_reg __fpu_ymmh15;
};
# 751 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_exception_state64
{
    __uint16_t __trapno;
    __uint16_t __cpu;
    __uint32_t __err;
    __uint64_t __faultvaddr;
};
# 771 "/usr/include/mach/i386/_structs.h" 3 4
struct __darwin_x86_debug_state64
{
 __uint64_t __dr0;
 __uint64_t __dr1;
 __uint64_t __dr2;
 __uint64_t __dr3;
 __uint64_t __dr4;
 __uint64_t __dr5;
 __uint64_t __dr6;
 __uint64_t __dr7;
};
# 34 "/usr/include/i386/_mcontext.h" 2 3 4
struct __darwin_mcontext32
{
 struct __darwin_i386_exception_state __es;
 struct __darwin_i386_thread_state __ss;
 struct __darwin_i386_float_state __fs;
};
struct __darwin_mcontext_avx32
{
 struct __darwin_i386_exception_state __es;
 struct __darwin_i386_thread_state __ss;
 struct __darwin_i386_avx_state __fs;
};
# 76 "/usr/include/i386/_mcontext.h" 3 4
struct __darwin_mcontext64
{
 struct __darwin_x86_exception_state64 __es;
 struct __darwin_x86_thread_state64 __ss;
 struct __darwin_x86_float_state64 __fs;
};
struct __darwin_mcontext_avx64
{
 struct __darwin_x86_exception_state64 __es;
 struct __darwin_x86_thread_state64 __ss;
 struct __darwin_x86_avx_state64 __fs;
};
# 115 "/usr/include/i386/_mcontext.h" 3 4
typedef struct __darwin_mcontext64 *mcontext_t;
# 30 "/usr/include/machine/_mcontext.h" 2 3 4
# 147 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_pthread/_pthread_attr_t.h" 1 3 4
# 30 "/usr/include/sys/_pthread/_pthread_attr_t.h" 3 4
typedef __darwin_pthread_attr_t pthread_attr_t;
# 149 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_sigaltstack.h" 1 3 4
# 36 "/usr/include/sys/_types/_sigaltstack.h" 3 4
struct __darwin_sigaltstack
{
 void *ss_sp;
 __darwin_size_t ss_size;
 int ss_flags;
};
typedef struct __darwin_sigaltstack stack_t;
# 151 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_ucontext.h" 1 3 4
# 34 "/usr/include/sys/_types/_ucontext.h" 3 4
struct __darwin_ucontext
{
 int uc_onstack;
 __darwin_sigset_t uc_sigmask;
 struct __darwin_sigaltstack uc_stack;
 struct __darwin_ucontext *uc_link;
 __darwin_size_t uc_mcsize;
 struct __darwin_mcontext64 *uc_mcontext;
};
typedef struct __darwin_ucontext ucontext_t;
# 152 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_sigset_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_sigset_t.h" 3 4
typedef __darwin_sigset_t sigset_t;
# 155 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_size_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_size_t.h" 3 4
typedef __darwin_size_t size_t;
# 156 "/usr/include/sys/signal.h" 2 3 4
# 1 "/usr/include/sys/_types/_uid_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_uid_t.h" 3 4
typedef __darwin_uid_t uid_t;
# 157 "/usr/include/sys/signal.h" 2 3 4
union sigval {
 int sival_int;
 void *sival_ptr;
};
struct sigevent {
 int sigev_notify;
 int sigev_signo;
 union sigval sigev_value;
 void (*sigev_notify_function)(union sigval);
 pthread_attr_t *sigev_notify_attributes;
};
typedef struct __siginfo {
 int si_signo;
 int si_errno;
 int si_code;
 pid_t si_pid;
 uid_t si_uid;
 int si_status;
 void *si_addr;
 union sigval si_value;
 long si_band;
 unsigned long __pad[7];
} siginfo_t;
# 269 "/usr/include/sys/signal.h" 3 4
union __sigaction_u {
 void (*__sa_handler)(int);
 void (*__sa_sigaction)(int, struct __siginfo *,
         void *);
};
struct __sigaction {
 union __sigaction_u __sigaction_u;
 void (*sa_tramp)(void *, int, int, siginfo_t *, void *);
 sigset_t sa_mask;
 int sa_flags;
};
struct sigaction {
 union __sigaction_u __sigaction_u;
 sigset_t sa_mask;
 int sa_flags;
};
# 331 "/usr/include/sys/signal.h" 3 4
typedef void (*sig_t)(int);
# 348 "/usr/include/sys/signal.h" 3 4
struct sigvec {
 void (*sv_handler)(int);
 int sv_mask;
 int sv_flags;
};
# 367 "/usr/include/sys/signal.h" 3 4
struct sigstack {
 char *ss_sp;
 int ss_onstack;
};
# 389 "/usr/include/sys/signal.h" 3 4
extern "C" {
void (*signal(int, void (*)(int)))(int);
}
# 110 "/usr/include/sys/wait.h" 2 3 4
# 1 "/usr/include/sys/resource.h" 1 3 4
# 72 "/usr/include/sys/resource.h" 3 4
# 1 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdint.h" 1 3 4
# 106 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdint.h" 3
# 119 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdint.h" 3
# 1 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 1 3 4
# 63 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 3 4
# 1 "/usr/include/stdint.h" 1 3 4
# 18 "/usr/include/stdint.h" 3 4
# 1 "/usr/include/sys/_types/_int8_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int8_t.h" 3 4
typedef signed char int8_t;
# 19 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int16_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int16_t.h" 3 4
typedef short int16_t;
# 20 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int32_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int32_t.h" 3 4
typedef int int32_t;
# 21 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_int64_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_int64_t.h" 3 4
typedef long long int64_t;
# 22 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint8_t.h" 1 3 4
# 31 "/usr/include/_types/_uint8_t.h" 3 4
typedef unsigned char uint8_t;
# 24 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint16_t.h" 1 3 4
# 31 "/usr/include/_types/_uint16_t.h" 3 4
typedef unsigned short uint16_t;
# 25 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint32_t.h" 1 3 4
# 31 "/usr/include/_types/_uint32_t.h" 3 4
typedef unsigned int uint32_t;
# 26 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uint64_t.h" 1 3 4
# 31 "/usr/include/_types/_uint64_t.h" 3 4
typedef unsigned long long uint64_t;
# 27 "/usr/include/stdint.h" 2 3 4
typedef int8_t int_least8_t;
typedef int16_t int_least16_t;
typedef int32_t int_least32_t;
typedef int64_t int_least64_t;
typedef uint8_t uint_least8_t;
typedef uint16_t uint_least16_t;
typedef uint32_t uint_least32_t;
typedef uint64_t uint_least64_t;
typedef int8_t int_fast8_t;
typedef int16_t int_fast16_t;
typedef int32_t int_fast32_t;
typedef int64_t int_fast64_t;
typedef uint8_t uint_fast8_t;
typedef uint16_t uint_fast16_t;
typedef uint32_t uint_fast32_t;
typedef uint64_t uint_fast64_t;
# 1 "/usr/include/sys/_types/_intptr_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_intptr_t.h" 3 4
typedef __darwin_intptr_t intptr_t;
# 54 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/sys/_types/_uintptr_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_uintptr_t.h" 3 4
typedef unsigned long uintptr_t;
# 55 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_intmax_t.h" 1 3 4
# 32 "/usr/include/_types/_intmax_t.h" 3 4
typedef long int intmax_t;
# 59 "/usr/include/stdint.h" 2 3 4
# 1 "/usr/include/_types/_uintmax_t.h" 1 3 4
# 32 "/usr/include/_types/_uintmax_t.h" 3 4
typedef long unsigned int uintmax_t;
# 60 "/usr/include/stdint.h" 2 3 4
# 64 "/usr/local/Cellar/llvm/6.0.0/lib/clang/6.0.0/include/stdint.h" 2 3 4
# 120 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdint.h" 2 3
# 73 "/usr/include/sys/resource.h" 2 3 4
# 1 "/usr/include/sys/_types/_timeval.h" 1 3 4
# 30 "/usr/include/sys/_types/_timeval.h" 3 4
struct timeval
{
 __darwin_time_t tv_sec;
 __darwin_suseconds_t tv_usec;
};
# 81 "/usr/include/sys/resource.h" 2 3 4
typedef __uint64_t rlim_t;
# 152 "/usr/include/sys/resource.h" 3 4
struct rusage {
 struct timeval ru_utime;
 struct timeval ru_stime;
# 163 "/usr/include/sys/resource.h" 3 4
 long ru_maxrss;
 long ru_ixrss;
 long ru_idrss;
 long ru_isrss;
 long ru_minflt;
 long ru_majflt;
 long ru_nswap;
 long ru_inblock;
 long ru_oublock;
 long ru_msgsnd;
 long ru_msgrcv;
 long ru_nsignals;
 long ru_nvcsw;
 long ru_nivcsw;
};
# 192 "/usr/include/sys/resource.h" 3 4
typedef void *rusage_info_t;
struct rusage_info_v0 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
};
struct rusage_info_v1 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
};
struct rusage_info_v2 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
 uint64_t ri_diskio_bytesread;
 uint64_t ri_diskio_byteswritten;
};
struct rusage_info_v3 {
 uint8_t ri_uuid[16];
 uint64_t ri_user_time;
 uint64_t ri_system_time;
 uint64_t ri_pkg_idle_wkups;
 uint64_t ri_interrupt_wkups;
 uint64_t ri_pageins;
 uint64_t ri_wired_size;
 uint64_t ri_resident_size;
 uint64_t ri_phys_footprint;
 uint64_t ri_proc_start_abstime;
 uint64_t ri_proc_exit_abstime;
 uint64_t ri_child_user_time;
 uint64_t ri_child_system_time;
 uint64_t ri_child_pkg_idle_wkups;
 uint64_t ri_child_interrupt_wkups;
 uint64_t ri_child_pageins;
 uint64_t ri_child_elapsed_abstime;
 uint64_t ri_diskio_bytesread;
 uint64_t ri_diskio_byteswritten;
 uint64_t ri_cpu_time_qos_default;
 uint64_t ri_cpu_time_qos_maintenance;
 uint64_t ri_cpu_time_qos_background;
 uint64_t ri_cpu_time_qos_utility;
 uint64_t ri_cpu_time_qos_legacy;
 uint64_t ri_cpu_time_qos_user_initiated;
 uint64_t ri_cpu_time_qos_user_interactive;
 uint64_t ri_billed_system_time;
 uint64_t ri_serviced_system_time;
};
typedef struct rusage_info_v3 rusage_info_current;
# 325 "/usr/include/sys/resource.h" 3 4
struct rlimit {
 rlim_t rlim_cur;
 rlim_t rlim_max;
};
# 353 "/usr/include/sys/resource.h" 3 4
struct proc_rlimit_control_wakeupmon {
 uint32_t wm_flags;
 int32_t wm_rate;
};
# 384 "/usr/include/sys/resource.h" 3 4
extern "C" {
int getpriority(int, id_t);
int getiopolicy_np(int, int) __attribute__((availability(macosx,introduced=10.5)));
int getrlimit(int, struct rlimit *) __asm("_" "getrlimit" );
int getrusage(int, struct rusage *);
int setpriority(int, id_t, int);
int setiopolicy_np(int, int, int) __attribute__((availability(macosx,introduced=10.5)));
int setrlimit(int, const struct rlimit *) __asm("_" "setrlimit" );
}
# 111 "/usr/include/sys/wait.h" 2 3 4
# 186 "/usr/include/sys/wait.h" 3 4
# 1 "/usr/include/machine/endian.h" 1 3 4
# 35 "/usr/include/machine/endian.h" 3 4
# 1 "/usr/include/i386/endian.h" 1 3 4
# 99 "/usr/include/i386/endian.h" 3 4
# 1 "/usr/include/sys/_endian.h" 1 3 4
# 130 "/usr/include/sys/_endian.h" 3 4
# 1 "/usr/include/libkern/_OSByteOrder.h" 1 3 4
# 66 "/usr/include/libkern/_OSByteOrder.h" 3 4
# 1 "/usr/include/libkern/i386/_OSByteOrder.h" 1 3 4
# 44 "/usr/include/libkern/i386/_OSByteOrder.h" 3 4
static inline
__uint16_t
_OSSwapInt16(
    __uint16_t _data
)
{
    return ((__uint16_t)((_data << 8) | (_data >> 8)));
}
static inline
__uint32_t
_OSSwapInt32(
    __uint32_t _data
)
{
    return __builtin_bswap32(_data);
}
static inline
__uint64_t
_OSSwapInt64(
    __uint64_t _data
)
{
    return __builtin_bswap64(_data);
}
# 67 "/usr/include/libkern/_OSByteOrder.h" 2 3 4
# 131 "/usr/include/sys/_endian.h" 2 3 4
# 100 "/usr/include/i386/endian.h" 2 3 4
# 36 "/usr/include/machine/endian.h" 2 3 4
# 187 "/usr/include/sys/wait.h" 2 3 4
union wait {
 int w_status;
 struct {
  unsigned int w_Termsig:7,
    w_Coredump:1,
    w_Retcode:8,
    w_Filler:16;
 } w_T;
 struct {
  unsigned int w_Stopval:8,
    w_Stopsig:8,
    w_Filler:16;
 } w_S;
};
# 247 "/usr/include/sys/wait.h" 3 4
extern "C" {
pid_t wait(int *) __asm("_" "wait" );
pid_t waitpid(pid_t, int *, int) __asm("_" "waitpid" );
int waitid(idtype_t, id_t, siginfo_t *, int) __asm("_" "waitid" );
pid_t wait3(int *, int, struct rusage *);
pid_t wait4(pid_t, int *, int, struct rusage *);
}
# 66 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/alloca.h" 1 3 4
# 31 "/usr/include/alloca.h" 3 4
extern "C" {
void *alloca(size_t);
}
# 68 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_ct_rune_t.h" 1 3 4
# 31 "/usr/include/sys/_types/_ct_rune_t.h" 3 4
typedef __darwin_ct_rune_t ct_rune_t;
# 77 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_rune_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_rune_t.h" 3 4
typedef __darwin_rune_t rune_t;
# 78 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_wchar_t.h" 1 3 4
# 81 "/usr/include/stdlib.h" 2 3 4
typedef struct {
 int quot;
 int rem;
} div_t;
typedef struct {
 long quot;
 long rem;
} ldiv_t;
typedef struct {
 long long quot;
 long long rem;
} lldiv_t;
# 1 "/usr/include/sys/_types/_null.h" 1 3 4
# 100 "/usr/include/stdlib.h" 2 3 4
# 117 "/usr/include/stdlib.h" 3 4
extern int __mb_cur_max;
# 127 "/usr/include/stdlib.h" 3 4
extern "C" {
void abort(void) __attribute__((noreturn));
int abs(int) __attribute__((const));
int atexit(void (* _Nonnull)(void));
double atof(const char *);
int atoi(const char *);
long atol(const char *);
long long
  atoll(const char *);
void *bsearch(const void *__key, const void *__base, size_t __nel,
     size_t __width, int (* _Nonnull __compar)(const void *, const void *));
void *calloc(size_t __count, size_t __size) __attribute__((__warn_unused_result__));
div_t div(int, int) __attribute__((const));
void exit(int) __attribute__((noreturn));
void free(void *);
char *getenv(const char *);
long labs(long) __attribute__((const));
ldiv_t ldiv(long, long) __attribute__((const));
long long
  llabs(long long);
lldiv_t lldiv(long long, long long);
void *malloc(size_t __size) __attribute__((__warn_unused_result__));
int mblen(const char *__s, size_t __n);
size_t mbstowcs(wchar_t * , const char * , size_t);
int mbtowc(wchar_t * , const char * , size_t);
int posix_memalign(void **__memptr, size_t __alignment, size_t __size) __attribute__((availability(macosx,introduced=10.6)));
void qsort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int rand(void) ;
void *realloc(void *__ptr, size_t __size) __attribute__((__warn_unused_result__));
void srand(unsigned) ;
double strtod(const char *, char **) __asm("_" "strtod" );
float strtof(const char *, char **) __asm("_" "strtof" );
long strtol(const char *__str, char **__endptr, int __base);
long double
  strtold(const char *, char **);
long long
  strtoll(const char *__str, char **__endptr, int __base);
unsigned long
  strtoul(const char *__str, char **__endptr, int __base);
unsigned long long
  strtoull(const char *__str, char **__endptr, int __base);
# 185 "/usr/include/stdlib.h" 3 4
__attribute__((availability(macosx,introduced=10.0)))
__attribute__((availability(watchos,unavailable))) __attribute__((availability(tvos,unavailable)))
int system(const char *) __asm("_" "system" );
size_t wcstombs(char * , const wchar_t * , size_t);
int wctomb(char *, wchar_t);
void _Exit(int) __attribute__((noreturn));
long a64l(const char *);
double drand48(void);
char *ecvt(double, int, int *, int *);
double erand48(unsigned short[3]);
char *fcvt(double, int, int *, int *);
char *gcvt(double, int, char *);
int getsubopt(char **, char * const *, char **);
int grantpt(int);
char *initstate(unsigned, char *, size_t);
long jrand48(unsigned short[3]) ;
char *l64a(long);
void lcong48(unsigned short[7]);
long lrand48(void) ;
char *mktemp(char *);
int mkstemp(char *);
long mrand48(void) ;
long nrand48(unsigned short[3]) ;
int posix_openpt(int);
char *ptsname(int);
int putenv(char *) __asm("_" "putenv" );
long random(void) ;
int rand_r(unsigned *) ;
char *realpath(const char * , char * ) __asm("_" "realpath" "$DARWIN_EXTSN");
unsigned short
 *seed48(unsigned short[3]);
int setenv(const char * __name, const char * __value, int __overwrite) __asm("_" "setenv" );
void setkey(const char *) __asm("_" "setkey" );
char *setstate(const char *);
void srand48(long);
void srandom(unsigned);
int unlockpt(int);
int unsetenv(const char *) __asm("_" "unsetenv" );
# 1 "/usr/include/machine/types.h" 1 3 4
# 35 "/usr/include/machine/types.h" 3 4
# 1 "/usr/include/i386/types.h" 1 3 4
# 81 "/usr/include/i386/types.h" 3 4
# 1 "/usr/include/sys/_types/_u_int8_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int8_t.h" 3 4
typedef unsigned char u_int8_t;
# 82 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int16_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int16_t.h" 3 4
typedef unsigned short u_int16_t;
# 83 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int32_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int32_t.h" 3 4
typedef unsigned int u_int32_t;
# 84 "/usr/include/i386/types.h" 2 3 4
# 1 "/usr/include/sys/_types/_u_int64_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_u_int64_t.h" 3 4
typedef unsigned long long u_int64_t;
# 85 "/usr/include/i386/types.h" 2 3 4
typedef int64_t register_t;
# 97 "/usr/include/i386/types.h" 3 4
typedef u_int64_t user_addr_t;
typedef u_int64_t user_size_t;
typedef int64_t user_ssize_t;
typedef int64_t user_long_t;
typedef u_int64_t user_ulong_t;
typedef int64_t user_time_t;
typedef int64_t user_off_t;
typedef u_int64_t syscall_arg_t;
# 36 "/usr/include/machine/types.h" 2 3 4
# 252 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_dev_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_dev_t.h" 3 4
typedef __darwin_dev_t dev_t;
# 254 "/usr/include/stdlib.h" 2 3 4
# 1 "/usr/include/sys/_types/_mode_t.h" 1 3 4
# 30 "/usr/include/sys/_types/_mode_t.h" 3 4
typedef __darwin_mode_t mode_t;
# 255 "/usr/include/stdlib.h" 2 3 4
uint32_t arc4random(void);
void arc4random_addrandom(unsigned char * , int )
    __attribute__((availability(macosx,introduced=10.0))) __attribute__((availability(macosx,deprecated=10.12,message="use arc4random_stir")))
    __attribute__((availability(ios,introduced=2.0))) __attribute__((availability(ios,deprecated=10.0,message="use arc4random_stir")))
    __attribute__((availability(tvos,introduced=2.0))) __attribute__((availability(tvos,deprecated=10.0,message="use arc4random_stir")))
    __attribute__((availability(watchos,introduced=1.0))) __attribute__((availability(watchos,deprecated=3.0,message="use arc4random_stir")));
void arc4random_buf(void * __buf, size_t __nbytes) __attribute__((availability(macosx,introduced=10.7)));
void arc4random_stir(void);
uint32_t
  arc4random_uniform(uint32_t __upper_bound) __attribute__((availability(macosx,introduced=10.7)));
int atexit_b(void (^ _Nonnull)(void)) __attribute__((availability(macosx,introduced=10.6)));
void *bsearch_b(const void *__key, const void *__base, size_t __nel,
     size_t __width, int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
char *cgetcap(char *, const char *, int);
int cgetclose(void);
int cgetent(char **, char **, const char *);
int cgetfirst(char **, char **);
int cgetmatch(const char *, const char *);
int cgetnext(char **, char **);
int cgetnum(char *, const char *, long *);
int cgetset(const char *);
int cgetstr(char *, const char *, char **);
int cgetustr(char *, const char *, char **);
int daemon(int, int) __asm("_" "daemon" "$1050") __attribute__((availability(macosx,introduced=10.0,deprecated=10.5,message="Use posix_spawn APIs instead."))) __attribute__((availability(watchos,unavailable))) __attribute__((availability(tvos,unavailable)));
char *devname(dev_t, mode_t);
char *devname_r(dev_t, mode_t, char *buf, int len);
char *getbsize(int *, long *);
int getloadavg(double [], int);
const char
 *getprogname(void);
int heapsort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int heapsort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
int mergesort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *));
int mergesort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort(void *__base, size_t __nel, size_t __width,
     int (* _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void psort_r(void *__base, size_t __nel, size_t __width, void *,
     int (* _Nonnull __compar)(void *, const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void qsort_b(void *__base, size_t __nel, size_t __width,
     int (^ _Nonnull __compar)(const void *, const void *)) __attribute__((availability(macosx,introduced=10.6)));
void qsort_r(void *__base, size_t __nel, size_t __width, void *,
     int (* _Nonnull __compar)(void *, const void *, const void *));
int radixsort(const unsigned char **__base, int __nel, const unsigned char *__table,
     unsigned __endbyte);
void setprogname(const char *);
int sradixsort(const unsigned char **__base, int __nel, const unsigned char *__table,
     unsigned __endbyte);
void sranddev(void);
void srandomdev(void);
void *reallocf(void *__ptr, size_t __size);
long long
  strtoq(const char *__str, char **__endptr, int __base);
unsigned long long
  strtouq(const char *__str, char **__endptr, int __base);
extern char *suboptarg;
void *valloc(size_t);
}
# 95 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdlib.h" 2 3
extern "C++" {
# 111 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/stdlib.h" 3
inline __attribute__ ((__visibility__("hidden"), __always_inline__)) long abs( long __x) noexcept {return labs(__x);}
inline __attribute__ ((__visibility__("hidden"), __always_inline__)) long long abs(long long __x) noexcept {return llabs(__x);}
inline __attribute__ ((__visibility__("hidden"), __always_inline__)) ldiv_t div( long __x, long __y) noexcept {return ldiv(__x, __y);}
inline __attribute__ ((__visibility__("hidden"), __always_inline__)) lldiv_t div(long long __x, long long __y) noexcept {return lldiv(__x, __y);}
}
# 87 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 2 3
# 90 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 3
namespace std {inline namespace __1 {
using ::size_t;
using ::div_t;
using ::ldiv_t;
using ::lldiv_t;
using ::atof;
using ::atoi;
using ::atol;
using ::atoll;
using ::strtod;
using ::strtof;
using ::strtold;
using ::strtol;
using ::strtoll;
using ::strtoul;
using ::strtoull;
using ::rand;
using ::srand;
using ::calloc;
using ::free;
using ::malloc;
using ::realloc;
using ::abort;
using ::atexit;
using ::exit;
using ::_Exit;
using ::getenv;
using ::system;
using ::bsearch;
using ::qsort;
using ::abs;
using ::labs;
using ::llabs;
using ::div;
using ::ldiv;
using ::lldiv;
using ::mblen;
using ::mbtowc;
using ::wctomb;
using ::mbstowcs;
using ::wcstombs;
# 162 "/usr/local/Cellar/llvm/6.0.0/include/c++/v1/cstdlib" 3
} }
# 4 "/tmp/petsc-mjVUVK/config.setCompilers/conftest.cc" 2
Preprocess stderr before filtering::
Preprocess stderr after filtering::
              Popping language Cxx
================================================================================
TEST checkFortranCompiler from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:947)
TESTING: checkFortranCompiler from config.setCompilers(config/BuildSystem/config/setCompilers.py:947)
  Locate a functional Fortran compiler
Executing: mpif90 --help
stdout:
Usage: gfortran [options] file...
Options:
  -pass-exit-codes         Exit with highest error code from a phase.
  --help                   Display this information.
  --target-help            Display target specific command line options.
  --help={common|optimizers|params|target|warnings|[^]{joined|separate|undocumented}}[,...].
                           Display specific types of command line options.
  (Use '-v --help' to display command line options of sub-processes).
  --version                Display compiler version information.
  -dumpspecs               Display all of the built in spec strings.
  -dumpversion             Display the version of the compiler.
  -dumpmachine             Display the compiler's target processor.
  -print-search-dirs       Display the directories in the compiler's search path.
  -print-libgcc-file-name  Display the name of the compiler's companion library.
  -print-file-name=<lib>   Display the full path to library <lib>.
  -print-prog-name=<prog>  Display the full path to compiler component <prog>.
  -print-multiarch         Display the target's normalized GNU triplet, used as
                           a component in the library path.
  -print-multi-directory   Display the root directory for versions of libgcc.
  -print-multi-lib         Display the mapping between command line options and
                           multiple library search directories.
  -print-multi-os-directory Display the relative path to OS libraries.
  -print-sysroot           Display the target libraries directory.
  -print-sysroot-headers-suffix Display the sysroot suffix used to find headers.
  -Wa,<options>            Pass comma-separated <options> on to the assembler.
  -Wp,<options>            Pass comma-separated <options> on to the preprocessor.
  -Wl,<options>            Pass comma-separated <options> on to the linker.
  -Xassembler <arg>        Pass <arg> on to the assembler.
  -Xpreprocessor <arg>     Pass <arg> on to the preprocessor.
  -Xlinker <arg>           Pass <arg> on to the linker.
  -save-temps              Do not delete intermediate files.
  -save-temps=<arg>        Do not delete intermediate files.
  -no-canonical-prefixes   Do not canonicalize paths when building relative
                           prefixes to other gcc components.
  -pipe                    Use pipes rather than intermediate files.
  -time                    Time the execution of each subprocess.
  -specs=<file>            Override built-in specs with the contents of <file>.
  -std=<standard>          Assume that the input sources are for <standard>.
  --sysroot=<directory>    Use <directory> as the root directory for headers
                           and libraries.
  -B <directory>           Add <directory> to the compiler's search paths.
  -v                       Display the programs invoked by the compiler.
  -###                     Like -v but options quoted and commands not executed.
  -E                       Preprocess only; do not compile, assemble or link.
  -S                       Compile only; do not assemble or link.
  -c                       Compile and assemble, but do not link.
  -o <file>                Place the output into <file>.
  -pie                     Create a position independent executable.
  -shared                  Create a shared library.
  -x <language>            Specify the language of the following input files.
                           Permissible languages include: c c++ assembler none
                           'none' means revert to the default behavior of
                           guessing the language based on the file's extension.
Options starting with -g, -f, -m, -O, -W, or --param are automatically
 passed on to the various sub-processes invoked by gfortran.  In order to pass
 other options on to these processes the -W<letter> options must be used.
For bug reporting instructions, please see:
<https://github.com/Homebrew/homebrew-core/issues>.
Checking for program /usr/local/bin/mpif90...found
                Defined make macro "FC" to "mpif90"
                Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main


      end
Executing: mpif90 -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main


      end
                                                Pushing language C
                                                Popping language C
                                                Pushing language CUDA
                                                Popping language CUDA
                                                Pushing language Cxx
                                                Popping language Cxx
                                                Pushing language FC
                                                Popping language FC
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Executing: mpif90 -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main


      end
                        Pushing language FC
                        Popping language FC
Executing: mpif90  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o  -lpetsc-ufod4vtr9mqHvKIQiVAm
Possible ERROR while running linker: exit code 256
stderr:
ld: library not found for -lpetsc-ufod4vtr9mqHvKIQiVAm
collect2: error: ld returned 1 exit status
Executing: mpif90 -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.F90 
Successful compile:
Source:
      program main


      end
                            Pushing language FC
                            Popping language FC
Executing: mpif90  -o /tmp/petsc-mjVUVK/config.setCompilers/conftest    /tmp/petsc-mjVUVK/config.setCompilers/conftest.o 
Testing executable /tmp/petsc-mjVUVK/config.setCompilers/conftest to see if it can be run
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
Executing: /tmp/petsc-mjVUVK/config.setCompilers/conftest
                Popping language FC
================================================================================
TEST checkFortranComments from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:968)
TESTING: checkFortranComments from config.setCompilers(config/BuildSystem/config/setCompilers.py:968)
  Make sure fortran comment "!" works
              Pushing language FC
Executing: mpif90 -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.F90 
Successful compile:
Source:
! comment
      program main


      end
            Fortran comments can use ! in column 1
              Popping language FC
================================================================================
TEST checkLargeFileIO from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1085)
TESTING: checkLargeFileIO from config.setCompilers(config/BuildSystem/config/setCompilers.py:1085)
================================================================================
TEST checkArchiver from config.setCompilers(/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py:1184)
TESTING: checkArchiver from config.setCompilers(config/BuildSystem/config/setCompilers.py:1184)
  Check that the archiver exists and can make a library usable by the compiler
              Pushing language C
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib -c"
Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
ar: temporary file: No such file or directory
            Archiver is not functional
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib"
Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
ar: temporary file: No such file or directory
            Archiver is not functional
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/true...not found
Checking for program /usr/bin/true...found
                Defined make macro "RANLIB" to "/usr/bin/true"
Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
ar: temporary file: No such file or directory
            Archiver is not functional
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib -c"
Executing: /usr/bin/ar -X64 cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: illegal option -- X
usage:  ar -d [-TLsv] archive file ...
	ar -m [-TLsv] archive file ...
	ar -m [-abiTLsv] position archive file ...
	ar -p [-TLsv] archive [file ...]
	ar -q [-cTLsv] archive file ...
	ar -r [-cuTLsv] archive file ...
	ar -r [-abciuTLsv] position archive file ...
	ar -t [-TLsv] archive [file ...]
	ar -x [-ouTLsv] archive [file ...]
            Archiver is not functional
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/ranlib...not found
Checking for program /usr/bin/ranlib...found
                Defined make macro "RANLIB" to "/usr/bin/ranlib"
Executing: /usr/bin/ar -X64 cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: illegal option -- X
usage:  ar -d [-TLsv] archive file ...
	ar -m [-TLsv] archive file ...
	ar -m [-abiTLsv] position archive file ...
	ar -p [-TLsv] archive [file ...]
	ar -q [-cTLsv] archive file ...
	ar -r [-cuTLsv] archive file ...
	ar -r [-abciuTLsv] position archive file ...
	ar -t [-TLsv] archive [file ...]
	ar -x [-ouTLsv] archive [file ...]
            Archiver is not functional
Executing: ar -V
Executing: ar -V
                  Defined make macro "FAST_AR_FLAGS" to "Scq"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/ar...not found
Checking for program /usr/bin/ar...found
                Defined make macro "AR" to "/usr/bin/ar"
Checking for program /usr/local/bin/true...not found
Checking for program /usr/bin/true...found
                Defined make macro "RANLIB" to "/usr/bin/true"
Executing: /usr/bin/ar -X64 cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar: illegal option -- X
usage:  ar -d [-TLsv] archive file ...
	ar -m [-TLsv] archive file ...
	ar -m [-abiTLsv] position archive file ...
	ar -p [-TLsv] archive [file ...]
	ar -q [-cTLsv] archive file ...
	ar -r [-cuTLsv] archive file ...
	ar -r [-abciuTLsv] position archive file ...
	ar -t [-TLsv] archive [file ...]
	ar -x [-ouTLsv] archive [file ...]
            Archiver is not functional
                  Defined make macro "FAST_AR_FLAGS" to "-a -P512"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/win32fe...not found
Checking for program /usr/bin/win32fe...not found
Checking for program /bin/win32fe...not found
Checking for program /usr/sbin/win32fe...not found
Checking for program /sbin/win32fe...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/win32fe...not found
Checking for program /Library/TeX/texbin/win32fe...not found
Checking for program /Users/zhihui/win32fe...not found
Checking for program /Users/zhihui/igx/deps/srcs/petsc-3.8.3/bin/win32fe/win32fe...not found
                  Defined make macro "FAST_AR_FLAGS" to "-a"
Executing: mpicc -c -o /tmp/petsc-mjVUVK/config.setCompilers/conftest.o -I/tmp/petsc-mjVUVK/config.setCompilers   /tmp/petsc-mjVUVK/config.setCompilers/conftest.c 
Successful compile:
Source:
#include "confdefs.h"
#include "conffix.h"
int foo(int a) {
  return a+1;
}


Checking for program /usr/local/bin/win32fe...not found
Checking for program /usr/bin/win32fe...not found
Checking for program /bin/win32fe...not found
Checking for program /usr/sbin/win32fe...not found
Checking for program /sbin/win32fe...not found
Checking for program /Applications/VMware Fusion.app/Contents/Public/win32fe...not found
Checking for program /Library/TeX/texbin/win32fe...not found
Checking for program /Users/zhihui/win32fe...not found
Checking for program /Users/zhihui/igx/deps/srcs/petsc-3.8.3/bin/win32fe/win32fe...not found
              Popping language C
*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Could not find a suitable archiver.  Use --with-ar to specify an archiver.
*******************************************************************************
  File "./config/configure.py", line 393, in petsc_configure
    framework.configure(out = sys.stdout)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
    self.processChildren()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
    self.serialEvaluation(self.childGraph)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
    child.configure()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py", line 1692, in configure
    self.executeTest(self.checkArchiver)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
    ret = test(*args,**kargs)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/setCompilers.py", line 1237, in checkArchiver
    raise RuntimeError('Could not find a suitable archiver.  Use --with-ar to specify an archiver.')
================================================================================
Finishing Configure Run at Thu Oct 25 16:25:32 2018
================================================================================





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Send time: Friday, Oct 26, 2018 6:23 AM
To: "avatar"<648934229 at qq.com>; 
Cc: "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto  specify an archiver"



Its best to stick with one list - so will follow up on petsc-maint.

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Hi,
> 
> When I try to configure PETSc to connect Matlab with the following command
> ./petsc-3.8.3/configure ??with-matlab
> , it prompts
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: checkArchiver from config.setCompilers(config/BuildSystem/config/setCompilers.py:1184)                  *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Could not find a suitable archiver.  Use --with-ar to specify an archiver.
> *******************************************************************************
> 
> 
> 
> I am a newbie on Petsc. Could you please tell me how I should specify an archiver?  I am on MacOX system. I have tried many ways but it still does not work.
> 
> 
> Thank you very much for your help.
> 
> 
> Best,
> Joe
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/956c06c3/attachment-0001.html>

From balay at mcs.anl.gov  Thu Oct 25 17:57:45 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 22:57:45 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_014E19233CF8B25F19D46D24@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
Message-ID: <alpine.LFD.2.21.1810251752400.7009@asterix>

On Fri, 26 Oct 2018, avatar wrote:

> Hi Satish,
> 
> 
> Thank you very much for your quick response.
> 
> 
> The log file is as follow:
> 

>>>>>>
Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
ar: temporary file: No such file or directory
            Archiver is not functional

<<<<<<
This is a strange error.

What do you get when you do the following:

balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
-rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
-rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
__.SYMDEF SORTED
sizeof.o

balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
-rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
-rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
__.SYMDEF SORTED
sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 

Satish

From 648934229 at qq.com  Thu Oct 25 18:03:07 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 07:03:07 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251752400.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
Message-ID: <tencent_61AA8CAC07A9C90569DB8D7A@qq.com>

I could not do all things you posted below. I get these:


Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
-rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
ar: temporary file: No such file or directory
Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
ar: temporary file: No such file or directory
Scott-Grad-MacBook-Pro:benchmarks zhihui$





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



On Fri, 26 Oct 2018, avatar wrote:

> Hi Satish,
> 
> 
> Thank you very much for your quick response.
> 
> 
> The log file is as follow:
> 

>>>>>>
Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
Possible ERROR while running archiver: exit code 256
stderr:
ar: temporary file: No such file or directory
            Archiver is not functional

<<<<<<
This is a strange error.

What do you get when you do the following:

balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
-rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
-rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
__.SYMDEF SORTED
sizeof.o

balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
-rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
-rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
__.SYMDEF SORTED
sizeof.o
balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 

Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/bc07e396/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 18:19:20 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 23:19:20 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
Message-ID: <alpine.LFD.2.21.1810251818430.7009@asterix>

What about:


mpicc -c sizeof.c
/usr/bin/ar cr libconf1.a sizeof.o

Satish

On Fri, 26 Oct 2018, avatar wrote:

> I could not do all things you posted below. I get these:
> 
> 
> Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> ar: temporary file: No such file or directory
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> ar: temporary file: No such file or directory
> Scott-Grad-MacBook-Pro:benchmarks zhihui$
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Hi Satish,
> > 
> > 
> > Thank you very much for your quick response.
> > 
> > 
> > The log file is as follow:
> > 
> 
> >>>>>>
> Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> Possible ERROR while running archiver: exit code 256
> stderr:
> ar: temporary file: No such file or directory
>             Archiver is not functional
> 
> <<<<<<
> This is a strange error.
> 
> What do you get when you do the following:
> 
> balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> __.SYMDEF SORTED
> sizeof.o
> 
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> __.SYMDEF SORTED
> sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> 
> Satish


From 648934229 at qq.com  Thu Oct 25 18:22:22 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 07:22:22 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251818430.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
Message-ID: <tencent_639577384C3EB7845D018642@qq.com>

Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
ar: temporary file: No such file or directory





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



What about:


mpicc -c sizeof.c
/usr/bin/ar cr libconf1.a sizeof.o

Satish

On Fri, 26 Oct 2018, avatar wrote:

> I could not do all things you posted below. I get these:
> 
> 
> Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> ar: temporary file: No such file or directory
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> ar: temporary file: No such file or directory
> Scott-Grad-MacBook-Pro:benchmarks zhihui$
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Hi Satish,
> > 
> > 
> > Thank you very much for your quick response.
> > 
> > 
> > The log file is as follow:
> > 
> 
> >>>>>>
> Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> Possible ERROR while running archiver: exit code 256
> stderr:
> ar: temporary file: No such file or directory
>             Archiver is not functional
> 
> <<<<<<
> This is a strange error.
> 
> What do you get when you do the following:
> 
> balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> __.SYMDEF SORTED
> sizeof.o
> 
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> __.SYMDEF SORTED
> sizeof.o
> balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> 
> Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/84682f54/attachment-0001.html>

From balay at mcs.anl.gov  Thu Oct 25 18:34:00 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 23:34:00 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_639577384C3EB7845D018642@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
Message-ID: <alpine.LFD.2.21.1810251833190.7009@asterix>

How about:

TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> ar: temporary file: No such file or directory
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> What about:
> 
> 
> mpicc -c sizeof.c
> /usr/bin/ar cr libconf1.a sizeof.o
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > I could not do all things you posted below. I get these:
> > 
> > 
> > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > ar: temporary file: No such file or directory
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > ar: temporary file: No such file or directory
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Hi Satish,
> > > 
> > > 
> > > Thank you very much for your quick response.
> > > 
> > > 
> > > The log file is as follow:
> > > 
> > 
> > >>>>>>
> > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > Possible ERROR while running archiver: exit code 256
> > stderr:
> > ar: temporary file: No such file or directory
> >             Archiver is not functional
> > 
> > <<<<<<
> > This is a strange error.
> > 
> > What do you get when you do the following:
> > 
> > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > __.SYMDEF SORTED
> > sizeof.o
> > 
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > __.SYMDEF SORTED
> > sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > 
> > Satish


From 648934229 at qq.com  Thu Oct 25 18:37:34 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 07:37:34 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251833190.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
Message-ID: <tencent_58A588D62B3D43D106CA78FE@qq.com>

As follow. Then, what I should do next?


Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
Scott-Grad-MacBook-Pro:benchmarks zhihui$





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



How about:

TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> ar: temporary file: No such file or directory
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> What about:
> 
> 
> mpicc -c sizeof.c
> /usr/bin/ar cr libconf1.a sizeof.o
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > I could not do all things you posted below. I get these:
> > 
> > 
> > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > ar: temporary file: No such file or directory
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > ar: temporary file: No such file or directory
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Hi Satish,
> > > 
> > > 
> > > Thank you very much for your quick response.
> > > 
> > > 
> > > The log file is as follow:
> > > 
> > 
> > >>>>>>
> > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > Possible ERROR while running archiver: exit code 256
> > stderr:
> > ar: temporary file: No such file or directory
> >             Archiver is not functional
> > 
> > <<<<<<
> > This is a strange error.
> > 
> > What do you get when you do the following:
> > 
> > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > __.SYMDEF SORTED
> > sizeof.o
> > 
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > __.SYMDEF SORTED
> > sizeof.o
> > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > 
> > Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/6faeffda/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 18:40:18 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 23:40:18 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_58A588D62B3D43D106CA78FE@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
Message-ID: <alpine.LFD.2.21.1810251839070.7009@asterix>

Looks like it worked.

What do you have for:

echo $TMPDIR
ls -l libconf1.a
/usr/bin/ar t libconf1.a
TMPDIR=$PWD /usr/bin/ar t libconf1.a

Satish


On Fri, 26 Oct 2018, avatar wrote:

> As follow. Then, what I should do next?
> 
> 
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> Scott-Grad-MacBook-Pro:benchmarks zhihui$
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> How about:
> 
> TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> > Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> > Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> > MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> > MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> > PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> > PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> > ar: temporary file: No such file or directory
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > What about:
> > 
> > 
> > mpicc -c sizeof.c
> > /usr/bin/ar cr libconf1.a sizeof.o
> > 
> > Satish
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > I could not do all things you posted below. I get these:
> > > 
> > > 
> > > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > > ar: temporary file: No such file or directory
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > ar: temporary file: No such file or directory
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Hi Satish,
> > > > 
> > > > 
> > > > Thank you very much for your quick response.
> > > > 
> > > > 
> > > > The log file is as follow:
> > > > 
> > > 
> > > >>>>>>
> > > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > > Possible ERROR while running archiver: exit code 256
> > > stderr:
> > > ar: temporary file: No such file or directory
> > >             Archiver is not functional
> > > 
> > > <<<<<<
> > > This is a strange error.
> > > 
> > > What do you get when you do the following:
> > > 
> > > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > > __.SYMDEF SORTED
> > > sizeof.o
> > > 
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > > __.SYMDEF SORTED
> > > sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > > 
> > > Satish


From 648934229 at qq.com  Thu Oct 25 18:43:30 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 07:43:30 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251839070.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
Message-ID: <tencent_69BFB583499C0AC36930F396@qq.com>

These are what I got:
Scott-Grad-MacBook-Pro:benchmarks zhihui$ echo $TMPDIR
tmp
Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l libconf1.a
-rw-r--r--  1 zhihui  staff  1624 Oct 25 17:36 libconf1.a
Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar t libconf1.a
__.SYMDEF SORTED
sizeof.o
Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar t libconf1.a
__.SYMDEF SORTED
sizeof.o





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



Looks like it worked.

What do you have for:

echo $TMPDIR
ls -l libconf1.a
/usr/bin/ar t libconf1.a
TMPDIR=$PWD /usr/bin/ar t libconf1.a

Satish


On Fri, 26 Oct 2018, avatar wrote:

> As follow. Then, what I should do next?
> 
> 
> Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> Scott-Grad-MacBook-Pro:benchmarks zhihui$
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> How about:
> 
> TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> > Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> > Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> > MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> > MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> > PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> > PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> > ar: temporary file: No such file or directory
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > What about:
> > 
> > 
> > mpicc -c sizeof.c
> > /usr/bin/ar cr libconf1.a sizeof.o
> > 
> > Satish
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > I could not do all things you posted below. I get these:
> > > 
> > > 
> > > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > > ar: temporary file: No such file or directory
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > ar: temporary file: No such file or directory
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Hi Satish,
> > > > 
> > > > 
> > > > Thank you very much for your quick response.
> > > > 
> > > > 
> > > > The log file is as follow:
> > > > 
> > > 
> > > >>>>>>
> > > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > > Possible ERROR while running archiver: exit code 256
> > > stderr:
> > > ar: temporary file: No such file or directory
> > >             Archiver is not functional
> > > 
> > > <<<<<<
> > > This is a strange error.
> > > 
> > > What do you get when you do the following:
> > > 
> > > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > > __.SYMDEF SORTED
> > > sizeof.o
> > > 
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > > __.SYMDEF SORTED
> > > sizeof.o
> > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > > 
> > > Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/080ca755/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 18:44:53 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 23:44:53 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251839070.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
Message-ID: <alpine.LFD.2.21.1810251841160.7009@asterix>

I see - you have:

>>
TMPDIR=tmp
<<

Did you set this in your ~/.bashrc or somewhere? This is wrong and is breaking tools.
OSX should setup something likethe following for you.

petsc-mini:~ balay$ echo $TMPDIR
/var/folders/lw/hyrlb1051p9fj96qkktfvhvm0000gn/T/

Remove it - and retry building PETSc.

Satish

On Thu, 25 Oct 2018, Satish Balay wrote:

> Looks like it worked.
> 
> What do you have for:
> 
> echo $TMPDIR
> ls -l libconf1.a
> /usr/bin/ar t libconf1.a
> TMPDIR=$PWD /usr/bin/ar t libconf1.a
> 
> Satish
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > As follow. Then, what I should do next?
> > 
> > 
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > How about:
> > 
> > TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > 
> > Satish
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> > > Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> > > Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> > > MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> > > MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> > > PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> > > PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> > > ar: temporary file: No such file or directory
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > What about:
> > > 
> > > 
> > > mpicc -c sizeof.c
> > > /usr/bin/ar cr libconf1.a sizeof.o
> > > 
> > > Satish
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > I could not do all things you posted below. I get these:
> > > > 
> > > > 
> > > > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > > > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > > > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > > > ar: temporary file: No such file or directory
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > ar: temporary file: No such file or directory
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > ------------------ Original ------------------
> > > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > > Date:  Oct 26, 2018
> > > > To:  "avatar"<648934229 at qq.com>; 
> > > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > > 
> > > > 
> > > > 
> > > > On Fri, 26 Oct 2018, avatar wrote:
> > > > 
> > > > > Hi Satish,
> > > > > 
> > > > > 
> > > > > Thank you very much for your quick response.
> > > > > 
> > > > > 
> > > > > The log file is as follow:
> > > > > 
> > > > 
> > > > >>>>>>
> > > > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > > > Possible ERROR while running archiver: exit code 256
> > > > stderr:
> > > > ar: temporary file: No such file or directory
> > > >             Archiver is not functional
> > > > 
> > > > <<<<<<
> > > > This is a strange error.
> > > > 
> > > > What do you get when you do the following:
> > > > 
> > > > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > > > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > > > __.SYMDEF SORTED
> > > > sizeof.o
> > > > 
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > > > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > > > __.SYMDEF SORTED
> > > > sizeof.o
> > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > > > 
> > > > Satish
> 
> 


From balay at mcs.anl.gov  Thu Oct 25 18:48:21 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Thu, 25 Oct 2018 23:48:21 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251841160.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
Message-ID: <alpine.LFD.2.21.1810251846440.7009@asterix>

To clarify, I meant: remove 'code that is changing the value of TMPDIR to 'tmp'

Satish

 On Thu, 25 Oct 2018, Satish Balay wrote:

> I see - you have:
> 
> >>
> TMPDIR=tmp
> <<
> 
> Did you set this in your ~/.bashrc or somewhere? This is wrong and is breaking tools.
> OSX should setup something likethe following for you.
> 
> petsc-mini:~ balay$ echo $TMPDIR
> /var/folders/lw/hyrlb1051p9fj96qkktfvhvm0000gn/T/
> 
> Remove it - and retry building PETSc.
> 
> Satish
> 
> On Thu, 25 Oct 2018, Satish Balay wrote:
> 
> > Looks like it worked.
> > 
> > What do you have for:
> > 
> > echo $TMPDIR
> > ls -l libconf1.a
> > /usr/bin/ar t libconf1.a
> > TMPDIR=$PWD /usr/bin/ar t libconf1.a
> > 
> > Satish
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > As follow. Then, what I should do next?
> > > 
> > > 
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > How about:
> > > 
> > > TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > > 
> > > Satish
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> > > > Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> > > > Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> > > > MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> > > > MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> > > > PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> > > > PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> > > > ar: temporary file: No such file or directory
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > ------------------ Original ------------------
> > > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > > Date:  Oct 26, 2018
> > > > To:  "avatar"<648934229 at qq.com>; 
> > > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > > 
> > > > 
> > > > 
> > > > What about:
> > > > 
> > > > 
> > > > mpicc -c sizeof.c
> > > > /usr/bin/ar cr libconf1.a sizeof.o
> > > > 
> > > > Satish
> > > > 
> > > > On Fri, 26 Oct 2018, avatar wrote:
> > > > 
> > > > > I could not do all things you posted below. I get these:
> > > > > 
> > > > > 
> > > > > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > > > > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > > > > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > > > > ar: temporary file: No such file or directory
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > > ar: temporary file: No such file or directory
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > ------------------ Original ------------------
> > > > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > > > Date:  Oct 26, 2018
> > > > > To:  "avatar"<648934229 at qq.com>; 
> > > > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > > > 
> > > > > 
> > > > > 
> > > > > On Fri, 26 Oct 2018, avatar wrote:
> > > > > 
> > > > > > Hi Satish,
> > > > > > 
> > > > > > 
> > > > > > Thank you very much for your quick response.
> > > > > > 
> > > > > > 
> > > > > > The log file is as follow:
> > > > > > 
> > > > > 
> > > > > >>>>>>
> > > > > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > > > > Possible ERROR while running archiver: exit code 256
> > > > > stderr:
> > > > > ar: temporary file: No such file or directory
> > > > >             Archiver is not functional
> > > > > 
> > > > > <<<<<<
> > > > > This is a strange error.
> > > > > 
> > > > > What do you get when you do the following:
> > > > > 
> > > > > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > > > > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > > > > __.SYMDEF SORTED
> > > > > sizeof.o
> > > > > 
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > > > > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > > > > __.SYMDEF SORTED
> > > > > sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > > > > 
> > > > > Satish
> > 
> > 
> 
> 


From 648934229 at qq.com  Thu Oct 25 18:53:33 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 07:53:33 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251846440.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
Message-ID: <tencent_290A75DD41C7D04B4B36A320@qq.com>

Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.




------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Cc:  "avatar"<648934229 at qq.com>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



To clarify, I meant: remove 'code that is changing the value of TMPDIR to 'tmp'

Satish

 On Thu, 25 Oct 2018, Satish Balay wrote:

> I see - you have:
> 
> >>
> TMPDIR=tmp
> <<
> 
> Did you set this in your ~/.bashrc or somewhere? This is wrong and is breaking tools.
> OSX should setup something likethe following for you.
> 
> petsc-mini:~ balay$ echo $TMPDIR
> /var/folders/lw/hyrlb1051p9fj96qkktfvhvm0000gn/T/
> 
> Remove it - and retry building PETSc.
> 
> Satish
> 
> On Thu, 25 Oct 2018, Satish Balay wrote:
> 
> > Looks like it worked.
> > 
> > What do you have for:
> > 
> > echo $TMPDIR
> > ls -l libconf1.a
> > /usr/bin/ar t libconf1.a
> > TMPDIR=$PWD /usr/bin/ar t libconf1.a
> > 
> > Satish
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > As follow. Then, what I should do next?
> > > 
> > > 
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > How about:
> > > 
> > > TMPDIR=$PWD /usr/bin/ar cr libconf1.a sizeof.o
> > > 
> > > Satish
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls
> > > > Index.c			PetscGetCPUTime.c	PetscMemcmp.c		PetscTime.c		benchmarkExample.py	sizeof.c
> > > > Index.c.html		PetscGetCPUTime.c.html	PetscMemcmp.c.html	PetscTime.c.html	daemon.py		sizeof.o
> > > > MPI_Wtime.c		PetscGetTime.c		PetscMemcpy.c		PetscVecNorm.c		index.html		streams
> > > > MPI_Wtime.c.html	PetscGetTime.c.html	PetscMemcpy.c.html	PetscVecNorm.c.html	libconf1.a
> > > > PLogEvent.c		PetscMalloc.c		PetscMemzero.c		benchmarkAssembly.py	makefile
> > > > PLogEvent.c.html	PetscMalloc.c.html	PetscMemzero.c.html	benchmarkBatch.py	makefile.html
> > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr libconf1.a sizeof.o
> > > > ar: temporary file: No such file or directory
> > > > 
> > > > 
> > > > 
> > > > 
> > > > 
> > > > ------------------ Original ------------------
> > > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > > Date:  Oct 26, 2018
> > > > To:  "avatar"<648934229 at qq.com>; 
> > > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > > 
> > > > 
> > > > 
> > > > What about:
> > > > 
> > > > 
> > > > mpicc -c sizeof.c
> > > > /usr/bin/ar cr libconf1.a sizeof.o
> > > > 
> > > > Satish
> > > > 
> > > > On Fri, 26 Oct 2018, avatar wrote:
> > > > 
> > > > > I could not do all things you posted below. I get these:
> > > > > 
> > > > > 
> > > > > Scott-Grad-MacBook-Pro:petsc-3.8.3 zhihui$ cd src
> > > > > Scott-Grad-MacBook-Pro:src zhihui$ cd benchmarks/
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ mpicc -c sizeof.c
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ ls -l sizeof.o
> > > > > -rw-r--r--  1 zhihui  staff  1452 Oct 25 17:00 sizeof.o
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a sizeof.o
> > > > > ar: temporary file: No such file or directory
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > > ar: temporary file: No such file or directory
> > > > > Scott-Grad-MacBook-Pro:benchmarks zhihui$
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > 
> > > > > ------------------ Original ------------------
> > > > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > > > Date:  Oct 26, 2018
> > > > > To:  "avatar"<648934229 at qq.com>; 
> > > > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > > > 
> > > > > 
> > > > > 
> > > > > On Fri, 26 Oct 2018, avatar wrote:
> > > > > 
> > > > > > Hi Satish,
> > > > > > 
> > > > > > 
> > > > > > Thank you very much for your quick response.
> > > > > > 
> > > > > > 
> > > > > > The log file is as follow:
> > > > > > 
> > > > > 
> > > > > >>>>>>
> > > > > Executing: /usr/bin/ar cr /tmp/petsc-mjVUVK/config.setCompilers/libconf1.a /tmp/petsc-mjVUVK/config.setCompilers/conf1.o
> > > > > Possible ERROR while running archiver: exit code 256
> > > > > stderr:
> > > > > ar: temporary file: No such file or directory
> > > > >             Archiver is not functional
> > > > > 
> > > > > <<<<<<
> > > > > This is a strange error.
> > > > > 
> > > > > What do you get when you do the following:
> > > > > 
> > > > > balay at jpro^~/petsc(maint-3.8) $ cd src/benchmarks/
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ mpicc -c sizeof.c 
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l sizeof.o
> > > > > -rw-r--r--  1 balay  staff  3036 Oct 25 17:54 sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf1.a  sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf1.a
> > > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf1.a
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf1.a
> > > > > __.SYMDEF SORTED
> > > > > sizeof.o
> > > > > 
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ cp sizeof.o /tmp/
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/sizeof.o 
> > > > > -rw-r--r--  1 balay  wheel  3036 Oct 25 17:55 /tmp/sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ /usr/bin/ar cr /tmp/libconf2.a /tmp/sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ls -l /tmp/libconf2.a
> > > > > -rw-r--r--  1 balay  wheel  3224 Oct 25 17:55 /tmp/libconf2.a
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ ar t /tmp/libconf2.a
> > > > > __.SYMDEF SORTED
> > > > > sizeof.o
> > > > > balay at jpro^~/petsc/src/benchmarks(maint-3.8) $ 
> > > > > 
> > > > > Satish
> > 
> > 
> 
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/f43967ed/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 19:05:03 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 00:05:03 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_290A75DD41C7D04B4B36A320@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
Message-ID: <alpine.LFD.2.21.1810251859390.7009@asterix>

On Fri, 26 Oct 2018, avatar wrote:

> Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.

I guess you need to read up on some basic unix and system admin.

You probably meant to use:

export TMPDIR=/tmp

However  - you have:

export TMPDIR=tmp

To overcome this problem - you do:

export TMPDIR=/tmp

Satish

From 648934229 at qq.com  Thu Oct 25 19:18:24 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 08:18:24 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251859390.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
Message-ID: <tencent_12E9BAB855E3989758F79B2A@qq.com>

Thank Satish. It worked as you said. But when I rebuild petsc using 
./configure --with-matlab
It prompts 
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)         *******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Could not find a functional Matlab
Run with --with-matlab-dir=Matlabrootdir if you know where it is
*******************************************************************************



Then, I did the following. But it shows that "/Users/zhihui/Applications/MATLAB_R2018a.app/bin/matlab" nonexistent. However, it does exist as
Scott-Grad-MBP:bin zhihui$ pwd
/Applications/MATLAB_R2018a.app/bin
Scott-Grad-MBP:bin zhihui$ ls
activate_matlab.sh	lcdata_utf8.xml		matlab-glselector.sh	mexsh
deactivate_matlab.sh	ldd			matopts.sh		optsetup.sh
engopts.sh		m3iregistry		mex			registry
lcdata.xml		maci64			mexext			util
lcdata.xsd		matlab			mexopts.sh



Do you know what it is going on here?




Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=./../../../../Applications/MATLAB_R2018a/bin/matlab
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
*******************************************************************************
                ERROR in COMMAND LINE ARGUMENT to ./configure
-------------------------------------------------------------------------------
Nonexistent directory: /Users/zhihui/Applications/MATLAB_R2018a/bin/matlab for key with-matlab-dir
*******************************************************************************




  File "./config/configure.py", line 390, in petsc_configure
    framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 110, in __init__
    self.createChildren()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 321, in createChildren
    self.getChild(moduleName)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 306, in getChild
    config.setupDependencies(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 107, in setupDependencies
    obj = self.registerPythonFile(package,'config.packages')
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 63, in registerPythonFile
    utilityObj                             = self.framework.require(directory+utilityName, self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 326, in require
    config = self.getChild(moduleName, keywordArgs)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 304, in getChild
    config.setup()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 101, in setup
    logger.Logger.setup(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/logger.py", line 85, in setup
    args.ArgumentProcessor.setup(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/args.py", line 75, in setup
    self.setupArguments(self.argDB)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 85, in setupArguments
    self.setupHelp(self.help)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 15, in setupHelp
    help.addArgument('MATLAB', '-with-matlab-dir=<root dir>', nargs.ArgDir(None, None, 'Specify the root directory of the Matlab installation'))
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/help.py", line 107, in addArgument
    self.argDB.setType(self.getArgName(name), argType, forceLocal = 1)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/RDict.py", line 213, in setType
    value.setValue(v.getValue())
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/nargs.py", line 324, in setValue
    raise ValueError('Nonexistent directory: '+str(value)+' for key '+str(self.key))







------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



On Fri, 26 Oct 2018, avatar wrote:

> Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.

I guess you need to read up on some basic unix and system admin.

You probably meant to use:

export TMPDIR=/tmp

However  - you have:

export TMPDIR=tmp

To overcome this problem - you do:

export TMPDIR=/tmp

Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/ccd6791d/attachment-0001.html>

From balay at mcs.anl.gov  Thu Oct 25 19:25:53 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 00:25:53 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_12E9BAB855E3989758F79B2A@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
Message-ID: <alpine.LFD.2.21.1810251924290.7009@asterix>

you need

--with-matlab-dir=/Users/zhihui/Applications/MATLAB_R2018a.app

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Thank Satish. It worked as you said. But when I rebuild petsc using 
> ./configure --with-matlab
> It prompts 
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)         *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Could not find a functional Matlab
> Run with --with-matlab-dir=Matlabrootdir if you know where it is
> *******************************************************************************
> 
> 
> 
> Then, I did the following. But it shows that "/Users/zhihui/Applications/MATLAB_R2018a.app/bin/matlab" nonexistent. However, it does exist as
> Scott-Grad-MBP:bin zhihui$ pwd
> /Applications/MATLAB_R2018a.app/bin
> Scott-Grad-MBP:bin zhihui$ ls
> activate_matlab.sh	lcdata_utf8.xml		matlab-glselector.sh	mexsh
> deactivate_matlab.sh	ldd			matopts.sh		optsetup.sh
> engopts.sh		m3iregistry		mex			registry
> lcdata.xml		maci64			mexext			util
> lcdata.xsd		matlab			mexopts.sh
> 
> 
> 
> Do you know what it is going on here?
> 
> 
> 
> 
> Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=./../../../../Applications/MATLAB_R2018a/bin/matlab
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> *******************************************************************************
>                 ERROR in COMMAND LINE ARGUMENT to ./configure
> -------------------------------------------------------------------------------
> Nonexistent directory: /Users/zhihui/Applications/MATLAB_R2018a/bin/matlab for key with-matlab-dir
> *******************************************************************************
> 
> 
> 
> 
>   File "./config/configure.py", line 390, in petsc_configure
>     framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 110, in __init__
>     self.createChildren()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 321, in createChildren
>     self.getChild(moduleName)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 306, in getChild
>     config.setupDependencies(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 107, in setupDependencies
>     obj = self.registerPythonFile(package,'config.packages')
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 63, in registerPythonFile
>     utilityObj                             = self.framework.require(directory+utilityName, self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 326, in require
>     config = self.getChild(moduleName, keywordArgs)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 304, in getChild
>     config.setup()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 101, in setup
>     logger.Logger.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/logger.py", line 85, in setup
>     args.ArgumentProcessor.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/args.py", line 75, in setup
>     self.setupArguments(self.argDB)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 85, in setupArguments
>     self.setupHelp(self.help)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 15, in setupHelp
>     help.addArgument('MATLAB', '-with-matlab-dir=<root dir>', nargs.ArgDir(None, None, 'Specify the root directory of the Matlab installation'))
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/help.py", line 107, in addArgument
>     self.argDB.setType(self.getArgName(name), argType, forceLocal = 1)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/RDict.py", line 213, in setType
>     value.setValue(v.getValue())
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/nargs.py", line 324, in setValue
>     raise ValueError('Nonexistent directory: '+str(value)+' for key '+str(self.key))
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.
> 
> I guess you need to read up on some basic unix and system admin.
> 
> You probably meant to use:
> 
> export TMPDIR=/tmp
> 
> However  - you have:
> 
> export TMPDIR=tmp
> 
> To overcome this problem - you do:
> 
> export TMPDIR=/tmp
> 
> Satish


From 648934229 at qq.com  Thu Oct 25 19:34:52 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 08:34:52 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251924290.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
Message-ID: <tencent_60C6C8AB4EB344A412F12A08@qq.com>

still shows these


Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Users/zhihui/Applications/MATLAB_R2018a.app
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
*******************************************************************************
                ERROR in COMMAND LINE ARGUMENT to ./configure
-------------------------------------------------------------------------------
Nonexistent directory: /Users/zhihui/Applications/MATLAB_R2018a.app for key with-matlab-dir
*******************************************************************************




  File "./config/configure.py", line 390, in petsc_configure
    framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 110, in __init__
    self.createChildren()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 321, in createChildren
    self.getChild(moduleName)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 306, in getChild
    config.setupDependencies(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 107, in setupDependencies
    obj = self.registerPythonFile(package,'config.packages')
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 63, in registerPythonFile
    utilityObj                             = self.framework.require(directory+utilityName, self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 326, in require
    config = self.getChild(moduleName, keywordArgs)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 304, in getChild
    config.setup()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 101, in setup
    logger.Logger.setup(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/logger.py", line 85, in setup
    args.ArgumentProcessor.setup(self)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/args.py", line 75, in setup
    self.setupArguments(self.argDB)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 85, in setupArguments
    self.setupHelp(self.help)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 15, in setupHelp
    help.addArgument('MATLAB', '-with-matlab-dir=<root dir>', nargs.ArgDir(None, None, 'Specify the root directory of the Matlab installation'))
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/help.py", line 107, in addArgument
    self.argDB.setType(self.getArgName(name), argType, forceLocal = 1)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/RDict.py", line 213, in setType
    value.setValue(v.getValue())
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/nargs.py", line 324, in setValue
    raise ValueError('Nonexistent directory: '+str(value)+' for key '+str(self.key))







------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



you need

--with-matlab-dir=/Users/zhihui/Applications/MATLAB_R2018a.app

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Thank Satish. It worked as you said. But when I rebuild petsc using 
> ./configure --with-matlab
> It prompts 
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)         *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Could not find a functional Matlab
> Run with --with-matlab-dir=Matlabrootdir if you know where it is
> *******************************************************************************
> 
> 
> 
> Then, I did the following. But it shows that "/Users/zhihui/Applications/MATLAB_R2018a.app/bin/matlab" nonexistent. However, it does exist as
> Scott-Grad-MBP:bin zhihui$ pwd
> /Applications/MATLAB_R2018a.app/bin
> Scott-Grad-MBP:bin zhihui$ ls
> activate_matlab.sh	lcdata_utf8.xml		matlab-glselector.sh	mexsh
> deactivate_matlab.sh	ldd			matopts.sh		optsetup.sh
> engopts.sh		m3iregistry		mex			registry
> lcdata.xml		maci64			mexext			util
> lcdata.xsd		matlab			mexopts.sh
> 
> 
> 
> Do you know what it is going on here?
> 
> 
> 
> 
> Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=./../../../../Applications/MATLAB_R2018a/bin/matlab
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> *******************************************************************************
>                 ERROR in COMMAND LINE ARGUMENT to ./configure
> -------------------------------------------------------------------------------
> Nonexistent directory: /Users/zhihui/Applications/MATLAB_R2018a/bin/matlab for key with-matlab-dir
> *******************************************************************************
> 
> 
> 
> 
>   File "./config/configure.py", line 390, in petsc_configure
>     framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 110, in __init__
>     self.createChildren()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 321, in createChildren
>     self.getChild(moduleName)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 306, in getChild
>     config.setupDependencies(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 107, in setupDependencies
>     obj = self.registerPythonFile(package,'config.packages')
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 63, in registerPythonFile
>     utilityObj                             = self.framework.require(directory+utilityName, self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 326, in require
>     config = self.getChild(moduleName, keywordArgs)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 304, in getChild
>     config.setup()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 101, in setup
>     logger.Logger.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/logger.py", line 85, in setup
>     args.ArgumentProcessor.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/args.py", line 75, in setup
>     self.setupArguments(self.argDB)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 85, in setupArguments
>     self.setupHelp(self.help)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 15, in setupHelp
>     help.addArgument('MATLAB', '-with-matlab-dir=<root dir>', nargs.ArgDir(None, None, 'Specify the root directory of the Matlab installation'))
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/help.py", line 107, in addArgument
>     self.argDB.setType(self.getArgName(name), argType, forceLocal = 1)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/RDict.py", line 213, in setType
>     value.setValue(v.getValue())
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/nargs.py", line 324, in setValue
>     raise ValueError('Nonexistent directory: '+str(value)+' for key '+str(self.key))
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.
> 
> I guess you need to read up on some basic unix and system admin.
> 
> You probably meant to use:
> 
> export TMPDIR=/tmp
> 
> However  - you have:
> 
> export TMPDIR=tmp
> 
> To overcome this problem - you do:
> 
> export TMPDIR=/tmp
> 
> Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/43629b25/attachment-0001.html>

From 648934229 at qq.com  Thu Oct 25 19:37:10 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 08:37:10 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251924290.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
Message-ID: <tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>

When I do the following, it shows
Scott-Grad-MBP:~ zhihui$ cd Applications/
Scott-Grad-MBP:Applications zhihui$ ls
Chrome Apps.localized
Scott-Grad-MBP:Applications zhihui$



From these, we can see there is no MATALB_R2018a.app in Applications. But when I get into bin, and do the following, it does show
Scott-Grad-MBP:bin zhihui$ pwd
/Applications/MATLAB_R2018a.app/bin
Scott-Grad-MBP:bin zhihui$ ls
activate_matlab.sh	lcdata_utf8.xml		matlab-glselector.sh	mexsh
deactivate_matlab.sh	ldd			matopts.sh		optsetup.sh
engopts.sh		m3iregistry		mex			registry
lcdata.xml		maci64			mexext			util
lcdata.xsd		matlab			mexopts.sh





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



you need

--with-matlab-dir=/Users/zhihui/Applications/MATLAB_R2018a.app

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Thank Satish. It worked as you said. But when I rebuild petsc using 
> ./configure --with-matlab
> It prompts 
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)         *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> Could not find a functional Matlab
> Run with --with-matlab-dir=Matlabrootdir if you know where it is
> *******************************************************************************
> 
> 
> 
> Then, I did the following. But it shows that "/Users/zhihui/Applications/MATLAB_R2018a.app/bin/matlab" nonexistent. However, it does exist as
> Scott-Grad-MBP:bin zhihui$ pwd
> /Applications/MATLAB_R2018a.app/bin
> Scott-Grad-MBP:bin zhihui$ ls
> activate_matlab.sh	lcdata_utf8.xml		matlab-glselector.sh	mexsh
> deactivate_matlab.sh	ldd			matopts.sh		optsetup.sh
> engopts.sh		m3iregistry		mex			registry
> lcdata.xml		maci64			mexext			util
> lcdata.xsd		matlab			mexopts.sh
> 
> 
> 
> Do you know what it is going on here?
> 
> 
> 
> 
> Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=./../../../../Applications/MATLAB_R2018a/bin/matlab
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> *******************************************************************************
>                 ERROR in COMMAND LINE ARGUMENT to ./configure
> -------------------------------------------------------------------------------
> Nonexistent directory: /Users/zhihui/Applications/MATLAB_R2018a/bin/matlab for key with-matlab-dir
> *******************************************************************************
> 
> 
> 
> 
>   File "./config/configure.py", line 390, in petsc_configure
>     framework = config.framework.Framework(['--configModules=PETSc.Configure','--optionsModule=config.compilerOptions']+sys.argv[1:], loadArgDB = 0)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 110, in __init__
>     self.createChildren()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 321, in createChildren
>     self.getChild(moduleName)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 306, in getChild
>     config.setupDependencies(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 107, in setupDependencies
>     obj = self.registerPythonFile(package,'config.packages')
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/PETSc/Configure.py", line 63, in registerPythonFile
>     utilityObj                             = self.framework.require(directory+utilityName, self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 326, in require
>     config = self.getChild(moduleName, keywordArgs)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 304, in getChild
>     config.setup()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 101, in setup
>     logger.Logger.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/logger.py", line 85, in setup
>     args.ArgumentProcessor.setup(self)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/args.py", line 75, in setup
>     self.setupArguments(self.argDB)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/script.py", line 85, in setupArguments
>     self.setupHelp(self.help)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 15, in setupHelp
>     help.addArgument('MATLAB', '-with-matlab-dir=<root dir>', nargs.ArgDir(None, None, 'Specify the root directory of the Matlab installation'))
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/help.py", line 107, in addArgument
>     self.argDB.setType(self.getArgName(name), argType, forceLocal = 1)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/RDict.py", line 213, in setType
>     value.setValue(v.getValue())
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/nargs.py", line 324, in setValue
>     raise ValueError('Nonexistent directory: '+str(value)+' for key '+str(self.key))
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Is there other way to overcome this problem? Because if I don't set TMPDIR as \tmp. My other project will break. And actually I don't even know where my project set up the TMPDIR value.
> 
> I guess you need to read up on some basic unix and system admin.
> 
> You probably meant to use:
> 
> export TMPDIR=/tmp
> 
> However  - you have:
> 
> export TMPDIR=tmp
> 
> To overcome this problem - you do:
> 
> export TMPDIR=/tmp
> 
> Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/bce77a8e/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 19:43:06 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 00:43:06 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
Message-ID: <alpine.LFD.2.21.1810251942320.7009@asterix>

On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MBP:bin zhihui$ pwd
> /Applications/MATLAB_R2018a.app/bin


Sorry - you need

--with-matlab-dir=/Applications/MATLAB_R2018a.app/

Satish


From 648934229 at qq.com  Thu Oct 25 19:53:45 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 08:53:45 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251942320.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
Message-ID: <tencent_59464B694F19FC1F3EF5F7E5@qq.com>

Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
*******************************************************************************



Part of the log file as follow


#ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
#define PETSC_HAVE_MPI_REDUCE_SCATTER 1
#endif


#ifndef PETSC_HAVE_MPI_COMBINER_DUP
#define PETSC_HAVE_MPI_COMBINER_DUP 1
#endif


#ifndef PETSC_HAVE_MPIIO
#define PETSC_HAVE_MPIIO 1
#endif


#ifndef PETSC_HAVE_MPI_COMM_SPAWN
#define PETSC_HAVE_MPI_COMM_SPAWN 1
#endif


#ifndef PETSC_HAVE_MPI_FINT
#define PETSC_HAVE_MPI_FINT 1
#endif


#ifndef PETSC_HAVE_MPI_IBARRIER
#define PETSC_HAVE_MPI_IBARRIER 1
#endif


#ifndef PETSC_HAVE_MPI_ALLTOALLW
#define PETSC_HAVE_MPI_ALLTOALLW 1
#endif


#ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
#define PETSC_HAVE_OMPI_RELEASE_VERSION 2
#endif


#ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
#define PETSC_HAVE_MPI_REDUCE_LOCAL 1
#endif


#ifndef PETSC_HAVE_MPI_REPLACE
#define PETSC_HAVE_MPI_REPLACE 1
#endif


#ifndef PETSC_HAVE_MPI_EXSCAN
#define PETSC_HAVE_MPI_EXSCAN 1
#endif


#ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
#define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
#endif


#ifndef PETSC_HAVE_MPI_FINALIZED
#define PETSC_HAVE_MPI_FINALIZED 1
#endif


#ifndef PETSC_USE_INFO
#define PETSC_USE_INFO 1
#endif


#ifndef PETSC_Alignx
#define PETSC_Alignx(a,b)   
#endif


#ifndef PETSC_USE_BACKWARD_LOOP
#define PETSC_USE_BACKWARD_LOOP 1
#endif


#ifndef PETSC_USE_DEBUG
#define PETSC_USE_DEBUG 1
#endif


#ifndef PETSC_USE_LOG
#define PETSC_USE_LOG 1
#endif


#ifndef PETSC_IS_COLOR_VALUE_TYPE_F
#define PETSC_IS_COLOR_VALUE_TYPE_F integer2
#endif


#ifndef PETSC_IS_COLOR_VALUE_TYPE
#define PETSC_IS_COLOR_VALUE_TYPE short
#endif


#ifndef PETSC_USE_CTABLE
#define PETSC_USE_CTABLE 1
#endif


#ifndef PETSC_MEMALIGN
#define PETSC_MEMALIGN 16
#endif


#ifndef PETSC_LEVEL1_DCACHE_LINESIZE
#define PETSC_LEVEL1_DCACHE_LINESIZE 32
#endif


#ifndef PETSC_LEVEL1_DCACHE_SIZE
#define PETSC_LEVEL1_DCACHE_SIZE 32768
#endif


#ifndef PETSC_LEVEL1_DCACHE_ASSOC
#define PETSC_LEVEL1_DCACHE_ASSOC 2
#endif


#ifndef PETSC_HAVE_CLOSURE
#define PETSC_HAVE_CLOSURE 1
#endif


#ifndef PETSC__BSD_SOURCE
#define PETSC__BSD_SOURCE 1
#endif


#ifndef PETSC__DEFAULT_SOURCE
#define PETSC__DEFAULT_SOURCE 1
#endif


#ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
#define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
#endif


#ifndef PETSC_HAVE_GFORTRAN_IARGC
#define PETSC_HAVE_GFORTRAN_IARGC 1
#endif


#ifndef PETSC_USE_BYTES_FOR_SIZE
#define PETSC_USE_BYTES_FOR_SIZE 1
#endif


#ifndef PETSC_HAVE_SYS_SYSCTL_H
#define PETSC_HAVE_SYS_SYSCTL_H 1
#endif


#endif
**** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
#if !defined(INCLUDED_UNKNOWN)
#define INCLUDED_UNKNOWN


#if defined(__cplusplus)
extern "C" {
}
#else
#endif
#endif
*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
*******************************************************************************
  File "./config/configure.py", line 393, in petsc_configure
    framework.configure(out = sys.stdout)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
    self.processChildren()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
    self.serialEvaluation(self.childGraph)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
    child.configure()
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
    self.executeTest(self.configureLibrary)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
    ret = test(*args,**kargs)
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
    for matlab in self.generateGuesses():
  File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
    raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
================================================================================
Finishing Configure Run at Thu Oct 25 18:48:02 2018
================================================================================







------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MBP:bin zhihui$ pwd
> /Applications/MATLAB_R2018a.app/bin


Sorry - you need

--with-matlab-dir=/Applications/MATLAB_R2018a.app/

Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/c165e862/attachment-0001.html>

From balay at mcs.anl.gov  Thu Oct 25 19:58:26 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 00:58:26 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_59464B694F19FC1F3EF5F7E5@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
Message-ID: <alpine.LFD.2.21.1810251957370.7009@asterix>

1. You need to send us the complete log.

2. Also use current release - petsc-3.10 [not 3.8]

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> *******************************************************************************
> 
> 
> 
> Part of the log file as follow
> 
> 
> #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> #define PETSC_HAVE_MPI_COMBINER_DUP 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPIIO
> #define PETSC_HAVE_MPIIO 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> #define PETSC_HAVE_MPI_COMM_SPAWN 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_FINT
> #define PETSC_HAVE_MPI_FINT 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_IBARRIER
> #define PETSC_HAVE_MPI_IBARRIER 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_ALLTOALLW
> #define PETSC_HAVE_MPI_ALLTOALLW 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_REPLACE
> #define PETSC_HAVE_MPI_REPLACE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_EXSCAN
> #define PETSC_HAVE_MPI_EXSCAN 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_FINALIZED
> #define PETSC_HAVE_MPI_FINALIZED 1
> #endif
> 
> 
> #ifndef PETSC_USE_INFO
> #define PETSC_USE_INFO 1
> #endif
> 
> 
> #ifndef PETSC_Alignx
> #define PETSC_Alignx(a,b)   
> #endif
> 
> 
> #ifndef PETSC_USE_BACKWARD_LOOP
> #define PETSC_USE_BACKWARD_LOOP 1
> #endif
> 
> 
> #ifndef PETSC_USE_DEBUG
> #define PETSC_USE_DEBUG 1
> #endif
> 
> 
> #ifndef PETSC_USE_LOG
> #define PETSC_USE_LOG 1
> #endif
> 
> 
> #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> #endif
> 
> 
> #ifndef PETSC_IS_COLOR_VALUE_TYPE
> #define PETSC_IS_COLOR_VALUE_TYPE short
> #endif
> 
> 
> #ifndef PETSC_USE_CTABLE
> #define PETSC_USE_CTABLE 1
> #endif
> 
> 
> #ifndef PETSC_MEMALIGN
> #define PETSC_MEMALIGN 16
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_SIZE
> #define PETSC_LEVEL1_DCACHE_SIZE 32768
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> #define PETSC_LEVEL1_DCACHE_ASSOC 2
> #endif
> 
> 
> #ifndef PETSC_HAVE_CLOSURE
> #define PETSC_HAVE_CLOSURE 1
> #endif
> 
> 
> #ifndef PETSC__BSD_SOURCE
> #define PETSC__BSD_SOURCE 1
> #endif
> 
> 
> #ifndef PETSC__DEFAULT_SOURCE
> #define PETSC__DEFAULT_SOURCE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_GFORTRAN_IARGC
> #define PETSC_HAVE_GFORTRAN_IARGC 1
> #endif
> 
> 
> #ifndef PETSC_USE_BYTES_FOR_SIZE
> #define PETSC_USE_BYTES_FOR_SIZE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_SYS_SYSCTL_H
> #define PETSC_HAVE_SYS_SYSCTL_H 1
> #endif
> 
> 
> #endif
> **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> #if !defined(INCLUDED_UNKNOWN)
> #define INCLUDED_UNKNOWN
> 
> 
> #if defined(__cplusplus)
> extern "C" {
> }
> #else
> #endif
> #endif
> *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> *******************************************************************************
>   File "./config/configure.py", line 393, in petsc_configure
>     framework.configure(out = sys.stdout)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
>     self.processChildren()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
>     self.serialEvaluation(self.childGraph)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
>     child.configure()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
>     self.executeTest(self.configureLibrary)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
>     ret = test(*args,**kargs)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
>     for matlab in self.generateGuesses():
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
>     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> ================================================================================
> Finishing Configure Run at Thu Oct 25 18:48:02 2018
> ================================================================================
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MBP:bin zhihui$ pwd
> > /Applications/MATLAB_R2018a.app/bin
> 
> 
> Sorry - you need
> 
> --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> 
> Satish


From 648934229 at qq.com  Thu Oct 25 20:02:58 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 09:02:58 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810251957370.7009@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
Message-ID: <tencent_6472241E3860D8990F94338F@qq.com>

Hi Satish, 


The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.


Thank you.




------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



1. You need to send us the complete log.

2. Also use current release - petsc-3.10 [not 3.8]

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> *******************************************************************************
> 
> 
> 
> Part of the log file as follow
> 
> 
> #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> #define PETSC_HAVE_MPI_COMBINER_DUP 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPIIO
> #define PETSC_HAVE_MPIIO 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> #define PETSC_HAVE_MPI_COMM_SPAWN 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_FINT
> #define PETSC_HAVE_MPI_FINT 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_IBARRIER
> #define PETSC_HAVE_MPI_IBARRIER 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_ALLTOALLW
> #define PETSC_HAVE_MPI_ALLTOALLW 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_REPLACE
> #define PETSC_HAVE_MPI_REPLACE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_EXSCAN
> #define PETSC_HAVE_MPI_EXSCAN 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_MPI_FINALIZED
> #define PETSC_HAVE_MPI_FINALIZED 1
> #endif
> 
> 
> #ifndef PETSC_USE_INFO
> #define PETSC_USE_INFO 1
> #endif
> 
> 
> #ifndef PETSC_Alignx
> #define PETSC_Alignx(a,b)   
> #endif
> 
> 
> #ifndef PETSC_USE_BACKWARD_LOOP
> #define PETSC_USE_BACKWARD_LOOP 1
> #endif
> 
> 
> #ifndef PETSC_USE_DEBUG
> #define PETSC_USE_DEBUG 1
> #endif
> 
> 
> #ifndef PETSC_USE_LOG
> #define PETSC_USE_LOG 1
> #endif
> 
> 
> #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> #endif
> 
> 
> #ifndef PETSC_IS_COLOR_VALUE_TYPE
> #define PETSC_IS_COLOR_VALUE_TYPE short
> #endif
> 
> 
> #ifndef PETSC_USE_CTABLE
> #define PETSC_USE_CTABLE 1
> #endif
> 
> 
> #ifndef PETSC_MEMALIGN
> #define PETSC_MEMALIGN 16
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_SIZE
> #define PETSC_LEVEL1_DCACHE_SIZE 32768
> #endif
> 
> 
> #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> #define PETSC_LEVEL1_DCACHE_ASSOC 2
> #endif
> 
> 
> #ifndef PETSC_HAVE_CLOSURE
> #define PETSC_HAVE_CLOSURE 1
> #endif
> 
> 
> #ifndef PETSC__BSD_SOURCE
> #define PETSC__BSD_SOURCE 1
> #endif
> 
> 
> #ifndef PETSC__DEFAULT_SOURCE
> #define PETSC__DEFAULT_SOURCE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_GFORTRAN_IARGC
> #define PETSC_HAVE_GFORTRAN_IARGC 1
> #endif
> 
> 
> #ifndef PETSC_USE_BYTES_FOR_SIZE
> #define PETSC_USE_BYTES_FOR_SIZE 1
> #endif
> 
> 
> #ifndef PETSC_HAVE_SYS_SYSCTL_H
> #define PETSC_HAVE_SYS_SYSCTL_H 1
> #endif
> 
> 
> #endif
> **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> #if !defined(INCLUDED_UNKNOWN)
> #define INCLUDED_UNKNOWN
> 
> 
> #if defined(__cplusplus)
> extern "C" {
> }
> #else
> #endif
> #endif
> *******************************************************************************
>          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> -------------------------------------------------------------------------------
> You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> *******************************************************************************
>   File "./config/configure.py", line 393, in petsc_configure
>     framework.configure(out = sys.stdout)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
>     self.processChildren()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
>     self.serialEvaluation(self.childGraph)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
>     child.configure()
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
>     self.executeTest(self.configureLibrary)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
>     ret = test(*args,**kargs)
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
>     for matlab in self.generateGuesses():
>   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
>     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> ================================================================================
> Finishing Configure Run at Thu Oct 25 18:48:02 2018
> ================================================================================
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MBP:bin zhihui$ pwd
> > /Applications/MATLAB_R2018a.app/bin
> 
> 
> Sorry - you need
> 
> --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> 
> Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/a611a5a3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2032950 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/a611a5a3/attachment-0001.obj>

From balay at mcs.anl.gov  Thu Oct 25 20:29:23 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 01:29:23 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_6472241E3860D8990F94338F@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
Message-ID: <alpine.LFD.2.21.1810252027470.2360@asterix>

>>>>>>>>>>>>>>
Testing Matlab at /Applications/MATLAB_R2018a.app
Executing: /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
stdout:
< M A T L A B (R) >
                  Copyright 1984-2018 The MathWorks, Inc.
                   R2018a (9.4.0.813654) 64-bit (maci64)
                             February 23, 2018

For online documentation, see http://www.mathworks.com/support
For product information, visit www.mathworks.com.

Version 9.4.0.813654 (R2018a)
WARNING: Found Matlab at /Applications/MATLAB_R2018a.app but unable to run
<<<<<<<<<<<<<<<<<


So configure attempted to run Matlab - and perhaps got an error. What do you get if you run this manually?


/Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
echo $?

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Hi Satish, 
> 
> 
> The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.
> 
> 
> Thank you.
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> 1. You need to send us the complete log.
> 
> 2. Also use current release - petsc-3.10 [not 3.8]
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > ===============================================================================
> >              Configuring PETSc to compile on your system
> > ===============================================================================
> > TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> > 
> > 
> > 
> > Part of the log file as follow
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> > #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> > #define PETSC_HAVE_MPI_COMBINER_DUP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPIIO
> > #define PETSC_HAVE_MPIIO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> > #define PETSC_HAVE_MPI_COMM_SPAWN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINT
> > #define PETSC_HAVE_MPI_FINT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_IBARRIER
> > #define PETSC_HAVE_MPI_IBARRIER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_ALLTOALLW
> > #define PETSC_HAVE_MPI_ALLTOALLW 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> > #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> > #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REPLACE
> > #define PETSC_HAVE_MPI_REPLACE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_EXSCAN
> > #define PETSC_HAVE_MPI_EXSCAN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> > #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINALIZED
> > #define PETSC_HAVE_MPI_FINALIZED 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_INFO
> > #define PETSC_USE_INFO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_Alignx
> > #define PETSC_Alignx(a,b)   
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BACKWARD_LOOP
> > #define PETSC_USE_BACKWARD_LOOP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_DEBUG
> > #define PETSC_USE_DEBUG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_LOG
> > #define PETSC_USE_LOG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> > #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE
> > #define PETSC_IS_COLOR_VALUE_TYPE short
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_CTABLE
> > #define PETSC_USE_CTABLE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_MEMALIGN
> > #define PETSC_MEMALIGN 16
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> > #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_SIZE
> > #define PETSC_LEVEL1_DCACHE_SIZE 32768
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> > #define PETSC_LEVEL1_DCACHE_ASSOC 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_CLOSURE
> > #define PETSC_HAVE_CLOSURE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__BSD_SOURCE
> > #define PETSC__BSD_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__DEFAULT_SOURCE
> > #define PETSC__DEFAULT_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> > #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_GFORTRAN_IARGC
> > #define PETSC_HAVE_GFORTRAN_IARGC 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BYTES_FOR_SIZE
> > #define PETSC_USE_BYTES_FOR_SIZE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_SYS_SYSCTL_H
> > #define PETSC_HAVE_SYS_SYSCTL_H 1
> > #endif
> > 
> > 
> > #endif
> > **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> > #if !defined(INCLUDED_UNKNOWN)
> > #define INCLUDED_UNKNOWN
> > 
> > 
> > #if defined(__cplusplus)
> > extern "C" {
> > }
> > #else
> > #endif
> > #endif
> > *******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> >   File "./config/configure.py", line 393, in petsc_configure
> >     framework.configure(out = sys.stdout)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
> >     self.processChildren()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
> >     self.serialEvaluation(self.childGraph)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
> >     child.configure()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
> >     self.executeTest(self.configureLibrary)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
> >     ret = test(*args,**kargs)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
> >     for matlab in self.generateGuesses():
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
> >     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> > ================================================================================
> > Finishing Configure Run at Thu Oct 25 18:48:02 2018
> > ================================================================================
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MBP:bin zhihui$ pwd
> > > /Applications/MATLAB_R2018a.app/bin
> > 
> > 
> > Sorry - you need
> > 
> > --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > 
> > Satish


From 648934229 at qq.com  Thu Oct 25 20:41:38 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 09:41:38 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810252027470.2360@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<alpine.LFD.2.21.1810252027470.2360X-QQ-STYLE: 
Message-ID: <tencent_559F5A872574F8B77A4ACAE6@qq.com>

If I run it directly, I got the following. But if I double click /Applications/MATLAB_R2018a.app/bin/matlab in the folder. The matlab start normally.


Last login: Thu Oct 25 19:38:28 on ttys002
Scott-Grad-MBP:~ zhihui$ /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"


                                       < M A T L A B (R) >
                             Copyright 1984-2018 The MathWorks, Inc.
                              R2018a (9.4.0.813654) 64-bit (maci64)
                                        February 23, 2018




For online documentation, see http://www.mathworks.com/support
For product information, visit www.mathworks.com.


Version 9.4.0.813654 (R2018a)


--------------------------------------------------------------------------------
       Segmentation violation detected at Thu Oct 25 19:39:13 2018 -0600
--------------------------------------------------------------------------------


Configuration:
  Crash Decoding           : Disabled - No sandbox or build area path
  Crash Mode               : continue (default)
  Default Encoding         : ISO-8859-1
  Deployed                 : false
  Graphics Driver          : Unknown software
  MATLAB Architecture      : maci64
  MATLAB Entitlement ID    : 789930
  MATLAB Root              : /Applications/MATLAB_R2018a.app
  MATLAB Version           : 9.4.0.813654 (R2018a)
  OpenGL                   : software
  Operating System         : Mac OS Version 10.12.6 (Build 16G29)
  Process ID               : 77227
  Processor ID             : x86 Family 6 Model 158 Stepping 9, GenuineIntel
  Session Key              : 3308a767-ffe8-4f64-954b-ed266f8dea9c
  Window System            : None


Fault Count: 1




Abnormal termination


Register State (from fault):
  RAX = 0000000000000000  RBX = 00007f922bf7de00
  RCX = 00007f922bf7dde0  RDX = 0000000000000000
  RSP = 0000000108315005  RBP = 000070000b082a10
  RSI = 000070000b082c60  RDI = 000070000b082a30


   R8 = 0000000000000000   R9 = 0000000000000000
  R10 = 000070000b082c60  R11 = 00007f922be0ffd8
  R12 = 000070000b082a50  R13 = 00000001082d9a7c
  R14 = 000070000b082a50  R15 = 00007fff9f6aa410


  RIP = 0000000000000000  RFL = 000070000b083bc0


   CS = 0000000000000000   FS = 00007f922bf7d550   GS = 000070000b082f70


Stack Trace (from fault):
[  0] 0x00000001070f6f54                           bin/maci64/libmwfl.dylib+00053076 _ZN10foundation4core4diag15stacktrace_base7captureERKNS1_14thread_contextEm+00000052
[  1] 0x00000001070fbe26                           bin/maci64/libmwfl.dylib+00073254 _ZN10foundation4core4test17terminate_handledEv+00003958
[  2] 0x00000001070fac49                           bin/maci64/libmwfl.dylib+00068681 _ZN10foundation4core4diag13terminate_logEPKcPK17__darwin_ucontext+00000185
[  3] 0x000000010b04a2f0                          bin/maci64/libmwmcr.dylib+00574192 _Z19mnPrintErrorMessageRKNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEE+00010208
[  4] 0x000000010b047f72                          bin/maci64/libmwmcr.dylib+00565106 _Z19mnPrintErrorMessageRKNSt3__112basic_stringIcNS_11char_traitsIcEENS_9allocatorIcEEEE+00001122
[  5] 0x000000010b046681                          bin/maci64/libmwmcr.dylib+00558721 mnFatalSignalHandler+00000145
[  6] 0x00007fffa0d40b3a           /usr/lib/system/libsystem_platform.dylib+00011066 _sigtramp+00000026
[  7] 0x000000000000002e                                   <unknown-module>+00000000
[  8] 0x0000000114092674                    bin/maci64/libmwddux_impl.dylib+00251508 _ZNK17DDUXConfigService16updateDDUXConfigEv+00000148
[  9] 0x000000011406d426                    bin/maci64/libmwddux_impl.dylib+00099366 _ZNK4ddux6detail30UsageDataCollectionServiceImpl15logSessionStartERKN26UsageDataCollectionService18SessionStartParamsE+00006518
[ 10] 0x000000011405d65f                    bin/maci64/libmwddux_impl.dylib+00034399 _ZN4ddux16DDUXServiceProxyclERKNS_23SendSessionStartMessageERKN10foundation7msg_svc8exchange7RoutingE+00001711
[ 11] 0x00000001140680a7                    bin/maci64/libmwddux_impl.dylib+00077991 _ZN4ddux23SendSessionStartMessageC1Ev+00000071
[ 12] 0x0000000114065da5                    bin/maci64/libmwddux_impl.dylib+00069029 _ZN4ddux16DDUXServiceProxyclERKNS_23LogFunctionUsageMessageERKN10foundation7msg_svc8exchange7RoutingE+00024853
[ 13] 0x0000000114065ab0                    bin/maci64/libmwddux_impl.dylib+00068272 _ZN4ddux16DDUXServiceProxyclERKNS_23LogFunctionUsageMessageERKN10foundation7msg_svc8exchange7RoutingE+00024096
[ 14] 0x0000000106d3fd68                           bin/maci64/libmwms.dylib+00785768 _ZN10foundation7msg_svc8exchange6detail23DefaultMessageQueueImpl15deliverMessagesEPNS1_19DefaultMessageQueueE+00000088
[ 15] 0x000000011406675c                    bin/maci64/libmwddux_impl.dylib+00071516 _ZN4ddux16DDUXServiceProxyclERKNS_23LogFunctionUsageMessageERKN10foundation7msg_svc8exchange7RoutingE+00027340
[ 16] 0x0000000114066574                    bin/maci64/libmwddux_impl.dylib+00071028 _ZN4ddux16DDUXServiceProxyclERKNS_23LogFunctionUsageMessageERKN10foundation7msg_svc8exchange7RoutingE+00026852
[ 17] 0x0000000104f11d41 /Applications/MATLAB_R2018a.app/bin/maci64/libmwboost_thread.dylib+00011585 _ZN7mwboost6thread21start_thread_noexceptEv+00000353
[ 18] 0x00007fffa0d4a93b            /usr/lib/system/libsystem_pthread.dylib+00014651 _pthread_body+00000180
[ 19] 0x00007fffa0d4a887            /usr/lib/system/libsystem_pthread.dylib+00014471 _pthread_body+00000000
[ 20] 0x00007fffa0d4a08d            /usr/lib/system/libsystem_pthread.dylib+00012429 thread_start+00000013


** This crash report has been saved to disk as /Users/zhihui/matlab_crash_dump.77227-1 **






MATLAB is exiting because of fatal error
Killed: 9
Scott-Grad-MBP:~ zhihui$ echo $?
137





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



>>>>>>>>>>>>>>
Testing Matlab at /Applications/MATLAB_R2018a.app
Executing: /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
stdout:
< M A T L A B (R) >
                  Copyright 1984-2018 The MathWorks, Inc.
                   R2018a (9.4.0.813654) 64-bit (maci64)
                             February 23, 2018

For online documentation, see http://www.mathworks.com/support
For product information, visit www.mathworks.com.

Version 9.4.0.813654 (R2018a)
WARNING: Found Matlab at /Applications/MATLAB_R2018a.app but unable to run
<<<<<<<<<<<<<<<<<


So configure attempted to run Matlab - and perhaps got an error. What do you get if you run this manually?


/Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
echo $?

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Hi Satish, 
> 
> 
> The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.
> 
> 
> Thank you.
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> 1. You need to send us the complete log.
> 
> 2. Also use current release - petsc-3.10 [not 3.8]
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > ===============================================================================
> >              Configuring PETSc to compile on your system
> > ===============================================================================
> > TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> > 
> > 
> > 
> > Part of the log file as follow
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> > #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> > #define PETSC_HAVE_MPI_COMBINER_DUP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPIIO
> > #define PETSC_HAVE_MPIIO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> > #define PETSC_HAVE_MPI_COMM_SPAWN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINT
> > #define PETSC_HAVE_MPI_FINT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_IBARRIER
> > #define PETSC_HAVE_MPI_IBARRIER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_ALLTOALLW
> > #define PETSC_HAVE_MPI_ALLTOALLW 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> > #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> > #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REPLACE
> > #define PETSC_HAVE_MPI_REPLACE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_EXSCAN
> > #define PETSC_HAVE_MPI_EXSCAN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> > #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINALIZED
> > #define PETSC_HAVE_MPI_FINALIZED 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_INFO
> > #define PETSC_USE_INFO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_Alignx
> > #define PETSC_Alignx(a,b)   
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BACKWARD_LOOP
> > #define PETSC_USE_BACKWARD_LOOP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_DEBUG
> > #define PETSC_USE_DEBUG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_LOG
> > #define PETSC_USE_LOG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> > #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE
> > #define PETSC_IS_COLOR_VALUE_TYPE short
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_CTABLE
> > #define PETSC_USE_CTABLE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_MEMALIGN
> > #define PETSC_MEMALIGN 16
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> > #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_SIZE
> > #define PETSC_LEVEL1_DCACHE_SIZE 32768
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> > #define PETSC_LEVEL1_DCACHE_ASSOC 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_CLOSURE
> > #define PETSC_HAVE_CLOSURE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__BSD_SOURCE
> > #define PETSC__BSD_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__DEFAULT_SOURCE
> > #define PETSC__DEFAULT_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> > #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_GFORTRAN_IARGC
> > #define PETSC_HAVE_GFORTRAN_IARGC 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BYTES_FOR_SIZE
> > #define PETSC_USE_BYTES_FOR_SIZE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_SYS_SYSCTL_H
> > #define PETSC_HAVE_SYS_SYSCTL_H 1
> > #endif
> > 
> > 
> > #endif
> > **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> > #if !defined(INCLUDED_UNKNOWN)
> > #define INCLUDED_UNKNOWN
> > 
> > 
> > #if defined(__cplusplus)
> > extern "C" {
> > }
> > #else
> > #endif
> > #endif
> > *******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> >   File "./config/configure.py", line 393, in petsc_configure
> >     framework.configure(out = sys.stdout)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
> >     self.processChildren()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
> >     self.serialEvaluation(self.childGraph)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
> >     child.configure()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
> >     self.executeTest(self.configureLibrary)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
> >     ret = test(*args,**kargs)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
> >     for matlab in self.generateGuesses():
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
> >     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> > ================================================================================
> > Finishing Configure Run at Thu Oct 25 18:48:02 2018
> > ================================================================================
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MBP:bin zhihui$ pwd
> > > /Applications/MATLAB_R2018a.app/bin
> > 
> > 
> > Sorry - you need
> > 
> > --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > 
> > Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/db9cc93a/attachment-0001.html>

From 648934229 at qq.com  Thu Oct 25 20:54:56 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 09:54:56 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810252027470.2360@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<alpine.LFD.2.21.1810252027470.2360X-QQ-STYLE: 
Message-ID: <tencent_6D26D3AE4B8930B42372B5E9@qq.com>

Satish,


If I just run, /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay, then Matlab can run normally in the terminal as
Scott-Grad-MBP:~ zhihui$ /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay


                                                 < M A T L A B (R) >
                                       Copyright 1984-2018 The MathWorks, Inc.
                                        R2018a (9.4.0.813654) 64-bit (maci64)
                                                  February 23, 2018




For online documentation, see http://www.mathworks.com/support
For product information, visit www.mathworks.com.


>>









------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



>>>>>>>>>>>>>>
Testing Matlab at /Applications/MATLAB_R2018a.app
Executing: /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
stdout:
< M A T L A B (R) >
                  Copyright 1984-2018 The MathWorks, Inc.
                   R2018a (9.4.0.813654) 64-bit (maci64)
                             February 23, 2018

For online documentation, see http://www.mathworks.com/support
For product information, visit www.mathworks.com.

Version 9.4.0.813654 (R2018a)
WARNING: Found Matlab at /Applications/MATLAB_R2018a.app but unable to run
<<<<<<<<<<<<<<<<<


So configure attempted to run Matlab - and perhaps got an error. What do you get if you run this manually?


/Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
echo $?

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Hi Satish, 
> 
> 
> The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.
> 
> 
> Thank you.
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> 1. You need to send us the complete log.
> 
> 2. Also use current release - petsc-3.10 [not 3.8]
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > ===============================================================================
> >              Configuring PETSc to compile on your system
> > ===============================================================================
> > TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> > 
> > 
> > 
> > Part of the log file as follow
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> > #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> > #define PETSC_HAVE_MPI_COMBINER_DUP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPIIO
> > #define PETSC_HAVE_MPIIO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> > #define PETSC_HAVE_MPI_COMM_SPAWN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINT
> > #define PETSC_HAVE_MPI_FINT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_IBARRIER
> > #define PETSC_HAVE_MPI_IBARRIER 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_ALLTOALLW
> > #define PETSC_HAVE_MPI_ALLTOALLW 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> > #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> > #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_REPLACE
> > #define PETSC_HAVE_MPI_REPLACE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_EXSCAN
> > #define PETSC_HAVE_MPI_EXSCAN 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> > #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_MPI_FINALIZED
> > #define PETSC_HAVE_MPI_FINALIZED 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_INFO
> > #define PETSC_USE_INFO 1
> > #endif
> > 
> > 
> > #ifndef PETSC_Alignx
> > #define PETSC_Alignx(a,b)   
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BACKWARD_LOOP
> > #define PETSC_USE_BACKWARD_LOOP 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_DEBUG
> > #define PETSC_USE_DEBUG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_LOG
> > #define PETSC_USE_LOG 1
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> > #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> > #endif
> > 
> > 
> > #ifndef PETSC_IS_COLOR_VALUE_TYPE
> > #define PETSC_IS_COLOR_VALUE_TYPE short
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_CTABLE
> > #define PETSC_USE_CTABLE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_MEMALIGN
> > #define PETSC_MEMALIGN 16
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> > #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_SIZE
> > #define PETSC_LEVEL1_DCACHE_SIZE 32768
> > #endif
> > 
> > 
> > #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> > #define PETSC_LEVEL1_DCACHE_ASSOC 2
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_CLOSURE
> > #define PETSC_HAVE_CLOSURE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__BSD_SOURCE
> > #define PETSC__BSD_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC__DEFAULT_SOURCE
> > #define PETSC__DEFAULT_SOURCE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> > #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_GFORTRAN_IARGC
> > #define PETSC_HAVE_GFORTRAN_IARGC 1
> > #endif
> > 
> > 
> > #ifndef PETSC_USE_BYTES_FOR_SIZE
> > #define PETSC_USE_BYTES_FOR_SIZE 1
> > #endif
> > 
> > 
> > #ifndef PETSC_HAVE_SYS_SYSCTL_H
> > #define PETSC_HAVE_SYS_SYSCTL_H 1
> > #endif
> > 
> > 
> > #endif
> > **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> > #if !defined(INCLUDED_UNKNOWN)
> > #define INCLUDED_UNKNOWN
> > 
> > 
> > #if defined(__cplusplus)
> > extern "C" {
> > }
> > #else
> > #endif
> > #endif
> > *******************************************************************************
> >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > -------------------------------------------------------------------------------
> > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > *******************************************************************************
> >   File "./config/configure.py", line 393, in petsc_configure
> >     framework.configure(out = sys.stdout)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
> >     self.processChildren()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
> >     self.serialEvaluation(self.childGraph)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
> >     child.configure()
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
> >     self.executeTest(self.configureLibrary)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
> >     ret = test(*args,**kargs)
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
> >     for matlab in self.generateGuesses():
> >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
> >     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> > ================================================================================
> > Finishing Configure Run at Thu Oct 25 18:48:02 2018
> > ================================================================================
> > 
> > 
> > 
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MBP:bin zhihui$ pwd
> > > /Applications/MATLAB_R2018a.app/bin
> > 
> > 
> > Sorry - you need
> > 
> > --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > 
> > Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/c1719ab0/attachment.html>

From balay at mcs.anl.gov  Thu Oct 25 21:35:42 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Fri, 26 Oct 2018 02:35:42 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_6D26D3AE4B8930B42372B5E9@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<alpine.LFD.2.21.1810252027470.2360X-QQ-STYLE:
	<tencent_6D26D3AE4B8930B42372B5E9@qq.com>
Message-ID: <alpine.LFD.2.21.1810252132220.2360@asterix>

Presumably the error occurs after you type 'exit' from the terminal.

I'm not sure what to suggest. Having tools that break - is not ideal.

You could edit configure script - and bypass this check.

But what is your requirement wrt petsc and matlab?

Perhaps other petsc developers can guide you towards a suitable path.

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Satish,
> 
> 
> If I just run, /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay, then Matlab can run normally in the terminal as
> Scott-Grad-MBP:~ zhihui$ /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay
> 
> 
>                                                  < M A T L A B (R) >
>                                        Copyright 1984-2018 The MathWorks, Inc.
>                                         R2018a (9.4.0.813654) 64-bit (maci64)
>                                                   February 23, 2018
> 
> 
> 
> 
> For online documentation, see http://www.mathworks.com/support
> For product information, visit www.mathworks.com.
> 
> 
> >>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> >>>>>>>>>>>>>>
> Testing Matlab at /Applications/MATLAB_R2018a.app
> Executing: /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
> stdout:
> < M A T L A B (R) >
>                   Copyright 1984-2018 The MathWorks, Inc.
>                    R2018a (9.4.0.813654) 64-bit (maci64)
>                              February 23, 2018
> 
> For online documentation, see http://www.mathworks.com/support
> For product information, visit www.mathworks.com.
> 
> Version 9.4.0.813654 (R2018a)
> WARNING: Found Matlab at /Applications/MATLAB_R2018a.app but unable to run
> <<<<<<<<<<<<<<<<<
> 
> 
> So configure attempted to run Matlab - and perhaps got an error. What do you get if you run this manually?
> 
> 
> /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
> echo $?
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Hi Satish, 
> > 
> > 
> > The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.
> > 
> > 
> > Thank you.
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > 1. You need to send us the complete log.
> > 
> > 2. Also use current release - petsc-3.10 [not 3.8]
> > 
> > Satish
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > > ===============================================================================
> > >              Configuring PETSc to compile on your system
> > > ===============================================================================
> > > TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
> > >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > > -------------------------------------------------------------------------------
> > > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > > *******************************************************************************
> > > 
> > > 
> > > 
> > > Part of the log file as follow
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> > > #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> > > #define PETSC_HAVE_MPI_COMBINER_DUP 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPIIO
> > > #define PETSC_HAVE_MPIIO 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> > > #define PETSC_HAVE_MPI_COMM_SPAWN 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_FINT
> > > #define PETSC_HAVE_MPI_FINT 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_IBARRIER
> > > #define PETSC_HAVE_MPI_IBARRIER 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_ALLTOALLW
> > > #define PETSC_HAVE_MPI_ALLTOALLW 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> > > #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> > > #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REPLACE
> > > #define PETSC_HAVE_MPI_REPLACE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_EXSCAN
> > > #define PETSC_HAVE_MPI_EXSCAN 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> > > #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_FINALIZED
> > > #define PETSC_HAVE_MPI_FINALIZED 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_INFO
> > > #define PETSC_USE_INFO 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_Alignx
> > > #define PETSC_Alignx(a,b)   
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_BACKWARD_LOOP
> > > #define PETSC_USE_BACKWARD_LOOP 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_DEBUG
> > > #define PETSC_USE_DEBUG 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_LOG
> > > #define PETSC_USE_LOG 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> > > #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_IS_COLOR_VALUE_TYPE
> > > #define PETSC_IS_COLOR_VALUE_TYPE short
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_CTABLE
> > > #define PETSC_USE_CTABLE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_MEMALIGN
> > > #define PETSC_MEMALIGN 16
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> > > #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_SIZE
> > > #define PETSC_LEVEL1_DCACHE_SIZE 32768
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> > > #define PETSC_LEVEL1_DCACHE_ASSOC 2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_CLOSURE
> > > #define PETSC_HAVE_CLOSURE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC__BSD_SOURCE
> > > #define PETSC__BSD_SOURCE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC__DEFAULT_SOURCE
> > > #define PETSC__DEFAULT_SOURCE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> > > #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_GFORTRAN_IARGC
> > > #define PETSC_HAVE_GFORTRAN_IARGC 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_BYTES_FOR_SIZE
> > > #define PETSC_USE_BYTES_FOR_SIZE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_SYS_SYSCTL_H
> > > #define PETSC_HAVE_SYS_SYSCTL_H 1
> > > #endif
> > > 
> > > 
> > > #endif
> > > **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> > > #if !defined(INCLUDED_UNKNOWN)
> > > #define INCLUDED_UNKNOWN
> > > 
> > > 
> > > #if defined(__cplusplus)
> > > extern "C" {
> > > }
> > > #else
> > > #endif
> > > #endif
> > > *******************************************************************************
> > >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > > -------------------------------------------------------------------------------
> > > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > > *******************************************************************************
> > >   File "./config/configure.py", line 393, in petsc_configure
> > >     framework.configure(out = sys.stdout)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
> > >     self.processChildren()
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
> > >     self.serialEvaluation(self.childGraph)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
> > >     child.configure()
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
> > >     self.executeTest(self.configureLibrary)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
> > >     ret = test(*args,**kargs)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
> > >     for matlab in self.generateGuesses():
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
> > >     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> > > ================================================================================
> > > Finishing Configure Run at Thu Oct 25 18:48:02 2018
> > > ================================================================================
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Scott-Grad-MBP:bin zhihui$ pwd
> > > > /Applications/MATLAB_R2018a.app/bin
> > > 
> > > 
> > > Sorry - you need
> > > 
> > > --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > > 
> > > Satish


From 648934229 at qq.com  Thu Oct 25 21:53:45 2018
From: 648934229 at qq.com (=?ISO-8859-1?B?YXZhdGFy?=)
Date: Fri, 26 Oct 2018 10:53:45 +0800
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810252132220.2360@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<alpine.LFD.2.21.1810252027470.23X-QQ-STYLE: 
Message-ID: <tencent_7D30EA706521772457B88AC8@qq.com>

Thank Satish. I would like to call Matlab function in Pestc. For example, calculating the condition number of a petsc matrix.


When I comment out the test from configuration script, I got the following problem. And the attached is the LOG file.
===============================================================================
             Configuring PETSc to compile on your system
===============================================================================
TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)          *******************************************************************************
        CONFIGURATION CRASH  (Please send configure.log to petsc-maint at mcs.anl.gov)
*******************************************************************************





------------------ Original ------------------
From:  "Balay, Satish";<balay at mcs.anl.gov>;
Date:  Oct 26, 2018
To:  "avatar"<648934229 at qq.com>; 
Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"



Presumably the error occurs after you type 'exit' from the terminal.

I'm not sure what to suggest. Having tools that break - is not ideal.

You could edit configure script - and bypass this check.

But what is your requirement wrt petsc and matlab?

Perhaps other petsc developers can guide you towards a suitable path.

Satish

On Fri, 26 Oct 2018, avatar wrote:

> Satish,
> 
> 
> If I just run, /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay, then Matlab can run normally in the terminal as
> Scott-Grad-MBP:~ zhihui$ /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay
> 
> 
>                                                  < M A T L A B (R) >
>                                        Copyright 1984-2018 The MathWorks, Inc.
>                                         R2018a (9.4.0.813654) 64-bit (maci64)
>                                                   February 23, 2018
> 
> 
> 
> 
> For online documentation, see http://www.mathworks.com/support
> For product information, visit www.mathworks.com.
> 
> 
> >>
> 
> 
> 
> 
> 
> 
> 
> 
> 
> ------------------ Original ------------------
> From:  "Balay, Satish";<balay at mcs.anl.gov>;
> Date:  Oct 26, 2018
> To:  "avatar"<648934229 at qq.com>; 
> Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> 
> 
> 
> >>>>>>>>>>>>>>
> Testing Matlab at /Applications/MATLAB_R2018a.app
> Executing: /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
> stdout:
> < M A T L A B (R) >
>                   Copyright 1984-2018 The MathWorks, Inc.
>                    R2018a (9.4.0.813654) 64-bit (maci64)
>                              February 23, 2018
> 
> For online documentation, see http://www.mathworks.com/support
> For product information, visit www.mathworks.com.
> 
> Version 9.4.0.813654 (R2018a)
> WARNING: Found Matlab at /Applications/MATLAB_R2018a.app but unable to run
> <<<<<<<<<<<<<<<<<
> 
> 
> So configure attempted to run Matlab - and perhaps got an error. What do you get if you run this manually?
> 
> 
> /Applications/MATLAB_R2018a.app/bin/matlab -nojvm -nodisplay -r "display(['Version ' version]); exit"
> echo $?
> 
> Satish
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
> > Hi Satish, 
> > 
> > 
> > The attached is the whole configure.log file. I could not past it here because it is too big and crash my webpage. I could not use the latest version right now, because the project is maintained by other people. If I update petsc, I will break the whole project. But we will use the latest version when the maintenance guys update the dependences.
> > 
> > 
> > Thank you.
> > 
> > 
> > 
> > 
> > ------------------ Original ------------------
> > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > Date:  Oct 26, 2018
> > To:  "avatar"<648934229 at qq.com>; 
> > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > 
> > 
> > 
> > 1. You need to send us the complete log.
> > 
> > 2. Also use current release - petsc-3.10 [not 3.8]
> > 
> > Satish
> > 
> > On Fri, 26 Oct 2018, avatar wrote:
> > 
> > > Scott-Grad-MBP:petsc-3.8.3 zhihui$ ./configure --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > > ===============================================================================
> > >              Configuring PETSc to compile on your system
> > > ===============================================================================
> > > TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/*******************************************************************************
> > >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > > -------------------------------------------------------------------------------
> > > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > > *******************************************************************************
> > > 
> > > 
> > > 
> > > Part of the log file as follow
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REDUCE_SCATTER
> > > #define PETSC_HAVE_MPI_REDUCE_SCATTER 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_COMBINER_DUP
> > > #define PETSC_HAVE_MPI_COMBINER_DUP 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPIIO
> > > #define PETSC_HAVE_MPIIO 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_COMM_SPAWN
> > > #define PETSC_HAVE_MPI_COMM_SPAWN 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_FINT
> > > #define PETSC_HAVE_MPI_FINT 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_IBARRIER
> > > #define PETSC_HAVE_MPI_IBARRIER 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_ALLTOALLW
> > > #define PETSC_HAVE_MPI_ALLTOALLW 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_OMPI_RELEASE_VERSION
> > > #define PETSC_HAVE_OMPI_RELEASE_VERSION 2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REDUCE_LOCAL
> > > #define PETSC_HAVE_MPI_REDUCE_LOCAL 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_REPLACE
> > > #define PETSC_HAVE_MPI_REPLACE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_EXSCAN
> > > #define PETSC_HAVE_MPI_EXSCAN 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_C_DOUBLE_COMPLEX
> > > #define PETSC_HAVE_MPI_C_DOUBLE_COMPLEX 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_MPI_FINALIZED
> > > #define PETSC_HAVE_MPI_FINALIZED 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_INFO
> > > #define PETSC_USE_INFO 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_Alignx
> > > #define PETSC_Alignx(a,b)   
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_BACKWARD_LOOP
> > > #define PETSC_USE_BACKWARD_LOOP 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_DEBUG
> > > #define PETSC_USE_DEBUG 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_LOG
> > > #define PETSC_USE_LOG 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_IS_COLOR_VALUE_TYPE_F
> > > #define PETSC_IS_COLOR_VALUE_TYPE_F integer2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_IS_COLOR_VALUE_TYPE
> > > #define PETSC_IS_COLOR_VALUE_TYPE short
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_CTABLE
> > > #define PETSC_USE_CTABLE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_MEMALIGN
> > > #define PETSC_MEMALIGN 16
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_LINESIZE
> > > #define PETSC_LEVEL1_DCACHE_LINESIZE 32
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_SIZE
> > > #define PETSC_LEVEL1_DCACHE_SIZE 32768
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_LEVEL1_DCACHE_ASSOC
> > > #define PETSC_LEVEL1_DCACHE_ASSOC 2
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_CLOSURE
> > > #define PETSC_HAVE_CLOSURE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC__BSD_SOURCE
> > > #define PETSC__BSD_SOURCE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC__DEFAULT_SOURCE
> > > #define PETSC__DEFAULT_SOURCE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT
> > > #define PETSC_HAVE_FORTRAN_GET_COMMAND_ARGUMENT 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_GFORTRAN_IARGC
> > > #define PETSC_HAVE_GFORTRAN_IARGC 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_USE_BYTES_FOR_SIZE
> > > #define PETSC_USE_BYTES_FOR_SIZE 1
> > > #endif
> > > 
> > > 
> > > #ifndef PETSC_HAVE_SYS_SYSCTL_H
> > > #define PETSC_HAVE_SYS_SYSCTL_H 1
> > > #endif
> > > 
> > > 
> > > #endif
> > > **** C specific Configure header /var/folders/z_/2vhmh9zx3kx3h80k5wbkjmtr0000gp/T/petsc-yrHkm7/conffix.h ****
> > > #if !defined(INCLUDED_UNKNOWN)
> > > #define INCLUDED_UNKNOWN
> > > 
> > > 
> > > #if defined(__cplusplus)
> > > extern "C" {
> > > }
> > > #else
> > > #endif
> > > #endif
> > > *******************************************************************************
> > >          UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
> > > -------------------------------------------------------------------------------
> > > You set a value for --with-matlab-dir, but /Applications/MATLAB_R2018a.app cannot be used
> > > *******************************************************************************
> > >   File "./config/configure.py", line 393, in petsc_configure
> > >     framework.configure(out = sys.stdout)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1097, in configure
> > >     self.processChildren()
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1086, in processChildren
> > >     self.serialEvaluation(self.childGraph)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/framework.py", line 1067, in serialEvaluation
> > >     child.configure()
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/package.py", line 857, in configure
> > >     self.executeTest(self.configureLibrary)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/base.py", line 126, in executeTest
> > >     ret = test(*args,**kargs)
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 40, in configureLibrary
> > >     for matlab in self.generateGuesses():
> > >   File "/Users/zhihui/igx/deps/srcs/petsc-3.8.3/config/BuildSystem/config/packages/Matlab.py", line 23, in generateGuesses
> > >     raise RuntimeError('You set a value for --with-matlab-dir, but '+self.argDB['with-matlab-dir']+' cannot be used\n')
> > > ================================================================================
> > > Finishing Configure Run at Thu Oct 25 18:48:02 2018
> > > ================================================================================
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > 
> > > ------------------ Original ------------------
> > > From:  "Balay, Satish";<balay at mcs.anl.gov>;
> > > Date:  Oct 26, 2018
> > > To:  "avatar"<648934229 at qq.com>; 
> > > Cc:  "petsc-users"<petsc-users at mcs.anl.gov>; 
> > > Subject:  Re: [petsc-users] "Could not find a suitable archiver. Use --with-arto specify an archiver"
> > > 
> > > 
> > > 
> > > On Fri, 26 Oct 2018, avatar wrote:
> > > 
> > > > Scott-Grad-MBP:bin zhihui$ pwd
> > > > /Applications/MATLAB_R2018a.app/bin
> > > 
> > > 
> > > Sorry - you need
> > > 
> > > --with-matlab-dir=/Applications/MATLAB_R2018a.app/
> > > 
> > > Satish
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/41781826/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: application/octet-stream
Size: 2032041 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/41781826/attachment-0001.obj>

From Peterspy at outlook.com  Fri Oct 26 04:16:14 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Fri, 26 Oct 2018 09:16:14 +0000
Subject: [petsc-users] Problems with fortran subroutine DMDAVecGetArrayF90
Message-ID: <HK0PR04MB2547CC42F3A0FF877E2126EAD0F00@HK0PR04MB2547.apcprd04.prod.outlook.com>

Hi all,
I am trying to change a KSP tutorial code ex34.c to ex34f.F90. I found that after I changed DMDAVecGetArray to DMDAVecGetArrayF90, and DMDAVecRestoreArray to DMDAVecRestoreArrayF90. Compilation errors say that there is no matching specific subroutine for this generic subroutine call. Can anyone help me with the problem?
The error massage is listed below and the fortran code is also attached. Many thanks.

ex34f.F90(83): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECGETARRAYF90]
      call DMDAVecGetArrayF90(x,array,ierr)
-----------^
ex34f.F90(91): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECRESTOREARRAYF90]
      call DMDAVecRestoreArrayF90(x,array,ierr)
-----------^
compilation aborted for ex34f.F90 (code 1)

----------------------------------------------------------------
Pengyang Song?????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86) 15762264643

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/94af7350/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex34f.F90
Type: application/octet-stream
Size: 9909 bytes
Desc: ex34f.F90
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/94af7350/attachment.obj>

From Peterspy at outlook.com  Fri Oct 26 04:54:48 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Fri, 26 Oct 2018 09:54:48 +0000
Subject: [petsc-users] Problems with fortran subroutine DMDAVecGetArrayF90,
 again.
Message-ID: <HK0PR04MB2547174884F2F351C756BBB9D0F00@HK0PR04MB2547.apcprd04.prod.outlook.com>

Sorry for bothering again.
I have solved the problem by looking through another fortran code, it seems that I was wrongly calling the subroutine DMDAVecGetArrayF90.
I attached my final fortran code to share with you guys, which is changed from ksp tutorial ex34.c to ex34f.F90
Have a good day.

----------------------------------------------------------------
Pengyang Song?????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86) 15762264643

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/c2f0e1e8/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex34f.F90
Type: application/octet-stream
Size: 9888 bytes
Desc: ex34f.F90
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/c2f0e1e8/attachment-0001.obj>

From phil.tooley at sheffield.ac.uk  Fri Oct 26 08:53:30 2018
From: phil.tooley at sheffield.ac.uk (Phil Tooley)
Date: Fri, 26 Oct 2018 14:53:30 +0100
Subject: [petsc-users] Valgrind Errors with KSPSolve
Message-ID: <10701fcd-bc1c-6113-8bea-2a4c0bcda78e@sheffield.ac.uk>

Hi All,

Running valgrind over my code reveals a huge number (it stops collecting
after 1e6) of uninitialised value errors occuring the KSPSolve routine.?
These don't occur with the ksp example scripts but I can't figure out
what I am doing wrong in my code.? My code works as expected.? The same
errors occur with both version 3.9.4 and 3.10.0.

I am developing in c++ and using smart pointers to allow autodestruction
of Petsc objects, but I don't think this should have any side effects.?

I do the ksp setup as:

KSP_unique m_ksp = create_unique_ksp();

perr = KSPCreate(m_comm, m_ksp.get());CHKERRABORT(m_comm, perr);

perr = KSPSetOperators(*m_ksp, *normmat, *normmat);CHKERRABORT(m_comm,
perr);
perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);

perr = KSPSetFromOptions(*m_ksp);CHKERRABORT(m_comm, perr);
perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);

perr = KSPSolve(*m_ksp, *m_rhs, *m_delta);CHKERRABORT(m_comm, perr);

Where normmat is a smart pointer to a previously created Mat object and
m_rhs and m_delta are smart pointers to Vec objects which are
initialised using MatCreateVecs on normmat.

I am using MPICH to provide a (hopefully) valgrind clean MPI implementation.

Any insights are appreciated.

Many Thanks

Phil Tooley

Typical errors are as follows:

==7368== Conditional jump or move depends on uninitialised value(s)
==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
==7368==??? by 0x5891702: MatLUFactorNumeric_SeqAIJ (aijfact.c:553)
==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
(bjacobi.c:621)
==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
==7368==??? by 0x10FCDD: Elastic::innerstep(double) (in
/home/telemin/repos/pfire_petsc/bin/pfire)
==7368==??? by 0x110600: Elastic::innerloop(int) (in
/home/telemin/repos/pfire_petsc/bin/pfire)
==7368==
==7368== Conditional jump or move depends on uninitialised value(s)
==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
==7368==??? by 0x588A75F: MatPivotCheck_none(_p_Mat*, _p_Mat*,
MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
const*, FactorShiftCtx*, int) (matimpl.h:727)
==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
(bjacobi.c:621)
==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
==7368==
==7368== Conditional jump or move depends on uninitialised value(s)
==7368==??? at 0x588A76D: MatPivotCheck_none(_p_Mat*, _p_Mat*,
MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
const*, FactorShiftCtx*, int) (matimpl.h:727)
==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
(bjacobi.c:621)
==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
==7368==??? by 0x10FCDD: Elastic::innerstep(double) (in
/home/telemin/repos/pfire_petsc/bin/pfire)
==7368==
==7368== Conditional jump or move depends on uninitialised value(s)
==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
==7368==??? by 0x5889528: PetscIsNanScalar(double) (petscmath.h:674)
==7368==??? by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
const*, FactorShiftCtx*, int) (matimpl.h:727)
==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
(bjacobi.c:621)
==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
==7368==
==7368== Conditional jump or move depends on uninitialised value(s)
==7368==??? at 0x4FDACE3: std::isnan(double) (cmath:627)
==7368==??? by 0x4FDAC8D: PetscIsNanReal (mathinf.c:92)
==7368==??? by 0x588952D: PetscIsNanScalar(double) (petscmath.h:674)
==7368==??? by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
const*, FactorShiftCtx*, int) (matimpl.h:727)
==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
(bjacobi.c:621)
==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)

-- 
Phil Tooley
Research Software Engineering
University of Sheffield


From knepley at gmail.com  Fri Oct 26 09:01:17 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 26 Oct 2018 10:01:17 -0400
Subject: [petsc-users] Valgrind Errors with KSPSolve
In-Reply-To: <10701fcd-bc1c-6113-8bea-2a4c0bcda78e@sheffield.ac.uk>
References: <10701fcd-bc1c-6113-8bea-2a4c0bcda78e@sheffield.ac.uk>
Message-ID: <CAMYG4GmESp2aEf0687VninbjEhZC+Z0WX0FOi=1QRYr6sYnACA@mail.gmail.com>

On Fri, Oct 26, 2018 at 9:54 AM Phil Tooley <phil.tooley at sheffield.ac.uk>
wrote:

> Hi All,
>
> Running valgrind over my code reveals a huge number (it stops collecting
> after 1e6) of uninitialised value errors occuring the KSPSolve routine.
> These don't occur with the ksp example scripts but I can't figure out
> what I am doing wrong in my code.  My code works as expected.  The same
> errors occur with both version 3.9.4 and 3.10.0.
>
> I am developing in c++ and using smart pointers to allow autodestruction
> of Petsc objects, but I don't think this should have any side effects.
>

>From casual inspection, it sure seems like the matrix you are passing KSP
has unitialized
values.

<Editorial comment>I used smart pointers for years, and work on packages
that had smart
pointers. In almost every case, we ended up ripping out the smart pointers
and going with
explicit destruction, the amount of work they save never outweighed the
problems with
detecting errors</Editorial comment>

  Thanks,

     Matt

  Thanks,

    Matt


> I do the ksp setup as:
>
> KSP_unique m_ksp = create_unique_ksp();
>
> perr = KSPCreate(m_comm, m_ksp.get());CHKERRABORT(m_comm, perr);
>
> perr = KSPSetOperators(*m_ksp, *normmat, *normmat);CHKERRABORT(m_comm,
> perr);
> perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);
>
> perr = KSPSetFromOptions(*m_ksp);CHKERRABORT(m_comm, perr);
> perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);
>
> perr = KSPSolve(*m_ksp, *m_rhs, *m_delta);CHKERRABORT(m_comm, perr);
>
> Where normmat is a smart pointer to a previously created Mat object and
> m_rhs and m_delta are smart pointers to Vec objects which are
> initialised using MatCreateVecs on normmat.
>
> I am using MPICH to provide a (hopefully) valgrind clean MPI
> implementation.
>
> Any insights are appreciated.
>
> Many Thanks
>
> Phil Tooley
>
> Typical errors are as follows:
>
> ==7368== Conditional jump or move depends on uninitialised value(s)
> ==7368==    at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
> ==7368==    by 0x5891702: MatLUFactorNumeric_SeqAIJ (aijfact.c:553)
> ==7368==    by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
> ==7368==    by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
> ==7368==    by 0x6192D4F: PCSetUp (precon.c:923)
> ==7368==    by 0x62EAE56: KSPSetUp (itfunc.c:381)
> ==7368==    by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
> (bjacobi.c:621)
> ==7368==    by 0x619352C: PCSetUpOnBlocks (precon.c:954)
> ==7368==    by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
> ==7368==    by 0x62ECCB1: KSPSolve (itfunc.c:613)
> ==7368==    by 0x10FCDD: Elastic::innerstep(double) (in
> /home/telemin/repos/pfire_petsc/bin/pfire)
> ==7368==    by 0x110600: Elastic::innerloop(int) (in
> /home/telemin/repos/pfire_petsc/bin/pfire)
> ==7368==
> ==7368== Conditional jump or move depends on uninitialised value(s)
> ==7368==    at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
> ==7368==    by 0x588A75F: MatPivotCheck_none(_p_Mat*, _p_Mat*,
> MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
> ==7368==    by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
> const*, FactorShiftCtx*, int) (matimpl.h:727)
> ==7368==    by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
> ==7368==    by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
> ==7368==    by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
> ==7368==    by 0x6192D4F: PCSetUp (precon.c:923)
> ==7368==    by 0x62EAE56: KSPSetUp (itfunc.c:381)
> ==7368==    by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
> (bjacobi.c:621)
> ==7368==    by 0x619352C: PCSetUpOnBlocks (precon.c:954)
> ==7368==    by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
> ==7368==    by 0x62ECCB1: KSPSolve (itfunc.c:613)
> ==7368==
> ==7368== Conditional jump or move depends on uninitialised value(s)
> ==7368==    at 0x588A76D: MatPivotCheck_none(_p_Mat*, _p_Mat*,
> MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
> ==7368==    by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
> const*, FactorShiftCtx*, int) (matimpl.h:727)
> ==7368==    by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
> ==7368==    by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
> ==7368==    by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
> ==7368==    by 0x6192D4F: PCSetUp (precon.c:923)
> ==7368==    by 0x62EAE56: KSPSetUp (itfunc.c:381)
> ==7368==    by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
> (bjacobi.c:621)
> ==7368==    by 0x619352C: PCSetUpOnBlocks (precon.c:954)
> ==7368==    by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
> ==7368==    by 0x62ECCB1: KSPSolve (itfunc.c:613)
> ==7368==    by 0x10FCDD: Elastic::innerstep(double) (in
> /home/telemin/repos/pfire_petsc/bin/pfire)
> ==7368==
> ==7368== Conditional jump or move depends on uninitialised value(s)
> ==7368==    at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
> ==7368==    by 0x5889528: PetscIsNanScalar(double) (petscmath.h:674)
> ==7368==    by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
> MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
> ==7368==    by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
> const*, FactorShiftCtx*, int) (matimpl.h:727)
> ==7368==    by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
> ==7368==    by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
> ==7368==    by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
> ==7368==    by 0x6192D4F: PCSetUp (precon.c:923)
> ==7368==    by 0x62EAE56: KSPSetUp (itfunc.c:381)
> ==7368==    by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
> (bjacobi.c:621)
> ==7368==    by 0x619352C: PCSetUpOnBlocks (precon.c:954)
> ==7368==    by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
> ==7368==
> ==7368== Conditional jump or move depends on uninitialised value(s)
> ==7368==    at 0x4FDACE3: std::isnan(double) (cmath:627)
> ==7368==    by 0x4FDAC8D: PetscIsNanReal (mathinf.c:92)
> ==7368==    by 0x588952D: PetscIsNanScalar(double) (petscmath.h:674)
> ==7368==    by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
> MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
> ==7368==    by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*, MatFactorInfo
> const*, FactorShiftCtx*, int) (matimpl.h:727)
> ==7368==    by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
> ==7368==    by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
> ==7368==    by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
> ==7368==    by 0x6192D4F: PCSetUp (precon.c:923)
> ==7368==    by 0x62EAE56: KSPSetUp (itfunc.c:381)
> ==7368==    by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
> (bjacobi.c:621)
> ==7368==    by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>
> --
> Phil Tooley
> Research Software Engineering
> University of Sheffield
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/9392a4e4/attachment.html>

From phil.tooley at sheffield.ac.uk  Fri Oct 26 09:46:57 2018
From: phil.tooley at sheffield.ac.uk (Phil Tooley)
Date: Fri, 26 Oct 2018 15:46:57 +0100
Subject: [petsc-users] Valgrind Errors with KSPSolve
In-Reply-To: <CAMYG4GmESp2aEf0687VninbjEhZC+Z0WX0FOi=1QRYr6sYnACA@mail.gmail.com>
References: <10701fcd-bc1c-6113-8bea-2a4c0bcda78e@sheffield.ac.uk>
	<CAMYG4GmESp2aEf0687VninbjEhZC+Z0WX0FOi=1QRYr6sYnACA@mail.gmail.com>
Message-ID: <a159275c-c511-d3fc-45a0-cd8374def173@sheffield.ac.uk>

Thanks Matt,

I got it figured out.? As with a lot of uninitialised value bugs it
started a long way away (in a scalar value) and propagated through.?
What is slightly bizarre is that valgrind didn't pick it up where I
define and first use it, even with --track-origins=yes.

Thank you for your help.

Phil


On 26/10/2018 15:01, Matthew Knepley wrote:
> On Fri, Oct 26, 2018 at 9:54 AM Phil Tooley
> <phil.tooley at sheffield.ac.uk <mailto:phil.tooley at sheffield.ac.uk>> wrote:
>
>     Hi All,
>
>     Running valgrind over my code reveals a huge number (it stops
>     collecting
>     after 1e6) of uninitialised value errors occuring the KSPSolve
>     routine.?
>     These don't occur with the ksp example scripts but I can't figure out
>     what I am doing wrong in my code.? My code works as expected.? The
>     same
>     errors occur with both version 3.9.4 and 3.10.0.
>
>     I am developing in c++ and using smart pointers to allow
>     autodestruction
>     of Petsc objects, but I don't think this should have any side
>     effects.?
>
>
> From casual inspection, it sure seems like the matrix you are passing
> KSP has unitialized
> values.
>
> <Editorial comment>I used smart pointers for years, and work on
> packages that had smart
> pointers. In almost every case, we ended up ripping out the smart
> pointers and going with
> explicit destruction, the amount of work they save never outweighed
> the problems with
> detecting errors</Editorial comment>
>
> ? Thanks,
>
> ? ? ?Matt
>
> ? Thanks,
>
> ? ? Matt
> ?
>
>     I do the ksp setup as:
>
>     KSP_unique m_ksp = create_unique_ksp();
>
>     perr = KSPCreate(m_comm, m_ksp.get());CHKERRABORT(m_comm, perr);
>
>     perr = KSPSetOperators(*m_ksp, *normmat, *normmat);CHKERRABORT(m_comm,
>     perr);
>     perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);
>
>     perr = KSPSetFromOptions(*m_ksp);CHKERRABORT(m_comm, perr);
>     perr = KSPSetUp(*m_ksp);CHKERRABORT(m_comm, perr);
>
>     perr = KSPSolve(*m_ksp, *m_rhs, *m_delta);CHKERRABORT(m_comm, perr);
>
>     Where normmat is a smart pointer to a previously created Mat
>     object and
>     m_rhs and m_delta are smart pointers to Vec objects which are
>     initialised using MatCreateVecs on normmat.
>
>     I am using MPICH to provide a (hopefully) valgrind clean MPI
>     implementation.
>
>     Any insights are appreciated.
>
>     Many Thanks
>
>     Phil Tooley
>
>     Typical errors are as follows:
>
>     ==7368== Conditional jump or move depends on uninitialised value(s)
>     ==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
>     ==7368==??? by 0x5891702: MatLUFactorNumeric_SeqAIJ (aijfact.c:553)
>     ==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
>     ==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
>     ==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
>     ==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
>     ==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
>     (bjacobi.c:621)
>     ==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>     ==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
>     ==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
>     ==7368==??? by 0x10FCDD: Elastic::innerstep(double) (in
>     /home/telemin/repos/pfire_petsc/bin/pfire)
>     ==7368==??? by 0x110600: Elastic::innerloop(int) (in
>     /home/telemin/repos/pfire_petsc/bin/pfire)
>     ==7368==
>     ==7368== Conditional jump or move depends on uninitialised value(s)
>     ==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
>     ==7368==??? by 0x588A75F: MatPivotCheck_none(_p_Mat*, _p_Mat*,
>     MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
>     ==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*,
>     MatFactorInfo
>     const*, FactorShiftCtx*, int) (matimpl.h:727)
>     ==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
>     ==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
>     ==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
>     ==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
>     ==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
>     ==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
>     (bjacobi.c:621)
>     ==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>     ==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
>     ==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
>     ==7368==
>     ==7368== Conditional jump or move depends on uninitialised value(s)
>     ==7368==??? at 0x588A76D: MatPivotCheck_none(_p_Mat*, _p_Mat*,
>     MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
>     ==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*,
>     MatFactorInfo
>     const*, FactorShiftCtx*, int) (matimpl.h:727)
>     ==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
>     ==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
>     ==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
>     ==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
>     ==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
>     ==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
>     (bjacobi.c:621)
>     ==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>     ==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
>     ==7368==??? by 0x62ECCB1: KSPSolve (itfunc.c:613)
>     ==7368==??? by 0x10FCDD: Elastic::innerstep(double) (in
>     /home/telemin/repos/pfire_petsc/bin/pfire)
>     ==7368==
>     ==7368== Conditional jump or move depends on uninitialised value(s)
>     ==7368==??? at 0x58894EE: PetscAbsScalar(double) (petscmath.h:389)
>     ==7368==??? by 0x5889528: PetscIsNanScalar(double) (petscmath.h:674)
>     ==7368==??? by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
>     MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
>     ==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*,
>     MatFactorInfo
>     const*, FactorShiftCtx*, int) (matimpl.h:727)
>     ==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
>     ==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
>     ==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
>     ==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
>     ==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
>     ==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
>     (bjacobi.c:621)
>     ==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>     ==7368==??? by 0x62E8F0A: KSPSetUpOnBlocks (itfunc.c:213)
>     ==7368==
>     ==7368== Conditional jump or move depends on uninitialised value(s)
>     ==7368==??? at 0x4FDACE3: std::isnan(double) (cmath:627)
>     ==7368==??? by 0x4FDAC8D: PetscIsNanReal (mathinf.c:92)
>     ==7368==??? by 0x588952D: PetscIsNanScalar(double) (petscmath.h:674)
>     ==7368==??? by 0x588A784: MatPivotCheck_none(_p_Mat*, _p_Mat*,
>     MatFactorInfo const*, FactorShiftCtx*, int) (matimpl.h:704)
>     ==7368==??? by 0x588AD6E: MatPivotCheck(_p_Mat*, _p_Mat*,
>     MatFactorInfo
>     const*, FactorShiftCtx*, int) (matimpl.h:727)
>     ==7368==??? by 0x589177D: MatLUFactorNumeric_SeqAIJ (aijfact.c:558)
>     ==7368==??? by 0x5406B83: MatLUFactorNumeric (matrix.c:3031)
>     ==7368==??? by 0x609F317: PCSetUp_ILU(_p_PC*) (ilu.c:176)
>     ==7368==??? by 0x6192D4F: PCSetUp (precon.c:923)
>     ==7368==??? by 0x62EAE56: KSPSetUp (itfunc.c:381)
>     ==7368==??? by 0x60190BB: PCSetUpOnBlocks_BJacobi_Singleblock(_p_PC*)
>     (bjacobi.c:621)
>     ==7368==??? by 0x619352C: PCSetUpOnBlocks (precon.c:954)
>
>     -- 
>     Phil Tooley
>     Research Software Engineering
>     University of Sheffield
>
>
>
> -- 
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>

-- 
Phil Tooley
Research Software Engineering
University of Sheffield

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181026/8c3d4fb5/attachment-0001.html>

From bsmith at mcs.anl.gov  Fri Oct 26 20:01:45 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 27 Oct 2018 01:01:45 +0000
Subject: [petsc-users] Problems with fortran subroutine
 DMDAVecGetArrayF90
In-Reply-To: <HK0PR04MB2547CC42F3A0FF877E2126EAD0F00@HK0PR04MB2547.apcprd04.prod.outlook.com>
References: <HK0PR04MB2547CC42F3A0FF877E2126EAD0F00@HK0PR04MB2547.apcprd04.prod.outlook.com>
Message-ID: <882252B5-6D38-447D-AAAA-E9ED6690EA7E@anl.gov>


   You are calling the routine incorrectly. Take a look at usage in src/dm/examples/tutorials/ex11f90.F90

  Barry


> On Oct 26, 2018, at 5:16 AM, SONG Pengyang <Peterspy at outlook.com> wrote:
> 
> Hi all,
> I am trying to change a KSP tutorial code ex34.c to ex34f.F90. I found that after I changed DMDAVecGetArray to DMDAVecGetArrayF90, and DMDAVecRestoreArray to DMDAVecRestoreArrayF90. Compilation errors say that there is no matching specific subroutine for this generic subroutine call. Can anyone help me with the problem?
> The error massage is listed below and the fortran code is also attached. Many thanks.
>  
> ex34f.F90(83): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECGETARRAYF90]
>       call DMDAVecGetArrayF90(x,array,ierr)
> -----------^
> ex34f.F90(91): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECRESTOREARRAYF90]
>       call DMDAVecRestoreArrayF90(x,array,ierr)
> -----------^
> compilation aborted for ex34f.F90 (code 1)
>  
> ----------------------------------------------------------------
> Pengyang Song?????
> MSc candidate in Physical Oceanography
> College of Oceanic and Atmospheric Sciences
> Ocean University of China
> No. 238 Songling Road, Qingdao 266100, Shandong, China
> TEL: (+86) 15762264643
>  
> <ex34f.F90>


From Peterspy at outlook.com  Fri Oct 26 20:24:27 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Sat, 27 Oct 2018 01:24:27 +0000
Subject: [petsc-users] Problems with fortran subroutine
 DMDAVecGetArrayF90
Message-ID: <HK0PR04MB254731618006BDC469CC4B56D0F10@HK0PR04MB2547.apcprd04.prod.outlook.com>

Thanks Barry.
Another question is about subroutine ComputeMatrix(ksp,JJ,jac,ctx,ierr), which is called by KSPSetComputeOperators(ksp,ComputeMatrix,ctx,ierr).
In the tutorials, it seems that Mat JJ, the linear operator, is not assigned in ComputeMatrix. Only Mat jac, which supposed to be preconditioning matrix, is set. I am confused with this part. Why is the linear operator used without being set?
Looking forward to your reply.
Regards,
Pengyang
----------------------------------------------------------------
Pengyang Song?????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86) 15762264643

-----????-----
???: Smith, Barry F. <bsmith at mcs.anl.gov> 
????: 2018?10?27? 9:02
???: SONG Pengyang <Peterspy at outlook.com>
??: petsc-users at mcs.anl.gov
??: Re: [petsc-users] Problems with fortran subroutine DMDAVecGetArrayF90


   You are calling the routine incorrectly. Take a look at usage in src/dm/examples/tutorials/ex11f90.F90

  Barry


> On Oct 26, 2018, at 5:16 AM, SONG Pengyang <Peterspy at outlook.com> wrote:
> 
> Hi all,
> I am trying to change a KSP tutorial code ex34.c to ex34f.F90. I found that after I changed DMDAVecGetArray to DMDAVecGetArrayF90, and DMDAVecRestoreArray to DMDAVecRestoreArrayF90. Compilation errors say that there is no matching specific subroutine for this generic subroutine call. Can anyone help me with the problem?
> The error massage is listed below and the fortran code is also attached. Many thanks.
>  
> ex34f.F90(83): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECGETARRAYF90]
>       call DMDAVecGetArrayF90(x,array,ierr) -----------^
> ex34f.F90(91): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECRESTOREARRAYF90]
>       call DMDAVecRestoreArrayF90(x,array,ierr)
> -----------^
> compilation aborted for ex34f.F90 (code 1)
>  
> ----------------------------------------------------------------
> Pengyang Song?????
> MSc candidate in Physical Oceanography College of Oceanic and 
> Atmospheric Sciences Ocean University of China No. 238 Songling Road, 
> Qingdao 266100, Shandong, China
> TEL: (+86) 15762264643
>  
> <ex34f.F90>


From bsmith at mcs.anl.gov  Fri Oct 26 20:33:11 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 27 Oct 2018 01:33:11 +0000
Subject: [petsc-users] Problems with fortran subroutine
 DMDAVecGetArrayF90
In-Reply-To: <HK0PR04MB25473776AFE99D5FBB6595D1D0F10@HK0PR04MB2547.apcprd04.prod.outlook.com>
References: <HK0PR04MB25473776AFE99D5FBB6595D1D0F10@HK0PR04MB2547.apcprd04.prod.outlook.com>
Message-ID: <0386EEA3-B108-437D-AE5F-692D85564609@mcs.anl.gov>


   In the two fortran examples that use this construct

src/ksp/ksp/examples/tutorials/ex22f.F90 and ex45f.F90 

the JJ and jac matrix are the same object so only one of them needs to have values set.

In general (for KSP and SNES) either 

1) the JJ and jac matrices are the same object

or 

2) the JJ is a matrix-free operator (obtained for example with -snes_mf_operator) so doesn't need to have its values set but just needs to know the current location that the Jacobian is computed at which is set with the code:

       if ( A .ne. B) then
         call MatAssemblyBegin(A,MAT_FINAL_ASSEMBLY,ierr)
         call MatAssemblyEnd(A,MAT_FINAL_ASSEMBLY,ierr)
       endif

  Note: that if you want JJ and jac to be two regular matrices with different numerical values (for example JJ is 4th order discretization and jac is 2nd order) then you are correct the one must set values (different of course) into each of the matrices.


  Barry

   

> On Oct 26, 2018, at 9:22 PM, SONG Pengyang <Peterspy at outlook.com> wrote:
> 
> Thanks Barry.
> Another question is about subroutine ComputeMatrix(ksp,JJ,jac,ctx,ierr), which is called by KSPSetComputeOperators(ksp,ComputeMatrix,ctx,ierr).
> In the tutorials, it seems that Mat JJ, the linear operator, is not assigned in ComputeMatrix. Only Mat jac, which supposed to be preconditioning matrix, is set. I am confused with this part. Why is the linear operator used without being set?
> Looking forward to your reply.
> Regards,
> Pengyang
> ----------------------------------------------------------------
> Pengyang Song?????
> MSc candidate in Physical Oceanography
> College of Oceanic and Atmospheric Sciences
> Ocean University of China
> No. 238 Songling Road, Qingdao 266100, Shandong, China
> TEL: (+86) 15762264643
> 
> -----????-----
> ???: Smith, Barry F. <bsmith at mcs.anl.gov> 
> ????: 2018?10?27? 9:02
> ???: SONG Pengyang <Peterspy at outlook.com>
> ??: petsc-users at mcs.anl.gov
> ??: Re: [petsc-users] Problems with fortran subroutine DMDAVecGetArrayF90
> 
> 
>   You are calling the routine incorrectly. Take a look at usage in src/dm/examples/tutorials/ex11f90.F90
> 
>  Barry
> 
> 
>> On Oct 26, 2018, at 5:16 AM, SONG Pengyang <Peterspy at outlook.com> wrote:
>> 
>> Hi all,
>> I am trying to change a KSP tutorial code ex34.c to ex34f.F90. I found that after I changed DMDAVecGetArray to DMDAVecGetArrayF90, and DMDAVecRestoreArray to DMDAVecRestoreArrayF90. Compilation errors say that there is no matching specific subroutine for this generic subroutine call. Can anyone help me with the problem?
>> The error massage is listed below and the fortran code is also attached. Many thanks.
>> 
>> ex34f.F90(83): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECGETARRAYF90]
>>      call DMDAVecGetArrayF90(x,array,ierr) -----------^
>> ex34f.F90(91): error #6285: There is no matching specific subroutine for this generic subroutine call.   [DMDAVECRESTOREARRAYF90]
>>      call DMDAVecRestoreArrayF90(x,array,ierr)
>> -----------^
>> compilation aborted for ex34f.F90 (code 1)
>> 
>> ----------------------------------------------------------------
>> Pengyang Song?????
>> MSc candidate in Physical Oceanography College of Oceanic and 
>> Atmospheric Sciences Ocean University of China No. 238 Songling Road, 
>> Qingdao 266100, Shandong, China
>> TEL: (+86) 15762264643
>> 
>> <ex34f.F90>
> 


From fwan at mek.dtu.dk  Sat Oct 27 01:02:26 2018
From: fwan at mek.dtu.dk (Fengwen Wang)
Date: Sat, 27 Oct 2018 06:02:26 +0000
Subject: [petsc-users] How to impose  boundary conditions using DMDA
Message-ID: <265c311721234abea1563b41642a13de@mek.dtu.dk>

Dear Colleagues,


I use the finite element method to solve my problem in Petsc.


Mesh is defined as a  regular mesh using DMDA.  I have a special boundary condition which I  do not know how to impose it in Petsc.


In a 2D problem, the domain is unit size, two degrees of freedom per node (u, v). I would like impose the following boundary condition:


u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .


How can I impose such a boundary condition in Petsc?


Thanks a lot.


Best regards

Fengwen



____________________________________________________
Senior Researcher
Department of Mechanical Engineering, DTU
Nils Koppels All?
Building 404
2800  Kgs. Lyngby
fwan at mek.dtu.dk
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181027/6d715692/attachment.html>

From balay at mcs.anl.gov  Sat Oct 27 09:45:03 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Sat, 27 Oct 2018 14:45:03 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <tencent_7D30EA706521772457B88AC8@qq.com>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<alpine.LFD.2.21.1810252027470.23X-QQ-STYLE:
	<tencent_7D30EA706521772457B88AC8@qq.com>
Message-ID: <alpine.LFD.2.21.1810270939560.2360@asterix>

On Fri, 26 Oct 2018, avatar wrote:

> Thank Satish. I would like to call Matlab function in Pestc. For example, calculating the condition number of a petsc matrix.

Hm  - sounds like you need --with-matlab-engine. I'm not sure if we are still supporting this code.


> When I comment out the test from configuration script, I got the following problem. And the attached is the LOG file.
> ===============================================================================
>              Configuring PETSc to compile on your system
> ===============================================================================
> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)          *******************************************************************************
>         CONFIGURATION CRASH  (Please send configure.log to petsc-maint at mcs.anl.gov)
> *******************************************************************************

What likes did you comment? Here is what you could do..

Satish

-------
diff --git a/config/BuildSystem/config/packages/Matlab.py b/config/BuildSystem/config/packages/Matlab.py
index 351c7d2..66f4b72 100644
--- a/config/BuildSystem/config/packages/Matlab.py
+++ b/config/BuildSystem/config/packages/Matlab.py
@@ -44,17 +44,6 @@ class Configure(config.package.Package):
         interpreter = interpreter+' -'+self.argDB['with-matlab-arch']
 
       output      = ''
-      try:
-        output,err,ret = config.package.Package.executeShellCommand(interpreter+' -nojvm -nodisplay -r "display([\'Version \' version]); exit"', log = self.log)
-      except:
-        self.log.write('WARNING: Found Matlab at '+matlab+' but unable to run\n')
-        continue
-
-      match  = versionPattern.search(output)
-      r = float(match.group(1))
-      if r < 6.0:
-        self.log.write('WARNING: Matlab version must be at least 6; yours is '+str(r))
-        continue
       # make sure this is true root of Matlab
       if not os.path.isdir(os.path.join(matlab,'extern','lib')):
         self.log.write('WARNING:'+matlab+' is not the root directory for Matlab\n')

From knepley at gmail.com  Sat Oct 27 10:23:34 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 27 Oct 2018 11:23:34 -0400
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <265c311721234abea1563b41642a13de@mek.dtu.dk>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
Message-ID: <CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>

On Sat, Oct 27, 2018 at 2:02 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:

> Dear Colleagues,
>
>
> I use the finite element method to solve my problem in Petsc.
>
>
> Mesh is defined as a  regular mesh using DMDA.  I have a special boundary
> condition which I  do not know how to impose it in Petsc.
>
>
> In a 2D problem, the domain is unit size, two degrees of freedom per node
> (u, v). I would like impose the following boundary condition:
>
>
> u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .
>
>
> How can I impose such a boundary condition in Petsc?
>
In a serial code, you could do this just by equating those variables, but
in parallel we have no support for such a boundary condition.

  Thanks,

     Matt

> Thanks a lot.
>
>
> Best regards
>
> Fengwen
>
>
>
> ____________________________________________________
> Senior Researcher
> Department of Mechanical Engineering, DTU
> Nils Koppels All?
> Building 404
> 2800  Kgs. Lyngby
> fwan at mek.dtu.dk
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181027/2bf4aff5/attachment.html>

From bsmith at mcs.anl.gov  Sat Oct 27 17:52:22 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sat, 27 Oct 2018 22:52:22 +0000
Subject: [petsc-users] "Could not find a suitable archiver. Use
 --with-arto specify an archiver"
In-Reply-To: <alpine.LFD.2.21.1810270939560.2360@asterix>
References: <tencent_66005EB97387AAF422486D42@qq.com>
	<alpine.LFD.2.21.1810251723260.7009@asterix>
	<tencent_014E19233CF8B25F19D46D24@qq.com>
	<alpine.LFD.2.21.1810251752400.7009@asterix>
	<tencent_61AA8CAC07A9C90569DB8D7A@qq.com>
	<alpine.LFD.2.21.1810251818430.7009@asterix>
	<tencent_639577384C3EB7845D018642@qq.com>
	<alpine.LFD.2.21.1810251833190.7009@asterix>
	<tencent_58A588D62B3D43D106CA78FE@qq.com>
	<alpine.LFD.2.21.1810251839070.7009@asterix>
	<alpine.LFD.2.21.1810251841160.7009@asterix>
	<alpine.LFD.2.21.1810251846440.7009@asterix>
	<tencent_290A75DD41C7D04B4B36A320@qq.com>
	<alpine.LFD.2.21.1810251859390.7009@asterix>
	<tencent_12E9BAB855E3989758F79B2A@qq.com>
	<alpine.LFD.2.21.1810251924290.7009@asterix>
	<tencent_0F80EE9D23AFAC0F5A4ECB07@qq.com>
	<alpine.LFD.2.21.1810251942320.7009@asterix>
	<tencent_59464B694F19FC1F3EF5F7E5@qq.com>
	<alpine.LFD.2.21.1810251957370.7009@asterix>
	<tencent_6472241E3860D8990F94338F@qq.com>
	<tencent_7D30EA706521772457B88AC8@qq.com>
	<alpine.LFD.2.21.1810270939560.2360@asterix>
Message-ID: <8928A458-3B15-43DA-8B6A-2842C18B0002@mcs.anl.gov>



> On Oct 27, 2018, at 9:45 AM, Balay, Satish <balay at mcs.anl.gov> wrote:
> 
> On Fri, 26 Oct 2018, avatar wrote:
> 
>> Thank Satish. I would like to call Matlab function in Pestc. For example, calculating the condition number of a petsc matrix.
> 
> Hm  - sounds like you need --with-matlab-engine. I'm not sure if we are still supporting this code.

  Yes you need --with-matlab-engine also. 
> 
> 
>> When I comment out the test from configuration script, I got the following problem. And the attached is the LOG file.
>> ===============================================================================
>>             Configuring PETSc to compile on your system
>> ===============================================================================
>> TESTING: configureLibrary from config.packages.Matlab(config/BuildSystem/config/packages/Matlab.py:35)          *******************************************************************************
>>        CONFIGURATION CRASH  (Please send configure.log to petsc-maint at mcs.anl.gov)
>> *******************************************************************************
> 
> What likes did you comment? Here is what you could do..
> 
> Satish
> 
> -------
> diff --git a/config/BuildSystem/config/packages/Matlab.py b/config/BuildSystem/config/packages/Matlab.py
> index 351c7d2..66f4b72 100644
> --- a/config/BuildSystem/config/packages/Matlab.py
> +++ b/config/BuildSystem/config/packages/Matlab.py
> @@ -44,17 +44,6 @@ class Configure(config.package.Package):
>         interpreter = interpreter+' -'+self.argDB['with-matlab-arch']
> 
>       output      = ''
> -      try:
> -        output,err,ret = config.package.Package.executeShellCommand(interpreter+' -nojvm -nodisplay -r "display([\'Version \' version]); exit"', log = self.log)
> -      except:
> -        self.log.write('WARNING: Found Matlab at '+matlab+' but unable to run\n')
> -        continue
> -
> -      match  = versionPattern.search(output)
> -      r = float(match.group(1))
> -      if r < 6.0:
> -        self.log.write('WARNING: Matlab version must be at least 6; yours is '+str(r))
> -        continue
>       # make sure this is true root of Matlab
>       if not os.path.isdir(os.path.join(matlab,'extern','lib')):
>         self.log.write('WARNING:'+matlab+' is not the root directory for Matlab\n')


From repepo at gmail.com  Sun Oct 28 03:35:46 2018
From: repepo at gmail.com (Santiago Andres Triana)
Date: Sun, 28 Oct 2018 09:35:46 +0100
Subject: [petsc-users] [SLEPc] ex5 fails, error in lapack
In-Reply-To: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
References: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
Message-ID: <CAOrd_Sci2U=S=PJtZK_Kgdq2atkrKrFKuG4n3yUK4wSW_tTJEQ@mail.gmail.com>

Hi petsc-users,

I am experiencing problems running ex5 and ex7 from the slepc tutorial.
This is after upgrade to petsc-3.10.2 and slepc-3.10.1. Has anyone run into
this problem? see the error message below. Any help or advice would be
highly appreciated. Thanks in advance!

Santiago



trianas at hpcb-n02:/home/trianas/slepc-3.10.1/src/eps/examples/tutorials>
./ex5 -eps_nev 4

Markov Model, N=120 (m=15)

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Error in external library
[0]PETSC ERROR: Error in LAPACK subroutine hseqr: info=0
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
[0]PETSC ERROR: ./ex5 on a arch-linux2-c-opt named hpcb-n02 by trianas Sun
Oct 28 09:30:18 2018
[0]PETSC ERROR: Configure options --known-level1-dcache-size=32768
--known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8
--known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2
--known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
--known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8
--known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4
--known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1
--known-mpi-c-double-complex=1 --known-has-attribute-aligned=1
--with-scalar-type=complex --download-mumps=1 --download-parmetis
--download-metis --download-scalapack=1 --download-fblaslapack=1
--with-debugging=0 --download-superlu_dist=1 --download-ptscotch=1
CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native"
COPTFLAGS="-O3 -march=native" --with-batch --known-64-bit-blas-indices=1
[0]PETSC ERROR: #1 DSSolve_NHEP() line 586 in
/space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/impls/nhep/dsnhep.c
[0]PETSC ERROR: #2 DSSolve() line 586 in
/space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/interface/dsops.c
[0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Default() line 275 in
/space/hpc-home/trianas/slepc-3.10.1/src/eps/impls/krylov/krylovschur/krylovschur.c
[0]PETSC ERROR: #4 EPSSolve() line 148 in
/space/hpc-home/trianas/slepc-3.10.1/src/eps/interface/epssolve.c
[0]PETSC ERROR: #5 main() line 90 in
/home/trianas/slepc-3.10.1/src/eps/examples/tutorials/ex5.c
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -eps_nev 4
[0]PETSC ERROR: ----------------End of Error Message -------send entire
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76
:
system msg for write_line failure : Bad file descriptor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/18a65a5a/attachment.html>

From dave.mayhem23 at gmail.com  Sun Oct 28 04:31:06 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sun, 28 Oct 2018 09:31:06 +0000
Subject: [petsc-users] [SLEPc] ex5 fails, error in lapack
In-Reply-To: <CAOrd_Sci2U=S=PJtZK_Kgdq2atkrKrFKuG4n3yUK4wSW_tTJEQ@mail.gmail.com>
References: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
	<CAOrd_Sci2U=S=PJtZK_Kgdq2atkrKrFKuG4n3yUK4wSW_tTJEQ@mail.gmail.com>
Message-ID: <CAJ98EDqPW-SNAQW0ZXJ=9U9uN33aA=YYWNbokttVCyjEBmEj2g@mail.gmail.com>

On Sun, 28 Oct 2018 at 09:37, Santiago Andres Triana <repepo at gmail.com>
wrote:

> Hi petsc-users,
>
> I am experiencing problems running ex5 and ex7 from the slepc tutorial.
> This is after upgrade to petsc-3.10.2 and slepc-3.10.1. Has anyone run into
> this problem? see the error message below. Any help or advice would be
> highly appreciated. Thanks in advance!
>
> Santiago
>
>
>
> trianas at hpcb-n02:/home/trianas/slepc-3.10.1/src/eps/examples/tutorials>
> ./ex5 -eps_nev 4
>
> Markov Model, N=120 (m=15)
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Error in external library
> [0]PETSC ERROR: Error in LAPACK subroutine hseqr: info=0
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> [0]PETSC ERROR: ./ex5 on a arch-linux2-c-opt named hpcb-n02 by trianas Sun
> Oct 28 09:30:18 2018
> [0]PETSC ERROR: Configure options --known-level1-dcache-size=32768
> --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8
> --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2
> --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
> --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8
> --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4
> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1
> --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1
> --with-scalar-type=complex --download-mumps=1 --download-parmetis
> --download-metis --download-scalapack=1 --download-fblaslapack=1
> --with-debugging=0 --download-superlu_dist=1 --download-ptscotch=1
> CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native"
> COPTFLAGS="-O3 -march=native" --with-batch --known-64-bit-blas-indices=1
>

I think this last arg is wrong if you use --download-fblaslapack.

Did you explicitly add this option yourself?


[0]PETSC ERROR: #1 DSSolve_NHEP() line 586 in
> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/impls/nhep/dsnhep.c
> [0]PETSC ERROR: #2 DSSolve() line 586 in
> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/interface/dsops.c
> [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Default() line 275 in
> /space/hpc-home/trianas/slepc-3.10.1/src/eps/impls/krylov/krylovschur/krylovschur.c
> [0]PETSC ERROR: #4 EPSSolve() line 148 in
> /space/hpc-home/trianas/slepc-3.10.1/src/eps/interface/epssolve.c
> [0]PETSC ERROR: #5 main() line 90 in
> /home/trianas/slepc-3.10.1/src/eps/examples/tutorials/ex5.c
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -eps_nev 4
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76
> :
> system msg for write_line failure : Bad file descriptor
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/0121e128/attachment.html>

From bsmith at mcs.anl.gov  Sun Oct 28 11:18:21 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 28 Oct 2018 16:18:21 +0000
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
Message-ID: <516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>


    Matt,

       How difficult would it be to impose such boundary conditions with DMPlex? Presumably you just connect the mesh up "properly" and it is straightforward?

   Barry


> On Oct 27, 2018, at 10:23 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Sat, Oct 27, 2018 at 2:02 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:
> Dear Colleagues,
> 
> 
> 
> I use the finite element method to solve my problem in Petsc.
> 
> 
> 
> Mesh is defined as a  regular mesh using DMDA.  I have a special boundary condition which I  do not know how to impose it in Petsc. 
> 
> 
> 
> In a 2D problem, the domain is unit size, two degrees of freedom per node (u, v). I would like impose the following boundary condition:
> 
> 
> 
> u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .
> 
> 
> 
> How can I impose such a boundary condition in Petsc?  
> 
> In a serial code, you could do this just by equating those variables, but in parallel we have no support for such a boundary condition.
> 
>   Thanks,
> 
>      Matt
> Thanks a lot. 
> 
> 
> 
> Best regards
> 
> Fengwen
> 
> 
> 
> 
> 
> ____________________________________________________
> Senior Researcher
> Department of Mechanical Engineering, DTU
> Nils Koppels All?       
> Building 404    
> 2800  Kgs. Lyngby 
> fwan at mek.dtu.dk
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/


From bhatiamanav at gmail.com  Sun Oct 28 12:16:19 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Sun, 28 Oct 2018 12:16:19 -0500
Subject: [petsc-users] Convergence of AMG
Message-ID: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>

Hi, 

   I am attempting to solve a Mindlin plate bending problem with AMG solver in petsc. This test case is with a mesh of 300x300 elements and 543,606 dofs. 

   The discretization includes 6 variables (u, v, w, tx, ty, tz), but only three are relevant for plate bending (w, tx, ty). 

   I am calling the solver with the following options: 

-pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view 

  And the convergence behavior is shown below, along with the ksp_view information. Based on notes in the manual, this seems to be subpar convergence rate. At the end of the solution the norm of each variable is : 

var: 0: u  : norm: 5.505909e-18
var: 1: v  : norm: 7.639640e-18
var: 2: w : norm: 3.901464e-03
var: 3: tx : norm: 4.403576e-02
var: 4: ty : norm: 4.403576e-02
var: 5: tz : norm: 1.148409e-16

  I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not make a lot of difference in the norms of (w, tx, ty). 
 
  I do provide the solver with 6 rigid-body vectors to approximate the null-space of the problem. Without these the solver shows very poor convergence. 

  I would appreciate advice on possible strategies to improve this behavior. 

Thanks,
Manav 

    0 KSP Residual norm 1.696304497261e+00 
    1 KSP Residual norm 1.120485505777e+00 
    2 KSP Residual norm 8.324222302402e-01 
    3 KSP Residual norm 6.477349534115e-01 
    4 KSP Residual norm 5.080936471292e-01 
    5 KSP Residual norm 4.051099646638e-01 
    6 KSP Residual norm 3.260432664653e-01 
    7 KSP Residual norm 2.560483838143e-01 
    8 KSP Residual norm 2.029943986124e-01 
    9 KSP Residual norm 1.560985741610e-01 
   10 KSP Residual norm 1.163720702140e-01 
   11 KSP Residual norm 8.488411085459e-02 
   12 KSP Residual norm 5.888041729034e-02 
   13 KSP Residual norm 4.027792209980e-02 
   14 KSP Residual norm 2.819048087304e-02 
   15 KSP Residual norm 1.904674196962e-02 
   16 KSP Residual norm 1.289302447822e-02 
   17 KSP Residual norm 9.162203296376e-03 
   18 KSP Residual norm 7.016781679507e-03 
   19 KSP Residual norm 5.399170865328e-03 
   20 KSP Residual norm 4.254385887482e-03 
   21 KSP Residual norm 3.530831740621e-03 
   22 KSP Residual norm 2.946780747923e-03 
   23 KSP Residual norm 2.339361361128e-03 
   24 KSP Residual norm 1.815072489282e-03 
   25 KSP Residual norm 1.408814185342e-03 
   26 KSP Residual norm 1.063795714320e-03 
   27 KSP Residual norm 7.828540233117e-04 
   28 KSP Residual norm 5.683910750067e-04 
   29 KSP Residual norm 4.131151010250e-04 
   30 KSP Residual norm 3.065608221019e-04 
   31 KSP Residual norm 2.634114273459e-04 
   32 KSP Residual norm 2.198180137626e-04 
   33 KSP Residual norm 1.748956510799e-04 
   34 KSP Residual norm 1.317539710010e-04 
   35 KSP Residual norm 9.790121566055e-05 
   36 KSP Residual norm 7.465935386094e-05 
   37 KSP Residual norm 5.689506626052e-05 
   38 KSP Residual norm 4.413136619126e-05 
   39 KSP Residual norm 3.512194236402e-05 
   40 KSP Residual norm 2.877755408287e-05 
   41 KSP Residual norm 2.340080556431e-05 
   42 KSP Residual norm 1.904544450345e-05 
   43 KSP Residual norm 1.504723478235e-05 
   44 KSP Residual norm 1.141381950576e-05 
   45 KSP Residual norm 8.206151384599e-06 
   46 KSP Residual norm 5.911426091276e-06 
   47 KSP Residual norm 4.233669089283e-06 
   48 KSP Residual norm 2.898052944223e-06 
   49 KSP Residual norm 2.023556779973e-06 
   50 KSP Residual norm 1.459108043935e-06 
   51 KSP Residual norm 1.097335545865e-06 
   52 KSP Residual norm 8.440457332262e-07 
   53 KSP Residual norm 6.705616854004e-07 
   54 KSP Residual norm 5.404888680234e-07 
   55 KSP Residual norm 4.391368084979e-07 
   56 KSP Residual norm 3.697063014621e-07 
   57 KSP Residual norm 3.021772094146e-07 
   58 KSP Residual norm 2.479354520792e-07 
   59 KSP Residual norm 2.013077841968e-07 
   60 KSP Residual norm 1.553159612793e-07 
   61 KSP Residual norm 1.400784224898e-07 
   62 KSP Residual norm 9.707453662195e-08 
   63 KSP Residual norm 7.263173080146e-08 
   64 KSP Residual norm 5.593723572132e-08 
   65 KSP Residual norm 4.448788809586e-08 
   66 KSP Residual norm 3.613992590778e-08 
   67 KSP Residual norm 2.946099051876e-08 
   68 KSP Residual norm 2.408053564170e-08 
   69 KSP Residual norm 1.945257374856e-08 
   70 KSP Residual norm 1.572494535110e-08 


KSP Object: 4 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=6 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.   0.   0.   0.  
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
          Symmetric graph false
          Number of levels to square graph 1
          Number smoothing steps 1
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 4 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 4 MPI processes
      type: bjacobi
        number of blocks = 4
        Local solve is same for all blocks, in the following KSP and PC objects:
      KSP Object: (mg_coarse_sub_) 1 MPI processes
        type: preonly
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (mg_coarse_sub_) 1 MPI processes
        type: lu
          out-of-place factorization
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
          matrix ordering: nd
          factor fill ratio given 5., needed 1.
            Factored matrix follows:
              Mat Object: 1 MPI processes
                type: seqaij
                rows=6, cols=6, bs=6
                package used to perform factorization: petsc
                total: nonzeros=36, allocated nonzeros=36
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 2 nodes, limit used is 5
        linear system matrix = precond matrix:
        Mat Object: 1 MPI processes
          type: seqaij
          rows=6, cols=6, bs=6
          total: nonzeros=36, allocated nonzeros=36
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 2 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6, cols=6, bs=6
        total: nonzeros=36, allocated nonzeros=36
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 2 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 4 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.099971, max = 1.09968
        eigenvalues estimate via gmres min 0.154032, max 0.99971
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_1_esteig_) 4 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=54, cols=54, bs=6
        total: nonzeros=2916, allocated nonzeros=2916
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 11 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 4 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.171388, max = 1.88526
        eigenvalues estimate via gmres min 0.0717873, max 1.71388
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_2_esteig_) 4 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=642, cols=642, bs=6
        total: nonzeros=99468, allocated nonzeros=99468
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 47 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 4 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.164216, max = 1.80637
        eigenvalues estimate via gmres min 0.0376323, max 1.64216
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_3_esteig_) 4 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6726, cols=6726, bs=6
        total: nonzeros=941796, allocated nonzeros=941796
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 552 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 4 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.163283, max = 1.79611
        eigenvalues estimate via gmres min 0.0350306, max 1.63283
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_4_esteig_) 4 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=41022, cols=41022, bs=6
        total: nonzeros=2852316, allocated nonzeros=2852316
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 3432 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 5 -------------------------------
    KSP Object: (mg_levels_5_) 4 MPI processes
      type: chebyshev
        eigenvalue estimates used:  min = 0.157236, max = 1.7296
        eigenvalues estimate via gmres min 0.0317897, max 1.57236
        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
        KSP Object: (mg_levels_5_esteig_) 4 MPI processes
          type: gmres
            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
          left preconditioning
          using PRECONDITIONED norm type for convergence test
        estimating eigenvalues using noisy right hand side
      maximum iterations=2, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_5_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: () 4 MPI processes
        type: mpiaij
        rows=543606, cols=543606, bs=6
        total: nonzeros=29224836, allocated nonzeros=29302596
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: () 4 MPI processes
    type: mpiaij
    rows=543606, cols=543606, bs=6
    total: nonzeros=29224836, allocated nonzeros=29302596
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      using I-node (on process 0) routines: found 45644 nodes, limit used is 5


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/f30fc716/attachment-0001.html>

From repepo at gmail.com  Sun Oct 28 15:46:19 2018
From: repepo at gmail.com (Santiago Andres Triana)
Date: Sun, 28 Oct 2018 21:46:19 +0100
Subject: [petsc-users] [SLEPc] ex5 fails, error in lapack
In-Reply-To: <CAJ98EDqPW-SNAQW0ZXJ=9U9uN33aA=YYWNbokttVCyjEBmEj2g@mail.gmail.com>
References: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
	<CAOrd_Sci2U=S=PJtZK_Kgdq2atkrKrFKuG4n3yUK4wSW_tTJEQ@mail.gmail.com>
	<CAJ98EDqPW-SNAQW0ZXJ=9U9uN33aA=YYWNbokttVCyjEBmEj2g@mail.gmail.com>
Message-ID: <CAOrd_SeaiH7kcAXJNDeJOF8jAWzE6F2f14RnfvbbYYonu5AoAw@mail.gmail.com>

Hi Dave,

Indeed, I added that last arg myself after the configure script asked for
it (--with-batch seems to need it). I just tried with petsc-3.9.1, without
the --with-batch and --known-64-blas-indices=1 options and everything is
working nicely. I will try again later with the latest version.

Thanks!

Santiago

On Sun, Oct 28, 2018 at 10:31 AM Dave May <dave.mayhem23 at gmail.com> wrote:

>
>
> On Sun, 28 Oct 2018 at 09:37, Santiago Andres Triana <repepo at gmail.com>
> wrote:
>
>> Hi petsc-users,
>>
>> I am experiencing problems running ex5 and ex7 from the slepc tutorial.
>> This is after upgrade to petsc-3.10.2 and slepc-3.10.1. Has anyone run into
>> this problem? see the error message below. Any help or advice would be
>> highly appreciated. Thanks in advance!
>>
>> Santiago
>>
>>
>>
>> trianas at hpcb-n02:/home/trianas/slepc-3.10.1/src/eps/examples/tutorials>
>> ./ex5 -eps_nev 4
>>
>> Markov Model, N=120 (m=15)
>>
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Error in external library
>> [0]PETSC ERROR: Error in LAPACK subroutine hseqr: info=0
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>> [0]PETSC ERROR: ./ex5 on a arch-linux2-c-opt named hpcb-n02 by trianas
>> Sun Oct 28 09:30:18 2018
>> [0]PETSC ERROR: Configure options --known-level1-dcache-size=32768
>> --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8
>> --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2
>> --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
>> --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8
>> --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4
>> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1
>> --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1
>> --with-scalar-type=complex --download-mumps=1 --download-parmetis
>> --download-metis --download-scalapack=1 --download-fblaslapack=1
>> --with-debugging=0 --download-superlu_dist=1 --download-ptscotch=1
>> CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native"
>> COPTFLAGS="-O3 -march=native" --with-batch --known-64-bit-blas-indices=1
>>
>
> I think this last arg is wrong if you use --download-fblaslapack.
>
> Did you explicitly add this option yourself?
>
>
> [0]PETSC ERROR: #1 DSSolve_NHEP() line 586 in
>> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/impls/nhep/dsnhep.c
>> [0]PETSC ERROR: #2 DSSolve() line 586 in
>> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/interface/dsops.c
>> [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Default() line 275 in
>> /space/hpc-home/trianas/slepc-3.10.1/src/eps/impls/krylov/krylovschur/krylovschur.c
>> [0]PETSC ERROR: #4 EPSSolve() line 148 in
>> /space/hpc-home/trianas/slepc-3.10.1/src/eps/interface/epssolve.c
>> [0]PETSC ERROR: #5 main() line 90 in
>> /home/trianas/slepc-3.10.1/src/eps/examples/tutorials/ex5.c
>> [0]PETSC ERROR: PETSc Option Table entries:
>> [0]PETSC ERROR: -eps_nev 4
>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>> error message to petsc-maint at mcs.anl.gov----------
>> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76
>> :
>> system msg for write_line failure : Bad file descriptor
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/b8922d4b/attachment.html>

From bsmith at mcs.anl.gov  Sun Oct 28 15:55:27 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Sun, 28 Oct 2018 20:55:27 +0000
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
Message-ID: <6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>



> On Oct 28, 2018, at 12:16 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Hi, 
> 
>    I am attempting to solve a Mindlin plate bending problem with AMG solver in petsc. This test case is with a mesh of 300x300 elements and 543,606 dofs. 
> 
>    The discretization includes 6 variables (u, v, w, tx, ty, tz), but only three are relevant for plate bending (w, tx, ty). 
> 
>    I am calling the solver with the following options: 
> 
> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view 
> 
>   And the convergence behavior is shown below, along with the ksp_view information. Based on notes in the manual, this seems to be subpar convergence rate. At the end of the solution the norm of each variable is : 
> 
> var: 0: u  : norm: 5.505909e-18
> var: 1: v  : norm: 7.639640e-18
> var: 2: w : norm: 3.901464e-03
> var: 3: tx : norm: 4.403576e-02
> var: 4: ty : norm: 4.403576e-02
> var: 5: tz : norm: 1.148409e-16

   What do you mean by var: 2: w : norm etc? Is this the norm of the error for that variable, the norm of the residual, something else? How exactly are you calculating it?

    Thanks


   Barry

> 
>   I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not make a lot of difference in the norms of (w, tx, ty). 
>  
>   I do provide the solver with 6 rigid-body vectors to approximate the null-space of the problem. Without these the solver shows very poor convergence. 
> 
>   I would appreciate advice on possible strategies to improve this behavior. 
> 
> Thanks,
> Manav 
> 
>     0 KSP Residual norm 1.696304497261e+00 
>     1 KSP Residual norm 1.120485505777e+00 
>     2 KSP Residual norm 8.324222302402e-01 
>     3 KSP Residual norm 6.477349534115e-01 
>     4 KSP Residual norm 5.080936471292e-01 
>     5 KSP Residual norm 4.051099646638e-01 
>     6 KSP Residual norm 3.260432664653e-01 
>     7 KSP Residual norm 2.560483838143e-01 
>     8 KSP Residual norm 2.029943986124e-01 
>     9 KSP Residual norm 1.560985741610e-01 
>    10 KSP Residual norm 1.163720702140e-01 
>    11 KSP Residual norm 8.488411085459e-02 
>    12 KSP Residual norm 5.888041729034e-02 
>    13 KSP Residual norm 4.027792209980e-02 
>    14 KSP Residual norm 2.819048087304e-02 
>    15 KSP Residual norm 1.904674196962e-02 
>    16 KSP Residual norm 1.289302447822e-02 
>    17 KSP Residual norm 9.162203296376e-03 
>    18 KSP Residual norm 7.016781679507e-03 
>    19 KSP Residual norm 5.399170865328e-03 
>    20 KSP Residual norm 4.254385887482e-03 
>    21 KSP Residual norm 3.530831740621e-03 
>    22 KSP Residual norm 2.946780747923e-03 
>    23 KSP Residual norm 2.339361361128e-03 
>    24 KSP Residual norm 1.815072489282e-03 
>    25 KSP Residual norm 1.408814185342e-03 
>    26 KSP Residual norm 1.063795714320e-03 
>    27 KSP Residual norm 7.828540233117e-04 
>    28 KSP Residual norm 5.683910750067e-04 
>    29 KSP Residual norm 4.131151010250e-04 
>    30 KSP Residual norm 3.065608221019e-04 
>    31 KSP Residual norm 2.634114273459e-04 
>    32 KSP Residual norm 2.198180137626e-04 
>    33 KSP Residual norm 1.748956510799e-04 
>    34 KSP Residual norm 1.317539710010e-04 
>    35 KSP Residual norm 9.790121566055e-05 
>    36 KSP Residual norm 7.465935386094e-05 
>    37 KSP Residual norm 5.689506626052e-05 
>    38 KSP Residual norm 4.413136619126e-05 
>    39 KSP Residual norm 3.512194236402e-05 
>    40 KSP Residual norm 2.877755408287e-05 
>    41 KSP Residual norm 2.340080556431e-05 
>    42 KSP Residual norm 1.904544450345e-05 
>    43 KSP Residual norm 1.504723478235e-05 
>    44 KSP Residual norm 1.141381950576e-05 
>    45 KSP Residual norm 8.206151384599e-06 
>    46 KSP Residual norm 5.911426091276e-06 
>    47 KSP Residual norm 4.233669089283e-06 
>    48 KSP Residual norm 2.898052944223e-06 
>    49 KSP Residual norm 2.023556779973e-06 
>    50 KSP Residual norm 1.459108043935e-06 
>    51 KSP Residual norm 1.097335545865e-06 
>    52 KSP Residual norm 8.440457332262e-07 
>    53 KSP Residual norm 6.705616854004e-07 
>    54 KSP Residual norm 5.404888680234e-07 
>    55 KSP Residual norm 4.391368084979e-07 
>    56 KSP Residual norm 3.697063014621e-07 
>    57 KSP Residual norm 3.021772094146e-07 
>    58 KSP Residual norm 2.479354520792e-07 
>    59 KSP Residual norm 2.013077841968e-07 
>    60 KSP Residual norm 1.553159612793e-07 
>    61 KSP Residual norm 1.400784224898e-07 
>    62 KSP Residual norm 9.707453662195e-08 
>    63 KSP Residual norm 7.263173080146e-08 
>    64 KSP Residual norm 5.593723572132e-08 
>    65 KSP Residual norm 4.448788809586e-08 
>    66 KSP Residual norm 3.613992590778e-08 
>    67 KSP Residual norm 2.946099051876e-08 
>    68 KSP Residual norm 2.408053564170e-08 
>    69 KSP Residual norm 1.945257374856e-08 
>    70 KSP Residual norm 1.572494535110e-08 
> 
> 
> KSP Object: 4 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 4 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=6 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.   0.   0.   0.  
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 4 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 4 MPI processes
>       type: bjacobi
>         number of blocks = 4
>         Local solve is same for all blocks, in the following KSP and PC objects:
>       KSP Object: (mg_coarse_sub_) 1 MPI processes
>         type: preonly
>         maximum iterations=1, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>         left preconditioning
>         using NONE norm type for convergence test
>       PC Object: (mg_coarse_sub_) 1 MPI processes
>         type: lu
>           out-of-place factorization
>           tolerance for zero pivot 2.22045e-14
>           using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>           matrix ordering: nd
>           factor fill ratio given 5., needed 1.
>             Factored matrix follows:
>               Mat Object: 1 MPI processes
>                 type: seqaij
>                 rows=6, cols=6, bs=6
>                 package used to perform factorization: petsc
>                 total: nonzeros=36, allocated nonzeros=36
>                 total number of mallocs used during MatSetValues calls =0
>                   using I-node routines: found 2 nodes, limit used is 5
>         linear system matrix = precond matrix:
>         Mat Object: 1 MPI processes
>           type: seqaij
>           rows=6, cols=6, bs=6
>           total: nonzeros=36, allocated nonzeros=36
>           total number of mallocs used during MatSetValues calls =0
>             using I-node routines: found 2 nodes, limit used is 5
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=6, cols=6, bs=6
>         total: nonzeros=36, allocated nonzeros=36
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 2 nodes, limit used is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.099971, max = 1.09968
>         eigenvalues estimate via gmres min 0.154032, max 0.99971
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>         KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=54, cols=54, bs=6
>         total: nonzeros=2916, allocated nonzeros=2916
>         total number of mallocs used during MatSetValues calls =0
>           using I-node (on process 0) routines: found 11 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.171388, max = 1.88526
>         eigenvalues estimate via gmres min 0.0717873, max 1.71388
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>         KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=642, cols=642, bs=6
>         total: nonzeros=99468, allocated nonzeros=99468
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 47 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.164216, max = 1.80637
>         eigenvalues estimate via gmres min 0.0376323, max 1.64216
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>         KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=6726, cols=6726, bs=6
>         total: nonzeros=941796, allocated nonzeros=941796
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 552 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.163283, max = 1.79611
>         eigenvalues estimate via gmres min 0.0350306, max 1.63283
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>         KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=41022, cols=41022, bs=6
>         total: nonzeros=2852316, allocated nonzeros=2852316
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 3432 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 5 -------------------------------
>     KSP Object: (mg_levels_5_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.157236, max = 1.7296
>         eigenvalues estimate via gmres min 0.0317897, max 1.57236
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>         KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_5_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: () 4 MPI processes
>         type: mpiaij
>         rows=543606, cols=543606, bs=6
>         total: nonzeros=29224836, allocated nonzeros=29302596
>         total number of mallocs used during MatSetValues calls =0
>           has attached near null space
>           using I-node (on process 0) routines: found 45644 nodes, limit used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: () 4 MPI processes
>     type: mpiaij
>     rows=543606, cols=543606, bs=6
>     total: nonzeros=29224836, allocated nonzeros=29302596
>     total number of mallocs used during MatSetValues calls =0
>       has attached near null space
>       using I-node (on process 0) routines: found 45644 nodes, limit used is 5
> 
> 


From dave.mayhem23 at gmail.com  Sun Oct 28 16:01:33 2018
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sun, 28 Oct 2018 21:01:33 +0000
Subject: [petsc-users] [SLEPc] ex5 fails, error in lapack
In-Reply-To: <CAOrd_SeaiH7kcAXJNDeJOF8jAWzE6F2f14RnfvbbYYonu5AoAw@mail.gmail.com>
References: <F39A5533-FD1B-4F22-94ED-61EF45E49949@umich.edu>
	<CAOrd_Sci2U=S=PJtZK_Kgdq2atkrKrFKuG4n3yUK4wSW_tTJEQ@mail.gmail.com>
	<CAJ98EDqPW-SNAQW0ZXJ=9U9uN33aA=YYWNbokttVCyjEBmEj2g@mail.gmail.com>
	<CAOrd_SeaiH7kcAXJNDeJOF8jAWzE6F2f14RnfvbbYYonu5AoAw@mail.gmail.com>
Message-ID: <CAJ98EDrWtP--ssSQGrM-pFi7tW-Rj7XzNDDtbXdA9PuZ67U2oQ@mail.gmail.com>

On Sun, 28 Oct 2018 at 21:46, Santiago Andres Triana <repepo at gmail.com>
wrote:

> Hi Dave,
>
> Indeed, I added that last arg myself after the configure script asked for
> it (--with-batch seems to need it). I just tried with petsc-3.9.1, without
> the --with-batch and --known-64-blas-indices=1 options and everything is
> working nicely.
>

Great.

I believe as general rule, flags such as -known-64-bit-xxx are only
required to be specified by the user when using system provided packages
(actually any package not installed by petsc' configure). If you use
--download-yyy then petscs' configure defines how package yyy is to be
configured and built, hence it knows whether it used 64 bit ints, or not -
the user does not (and probably should not) provide a flag to indicate what
petsc configuration already knows


Thanks,
  Dave

I will try again later with the latest version.
>

Ok.



> Thanks!
>
> Santiago
>
> On Sun, Oct 28, 2018 at 10:31 AM Dave May <dave.mayhem23 at gmail.com> wrote:
>
>>
>>
>> On Sun, 28 Oct 2018 at 09:37, Santiago Andres Triana <repepo at gmail.com>
>> wrote:
>>
>>> Hi petsc-users,
>>>
>>> I am experiencing problems running ex5 and ex7 from the slepc tutorial.
>>> This is after upgrade to petsc-3.10.2 and slepc-3.10.1. Has anyone run into
>>> this problem? see the error message below. Any help or advice would be
>>> highly appreciated. Thanks in advance!
>>>
>>> Santiago
>>>
>>>
>>>
>>> trianas at hpcb-n02:/home/trianas/slepc-3.10.1/src/eps/examples/tutorials>
>>> ./ex5 -eps_nev 4
>>>
>>> Markov Model, N=120 (m=15)
>>>
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Error in external library
>>> [0]PETSC ERROR: Error in LAPACK subroutine hseqr: info=0
>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>> for trouble shooting.
>>> [0]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>>> [0]PETSC ERROR: ./ex5 on a arch-linux2-c-opt named hpcb-n02 by trianas
>>> Sun Oct 28 09:30:18 2018
>>> [0]PETSC ERROR: Configure options --known-level1-dcache-size=32768
>>> --known-level1-dcache-linesize=64 --known-level1-dcache-assoc=8
>>> --known-sizeof-char=1 --known-sizeof-void-p=8 --known-sizeof-short=2
>>> --known-sizeof-int=4 --known-sizeof-long=8 --known-sizeof-long-long=8
>>> --known-sizeof-float=4 --known-sizeof-double=8 --known-sizeof-size_t=8
>>> --known-bits-per-byte=8 --known-memcmp-ok=1 --known-sizeof-MPI_Comm=4
>>> --known-sizeof-MPI_Fint=4 --known-mpi-long-double=1 --known-mpi-int64_t=1
>>> --known-mpi-c-double-complex=1 --known-has-attribute-aligned=1
>>> --with-scalar-type=complex --download-mumps=1 --download-parmetis
>>> --download-metis --download-scalapack=1 --download-fblaslapack=1
>>> --with-debugging=0 --download-superlu_dist=1 --download-ptscotch=1
>>> CXXOPTFLAGS="-O3 -march=native" FOPTFLAGS="-O3 -march=native"
>>> COPTFLAGS="-O3 -march=native" --with-batch --known-64-bit-blas-indices=1
>>>
>>
>> I think this last arg is wrong if you use --download-fblaslapack.
>>
>> Did you explicitly add this option yourself?
>>
>>
>> [0]PETSC ERROR: #1 DSSolve_NHEP() line 586 in
>>> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/impls/nhep/dsnhep.c
>>> [0]PETSC ERROR: #2 DSSolve() line 586 in
>>> /space/hpc-home/trianas/slepc-3.10.1/src/sys/classes/ds/interface/dsops.c
>>> [0]PETSC ERROR: #3 EPSSolve_KrylovSchur_Default() line 275 in
>>> /space/hpc-home/trianas/slepc-3.10.1/src/eps/impls/krylov/krylovschur/krylovschur.c
>>> [0]PETSC ERROR: #4 EPSSolve() line 148 in
>>> /space/hpc-home/trianas/slepc-3.10.1/src/eps/interface/epssolve.c
>>> [0]PETSC ERROR: #5 main() line 90 in
>>> /home/trianas/slepc-3.10.1/src/eps/examples/tutorials/ex5.c
>>> [0]PETSC ERROR: PETSc Option Table entries:
>>> [0]PETSC ERROR: -eps_nev 4
>>> [0]PETSC ERROR: ----------------End of Error Message -------send entire
>>> error message to petsc-maint at mcs.anl.gov----------
>>> application called MPI_Abort(MPI_COMM_WORLD, 76) - process 0
>>> [unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=76
>>> :
>>> system msg for write_line failure : Bad file descriptor
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/c45c97ca/attachment-0001.html>

From bhatiamanav at gmail.com  Sun Oct 28 16:12:05 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Sun, 28 Oct 2018 16:12:05 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
Message-ID: <4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>

Var: 0,?,5  are the 6 variables that I am solving for: u, v, w, theta_x, theta_y, theta_z. 

The norms identified in my email are the L2 norms of all dofs corresponding to each variable in the solution vector. So, var: 0: u: norm is the L2 norm of the dofs for u only, and so on. 

I expect u, v, theta_z to be zero for the solution, which ends up being the case. 

If I plot the solution, they look sensible, but the reduction of KSP norm is slow. 


Thanks,
Manav

> On Oct 28, 2018, at 3:55 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
> 
>> On Oct 28, 2018, at 12:16 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Hi, 
>> 
>>   I am attempting to solve a Mindlin plate bending problem with AMG solver in petsc. This test case is with a mesh of 300x300 elements and 543,606 dofs. 
>> 
>>   The discretization includes 6 variables (u, v, w, tx, ty, tz), but only three are relevant for plate bending (w, tx, ty). 
>> 
>>   I am calling the solver with the following options: 
>> 
>> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view 
>> 
>>  And the convergence behavior is shown below, along with the ksp_view information. Based on notes in the manual, this seems to be subpar convergence rate. At the end of the solution the norm of each variable is : 
>> 
>> var: 0: u  : norm: 5.505909e-18
>> var: 1: v  : norm: 7.639640e-18
>> var: 2: w : norm: 3.901464e-03
>> var: 3: tx : norm: 4.403576e-02
>> var: 4: ty : norm: 4.403576e-02
>> var: 5: tz : norm: 1.148409e-16
> 
>   What do you mean by var: 2: w : norm etc? Is this the norm of the error for that variable, the norm of the residual, something else? How exactly are you calculating it?
> 
>    Thanks
> 
> 
>   Barry
> 
>> 
>>  I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not make a lot of difference in the norms of (w, tx, ty). 
>> 
>>  I do provide the solver with 6 rigid-body vectors to approximate the null-space of the problem. Without these the solver shows very poor convergence. 
>> 
>>  I would appreciate advice on possible strategies to improve this behavior. 
>> 
>> Thanks,
>> Manav 
>> 
>>    0 KSP Residual norm 1.696304497261e+00 
>>    1 KSP Residual norm 1.120485505777e+00 
>>    2 KSP Residual norm 8.324222302402e-01 
>>    3 KSP Residual norm 6.477349534115e-01 
>>    4 KSP Residual norm 5.080936471292e-01 
>>    5 KSP Residual norm 4.051099646638e-01 
>>    6 KSP Residual norm 3.260432664653e-01 
>>    7 KSP Residual norm 2.560483838143e-01 
>>    8 KSP Residual norm 2.029943986124e-01 
>>    9 KSP Residual norm 1.560985741610e-01 
>>   10 KSP Residual norm 1.163720702140e-01 
>>   11 KSP Residual norm 8.488411085459e-02 
>>   12 KSP Residual norm 5.888041729034e-02 
>>   13 KSP Residual norm 4.027792209980e-02 
>>   14 KSP Residual norm 2.819048087304e-02 
>>   15 KSP Residual norm 1.904674196962e-02 
>>   16 KSP Residual norm 1.289302447822e-02 
>>   17 KSP Residual norm 9.162203296376e-03 
>>   18 KSP Residual norm 7.016781679507e-03 
>>   19 KSP Residual norm 5.399170865328e-03 
>>   20 KSP Residual norm 4.254385887482e-03 
>>   21 KSP Residual norm 3.530831740621e-03 
>>   22 KSP Residual norm 2.946780747923e-03 
>>   23 KSP Residual norm 2.339361361128e-03 
>>   24 KSP Residual norm 1.815072489282e-03 
>>   25 KSP Residual norm 1.408814185342e-03 
>>   26 KSP Residual norm 1.063795714320e-03 
>>   27 KSP Residual norm 7.828540233117e-04 
>>   28 KSP Residual norm 5.683910750067e-04 
>>   29 KSP Residual norm 4.131151010250e-04 
>>   30 KSP Residual norm 3.065608221019e-04 
>>   31 KSP Residual norm 2.634114273459e-04 
>>   32 KSP Residual norm 2.198180137626e-04 
>>   33 KSP Residual norm 1.748956510799e-04 
>>   34 KSP Residual norm 1.317539710010e-04 
>>   35 KSP Residual norm 9.790121566055e-05 
>>   36 KSP Residual norm 7.465935386094e-05 
>>   37 KSP Residual norm 5.689506626052e-05 
>>   38 KSP Residual norm 4.413136619126e-05 
>>   39 KSP Residual norm 3.512194236402e-05 
>>   40 KSP Residual norm 2.877755408287e-05 
>>   41 KSP Residual norm 2.340080556431e-05 
>>   42 KSP Residual norm 1.904544450345e-05 
>>   43 KSP Residual norm 1.504723478235e-05 
>>   44 KSP Residual norm 1.141381950576e-05 
>>   45 KSP Residual norm 8.206151384599e-06 
>>   46 KSP Residual norm 5.911426091276e-06 
>>   47 KSP Residual norm 4.233669089283e-06 
>>   48 KSP Residual norm 2.898052944223e-06 
>>   49 KSP Residual norm 2.023556779973e-06 
>>   50 KSP Residual norm 1.459108043935e-06 
>>   51 KSP Residual norm 1.097335545865e-06 
>>   52 KSP Residual norm 8.440457332262e-07 
>>   53 KSP Residual norm 6.705616854004e-07 
>>   54 KSP Residual norm 5.404888680234e-07 
>>   55 KSP Residual norm 4.391368084979e-07 
>>   56 KSP Residual norm 3.697063014621e-07 
>>   57 KSP Residual norm 3.021772094146e-07 
>>   58 KSP Residual norm 2.479354520792e-07 
>>   59 KSP Residual norm 2.013077841968e-07 
>>   60 KSP Residual norm 1.553159612793e-07 
>>   61 KSP Residual norm 1.400784224898e-07 
>>   62 KSP Residual norm 9.707453662195e-08 
>>   63 KSP Residual norm 7.263173080146e-08 
>>   64 KSP Residual norm 5.593723572132e-08 
>>   65 KSP Residual norm 4.448788809586e-08 
>>   66 KSP Residual norm 3.613992590778e-08 
>>   67 KSP Residual norm 2.946099051876e-08 
>>   68 KSP Residual norm 2.408053564170e-08 
>>   69 KSP Residual norm 1.945257374856e-08 
>>   70 KSP Residual norm 1.572494535110e-08 
>> 
>> 
>> KSP Object: 4 MPI processes
>>  type: gmres
>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>    happy breakdown tolerance 1e-30
>>  maximum iterations=10000, initial guess is zero
>>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>>  left preconditioning
>>  using PRECONDITIONED norm type for convergence test
>> PC Object: 4 MPI processes
>>  type: gamg
>>    type is MULTIPLICATIVE, levels=6 cycles=v
>>      Cycles per PCApply=1
>>      Using externally compute Galerkin coarse grid matrices
>>      GAMG specific options
>>        Threshold for dropping small values in graph on each level =   0.   0.   0.   0.  
>>        Threshold scaling factor for each level not specified = 1.
>>        AGG specific options
>>          Symmetric graph false
>>          Number of levels to square graph 1
>>          Number smoothing steps 1
>>  Coarse grid solver -- level -------------------------------
>>    KSP Object: (mg_coarse_) 4 MPI processes
>>      type: preonly
>>      maximum iterations=10000, initial guess is zero
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_coarse_) 4 MPI processes
>>      type: bjacobi
>>        number of blocks = 4
>>        Local solve is same for all blocks, in the following KSP and PC objects:
>>      KSP Object: (mg_coarse_sub_) 1 MPI processes
>>        type: preonly
>>        maximum iterations=1, initial guess is zero
>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>        left preconditioning
>>        using NONE norm type for convergence test
>>      PC Object: (mg_coarse_sub_) 1 MPI processes
>>        type: lu
>>          out-of-place factorization
>>          tolerance for zero pivot 2.22045e-14
>>          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>          matrix ordering: nd
>>          factor fill ratio given 5., needed 1.
>>            Factored matrix follows:
>>              Mat Object: 1 MPI processes
>>                type: seqaij
>>                rows=6, cols=6, bs=6
>>                package used to perform factorization: petsc
>>                total: nonzeros=36, allocated nonzeros=36
>>                total number of mallocs used during MatSetValues calls =0
>>                  using I-node routines: found 2 nodes, limit used is 5
>>        linear system matrix = precond matrix:
>>        Mat Object: 1 MPI processes
>>          type: seqaij
>>          rows=6, cols=6, bs=6
>>          total: nonzeros=36, allocated nonzeros=36
>>          total number of mallocs used during MatSetValues calls =0
>>            using I-node routines: found 2 nodes, limit used is 5
>>      linear system matrix = precond matrix:
>>      Mat Object: 4 MPI processes
>>        type: mpiaij
>>        rows=6, cols=6, bs=6
>>        total: nonzeros=36, allocated nonzeros=36
>>        total number of mallocs used during MatSetValues calls =0
>>          using nonscalable MatPtAP() implementation
>>          using I-node (on process 0) routines: found 2 nodes, limit used is 5
>>  Down solver (pre-smoother) on level 1 -------------------------------
>>    KSP Object: (mg_levels_1_) 4 MPI processes
>>      type: chebyshev
>>        eigenvalue estimates used:  min = 0.099971, max = 1.09968
>>        eigenvalues estimate via gmres min 0.154032, max 0.99971
>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>        KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>>          type: gmres
>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>            happy breakdown tolerance 1e-30
>>          maximum iterations=10, initial guess is zero
>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>          left preconditioning
>>          using PRECONDITIONED norm type for convergence test
>>        estimating eigenvalues using noisy right hand side
>>      maximum iterations=2, nonzero initial guess
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_levels_1_) 4 MPI processes
>>      type: sor
>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>      linear system matrix = precond matrix:
>>      Mat Object: 4 MPI processes
>>        type: mpiaij
>>        rows=54, cols=54, bs=6
>>        total: nonzeros=2916, allocated nonzeros=2916
>>        total number of mallocs used during MatSetValues calls =0
>>          using I-node (on process 0) routines: found 11 nodes, limit used is 5
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  Down solver (pre-smoother) on level 2 -------------------------------
>>    KSP Object: (mg_levels_2_) 4 MPI processes
>>      type: chebyshev
>>        eigenvalue estimates used:  min = 0.171388, max = 1.88526
>>        eigenvalues estimate via gmres min 0.0717873, max 1.71388
>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>        KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>>          type: gmres
>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>            happy breakdown tolerance 1e-30
>>          maximum iterations=10, initial guess is zero
>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>          left preconditioning
>>          using PRECONDITIONED norm type for convergence test
>>        estimating eigenvalues using noisy right hand side
>>      maximum iterations=2, nonzero initial guess
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_levels_2_) 4 MPI processes
>>      type: sor
>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>      linear system matrix = precond matrix:
>>      Mat Object: 4 MPI processes
>>        type: mpiaij
>>        rows=642, cols=642, bs=6
>>        total: nonzeros=99468, allocated nonzeros=99468
>>        total number of mallocs used during MatSetValues calls =0
>>          using nonscalable MatPtAP() implementation
>>          using I-node (on process 0) routines: found 47 nodes, limit used is 5
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  Down solver (pre-smoother) on level 3 -------------------------------
>>    KSP Object: (mg_levels_3_) 4 MPI processes
>>      type: chebyshev
>>        eigenvalue estimates used:  min = 0.164216, max = 1.80637
>>        eigenvalues estimate via gmres min 0.0376323, max 1.64216
>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>        KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>>          type: gmres
>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>            happy breakdown tolerance 1e-30
>>          maximum iterations=10, initial guess is zero
>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>          left preconditioning
>>          using PRECONDITIONED norm type for convergence test
>>        estimating eigenvalues using noisy right hand side
>>      maximum iterations=2, nonzero initial guess
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_levels_3_) 4 MPI processes
>>      type: sor
>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>      linear system matrix = precond matrix:
>>      Mat Object: 4 MPI processes
>>        type: mpiaij
>>        rows=6726, cols=6726, bs=6
>>        total: nonzeros=941796, allocated nonzeros=941796
>>        total number of mallocs used during MatSetValues calls =0
>>          using nonscalable MatPtAP() implementation
>>          using I-node (on process 0) routines: found 552 nodes, limit used is 5
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  Down solver (pre-smoother) on level 4 -------------------------------
>>    KSP Object: (mg_levels_4_) 4 MPI processes
>>      type: chebyshev
>>        eigenvalue estimates used:  min = 0.163283, max = 1.79611
>>        eigenvalues estimate via gmres min 0.0350306, max 1.63283
>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>        KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>>          type: gmres
>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>            happy breakdown tolerance 1e-30
>>          maximum iterations=10, initial guess is zero
>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>          left preconditioning
>>          using PRECONDITIONED norm type for convergence test
>>        estimating eigenvalues using noisy right hand side
>>      maximum iterations=2, nonzero initial guess
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_levels_4_) 4 MPI processes
>>      type: sor
>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>      linear system matrix = precond matrix:
>>      Mat Object: 4 MPI processes
>>        type: mpiaij
>>        rows=41022, cols=41022, bs=6
>>        total: nonzeros=2852316, allocated nonzeros=2852316
>>        total number of mallocs used during MatSetValues calls =0
>>          using nonscalable MatPtAP() implementation
>>          using I-node (on process 0) routines: found 3432 nodes, limit used is 5
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  Down solver (pre-smoother) on level 5 -------------------------------
>>    KSP Object: (mg_levels_5_) 4 MPI processes
>>      type: chebyshev
>>        eigenvalue estimates used:  min = 0.157236, max = 1.7296
>>        eigenvalues estimate via gmres min 0.0317897, max 1.57236
>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>        KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>>          type: gmres
>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>            happy breakdown tolerance 1e-30
>>          maximum iterations=10, initial guess is zero
>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>          left preconditioning
>>          using PRECONDITIONED norm type for convergence test
>>        estimating eigenvalues using noisy right hand side
>>      maximum iterations=2, nonzero initial guess
>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>      left preconditioning
>>      using NONE norm type for convergence test
>>    PC Object: (mg_levels_5_) 4 MPI processes
>>      type: sor
>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>      linear system matrix = precond matrix:
>>      Mat Object: () 4 MPI processes
>>        type: mpiaij
>>        rows=543606, cols=543606, bs=6
>>        total: nonzeros=29224836, allocated nonzeros=29302596
>>        total number of mallocs used during MatSetValues calls =0
>>          has attached near null space
>>          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>  linear system matrix = precond matrix:
>>  Mat Object: () 4 MPI processes
>>    type: mpiaij
>>    rows=543606, cols=543606, bs=6
>>    total: nonzeros=29224836, allocated nonzeros=29302596
>>    total number of mallocs used during MatSetValues calls =0
>>      has attached near null space
>>      using I-node (on process 0) routines: found 45644 nodes, limit used is 5

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181028/ec3929ca/attachment-0001.html>

From knepley at gmail.com  Mon Oct 29 04:24:36 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Oct 2018 05:24:36 -0400
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
Message-ID: <CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>

On Sun, Oct 28, 2018 at 12:18 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote

>     Matt,
>
>        How difficult would it be to impose such boundary conditions with
> DMPlex? Presumably you just connect the mesh up "properly" and it is
> straightforward?
>

If you figure out what topology is meant by this, it should not be hard.
However, I
cannot figure out what they mean right now.

  Thanks,

    Matt


>    Barry
>
>
> > On Oct 27, 2018, at 10:23 AM, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > On Sat, Oct 27, 2018 at 2:02 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:
> > Dear Colleagues,
> >
> >
> >
> > I use the finite element method to solve my problem in Petsc.
> >
> >
> >
> > Mesh is defined as a  regular mesh using DMDA.  I have a special
> boundary condition which I  do not know how to impose it in Petsc.
> >
> >
> >
> > In a 2D problem, the domain is unit size, two degrees of freedom per
> node (u, v). I would like impose the following boundary condition:
> >
> >
> >
> > u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .
> >
> >
> >
> > How can I impose such a boundary condition in Petsc?
> >
> > In a serial code, you could do this just by equating those variables,
> but in parallel we have no support for such a boundary condition.
> >
> >   Thanks,
> >
> >      Matt
> > Thanks a lot.
> >
> >
> >
> > Best regards
> >
> > Fengwen
> >
> >
> >
> >
> >
> > ____________________________________________________
> > Senior Researcher
> > Department of Mechanical Engineering, DTU
> > Nils Koppels All?
> > Building 404
> > 2800  Kgs. Lyngby
> > fwan at mek.dtu.dk
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> > https://www.cse.buffalo.edu/~knepley/
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/3fd82c81/attachment.html>

From fwan at mek.dtu.dk  Mon Oct 29 04:38:08 2018
From: fwan at mek.dtu.dk (Fengwen Wang)
Date: Mon, 29 Oct 2018 09:38:08 +0000
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>,
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
Message-ID: <e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>


Hi Matt and Barry,


I only have  a regular 2D  square domain of a unit cell.


How can I use DMPlex to apply such a boundary condition?


Thanks a lot.


Best regards

Fengwen



____________________________________________________
Senior Researcher
Department of Mechanical Engineering, DTU
Nils Koppels All?
Building 404
2800  Kgs. Lyngby
fwan at mek.dtu.dk
________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Monday, October 29, 2018 10:24:36 AM
To: Barry Smith
Cc: Fengwen Wang; PETSc; PETSc
Subject: Re: [petsc-maint] How to impose boundary conditions using DMDA


On Sun, Oct 28, 2018 at 12:18 PM Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>> wrote
    Matt,

       How difficult would it be to impose such boundary conditions with DMPlex? Presumably you just connect the mesh up "properly" and it is straightforward?

If you figure out what topology is meant by this, it should not be hard. However, I
cannot figure out what they mean right now.

  Thanks,

    Matt

   Barry


> On Oct 27, 2018, at 10:23 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
>
> On Sat, Oct 27, 2018 at 2:02 AM Fengwen Wang <fwan at mek.dtu.dk<mailto:fwan at mek.dtu.dk>> wrote:
> Dear Colleagues,
>
>
>
> I use the finite element method to solve my problem in Petsc.
>
>
>
> Mesh is defined as a  regular mesh using DMDA.  I have a special boundary condition which I  do not know how to impose it in Petsc.
>
>
>
> In a 2D problem, the domain is unit size, two degrees of freedom per node (u, v). I would like impose the following boundary condition:
>
>
>
> u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .
>
>
>
> How can I impose such a boundary condition in Petsc?
>
> In a serial code, you could do this just by equating those variables, but in parallel we have no support for such a boundary condition.
>
>   Thanks,
>
>      Matt
> Thanks a lot.
>
>
>
> Best regards
>
> Fengwen
>
>
>
>
>
> ____________________________________________________
> Senior Researcher
> Department of Mechanical Engineering, DTU
> Nils Koppels All?
> Building 404
> 2800  Kgs. Lyngby
> fwan at mek.dtu.dk<mailto:fwan at mek.dtu.dk>
>
>
> --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/



--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/<http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/eacaea4f/attachment.html>

From knepley at gmail.com  Mon Oct 29 05:09:16 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Oct 2018 06:09:16 -0400
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
Message-ID: <CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>

On Mon, Oct 29, 2018 at 5:39 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:

> Hi Matt and Barry,
>
> I only have  a regular 2D  square domain of a unit cell.
>
The boundary condition implies some sort of topology. For example, if you
condition was

  u, v (x = 0) = u, v (x = 1)

you are on a cylinder. And if you add

  u, v (y = 0) = u, v (y = 1)

you are on a torus. However, you are hooking the right edge to the top edge
and also transforming the basis. I cannot understand what is meant.

  Thanks,

     Matt

> How can I use DMPlex to apply such a boundary condition?
>
>
> Thanks a lot.
>
>
> Best regards
>
> Fengwen
>
>
>
> ____________________________________________________
> Senior Researcher
> Department of Mechanical Engineering, DTU
> Nils Koppels All?
> Building 404
> 2800  Kgs. Lyngby
> fwan at mek.dtu.dk
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Monday, October 29, 2018 10:24:36 AM
> *To:* Barry Smith
> *Cc:* Fengwen Wang; PETSc; PETSc
> *Subject:* Re: [petsc-maint] How to impose boundary conditions using DMDA
>
>
> On Sun, Oct 28, 2018 at 12:18 PM Smith, Barry F. <bsmith at mcs.anl.gov>
> wrote
>
>>     Matt,
>>
>>        How difficult would it be to impose such boundary conditions with
>> DMPlex? Presumably you just connect the mesh up "properly" and it is
>> straightforward?
>>
>
> If you figure out what topology is meant by this, it should not be hard.
> However, I
> cannot figure out what they mean right now.
>
>   Thanks,
>
>     Matt
>
>
>>    Barry
>>
>>
>> > On Oct 27, 2018, at 10:23 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> >
>> > On Sat, Oct 27, 2018 at 2:02 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:
>> > Dear Colleagues,
>> >
>> >
>> >
>> > I use the finite element method to solve my problem in Petsc.
>> >
>> >
>> >
>> > Mesh is defined as a  regular mesh using DMDA.  I have a special
>> boundary condition which I  do not know how to impose it in Petsc.
>> >
>> >
>> >
>> > In a 2D problem, the domain is unit size, two degrees of freedom per
>> node (u, v). I would like impose the following boundary condition:
>> >
>> >
>> >
>> > u(x=1) = -v ( y=1) and v(x=1 )= -u (y=1) .
>> >
>> >
>> >
>> > How can I impose such a boundary condition in Petsc?
>> >
>> > In a serial code, you could do this just by equating those variables,
>> but in parallel we have no support for such a boundary condition.
>> >
>> >   Thanks,
>> >
>> >      Matt
>> > Thanks a lot.
>> >
>> >
>> >
>> > Best regards
>> >
>> > Fengwen
>> >
>> >
>> >
>> >
>> >
>> > ____________________________________________________
>> > Senior Researcher
>> > Department of Mechanical Engineering, DTU
>> > Nils Koppels All?
>> > Building 404
>> > 2800  Kgs. Lyngby
>> > fwan at mek.dtu.dk
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> > https://www.cse.buffalo.edu/~knepley/
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/41cf7c80/attachment.html>

From wencel at gmail.com  Mon Oct 29 05:22:10 2018
From: wencel at gmail.com (Lawrence Mitchell)
Date: Mon, 29 Oct 2018 10:22:10 +0000
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
	<CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
Message-ID: <2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>



> On 29 Oct 2018, at 10:09, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Mon, Oct 29, 2018 at 5:39 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:
> Hi Matt and Barry,
> 
> I only have  a regular 2D  square domain of a unit cell. 
> 
> The boundary condition implies some sort of topology. For example, if you condition was
> 
>   u, v (x = 0) = u, v (x = 1)
> 
> you are on a cylinder. And if you add
> 
>   u, v (y = 0) = u, v (y = 1)
> 
> you are on a torus. However, you are hooking the right edge to the top edge
> and also transforming the basis. I cannot understand what is meant.

I'm no good at drawing embedded things, I read the bc like the attached picture. You sort of fold the top and right edges together "inwards", which means that stuff that flows out of the right edge flows in through the top edge, and stuff that is flowing up along the right edge ends up flowing left along the top edge?

Lawrence
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/17060209/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_1343.jpg
Type: image/jpeg
Size: 170075 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/17060209/attachment-0001.jpg>

From knepley at gmail.com  Mon Oct 29 05:56:42 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Oct 2018 06:56:42 -0400
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
	<CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
	<2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>
Message-ID: <CAMYG4GmRt8x8hVopoeDKMyNaXE1SotFOZ1xwRCHzj+rH+wbt-A@mail.gmail.com>

On Mon, Oct 29, 2018 at 6:22 AM Lawrence Mitchell <wencel at gmail.com> wrote:

>
>
> On 29 Oct 2018, at 10:09, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Mon, Oct 29, 2018 at 5:39 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:
> Hi Matt and Barry,
>
> I only have  a regular 2D  square domain of a unit cell.
>
> The boundary condition implies some sort of topology. For example, if you
> condition was
>
>   u, v (x = 0) = u, v (x = 1)
>
> you are on a cylinder. And if you add
>
>   u, v (y = 0) = u, v (y = 1)
>
> you are on a torus. However, you are hooking the right edge to the top edge
> and also transforming the basis. I cannot understand what is meant.
>
>
> I'm no good at drawing embedded things, I read the bc like the attached
> picture. You sort of fold the top and right edges together "inwards", which
> means that stuff that flows out of the right edge flows in through the top
> edge, and stuff that is flowing up along the right edge ends up flowing
> left along the top edge?
>

You can certainly map the right edge to the top edge (its topologically a
square again), but that
mapping is not smooth, and I do not know how you would make a global basis
for the approximation space.

   Matt


> Lawrence
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/5fd04a9d/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_1343.jpg
Type: image/jpeg
Size: 170075 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/5fd04a9d/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: IMG_1343.jpg
Type: image/jpeg
Size: 170075 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/5fd04a9d/attachment-0003.jpg>

From wence at gmx.li  Mon Oct 29 07:17:34 2018
From: wence at gmx.li (Lawrence Mitchell)
Date: Mon, 29 Oct 2018 12:17:34 +0000
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <CAMYG4GmRt8x8hVopoeDKMyNaXE1SotFOZ1xwRCHzj+rH+wbt-A@mail.gmail.com>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
	<CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
	<2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>
	<CAMYG4GmRt8x8hVopoeDKMyNaXE1SotFOZ1xwRCHzj+rH+wbt-A@mail.gmail.com>
Message-ID: <ED2A5E73-4A90-47A5-8820-6511CF5174F7@gmx.li>



> On 29 Oct 2018, at 10:56, Matthew Knepley <knepley at gmail.com> wrote:
> 
> You can certainly map the right edge to the top edge (its topologically a square again), but that
> mapping is not smooth, and I do not know how you would make a global basis for the approximation space.

Yeah, I thought this was topologically just a cone, but that would have:

u(x=1) = -v(y=1)

and

v(x=1) = u(y=1) (note matching sign).

So if you wanted the v(x=1) = -u(y=1) there would be a twist somehow, and then a discontinuity in the mapping.

Lawrence

From mfadams at lbl.gov  Mon Oct 29 08:14:07 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 29 Oct 2018 09:14:07 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
Message-ID: <CADOhEh6RLbdhQDdH64PdbWMwv6LpFEnaF=xQJ5LH87=PQLjRVg@mail.gmail.com>

On Sun, Oct 28, 2018 at 1:17 PM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Hi,
>
>    I am attempting to solve a Mindlin plate bending problem with AMG
> solver in petsc. This test case is with a mesh of 300x300 elements and
> 543,606 dofs.
>
>    The discretization includes 6 variables (u, v, w, tx, ty, tz), but only
> three are relevant for plate bending (w, tx, ty).
>
>    I am calling the solver with the following options:
>
> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6
> -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view
>
>   And the convergence behavior is shown below, along with the ksp_view
> information. Based on notes in the manual, this seems to be subpar
> convergence rate. At the end of the solution the norm of each variable is :
>
> var: 0: u  : norm: 5.505909e-18
> var: 1: v  : norm: 7.639640e-18
> var: 2: w : norm: 3.901464e-03
> var: 3: tx : norm: 4.403576e-02
> var: 4: ty : norm: 4.403576e-02
> var: 5: tz : norm: 1.148409e-16
>

So your u,v,tz are inactive? I assume that this looks ok to you.


>
>   I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does
> not make a lot of difference in the norms of (w, tx, ty).
>
>   I do provide the solver with 6 rigid-body vectors to approximate the
> null-space of the problem. Without these the solver shows very poor
> convergence.
>

So the RBMs helped, that tells me that you did hook it in correctly.

Shells are hard and I don't know what your test case is, so I'm not sure
how bad this is. I can recommend using:

-mg_levels_esteig_ksp_type cg

This will give better eigen estimates for symmetric problems. You want to
coarsen slowly for plates. Start with:

-pc_gamg_square_graph 0

 Mark


>   I would appreciate advice on possible strategies to improve this
> behavior.
>
> Thanks,
> Manav
>
>     0 KSP Residual norm 1.696304497261e+00
>     1 KSP Residual norm 1.120485505777e+00
>     2 KSP Residual norm 8.324222302402e-01
>     3 KSP Residual norm 6.477349534115e-01
>     4 KSP Residual norm 5.080936471292e-01
>     5 KSP Residual norm 4.051099646638e-01
>     6 KSP Residual norm 3.260432664653e-01
>     7 KSP Residual norm 2.560483838143e-01
>     8 KSP Residual norm 2.029943986124e-01
>     9 KSP Residual norm 1.560985741610e-01
>    10 KSP Residual norm 1.163720702140e-01
>    11 KSP Residual norm 8.488411085459e-02
>    12 KSP Residual norm 5.888041729034e-02
>    13 KSP Residual norm 4.027792209980e-02
>    14 KSP Residual norm 2.819048087304e-02
>    15 KSP Residual norm 1.904674196962e-02
>    16 KSP Residual norm 1.289302447822e-02
>    17 KSP Residual norm 9.162203296376e-03
>    18 KSP Residual norm 7.016781679507e-03
>    19 KSP Residual norm 5.399170865328e-03
>    20 KSP Residual norm 4.254385887482e-03
>    21 KSP Residual norm 3.530831740621e-03
>    22 KSP Residual norm 2.946780747923e-03
>    23 KSP Residual norm 2.339361361128e-03
>    24 KSP Residual norm 1.815072489282e-03
>    25 KSP Residual norm 1.408814185342e-03
>    26 KSP Residual norm 1.063795714320e-03
>    27 KSP Residual norm 7.828540233117e-04
>    28 KSP Residual norm 5.683910750067e-04
>    29 KSP Residual norm 4.131151010250e-04
>    30 KSP Residual norm 3.065608221019e-04
>    31 KSP Residual norm 2.634114273459e-04
>    32 KSP Residual norm 2.198180137626e-04
>    33 KSP Residual norm 1.748956510799e-04
>    34 KSP Residual norm 1.317539710010e-04
>    35 KSP Residual norm 9.790121566055e-05
>    36 KSP Residual norm 7.465935386094e-05
>    37 KSP Residual norm 5.689506626052e-05
>    38 KSP Residual norm 4.413136619126e-05
>    39 KSP Residual norm 3.512194236402e-05
>    40 KSP Residual norm 2.877755408287e-05
>    41 KSP Residual norm 2.340080556431e-05
>    42 KSP Residual norm 1.904544450345e-05
>    43 KSP Residual norm 1.504723478235e-05
>    44 KSP Residual norm 1.141381950576e-05
>    45 KSP Residual norm 8.206151384599e-06
>    46 KSP Residual norm 5.911426091276e-06
>    47 KSP Residual norm 4.233669089283e-06
>    48 KSP Residual norm 2.898052944223e-06
>    49 KSP Residual norm 2.023556779973e-06
>    50 KSP Residual norm 1.459108043935e-06
>    51 KSP Residual norm 1.097335545865e-06
>    52 KSP Residual norm 8.440457332262e-07
>    53 KSP Residual norm 6.705616854004e-07
>    54 KSP Residual norm 5.404888680234e-07
>    55 KSP Residual norm 4.391368084979e-07
>    56 KSP Residual norm 3.697063014621e-07
>    57 KSP Residual norm 3.021772094146e-07
>    58 KSP Residual norm 2.479354520792e-07
>    59 KSP Residual norm 2.013077841968e-07
>    60 KSP Residual norm 1.553159612793e-07
>    61 KSP Residual norm 1.400784224898e-07
>    62 KSP Residual norm 9.707453662195e-08
>    63 KSP Residual norm 7.263173080146e-08
>    64 KSP Residual norm 5.593723572132e-08
>    65 KSP Residual norm 4.448788809586e-08
>    66 KSP Residual norm 3.613992590778e-08
>    67 KSP Residual norm 2.946099051876e-08
>    68 KSP Residual norm 2.408053564170e-08
>    69 KSP Residual norm 1.945257374856e-08
>    70 KSP Residual norm 1.572494535110e-08
>
>
> KSP Object: 4 MPI processes
>   type: gmres
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 4 MPI processes
>   type: gamg
>     type is MULTIPLICATIVE, levels=6 cycles=v
>       Cycles per PCApply=1
>       Using externally compute Galerkin coarse grid matrices
>       GAMG specific options
>         Threshold for dropping small values in graph on each level =   0.
>   0.   0.   0.
>         Threshold scaling factor for each level not specified = 1.
>         AGG specific options
>           Symmetric graph false
>           Number of levels to square graph 1
>           Number smoothing steps 1
>   Coarse grid solver -- level -------------------------------
>     KSP Object: (mg_coarse_) 4 MPI processes
>       type: preonly
>       maximum iterations=10000, initial guess is zero
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_coarse_) 4 MPI processes
>       type: bjacobi
>         number of blocks = 4
>         Local solve is same for all blocks, in the following KSP and PC
> objects:
>       KSP Object: (mg_coarse_sub_) 1 MPI processes
>         type: preonly
>         maximum iterations=1, initial guess is zero
>         tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>         left preconditioning
>         using NONE norm type for convergence test
>       PC Object: (mg_coarse_sub_) 1 MPI processes
>         type: lu
>           out-of-place factorization
>           tolerance for zero pivot 2.22045e-14
>           using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>           matrix ordering: nd
>           factor fill ratio given 5., needed 1.
>             Factored matrix follows:
>               Mat Object: 1 MPI processes
>                 type: seqaij
>                 rows=6, cols=6, bs=6
>                 package used to perform factorization: petsc
>                 total: nonzeros=36, allocated nonzeros=36
>                 total number of mallocs used during MatSetValues calls =0
>                   using I-node routines: found 2 nodes, limit used is 5
>         linear system matrix = precond matrix:
>         Mat Object: 1 MPI processes
>           type: seqaij
>           rows=6, cols=6, bs=6
>           total: nonzeros=36, allocated nonzeros=36
>           total number of mallocs used during MatSetValues calls =0
>             using I-node routines: found 2 nodes, limit used is 5
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=6, cols=6, bs=6
>         total: nonzeros=36, allocated nonzeros=36
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 2 nodes, limit used
> is 5
>   Down solver (pre-smoother) on level 1 -------------------------------
>     KSP Object: (mg_levels_1_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.099971, max = 1.09968
>         eigenvalues estimate via gmres min 0.154032, max 0.99971
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>         KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_1_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=54, cols=54, bs=6
>         total: nonzeros=2916, allocated nonzeros=2916
>         total number of mallocs used during MatSetValues calls =0
>           using I-node (on process 0) routines: found 11 nodes, limit used
> is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 2 -------------------------------
>     KSP Object: (mg_levels_2_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.171388, max = 1.88526
>         eigenvalues estimate via gmres min 0.0717873, max 1.71388
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>         KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_2_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=642, cols=642, bs=6
>         total: nonzeros=99468, allocated nonzeros=99468
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 47 nodes, limit used
> is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 3 -------------------------------
>     KSP Object: (mg_levels_3_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.164216, max = 1.80637
>         eigenvalues estimate via gmres min 0.0376323, max 1.64216
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>         KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_3_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=6726, cols=6726, bs=6
>         total: nonzeros=941796, allocated nonzeros=941796
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 552 nodes, limit
> used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 4 -------------------------------
>     KSP Object: (mg_levels_4_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.163283, max = 1.79611
>         eigenvalues estimate via gmres min 0.0350306, max 1.63283
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>         KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_4_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: 4 MPI processes
>         type: mpiaij
>         rows=41022, cols=41022, bs=6
>         total: nonzeros=2852316, allocated nonzeros=2852316
>         total number of mallocs used during MatSetValues calls =0
>           using nonscalable MatPtAP() implementation
>           using I-node (on process 0) routines: found 3432 nodes, limit
> used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   Down solver (pre-smoother) on level 5 -------------------------------
>     KSP Object: (mg_levels_5_) 4 MPI processes
>       type: chebyshev
>         eigenvalue estimates used:  min = 0.157236, max = 1.7296
>         eigenvalues estimate via gmres min 0.0317897, max 1.57236
>         eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>         KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>           type: gmres
>             restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>             happy breakdown tolerance 1e-30
>           maximum iterations=10, initial guess is zero
>           tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>           left preconditioning
>           using PRECONDITIONED norm type for convergence test
>         estimating eigenvalues using noisy right hand side
>       maximum iterations=2, nonzero initial guess
>       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>       left preconditioning
>       using NONE norm type for convergence test
>     PC Object: (mg_levels_5_) 4 MPI processes
>       type: sor
>         type = local_symmetric, iterations = 1, local iterations = 1,
> omega = 1.
>       linear system matrix = precond matrix:
>       Mat Object: () 4 MPI processes
>         type: mpiaij
>         rows=543606, cols=543606, bs=6
>         total: nonzeros=29224836, allocated nonzeros=29302596
>         total number of mallocs used during MatSetValues calls =0
>           has attached near null space
>           using I-node (on process 0) routines: found 45644 nodes, limit
> used is 5
>   Up solver (post-smoother) same as down solver (pre-smoother)
>   linear system matrix = precond matrix:
>   Mat Object: () 4 MPI processes
>     type: mpiaij
>     rows=543606, cols=543606, bs=6
>     total: nonzeros=29224836, allocated nonzeros=29302596
>     total number of mallocs used during MatSetValues calls =0
>       has attached near null space
>       using I-node (on process 0) routines: found 45644 nodes, limit used
> is 5
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/ea9d10b2/attachment-0001.html>

From mfadams at lbl.gov  Mon Oct 29 08:28:02 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 29 Oct 2018 09:28:02 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
Message-ID: <CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>

I would recommend using '-mg_levels 2' and check that that gives you two
levels. I would also run this on one processor just to start.

Use -mg_levels_ksp_max_it 4.  And '-pc_gamg_threshold 0.04'

These parameters are meant to increase convergence rate at all costs. Once
we our best rate we should be able to back off of some of this without much
degradation or play around with the parameters to optimize run time.


On Sun, Oct 28, 2018 at 5:13 PM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Var: 0,?,5  are the 6 variables that I am solving for: u, v, w, theta_x,
> theta_y, theta_z.
>
> The norms identified in my email are the L2 norms of all dofs
> corresponding to each variable in the solution vector. So, var: 0: u: norm
> is the L2 norm of the dofs for u only, and so on.
>
> I expect u, v, theta_z to be zero for the solution, which ends up being
> the case.
>
> If I plot the solution, they look sensible, but the reduction of KSP norm
> is slow.
>
>
> Thanks,
> Manav
>
> On Oct 28, 2018, at 3:55 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>
>
>
> On Oct 28, 2018, at 12:16 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Hi,
>
>   I am attempting to solve a Mindlin plate bending problem with AMG solver
> in petsc. This test case is with a mesh of 300x300 elements and 543,606
> dofs.
>
>   The discretization includes 6 variables (u, v, w, tx, ty, tz), but only
> three are relevant for plate bending (w, tx, ty).
>
>   I am calling the solver with the following options:
>
> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6
> -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view
>
>  And the convergence behavior is shown below, along with the ksp_view
> information. Based on notes in the manual, this seems to be subpar
> convergence rate. At the end of the solution the norm of each variable is :
>
>
> var: 0: u  : norm: 5.505909e-18
> var: 1: v  : norm: 7.639640e-18
> var: 2: w : norm: 3.901464e-03
> var: 3: tx : norm: 4.403576e-02
> var: 4: ty : norm: 4.403576e-02
> var: 5: tz : norm: 1.148409e-16
>
>
>   What do you mean by var: 2: w : norm etc? Is this the norm of the error
> for that variable, the norm of the residual, something else? How exactly
> are you calculating it?
>
>    Thanks
>
>
>   Barry
>
>
>  I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not
> make a lot of difference in the norms of (w, tx, ty).
>
>  I do provide the solver with 6 rigid-body vectors to approximate the
> null-space of the problem. Without these the solver shows very poor
> convergence.
>
>  I would appreciate advice on possible strategies to improve this behavior.
>
>
> Thanks,
> Manav
>
>    0 KSP Residual norm 1.696304497261e+00
>    1 KSP Residual norm 1.120485505777e+00
>    2 KSP Residual norm 8.324222302402e-01
>    3 KSP Residual norm 6.477349534115e-01
>    4 KSP Residual norm 5.080936471292e-01
>    5 KSP Residual norm 4.051099646638e-01
>    6 KSP Residual norm 3.260432664653e-01
>    7 KSP Residual norm 2.560483838143e-01
>    8 KSP Residual norm 2.029943986124e-01
>    9 KSP Residual norm 1.560985741610e-01
>   10 KSP Residual norm 1.163720702140e-01
>   11 KSP Residual norm 8.488411085459e-02
>   12 KSP Residual norm 5.888041729034e-02
>   13 KSP Residual norm 4.027792209980e-02
>   14 KSP Residual norm 2.819048087304e-02
>   15 KSP Residual norm 1.904674196962e-02
>   16 KSP Residual norm 1.289302447822e-02
>   17 KSP Residual norm 9.162203296376e-03
>   18 KSP Residual norm 7.016781679507e-03
>   19 KSP Residual norm 5.399170865328e-03
>   20 KSP Residual norm 4.254385887482e-03
>   21 KSP Residual norm 3.530831740621e-03
>   22 KSP Residual norm 2.946780747923e-03
>   23 KSP Residual norm 2.339361361128e-03
>   24 KSP Residual norm 1.815072489282e-03
>   25 KSP Residual norm 1.408814185342e-03
>   26 KSP Residual norm 1.063795714320e-03
>   27 KSP Residual norm 7.828540233117e-04
>   28 KSP Residual norm 5.683910750067e-04
>   29 KSP Residual norm 4.131151010250e-04
>   30 KSP Residual norm 3.065608221019e-04
>   31 KSP Residual norm 2.634114273459e-04
>   32 KSP Residual norm 2.198180137626e-04
>   33 KSP Residual norm 1.748956510799e-04
>   34 KSP Residual norm 1.317539710010e-04
>   35 KSP Residual norm 9.790121566055e-05
>   36 KSP Residual norm 7.465935386094e-05
>   37 KSP Residual norm 5.689506626052e-05
>   38 KSP Residual norm 4.413136619126e-05
>   39 KSP Residual norm 3.512194236402e-05
>   40 KSP Residual norm 2.877755408287e-05
>   41 KSP Residual norm 2.340080556431e-05
>   42 KSP Residual norm 1.904544450345e-05
>   43 KSP Residual norm 1.504723478235e-05
>   44 KSP Residual norm 1.141381950576e-05
>   45 KSP Residual norm 8.206151384599e-06
>   46 KSP Residual norm 5.911426091276e-06
>   47 KSP Residual norm 4.233669089283e-06
>   48 KSP Residual norm 2.898052944223e-06
>   49 KSP Residual norm 2.023556779973e-06
>   50 KSP Residual norm 1.459108043935e-06
>   51 KSP Residual norm 1.097335545865e-06
>   52 KSP Residual norm 8.440457332262e-07
>   53 KSP Residual norm 6.705616854004e-07
>   54 KSP Residual norm 5.404888680234e-07
>   55 KSP Residual norm 4.391368084979e-07
>   56 KSP Residual norm 3.697063014621e-07
>   57 KSP Residual norm 3.021772094146e-07
>   58 KSP Residual norm 2.479354520792e-07
>   59 KSP Residual norm 2.013077841968e-07
>   60 KSP Residual norm 1.553159612793e-07
>   61 KSP Residual norm 1.400784224898e-07
>   62 KSP Residual norm 9.707453662195e-08
>   63 KSP Residual norm 7.263173080146e-08
>   64 KSP Residual norm 5.593723572132e-08
>   65 KSP Residual norm 4.448788809586e-08
>   66 KSP Residual norm 3.613992590778e-08
>   67 KSP Residual norm 2.946099051876e-08
>   68 KSP Residual norm 2.408053564170e-08
>   69 KSP Residual norm 1.945257374856e-08
>   70 KSP Residual norm 1.572494535110e-08
>
>
> KSP Object: 4 MPI processes
>  type: gmres
>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization
> with no iterative refinement
>    happy breakdown tolerance 1e-30
>  maximum iterations=10000, initial guess is zero
>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>  left preconditioning
>  using PRECONDITIONED norm type for convergence test
> PC Object: 4 MPI processes
>  type: gamg
>    type is MULTIPLICATIVE, levels=6 cycles=v
>      Cycles per PCApply=1
>      Using externally compute Galerkin coarse grid matrices
>      GAMG specific options
>        Threshold for dropping small values in graph on each level =   0.
>   0.   0.   0.
>        Threshold scaling factor for each level not specified = 1.
>        AGG specific options
>          Symmetric graph false
>          Number of levels to square graph 1
>          Number smoothing steps 1
>  Coarse grid solver -- level -------------------------------
>    KSP Object: (mg_coarse_) 4 MPI processes
>      type: preonly
>      maximum iterations=10000, initial guess is zero
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_coarse_) 4 MPI processes
>      type: bjacobi
>        number of blocks = 4
>        Local solve is same for all blocks, in the following KSP and PC
> objects:
>      KSP Object: (mg_coarse_sub_) 1 MPI processes
>        type: preonly
>        maximum iterations=1, initial guess is zero
>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>        left preconditioning
>        using NONE norm type for convergence test
>      PC Object: (mg_coarse_sub_) 1 MPI processes
>        type: lu
>          out-of-place factorization
>          tolerance for zero pivot 2.22045e-14
>          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>          matrix ordering: nd
>          factor fill ratio given 5., needed 1.
>            Factored matrix follows:
>              Mat Object: 1 MPI processes
>                type: seqaij
>                rows=6, cols=6, bs=6
>                package used to perform factorization: petsc
>                total: nonzeros=36, allocated nonzeros=36
>                total number of mallocs used during MatSetValues calls =0
>                  using I-node routines: found 2 nodes, limit used is 5
>        linear system matrix = precond matrix:
>        Mat Object: 1 MPI processes
>          type: seqaij
>          rows=6, cols=6, bs=6
>          total: nonzeros=36, allocated nonzeros=36
>          total number of mallocs used during MatSetValues calls =0
>            using I-node routines: found 2 nodes, limit used is 5
>      linear system matrix = precond matrix:
>      Mat Object: 4 MPI processes
>        type: mpiaij
>        rows=6, cols=6, bs=6
>        total: nonzeros=36, allocated nonzeros=36
>        total number of mallocs used during MatSetValues calls =0
>          using nonscalable MatPtAP() implementation
>          using I-node (on process 0) routines: found 2 nodes, limit used
> is 5
>  Down solver (pre-smoother) on level 1 -------------------------------
>    KSP Object: (mg_levels_1_) 4 MPI processes
>      type: chebyshev
>        eigenvalue estimates used:  min = 0.099971, max = 1.09968
>        eigenvalues estimate via gmres min 0.154032, max 0.99971
>        eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>        KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>          type: gmres
>            restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>          left preconditioning
>          using PRECONDITIONED norm type for convergence test
>        estimating eigenvalues using noisy right hand side
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_1_) 4 MPI processes
>      type: sor
>        type = local_symmetric, iterations = 1, local iterations = 1, omega
> = 1.
>      linear system matrix = precond matrix:
>      Mat Object: 4 MPI processes
>        type: mpiaij
>        rows=54, cols=54, bs=6
>        total: nonzeros=2916, allocated nonzeros=2916
>        total number of mallocs used during MatSetValues calls =0
>          using I-node (on process 0) routines: found 11 nodes, limit used
> is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 2 -------------------------------
>    KSP Object: (mg_levels_2_) 4 MPI processes
>      type: chebyshev
>        eigenvalue estimates used:  min = 0.171388, max = 1.88526
>        eigenvalues estimate via gmres min 0.0717873, max 1.71388
>        eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>        KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>          type: gmres
>            restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>          left preconditioning
>          using PRECONDITIONED norm type for convergence test
>        estimating eigenvalues using noisy right hand side
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_2_) 4 MPI processes
>      type: sor
>        type = local_symmetric, iterations = 1, local iterations = 1, omega
> = 1.
>      linear system matrix = precond matrix:
>      Mat Object: 4 MPI processes
>        type: mpiaij
>        rows=642, cols=642, bs=6
>        total: nonzeros=99468, allocated nonzeros=99468
>        total number of mallocs used during MatSetValues calls =0
>          using nonscalable MatPtAP() implementation
>          using I-node (on process 0) routines: found 47 nodes, limit used
> is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 3 -------------------------------
>    KSP Object: (mg_levels_3_) 4 MPI processes
>      type: chebyshev
>        eigenvalue estimates used:  min = 0.164216, max = 1.80637
>        eigenvalues estimate via gmres min 0.0376323, max 1.64216
>        eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>        KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>          type: gmres
>            restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>          left preconditioning
>          using PRECONDITIONED norm type for convergence test
>        estimating eigenvalues using noisy right hand side
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_3_) 4 MPI processes
>      type: sor
>        type = local_symmetric, iterations = 1, local iterations = 1, omega
> = 1.
>      linear system matrix = precond matrix:
>      Mat Object: 4 MPI processes
>        type: mpiaij
>        rows=6726, cols=6726, bs=6
>        total: nonzeros=941796, allocated nonzeros=941796
>        total number of mallocs used during MatSetValues calls =0
>          using nonscalable MatPtAP() implementation
>          using I-node (on process 0) routines: found 552 nodes, limit used
> is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 4 -------------------------------
>    KSP Object: (mg_levels_4_) 4 MPI processes
>      type: chebyshev
>        eigenvalue estimates used:  min = 0.163283, max = 1.79611
>        eigenvalues estimate via gmres min 0.0350306, max 1.63283
>        eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>        KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>          type: gmres
>            restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>          left preconditioning
>          using PRECONDITIONED norm type for convergence test
>        estimating eigenvalues using noisy right hand side
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_4_) 4 MPI processes
>      type: sor
>        type = local_symmetric, iterations = 1, local iterations = 1, omega
> = 1.
>      linear system matrix = precond matrix:
>      Mat Object: 4 MPI processes
>        type: mpiaij
>        rows=41022, cols=41022, bs=6
>        total: nonzeros=2852316, allocated nonzeros=2852316
>        total number of mallocs used during MatSetValues calls =0
>          using nonscalable MatPtAP() implementation
>          using I-node (on process 0) routines: found 3432 nodes, limit
> used is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  Down solver (pre-smoother) on level 5 -------------------------------
>    KSP Object: (mg_levels_5_) 4 MPI processes
>      type: chebyshev
>        eigenvalue estimates used:  min = 0.157236, max = 1.7296
>        eigenvalues estimate via gmres min 0.0317897, max 1.57236
>        eigenvalues estimated using gmres with translations  [0. 0.1; 0.
> 1.1]
>        KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>          type: gmres
>            restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>            happy breakdown tolerance 1e-30
>          maximum iterations=10, initial guess is zero
>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>          left preconditioning
>          using PRECONDITIONED norm type for convergence test
>        estimating eigenvalues using noisy right hand side
>      maximum iterations=2, nonzero initial guess
>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>      left preconditioning
>      using NONE norm type for convergence test
>    PC Object: (mg_levels_5_) 4 MPI processes
>      type: sor
>        type = local_symmetric, iterations = 1, local iterations = 1, omega
> = 1.
>      linear system matrix = precond matrix:
>      Mat Object: () 4 MPI processes
>        type: mpiaij
>        rows=543606, cols=543606, bs=6
>        total: nonzeros=29224836, allocated nonzeros=29302596
>        total number of mallocs used during MatSetValues calls =0
>          has attached near null space
>          using I-node (on process 0) routines: found 45644 nodes, limit
> used is 5
>  Up solver (post-smoother) same as down solver (pre-smoother)
>  linear system matrix = precond matrix:
>  Mat Object: () 4 MPI processes
>    type: mpiaij
>    rows=543606, cols=543606, bs=6
>    total: nonzeros=29224836, allocated nonzeros=29302596
>    total number of mallocs used during MatSetValues calls =0
>      has attached near null space
>      using I-node (on process 0) routines: found 45644 nodes, limit used
> is 5
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/c32de421/attachment-0001.html>

From fwan at mek.dtu.dk  Mon Oct 29 08:35:56 2018
From: fwan at mek.dtu.dk (Fengwen Wang)
Date: Mon, 29 Oct 2018 13:35:56 +0000
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <ED2A5E73-4A90-47A5-8820-6511CF5174F7@gmx.li>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
	<CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
	<2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>
	<CAMYG4GmRt8x8hVopoeDKMyNaXE1SotFOZ1xwRCHzj+rH+wbt-A@mail.gmail.com>,
	<ED2A5E73-4A90-47A5-8820-6511CF5174F7@gmx.li>
Message-ID: <1e4a8501e44d4fdf8727d7ef7a217fff@mek.dtu.dk>

Yes.  I would like to a rotational BCs(Twist bcs).


Such a boundary is needed to reduce the computational domain.


Thanks lot


Best regards

Fengwen




____________________________________________________
Senior Researcher
Department of Mechanical Engineering, DTU
Nils Koppels All?
Building 404
2800  Kgs. Lyngby
fwan at mek.dtu.dk
________________________________
From: Lawrence Mitchell <wence at gmx.li>
Sent: Monday, October 29, 2018 1:17:34 PM
To: Matt Knepley
Cc: Fengwen Wang; PETSc; PETSc
Subject: Re: [petsc-users] [petsc-maint] How to impose boundary conditions using DMDA



> On 29 Oct 2018, at 10:56, Matthew Knepley <knepley at gmail.com> wrote:
>
> You can certainly map the right edge to the top edge (its topologically a square again), but that
> mapping is not smooth, and I do not know how you would make a global basis for the approximation space.

Yeah, I thought this was topologically just a cone, but that would have:

u(x=1) = -v(y=1)

and

v(x=1) = u(y=1) (note matching sign).

So if you wanted the v(x=1) = -u(y=1) there would be a twist somehow, and then a discontinuity in the mapping.

Lawrence
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/8c86859c/attachment.html>

From bhatiamanav at gmail.com  Mon Oct 29 10:55:01 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 10:55:01 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
Message-ID: <C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>

Hi Mark, 

  Here are some results (still running with 4 cpus): 


With the default options the convergence is slow
-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8
    0 KSP Residual norm 1.696304497263e+00 
    1 KSP Residual norm 1.120485505766e+00 
    2 KSP Residual norm 8.324222302220e-01 
    3 KSP Residual norm 6.477349533922e-01 
    4 KSP Residual norm 5.080936471094e-01 
    5 KSP Residual norm 4.051099646451e-01 
    6 KSP Residual norm 3.260432664484e-01 
    7 KSP Residual norm 2.560483838000e-01 
    8 KSP Residual norm 2.029943986006e-01 
    9 KSP Residual norm 1.560985741519e-01 
   10 KSP Residual norm 1.163720702074e-01 
   11 KSP Residual norm 8.488411084998e-02 
   12 KSP Residual norm 5.888041728730e-02 
   13 KSP Residual norm 4.027792209782e-02 
   14 KSP Residual norm 2.819048087173e-02 
   15 KSP Residual norm 1.904674196882e-02 
   16 KSP Residual norm 1.289302447775e-02 
   17 KSP Residual norm 9.162203296105e-03 
   18 KSP Residual norm 7.016781679348e-03 
   19 KSP Residual norm 5.399170865246e-03 
   20 KSP Residual norm 4.254385887447e-03 
   21 KSP Residual norm 3.530831740603e-03 
   22 KSP Residual norm 2.946780747904e-03 
   23 KSP Residual norm 2.339361361103e-03 
   24 KSP Residual norm 1.815072489251e-03 
   25 KSP Residual norm 1.408814185309e-03 
   26 KSP Residual norm 1.063795714289e-03 
   27 KSP Residual norm 7.828540232832e-04 
   28 KSP Residual norm 5.683910749829e-04 
   29 KSP Residual norm 4.131151010060e-04 
   30 KSP Residual norm 3.065608169121e-04 
   31 KSP Residual norm 2.634114212906e-04 
   32 KSP Residual norm 2.198180088890e-04 
   33 KSP Residual norm 1.748956465770e-04 
   34 KSP Residual norm 1.317539664398e-04 
   35 KSP Residual norm 9.790121191782e-05 
   36 KSP Residual norm 7.465935116526e-05 
   37 KSP Residual norm 5.689506439547e-05 
   38 KSP Residual norm 4.413136465026e-05 
   39 KSP Residual norm 3.512194107520e-05 
   40 KSP Residual norm 2.877755304955e-05 
   41 KSP Residual norm 2.340080488088e-05 
   42 KSP Residual norm 1.904544419876e-05 
   43 KSP Residual norm 1.504723479640e-05 
   44 KSP Residual norm 1.141381974873e-05 
   45 KSP Residual norm 8.206151668656e-06 
   46 KSP Residual norm 5.911426282047e-06 
   47 KSP Residual norm 4.233669179704e-06 
   48 KSP Residual norm 2.898052992624e-06 
   49 KSP Residual norm 2.023556817205e-06 
   50 KSP Residual norm 1.459108072651e-06 
   51 KSP Residual norm 1.097335572735e-06 
   52 KSP Residual norm 8.440457530634e-07 
   53 KSP Residual norm 6.705616952049e-07 
   54 KSP Residual norm 5.404888697309e-07 
   55 KSP Residual norm 4.391368066975e-07 
   56 KSP Residual norm 3.697063001345e-07 
   57 KSP Residual norm 3.021772076055e-07 
   58 KSP Residual norm 2.479354498371e-07 
   59 KSP Residual norm 2.013077815815e-07 
   60 KSP Residual norm 1.553178802459e-07 
   61 KSP Residual norm 1.400798352748e-07 
   62 KSP Residual norm 9.707215027303e-08 
   63 KSP Residual norm 7.262869538195e-08 
   64 KSP Residual norm 5.593398375649e-08 
   65 KSP Residual norm 4.448475420166e-08 
   66 KSP Residual norm 3.613734113472e-08 
   67 KSP Residual norm 2.945927212825e-08 
   68 KSP Residual norm 2.407949632330e-08 
   69 KSP Residual norm 1.945210209951e-08 
   70 KSP Residual norm 1.572500364747e-08 



Adding ?-pc_mg_levels 2? gives a significantly improved performance: 
-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2
    0 KSP Residual norm 2.980714123240e+01 
    1 KSP Residual norm 1.213007759874e+00 
    2 KSP Residual norm 1.543794101059e-01 
    3 KSP Residual norm 3.522492126064e-02 
    4 KSP Residual norm 7.453170557576e-03 
    5 KSP Residual norm 1.828043480467e-03 
    6 KSP Residual norm 4.779250859781e-04 
    7 KSP Residual norm 1.099093020733e-04 
    8 KSP Residual norm 2.806438906374e-05 
    9 KSP Residual norm 7.416077106013e-06 
   10 KSP Residual norm 1.669576855922e-06 
   11 KSP Residual norm 6.138913423983e-07 
   12 KSP Residual norm 3.914982893935e-07 
   13 KSP Residual norm 2.491167256452e-07 


I did not see an option in the ?-help? output by the name of ?-mg_levels_ksp_max_it?, so I added one for each level. Adding ?-mg_levels_1_ksp_max_it 4  -mg_levels_2_ksp_max_it 4 ? gives the following convergence rate: 
-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -mg_levels_1_ksp_max_it 4 -mg_levels_2_ksp_max_it 4
    0 KSP Residual norm 2.980759132912e+01 
    1 KSP Residual norm 1.268404746671e+00 
    2 KSP Residual norm 1.420311012425e-01 
    3 KSP Residual norm 3.026536678757e-02 
    4 KSP Residual norm 6.511170312990e-03 
    5 KSP Residual norm 1.539620841789e-03 
    6 KSP Residual norm 3.655528499924e-04 
    7 KSP Residual norm 8.111524453983e-05 
    8 KSP Residual norm 1.995956470676e-05 
    9 KSP Residual norm 4.397662980841e-06 
   10 KSP Residual norm 9.636956929342e-07 
   11 KSP Residual norm 3.013384202116e-07 
   12 KSP Residual norm 1.867699579369e-07 


Adding ?-pc_gamg_threshold 0.04? gives: 
-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -mg_levels_1_ksp_max_it 4 -mg_levels_2_ksp_max_it 4 -pc_gamg_threshold 0.04
    0 KSP Residual norm 2.980759132913e+01 
    1 KSP Residual norm 1.268404746942e+00 
    2 KSP Residual norm 1.420311012570e-01 
    3 KSP Residual norm 3.026536679076e-02 
    4 KSP Residual norm 6.511170313879e-03 
    5 KSP Residual norm 1.539620841827e-03 
    6 KSP Residual norm 3.655528500623e-04 
    7 KSP Residual norm 8.111524453279e-05 
    8 KSP Residual norm 1.995956474349e-05 
    9 KSP Residual norm 4.397662966260e-06 
   10 KSP Residual norm 9.636957102472e-07 
   11 KSP Residual norm 3.013383993102e-07 
   12 KSP Residual norm 1.867700181613e-07 


Using just the ?-pc_gamg_square_graph 0? option: 
-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_gamg_square_graph 0
    0 KSP Residual norm 2.064808848243e+00 
    1 KSP Residual norm 1.436709062429e+00 
    2 KSP Residual norm 1.043690492056e+00 
    3 KSP Residual norm 7.636589301482e-01 
    4 KSP Residual norm 5.733849171144e-01 
    5 KSP Residual norm 4.416033588916e-01 
    6 KSP Residual norm 3.390846779861e-01 
    7 KSP Residual norm 2.567469096297e-01 
    8 KSP Residual norm 1.874940743028e-01 
    9 KSP Residual norm 1.322422119221e-01 
   10 KSP Residual norm 9.157000749016e-02 
   11 KSP Residual norm 6.376782657530e-02 
   12 KSP Residual norm 4.456316146538e-02 
   13 KSP Residual norm 3.101613919753e-02 
   14 KSP Residual norm 2.167127331495e-02 
   15 KSP Residual norm 1.498469528896e-02 
   16 KSP Residual norm 1.075794635819e-02 
   17 KSP Residual norm 7.764685216272e-03 
   18 KSP Residual norm 5.435228207429e-03 
   19 KSP Residual norm 3.942376675316e-03 
   20 KSP Residual norm 2.846234513499e-03 
   21 KSP Residual norm 1.914323559680e-03 
   22 KSP Residual norm 1.332049518265e-03 
   23 KSP Residual norm 9.414825222665e-04 
   24 KSP Residual norm 6.600086167534e-04 
   25 KSP Residual norm 4.679306216706e-04 
   26 KSP Residual norm 3.264136607741e-04 
   27 KSP Residual norm 2.316366100549e-04 
   28 KSP Residual norm 1.642594683886e-04 
   29 KSP Residual norm 1.103783108795e-04 
   30 KSP Residual norm 7.740619320447e-05 
   31 KSP Residual norm 6.389704142174e-05 
   32 KSP Residual norm 5.006485093406e-05 
   33 KSP Residual norm 3.758456789529e-05 
   34 KSP Residual norm 2.830743556147e-05 
   35 KSP Residual norm 2.062379654606e-05 
   36 KSP Residual norm 1.490349770670e-05 
   37 KSP Residual norm 1.051767740994e-05 
   38 KSP Residual norm 7.040814709680e-06 
   39 KSP Residual norm 4.931079567116e-06 
   40 KSP Residual norm 3.385386183658e-06 
   41 KSP Residual norm 2.274379328203e-06 
   42 KSP Residual norm 1.576979495342e-06 
   43 KSP Residual norm 1.134465108080e-06 
   44 KSP Residual norm 8.164007819038e-07 
   45 KSP Residual norm 5.697298265561e-07 
   46 KSP Residual norm 4.079302286082e-07 
   47 KSP Residual norm 3.032840167758e-07 
   48 KSP Residual norm 2.118663896145e-07 
   49 KSP Residual norm 1.531268272774e-07 
   50 KSP Residual norm 1.155662360224e-07 
   51 KSP Residual norm 8.763548545594e-08 
   52 KSP Residual norm 6.411338426040e-08 
   53 KSP Residual norm 4.720820815019e-08 
   54 KSP Residual norm 3.694420088192e-08 
   55 KSP Residual norm 2.924088208699e-08 
   56 KSP Residual norm 2.365173325198e-08 
   57 KSP Residual norm 1.891630286373e-08 



Without the threshold and max it modifications, and adding ?-pc_gamg_square_graph 0? : 

-pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -pc_gamg_square_graph 0
    0 KSP Residual norm 4.658705362181e+01 
    1 KSP Residual norm 6.723806072355e-01 
    2 KSP Residual norm 4.063455422565e-02 
    3 KSP Residual norm 2.311496772987e-03 
    4 KSP Residual norm 2.337388101209e-04 
    5 KSP Residual norm 2.541042271307e-05 
    6 KSP Residual norm 5.461281412935e-06 
    7 KSP Residual norm 2.718337804133e-06 
    8 KSP Residual norm 1.223645122249e-06 
    9 KSP Residual norm 7.877002862516e-07 
   10 KSP Residual norm 4.742159201655e-07 
   11 KSP Residual norm 2.849093170324e-07 


So, it appears that the number of MG levels has the most significant impact on the convergence rate. 

What would be the reason for this? Is there a general recommendation on MG levels (default options gave 4)? I would suspect that this is problem dependent. 

Regards,
Manav

> On Oct 29, 2018, at 8:28 AM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> 
> I would recommend using '-mg_levels 2' and check that that gives you two levels. I would also run this on one processor just to start. 
> 
> Use -mg_levels_ksp_max_it 4.  And '-pc_gamg_threshold 0.04'
> 
> These parameters are meant to increase convergence rate at all costs. Once we our best rate we should be able to back off of some of this without much degradation or play around with the parameters to optimize run time. 
> 
> 
> On Sun, Oct 28, 2018 at 5:13 PM Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
> Var: 0,?,5  are the 6 variables that I am solving for: u, v, w, theta_x, theta_y, theta_z. 
> 
> The norms identified in my email are the L2 norms of all dofs corresponding to each variable in the solution vector. So, var: 0: u: norm is the L2 norm of the dofs for u only, and so on. 
> 
> I expect u, v, theta_z to be zero for the solution, which ends up being the case. 
> 
> If I plot the solution, they look sensible, but the reduction of KSP norm is slow. 
> 
> 
> Thanks,
> Manav
> 
>> On Oct 28, 2018, at 3:55 PM, Smith, Barry F. <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>> 
>> 
>> 
>>> On Oct 28, 2018, at 12:16 PM, Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
>>> 
>>> Hi, 
>>> 
>>>   I am attempting to solve a Mindlin plate bending problem with AMG solver in petsc. This test case is with a mesh of 300x300 elements and 543,606 dofs. 
>>> 
>>>   The discretization includes 6 variables (u, v, w, tx, ty, tz), but only three are relevant for plate bending (w, tx, ty). 
>>> 
>>>   I am calling the solver with the following options: 
>>> 
>>> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view 
>>> 
>>>  And the convergence behavior is shown below, along with the ksp_view information. Based on notes in the manual, this seems to be subpar convergence rate. At the end of the solution the norm of each variable is : 
>>> 
>>> var: 0: u  : norm: 5.505909e-18
>>> var: 1: v  : norm: 7.639640e-18
>>> var: 2: w : norm: 3.901464e-03
>>> var: 3: tx : norm: 4.403576e-02
>>> var: 4: ty : norm: 4.403576e-02
>>> var: 5: tz : norm: 1.148409e-16
>> 
>>   What do you mean by var: 2: w : norm etc? Is this the norm of the error for that variable, the norm of the residual, something else? How exactly are you calculating it?
>> 
>>    Thanks
>> 
>> 
>>   Barry
>> 
>>> 
>>>  I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not make a lot of difference in the norms of (w, tx, ty). 
>>> 
>>>  I do provide the solver with 6 rigid-body vectors to approximate the null-space of the problem. Without these the solver shows very poor convergence. 
>>> 
>>>  I would appreciate advice on possible strategies to improve this behavior. 
>>> 
>>> Thanks,
>>> Manav 
>>> 
>>>    0 KSP Residual norm 1.696304497261e+00 
>>>    1 KSP Residual norm 1.120485505777e+00 
>>>    2 KSP Residual norm 8.324222302402e-01 
>>>    3 KSP Residual norm 6.477349534115e-01 
>>>    4 KSP Residual norm 5.080936471292e-01 
>>>    5 KSP Residual norm 4.051099646638e-01 
>>>    6 KSP Residual norm 3.260432664653e-01 
>>>    7 KSP Residual norm 2.560483838143e-01 
>>>    8 KSP Residual norm 2.029943986124e-01 
>>>    9 KSP Residual norm 1.560985741610e-01 
>>>   10 KSP Residual norm 1.163720702140e-01 
>>>   11 KSP Residual norm 8.488411085459e-02 
>>>   12 KSP Residual norm 5.888041729034e-02 
>>>   13 KSP Residual norm 4.027792209980e-02 
>>>   14 KSP Residual norm 2.819048087304e-02 
>>>   15 KSP Residual norm 1.904674196962e-02 
>>>   16 KSP Residual norm 1.289302447822e-02 
>>>   17 KSP Residual norm 9.162203296376e-03 
>>>   18 KSP Residual norm 7.016781679507e-03 
>>>   19 KSP Residual norm 5.399170865328e-03 
>>>   20 KSP Residual norm 4.254385887482e-03 
>>>   21 KSP Residual norm 3.530831740621e-03 
>>>   22 KSP Residual norm 2.946780747923e-03 
>>>   23 KSP Residual norm 2.339361361128e-03 
>>>   24 KSP Residual norm 1.815072489282e-03 
>>>   25 KSP Residual norm 1.408814185342e-03 
>>>   26 KSP Residual norm 1.063795714320e-03 
>>>   27 KSP Residual norm 7.828540233117e-04 
>>>   28 KSP Residual norm 5.683910750067e-04 
>>>   29 KSP Residual norm 4.131151010250e-04 
>>>   30 KSP Residual norm 3.065608221019e-04 
>>>   31 KSP Residual norm 2.634114273459e-04 
>>>   32 KSP Residual norm 2.198180137626e-04 
>>>   33 KSP Residual norm 1.748956510799e-04 
>>>   34 KSP Residual norm 1.317539710010e-04 
>>>   35 KSP Residual norm 9.790121566055e-05 
>>>   36 KSP Residual norm 7.465935386094e-05 
>>>   37 KSP Residual norm 5.689506626052e-05 
>>>   38 KSP Residual norm 4.413136619126e-05 
>>>   39 KSP Residual norm 3.512194236402e-05 
>>>   40 KSP Residual norm 2.877755408287e-05 
>>>   41 KSP Residual norm 2.340080556431e-05 
>>>   42 KSP Residual norm 1.904544450345e-05 
>>>   43 KSP Residual norm 1.504723478235e-05 
>>>   44 KSP Residual norm 1.141381950576e-05 
>>>   45 KSP Residual norm 8.206151384599e-06 
>>>   46 KSP Residual norm 5.911426091276e-06 
>>>   47 KSP Residual norm 4.233669089283e-06 
>>>   48 KSP Residual norm 2.898052944223e-06 
>>>   49 KSP Residual norm 2.023556779973e-06 
>>>   50 KSP Residual norm 1.459108043935e-06 
>>>   51 KSP Residual norm 1.097335545865e-06 
>>>   52 KSP Residual norm 8.440457332262e-07 
>>>   53 KSP Residual norm 6.705616854004e-07 
>>>   54 KSP Residual norm 5.404888680234e-07 
>>>   55 KSP Residual norm 4.391368084979e-07 
>>>   56 KSP Residual norm 3.697063014621e-07 
>>>   57 KSP Residual norm 3.021772094146e-07 
>>>   58 KSP Residual norm 2.479354520792e-07 
>>>   59 KSP Residual norm 2.013077841968e-07 
>>>   60 KSP Residual norm 1.553159612793e-07 
>>>   61 KSP Residual norm 1.400784224898e-07 
>>>   62 KSP Residual norm 9.707453662195e-08 
>>>   63 KSP Residual norm 7.263173080146e-08 
>>>   64 KSP Residual norm 5.593723572132e-08 
>>>   65 KSP Residual norm 4.448788809586e-08 
>>>   66 KSP Residual norm 3.613992590778e-08 
>>>   67 KSP Residual norm 2.946099051876e-08 
>>>   68 KSP Residual norm 2.408053564170e-08 
>>>   69 KSP Residual norm 1.945257374856e-08 
>>>   70 KSP Residual norm 1.572494535110e-08 
>>> 
>>> 
>>> KSP Object: 4 MPI processes
>>>  type: gmres
>>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>    happy breakdown tolerance 1e-30
>>>  maximum iterations=10000, initial guess is zero
>>>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>>>  left preconditioning
>>>  using PRECONDITIONED norm type for convergence test
>>> PC Object: 4 MPI processes
>>>  type: gamg
>>>    type is MULTIPLICATIVE, levels=6 cycles=v
>>>      Cycles per PCApply=1
>>>      Using externally compute Galerkin coarse grid matrices
>>>      GAMG specific options
>>>        Threshold for dropping small values in graph on each level =   0.   0.   0.   0.  
>>>        Threshold scaling factor for each level not specified = 1.
>>>        AGG specific options
>>>          Symmetric graph false
>>>          Number of levels to square graph 1
>>>          Number smoothing steps 1
>>>  Coarse grid solver -- level -------------------------------
>>>    KSP Object: (mg_coarse_) 4 MPI processes
>>>      type: preonly
>>>      maximum iterations=10000, initial guess is zero
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_coarse_) 4 MPI processes
>>>      type: bjacobi
>>>        number of blocks = 4
>>>        Local solve is same for all blocks, in the following KSP and PC objects:
>>>      KSP Object: (mg_coarse_sub_) 1 MPI processes
>>>        type: preonly
>>>        maximum iterations=1, initial guess is zero
>>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>        left preconditioning
>>>        using NONE norm type for convergence test
>>>      PC Object: (mg_coarse_sub_) 1 MPI processes
>>>        type: lu
>>>          out-of-place factorization
>>>          tolerance for zero pivot 2.22045e-14
>>>          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>          matrix ordering: nd
>>>          factor fill ratio given 5., needed 1.
>>>            Factored matrix follows:
>>>              Mat Object: 1 MPI processes
>>>                type: seqaij
>>>                rows=6, cols=6, bs=6
>>>                package used to perform factorization: petsc
>>>                total: nonzeros=36, allocated nonzeros=36
>>>                total number of mallocs used during MatSetValues calls =0
>>>                  using I-node routines: found 2 nodes, limit used is 5
>>>        linear system matrix = precond matrix:
>>>        Mat Object: 1 MPI processes
>>>          type: seqaij
>>>          rows=6, cols=6, bs=6
>>>          total: nonzeros=36, allocated nonzeros=36
>>>          total number of mallocs used during MatSetValues calls =0
>>>            using I-node routines: found 2 nodes, limit used is 5
>>>      linear system matrix = precond matrix:
>>>      Mat Object: 4 MPI processes
>>>        type: mpiaij
>>>        rows=6, cols=6, bs=6
>>>        total: nonzeros=36, allocated nonzeros=36
>>>        total number of mallocs used during MatSetValues calls =0
>>>          using nonscalable MatPtAP() implementation
>>>          using I-node (on process 0) routines: found 2 nodes, limit used is 5
>>>  Down solver (pre-smoother) on level 1 -------------------------------
>>>    KSP Object: (mg_levels_1_) 4 MPI processes
>>>      type: chebyshev
>>>        eigenvalue estimates used:  min = 0.099971, max = 1.09968
>>>        eigenvalues estimate via gmres min 0.154032, max 0.99971
>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>        KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>>>          type: gmres
>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>            happy breakdown tolerance 1e-30
>>>          maximum iterations=10, initial guess is zero
>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>          left preconditioning
>>>          using PRECONDITIONED norm type for convergence test
>>>        estimating eigenvalues using noisy right hand side
>>>      maximum iterations=2, nonzero initial guess
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_levels_1_) 4 MPI processes
>>>      type: sor
>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>      linear system matrix = precond matrix:
>>>      Mat Object: 4 MPI processes
>>>        type: mpiaij
>>>        rows=54, cols=54, bs=6
>>>        total: nonzeros=2916, allocated nonzeros=2916
>>>        total number of mallocs used during MatSetValues calls =0
>>>          using I-node (on process 0) routines: found 11 nodes, limit used is 5
>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>  Down solver (pre-smoother) on level 2 -------------------------------
>>>    KSP Object: (mg_levels_2_) 4 MPI processes
>>>      type: chebyshev
>>>        eigenvalue estimates used:  min = 0.171388, max = 1.88526
>>>        eigenvalues estimate via gmres min 0.0717873, max 1.71388
>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>        KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>>>          type: gmres
>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>            happy breakdown tolerance 1e-30
>>>          maximum iterations=10, initial guess is zero
>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>          left preconditioning
>>>          using PRECONDITIONED norm type for convergence test
>>>        estimating eigenvalues using noisy right hand side
>>>      maximum iterations=2, nonzero initial guess
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_levels_2_) 4 MPI processes
>>>      type: sor
>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>      linear system matrix = precond matrix:
>>>      Mat Object: 4 MPI processes
>>>        type: mpiaij
>>>        rows=642, cols=642, bs=6
>>>        total: nonzeros=99468, allocated nonzeros=99468
>>>        total number of mallocs used during MatSetValues calls =0
>>>          using nonscalable MatPtAP() implementation
>>>          using I-node (on process 0) routines: found 47 nodes, limit used is 5
>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>  Down solver (pre-smoother) on level 3 -------------------------------
>>>    KSP Object: (mg_levels_3_) 4 MPI processes
>>>      type: chebyshev
>>>        eigenvalue estimates used:  min = 0.164216, max = 1.80637
>>>        eigenvalues estimate via gmres min 0.0376323, max 1.64216
>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>        KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>>>          type: gmres
>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>            happy breakdown tolerance 1e-30
>>>          maximum iterations=10, initial guess is zero
>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>          left preconditioning
>>>          using PRECONDITIONED norm type for convergence test
>>>        estimating eigenvalues using noisy right hand side
>>>      maximum iterations=2, nonzero initial guess
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_levels_3_) 4 MPI processes
>>>      type: sor
>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>      linear system matrix = precond matrix:
>>>      Mat Object: 4 MPI processes
>>>        type: mpiaij
>>>        rows=6726, cols=6726, bs=6
>>>        total: nonzeros=941796, allocated nonzeros=941796
>>>        total number of mallocs used during MatSetValues calls =0
>>>          using nonscalable MatPtAP() implementation
>>>          using I-node (on process 0) routines: found 552 nodes, limit used is 5
>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>  Down solver (pre-smoother) on level 4 -------------------------------
>>>    KSP Object: (mg_levels_4_) 4 MPI processes
>>>      type: chebyshev
>>>        eigenvalue estimates used:  min = 0.163283, max = 1.79611
>>>        eigenvalues estimate via gmres min 0.0350306, max 1.63283
>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>        KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>>>          type: gmres
>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>            happy breakdown tolerance 1e-30
>>>          maximum iterations=10, initial guess is zero
>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>          left preconditioning
>>>          using PRECONDITIONED norm type for convergence test
>>>        estimating eigenvalues using noisy right hand side
>>>      maximum iterations=2, nonzero initial guess
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_levels_4_) 4 MPI processes
>>>      type: sor
>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>      linear system matrix = precond matrix:
>>>      Mat Object: 4 MPI processes
>>>        type: mpiaij
>>>        rows=41022, cols=41022, bs=6
>>>        total: nonzeros=2852316, allocated nonzeros=2852316
>>>        total number of mallocs used during MatSetValues calls =0
>>>          using nonscalable MatPtAP() implementation
>>>          using I-node (on process 0) routines: found 3432 nodes, limit used is 5
>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>  Down solver (pre-smoother) on level 5 -------------------------------
>>>    KSP Object: (mg_levels_5_) 4 MPI processes
>>>      type: chebyshev
>>>        eigenvalue estimates used:  min = 0.157236, max = 1.7296
>>>        eigenvalues estimate via gmres min 0.0317897, max 1.57236
>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>        KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>>>          type: gmres
>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>            happy breakdown tolerance 1e-30
>>>          maximum iterations=10, initial guess is zero
>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>          left preconditioning
>>>          using PRECONDITIONED norm type for convergence test
>>>        estimating eigenvalues using noisy right hand side
>>>      maximum iterations=2, nonzero initial guess
>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>      left preconditioning
>>>      using NONE norm type for convergence test
>>>    PC Object: (mg_levels_5_) 4 MPI processes
>>>      type: sor
>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>      linear system matrix = precond matrix:
>>>      Mat Object: () 4 MPI processes
>>>        type: mpiaij
>>>        rows=543606, cols=543606, bs=6
>>>        total: nonzeros=29224836, allocated nonzeros=29302596
>>>        total number of mallocs used during MatSetValues calls =0
>>>          has attached near null space
>>>          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>  linear system matrix = precond matrix:
>>>  Mat Object: () 4 MPI processes
>>>    type: mpiaij
>>>    rows=543606, cols=543606, bs=6
>>>    total: nonzeros=29224836, allocated nonzeros=29302596
>>>    total number of mallocs used during MatSetValues calls =0
>>>      has attached near null space
>>>      using I-node (on process 0) routines: found 45644 nodes, limit used is 5
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/493251c6/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Oct 29 10:59:28 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 10:59:28 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
Message-ID: <771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>

Combining all options gives the best convergence rate so far: 

-pc_type gamg --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_1_ksp_max_it 4 -mg_levels_2_ksp_max_it 4 -pc_gamg_threshold 0.04
    0 KSP Residual norm 4.658748806833e+01 
    1 KSP Residual norm 6.825811662732e-01 
    2 KSP Residual norm 3.642264110308e-02 
    3 KSP Residual norm 2.013016231184e-03 
    4 KSP Residual norm 2.013432242047e-04 
    5 KSP Residual norm 2.105457376565e-05 
    6 KSP Residual norm 3.003346326867e-06 
    7 KSP Residual norm 1.445664725375e-06 
    8 KSP Residual norm 6.952678589683e-07 
    9 KSP Residual norm 3.813304706946e-07 

-Manav

> On Oct 29, 2018, at 10:55 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Hi Mark, 
> 
>   Here are some results (still running with 4 cpus): 
> 
> 
> With the default options the convergence is slow
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8
>     0 KSP Residual norm 1.696304497263e+00 
>     1 KSP Residual norm 1.120485505766e+00 
>     2 KSP Residual norm 8.324222302220e-01 
>     3 KSP Residual norm 6.477349533922e-01 
>     4 KSP Residual norm 5.080936471094e-01 
>     5 KSP Residual norm 4.051099646451e-01 
>     6 KSP Residual norm 3.260432664484e-01 
>     7 KSP Residual norm 2.560483838000e-01 
>     8 KSP Residual norm 2.029943986006e-01 
>     9 KSP Residual norm 1.560985741519e-01 
>    10 KSP Residual norm 1.163720702074e-01 
>    11 KSP Residual norm 8.488411084998e-02 
>    12 KSP Residual norm 5.888041728730e-02 
>    13 KSP Residual norm 4.027792209782e-02 
>    14 KSP Residual norm 2.819048087173e-02 
>    15 KSP Residual norm 1.904674196882e-02 
>    16 KSP Residual norm 1.289302447775e-02 
>    17 KSP Residual norm 9.162203296105e-03 
>    18 KSP Residual norm 7.016781679348e-03 
>    19 KSP Residual norm 5.399170865246e-03 
>    20 KSP Residual norm 4.254385887447e-03 
>    21 KSP Residual norm 3.530831740603e-03 
>    22 KSP Residual norm 2.946780747904e-03 
>    23 KSP Residual norm 2.339361361103e-03 
>    24 KSP Residual norm 1.815072489251e-03 
>    25 KSP Residual norm 1.408814185309e-03 
>    26 KSP Residual norm 1.063795714289e-03 
>    27 KSP Residual norm 7.828540232832e-04 
>    28 KSP Residual norm 5.683910749829e-04 
>    29 KSP Residual norm 4.131151010060e-04 
>    30 KSP Residual norm 3.065608169121e-04 
>    31 KSP Residual norm 2.634114212906e-04 
>    32 KSP Residual norm 2.198180088890e-04 
>    33 KSP Residual norm 1.748956465770e-04 
>    34 KSP Residual norm 1.317539664398e-04 
>    35 KSP Residual norm 9.790121191782e-05 
>    36 KSP Residual norm 7.465935116526e-05 
>    37 KSP Residual norm 5.689506439547e-05 
>    38 KSP Residual norm 4.413136465026e-05 
>    39 KSP Residual norm 3.512194107520e-05 
>    40 KSP Residual norm 2.877755304955e-05 
>    41 KSP Residual norm 2.340080488088e-05 
>    42 KSP Residual norm 1.904544419876e-05 
>    43 KSP Residual norm 1.504723479640e-05 
>    44 KSP Residual norm 1.141381974873e-05 
>    45 KSP Residual norm 8.206151668656e-06 
>    46 KSP Residual norm 5.911426282047e-06 
>    47 KSP Residual norm 4.233669179704e-06 
>    48 KSP Residual norm 2.898052992624e-06 
>    49 KSP Residual norm 2.023556817205e-06 
>    50 KSP Residual norm 1.459108072651e-06 
>    51 KSP Residual norm 1.097335572735e-06 
>    52 KSP Residual norm 8.440457530634e-07 
>    53 KSP Residual norm 6.705616952049e-07 
>    54 KSP Residual norm 5.404888697309e-07 
>    55 KSP Residual norm 4.391368066975e-07 
>    56 KSP Residual norm 3.697063001345e-07 
>    57 KSP Residual norm 3.021772076055e-07 
>    58 KSP Residual norm 2.479354498371e-07 
>    59 KSP Residual norm 2.013077815815e-07 
>    60 KSP Residual norm 1.553178802459e-07 
>    61 KSP Residual norm 1.400798352748e-07 
>    62 KSP Residual norm 9.707215027303e-08 
>    63 KSP Residual norm 7.262869538195e-08 
>    64 KSP Residual norm 5.593398375649e-08 
>    65 KSP Residual norm 4.448475420166e-08 
>    66 KSP Residual norm 3.613734113472e-08 
>    67 KSP Residual norm 2.945927212825e-08 
>    68 KSP Residual norm 2.407949632330e-08 
>    69 KSP Residual norm 1.945210209951e-08 
>    70 KSP Residual norm 1.572500364747e-08 
> 
> 
> 
> Adding ?-pc_mg_levels 2? gives a significantly improved performance: 
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2
>     0 KSP Residual norm 2.980714123240e+01 
>     1 KSP Residual norm 1.213007759874e+00 
>     2 KSP Residual norm 1.543794101059e-01 
>     3 KSP Residual norm 3.522492126064e-02 
>     4 KSP Residual norm 7.453170557576e-03 
>     5 KSP Residual norm 1.828043480467e-03 
>     6 KSP Residual norm 4.779250859781e-04 
>     7 KSP Residual norm 1.099093020733e-04 
>     8 KSP Residual norm 2.806438906374e-05 
>     9 KSP Residual norm 7.416077106013e-06 
>    10 KSP Residual norm 1.669576855922e-06 
>    11 KSP Residual norm 6.138913423983e-07 
>    12 KSP Residual norm 3.914982893935e-07 
>    13 KSP Residual norm 2.491167256452e-07 
> 
> 
> I did not see an option in the ?-help? output by the name of ?-mg_levels_ksp_max_it?, so I added one for each level. Adding ?-mg_levels_1_ksp_max_it 4  -mg_levels_2_ksp_max_it 4 ? gives the following convergence rate: 
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -mg_levels_1_ksp_max_it 4 -mg_levels_2_ksp_max_it 4
>     0 KSP Residual norm 2.980759132912e+01 
>     1 KSP Residual norm 1.268404746671e+00 
>     2 KSP Residual norm 1.420311012425e-01 
>     3 KSP Residual norm 3.026536678757e-02 
>     4 KSP Residual norm 6.511170312990e-03 
>     5 KSP Residual norm 1.539620841789e-03 
>     6 KSP Residual norm 3.655528499924e-04 
>     7 KSP Residual norm 8.111524453983e-05 
>     8 KSP Residual norm 1.995956470676e-05 
>     9 KSP Residual norm 4.397662980841e-06 
>    10 KSP Residual norm 9.636956929342e-07 
>    11 KSP Residual norm 3.013384202116e-07 
>    12 KSP Residual norm 1.867699579369e-07 
> 
> 
> Adding ?-pc_gamg_threshold 0.04? gives: 
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -mg_levels_1_ksp_max_it 4 -mg_levels_2_ksp_max_it 4 -pc_gamg_threshold 0.04
>     0 KSP Residual norm 2.980759132913e+01 
>     1 KSP Residual norm 1.268404746942e+00 
>     2 KSP Residual norm 1.420311012570e-01 
>     3 KSP Residual norm 3.026536679076e-02 
>     4 KSP Residual norm 6.511170313879e-03 
>     5 KSP Residual norm 1.539620841827e-03 
>     6 KSP Residual norm 3.655528500623e-04 
>     7 KSP Residual norm 8.111524453279e-05 
>     8 KSP Residual norm 1.995956474349e-05 
>     9 KSP Residual norm 4.397662966260e-06 
>    10 KSP Residual norm 9.636957102472e-07 
>    11 KSP Residual norm 3.013383993102e-07 
>    12 KSP Residual norm 1.867700181613e-07 
> 
> 
> Using just the ?-pc_gamg_square_graph 0? option: 
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_gamg_square_graph 0
>     0 KSP Residual norm 2.064808848243e+00 
>     1 KSP Residual norm 1.436709062429e+00 
>     2 KSP Residual norm 1.043690492056e+00 
>     3 KSP Residual norm 7.636589301482e-01 
>     4 KSP Residual norm 5.733849171144e-01 
>     5 KSP Residual norm 4.416033588916e-01 
>     6 KSP Residual norm 3.390846779861e-01 
>     7 KSP Residual norm 2.567469096297e-01 
>     8 KSP Residual norm 1.874940743028e-01 
>     9 KSP Residual norm 1.322422119221e-01 
>    10 KSP Residual norm 9.157000749016e-02 
>    11 KSP Residual norm 6.376782657530e-02 
>    12 KSP Residual norm 4.456316146538e-02 
>    13 KSP Residual norm 3.101613919753e-02 
>    14 KSP Residual norm 2.167127331495e-02 
>    15 KSP Residual norm 1.498469528896e-02 
>    16 KSP Residual norm 1.075794635819e-02 
>    17 KSP Residual norm 7.764685216272e-03 
>    18 KSP Residual norm 5.435228207429e-03 
>    19 KSP Residual norm 3.942376675316e-03 
>    20 KSP Residual norm 2.846234513499e-03 
>    21 KSP Residual norm 1.914323559680e-03 
>    22 KSP Residual norm 1.332049518265e-03 
>    23 KSP Residual norm 9.414825222665e-04 
>    24 KSP Residual norm 6.600086167534e-04 
>    25 KSP Residual norm 4.679306216706e-04 
>    26 KSP Residual norm 3.264136607741e-04 
>    27 KSP Residual norm 2.316366100549e-04 
>    28 KSP Residual norm 1.642594683886e-04 
>    29 KSP Residual norm 1.103783108795e-04 
>    30 KSP Residual norm 7.740619320447e-05 
>    31 KSP Residual norm 6.389704142174e-05 
>    32 KSP Residual norm 5.006485093406e-05 
>    33 KSP Residual norm 3.758456789529e-05 
>    34 KSP Residual norm 2.830743556147e-05 
>    35 KSP Residual norm 2.062379654606e-05 
>    36 KSP Residual norm 1.490349770670e-05 
>    37 KSP Residual norm 1.051767740994e-05 
>    38 KSP Residual norm 7.040814709680e-06 
>    39 KSP Residual norm 4.931079567116e-06 
>    40 KSP Residual norm 3.385386183658e-06 
>    41 KSP Residual norm 2.274379328203e-06 
>    42 KSP Residual norm 1.576979495342e-06 
>    43 KSP Residual norm 1.134465108080e-06 
>    44 KSP Residual norm 8.164007819038e-07 
>    45 KSP Residual norm 5.697298265561e-07 
>    46 KSP Residual norm 4.079302286082e-07 
>    47 KSP Residual norm 3.032840167758e-07 
>    48 KSP Residual norm 2.118663896145e-07 
>    49 KSP Residual norm 1.531268272774e-07 
>    50 KSP Residual norm 1.155662360224e-07 
>    51 KSP Residual norm 8.763548545594e-08 
>    52 KSP Residual norm 6.411338426040e-08 
>    53 KSP Residual norm 4.720820815019e-08 
>    54 KSP Residual norm 3.694420088192e-08 
>    55 KSP Residual norm 2.924088208699e-08 
>    56 KSP Residual norm 2.365173325198e-08 
>    57 KSP Residual norm 1.891630286373e-08 
> 
> 
> 
> Without the threshold and max it modifications, and adding ?-pc_gamg_square_graph 0? : 
> 
> -pc_type gamg -ksp_view --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -pc_mg_levels 2 -pc_gamg_square_graph 0
>     0 KSP Residual norm 4.658705362181e+01 
>     1 KSP Residual norm 6.723806072355e-01 
>     2 KSP Residual norm 4.063455422565e-02 
>     3 KSP Residual norm 2.311496772987e-03 
>     4 KSP Residual norm 2.337388101209e-04 
>     5 KSP Residual norm 2.541042271307e-05 
>     6 KSP Residual norm 5.461281412935e-06 
>     7 KSP Residual norm 2.718337804133e-06 
>     8 KSP Residual norm 1.223645122249e-06 
>     9 KSP Residual norm 7.877002862516e-07 
>    10 KSP Residual norm 4.742159201655e-07 
>    11 KSP Residual norm 2.849093170324e-07 
> 
> 
> So, it appears that the number of MG levels has the most significant impact on the convergence rate. 
> 
> What would be the reason for this? Is there a general recommendation on MG levels (default options gave 4)? I would suspect that this is problem dependent. 
> 
> Regards,
> Manav
> 
>> On Oct 29, 2018, at 8:28 AM, Mark Adams <mfadams at lbl.gov <mailto:mfadams at lbl.gov>> wrote:
>> 
>> 
>> I would recommend using '-mg_levels 2' and check that that gives you two levels. I would also run this on one processor just to start. 
>> 
>> Use -mg_levels_ksp_max_it 4.  And '-pc_gamg_threshold 0.04'
>> 
>> These parameters are meant to increase convergence rate at all costs. Once we our best rate we should be able to back off of some of this without much degradation or play around with the parameters to optimize run time. 
>> 
>> 
>> On Sun, Oct 28, 2018 at 5:13 PM Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
>> Var: 0,?,5  are the 6 variables that I am solving for: u, v, w, theta_x, theta_y, theta_z. 
>> 
>> The norms identified in my email are the L2 norms of all dofs corresponding to each variable in the solution vector. So, var: 0: u: norm is the L2 norm of the dofs for u only, and so on. 
>> 
>> I expect u, v, theta_z to be zero for the solution, which ends up being the case. 
>> 
>> If I plot the solution, they look sensible, but the reduction of KSP norm is slow. 
>> 
>> 
>> Thanks,
>> Manav
>> 
>>> On Oct 28, 2018, at 3:55 PM, Smith, Barry F. <bsmith at mcs.anl.gov <mailto:bsmith at mcs.anl.gov>> wrote:
>>> 
>>> 
>>> 
>>>> On Oct 28, 2018, at 12:16 PM, Manav Bhatia <bhatiamanav at gmail.com <mailto:bhatiamanav at gmail.com>> wrote:
>>>> 
>>>> Hi, 
>>>> 
>>>>   I am attempting to solve a Mindlin plate bending problem with AMG solver in petsc. This test case is with a mesh of 300x300 elements and 543,606 dofs. 
>>>> 
>>>>   The discretization includes 6 variables (u, v, w, tx, ty, tz), but only three are relevant for plate bending (w, tx, ty). 
>>>> 
>>>>   I am calling the solver with the following options: 
>>>> 
>>>> -pc_type gamg -pc_gamg_threshold 0. --node-major-dofs -mat_block_size 6 -ksp_rtol 1.e-8 -ksp_monitor -ksp_converged_reason -ksp_view 
>>>> 
>>>>  And the convergence behavior is shown below, along with the ksp_view information. Based on notes in the manual, this seems to be subpar convergence rate. At the end of the solution the norm of each variable is : 
>>>> 
>>>> var: 0: u  : norm: 5.505909e-18
>>>> var: 1: v  : norm: 7.639640e-18
>>>> var: 2: w : norm: 3.901464e-03
>>>> var: 3: tx : norm: 4.403576e-02
>>>> var: 4: ty : norm: 4.403576e-02
>>>> var: 5: tz : norm: 1.148409e-16
>>> 
>>>   What do you mean by var: 2: w : norm etc? Is this the norm of the error for that variable, the norm of the residual, something else? How exactly are you calculating it?
>>> 
>>>    Thanks
>>> 
>>> 
>>>   Barry
>>> 
>>>> 
>>>>  I tried different values of -ksp_rtol from 1e-1 to 1e-8 and this does not make a lot of difference in the norms of (w, tx, ty). 
>>>> 
>>>>  I do provide the solver with 6 rigid-body vectors to approximate the null-space of the problem. Without these the solver shows very poor convergence. 
>>>> 
>>>>  I would appreciate advice on possible strategies to improve this behavior. 
>>>> 
>>>> Thanks,
>>>> Manav 
>>>> 
>>>>    0 KSP Residual norm 1.696304497261e+00 
>>>>    1 KSP Residual norm 1.120485505777e+00 
>>>>    2 KSP Residual norm 8.324222302402e-01 
>>>>    3 KSP Residual norm 6.477349534115e-01 
>>>>    4 KSP Residual norm 5.080936471292e-01 
>>>>    5 KSP Residual norm 4.051099646638e-01 
>>>>    6 KSP Residual norm 3.260432664653e-01 
>>>>    7 KSP Residual norm 2.560483838143e-01 
>>>>    8 KSP Residual norm 2.029943986124e-01 
>>>>    9 KSP Residual norm 1.560985741610e-01 
>>>>   10 KSP Residual norm 1.163720702140e-01 
>>>>   11 KSP Residual norm 8.488411085459e-02 
>>>>   12 KSP Residual norm 5.888041729034e-02 
>>>>   13 KSP Residual norm 4.027792209980e-02 
>>>>   14 KSP Residual norm 2.819048087304e-02 
>>>>   15 KSP Residual norm 1.904674196962e-02 
>>>>   16 KSP Residual norm 1.289302447822e-02 
>>>>   17 KSP Residual norm 9.162203296376e-03 
>>>>   18 KSP Residual norm 7.016781679507e-03 
>>>>   19 KSP Residual norm 5.399170865328e-03 
>>>>   20 KSP Residual norm 4.254385887482e-03 
>>>>   21 KSP Residual norm 3.530831740621e-03 
>>>>   22 KSP Residual norm 2.946780747923e-03 
>>>>   23 KSP Residual norm 2.339361361128e-03 
>>>>   24 KSP Residual norm 1.815072489282e-03 
>>>>   25 KSP Residual norm 1.408814185342e-03 
>>>>   26 KSP Residual norm 1.063795714320e-03 
>>>>   27 KSP Residual norm 7.828540233117e-04 
>>>>   28 KSP Residual norm 5.683910750067e-04 
>>>>   29 KSP Residual norm 4.131151010250e-04 
>>>>   30 KSP Residual norm 3.065608221019e-04 
>>>>   31 KSP Residual norm 2.634114273459e-04 
>>>>   32 KSP Residual norm 2.198180137626e-04 
>>>>   33 KSP Residual norm 1.748956510799e-04 
>>>>   34 KSP Residual norm 1.317539710010e-04 
>>>>   35 KSP Residual norm 9.790121566055e-05 
>>>>   36 KSP Residual norm 7.465935386094e-05 
>>>>   37 KSP Residual norm 5.689506626052e-05 
>>>>   38 KSP Residual norm 4.413136619126e-05 
>>>>   39 KSP Residual norm 3.512194236402e-05 
>>>>   40 KSP Residual norm 2.877755408287e-05 
>>>>   41 KSP Residual norm 2.340080556431e-05 
>>>>   42 KSP Residual norm 1.904544450345e-05 
>>>>   43 KSP Residual norm 1.504723478235e-05 
>>>>   44 KSP Residual norm 1.141381950576e-05 
>>>>   45 KSP Residual norm 8.206151384599e-06 
>>>>   46 KSP Residual norm 5.911426091276e-06 
>>>>   47 KSP Residual norm 4.233669089283e-06 
>>>>   48 KSP Residual norm 2.898052944223e-06 
>>>>   49 KSP Residual norm 2.023556779973e-06 
>>>>   50 KSP Residual norm 1.459108043935e-06 
>>>>   51 KSP Residual norm 1.097335545865e-06 
>>>>   52 KSP Residual norm 8.440457332262e-07 
>>>>   53 KSP Residual norm 6.705616854004e-07 
>>>>   54 KSP Residual norm 5.404888680234e-07 
>>>>   55 KSP Residual norm 4.391368084979e-07 
>>>>   56 KSP Residual norm 3.697063014621e-07 
>>>>   57 KSP Residual norm 3.021772094146e-07 
>>>>   58 KSP Residual norm 2.479354520792e-07 
>>>>   59 KSP Residual norm 2.013077841968e-07 
>>>>   60 KSP Residual norm 1.553159612793e-07 
>>>>   61 KSP Residual norm 1.400784224898e-07 
>>>>   62 KSP Residual norm 9.707453662195e-08 
>>>>   63 KSP Residual norm 7.263173080146e-08 
>>>>   64 KSP Residual norm 5.593723572132e-08 
>>>>   65 KSP Residual norm 4.448788809586e-08 
>>>>   66 KSP Residual norm 3.613992590778e-08 
>>>>   67 KSP Residual norm 2.946099051876e-08 
>>>>   68 KSP Residual norm 2.408053564170e-08 
>>>>   69 KSP Residual norm 1.945257374856e-08 
>>>>   70 KSP Residual norm 1.572494535110e-08 
>>>> 
>>>> 
>>>> KSP Object: 4 MPI processes
>>>>  type: gmres
>>>>    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>    happy breakdown tolerance 1e-30
>>>>  maximum iterations=10000, initial guess is zero
>>>>  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>>>>  left preconditioning
>>>>  using PRECONDITIONED norm type for convergence test
>>>> PC Object: 4 MPI processes
>>>>  type: gamg
>>>>    type is MULTIPLICATIVE, levels=6 cycles=v
>>>>      Cycles per PCApply=1
>>>>      Using externally compute Galerkin coarse grid matrices
>>>>      GAMG specific options
>>>>        Threshold for dropping small values in graph on each level =   0.   0.   0.   0.  
>>>>        Threshold scaling factor for each level not specified = 1.
>>>>        AGG specific options
>>>>          Symmetric graph false
>>>>          Number of levels to square graph 1
>>>>          Number smoothing steps 1
>>>>  Coarse grid solver -- level -------------------------------
>>>>    KSP Object: (mg_coarse_) 4 MPI processes
>>>>      type: preonly
>>>>      maximum iterations=10000, initial guess is zero
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_coarse_) 4 MPI processes
>>>>      type: bjacobi
>>>>        number of blocks = 4
>>>>        Local solve is same for all blocks, in the following KSP and PC objects:
>>>>      KSP Object: (mg_coarse_sub_) 1 MPI processes
>>>>        type: preonly
>>>>        maximum iterations=1, initial guess is zero
>>>>        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>        left preconditioning
>>>>        using NONE norm type for convergence test
>>>>      PC Object: (mg_coarse_sub_) 1 MPI processes
>>>>        type: lu
>>>>          out-of-place factorization
>>>>          tolerance for zero pivot 2.22045e-14
>>>>          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
>>>>          matrix ordering: nd
>>>>          factor fill ratio given 5., needed 1.
>>>>            Factored matrix follows:
>>>>              Mat Object: 1 MPI processes
>>>>                type: seqaij
>>>>                rows=6, cols=6, bs=6
>>>>                package used to perform factorization: petsc
>>>>                total: nonzeros=36, allocated nonzeros=36
>>>>                total number of mallocs used during MatSetValues calls =0
>>>>                  using I-node routines: found 2 nodes, limit used is 5
>>>>        linear system matrix = precond matrix:
>>>>        Mat Object: 1 MPI processes
>>>>          type: seqaij
>>>>          rows=6, cols=6, bs=6
>>>>          total: nonzeros=36, allocated nonzeros=36
>>>>          total number of mallocs used during MatSetValues calls =0
>>>>            using I-node routines: found 2 nodes, limit used is 5
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=6, cols=6, bs=6
>>>>        total: nonzeros=36, allocated nonzeros=36
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          using nonscalable MatPtAP() implementation
>>>>          using I-node (on process 0) routines: found 2 nodes, limit used is 5
>>>>  Down solver (pre-smoother) on level 1 -------------------------------
>>>>    KSP Object: (mg_levels_1_) 4 MPI processes
>>>>      type: chebyshev
>>>>        eigenvalue estimates used:  min = 0.099971, max = 1.09968
>>>>        eigenvalues estimate via gmres min 0.154032, max 0.99971
>>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>>        KSP Object: (mg_levels_1_esteig_) 4 MPI processes
>>>>          type: gmres
>>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>            happy breakdown tolerance 1e-30
>>>>          maximum iterations=10, initial guess is zero
>>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>          left preconditioning
>>>>          using PRECONDITIONED norm type for convergence test
>>>>        estimating eigenvalues using noisy right hand side
>>>>      maximum iterations=2, nonzero initial guess
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_levels_1_) 4 MPI processes
>>>>      type: sor
>>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=54, cols=54, bs=6
>>>>        total: nonzeros=2916, allocated nonzeros=2916
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          using I-node (on process 0) routines: found 11 nodes, limit used is 5
>>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>  Down solver (pre-smoother) on level 2 -------------------------------
>>>>    KSP Object: (mg_levels_2_) 4 MPI processes
>>>>      type: chebyshev
>>>>        eigenvalue estimates used:  min = 0.171388, max = 1.88526
>>>>        eigenvalues estimate via gmres min 0.0717873, max 1.71388
>>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>>        KSP Object: (mg_levels_2_esteig_) 4 MPI processes
>>>>          type: gmres
>>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>            happy breakdown tolerance 1e-30
>>>>          maximum iterations=10, initial guess is zero
>>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>          left preconditioning
>>>>          using PRECONDITIONED norm type for convergence test
>>>>        estimating eigenvalues using noisy right hand side
>>>>      maximum iterations=2, nonzero initial guess
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_levels_2_) 4 MPI processes
>>>>      type: sor
>>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=642, cols=642, bs=6
>>>>        total: nonzeros=99468, allocated nonzeros=99468
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          using nonscalable MatPtAP() implementation
>>>>          using I-node (on process 0) routines: found 47 nodes, limit used is 5
>>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>  Down solver (pre-smoother) on level 3 -------------------------------
>>>>    KSP Object: (mg_levels_3_) 4 MPI processes
>>>>      type: chebyshev
>>>>        eigenvalue estimates used:  min = 0.164216, max = 1.80637
>>>>        eigenvalues estimate via gmres min 0.0376323, max 1.64216
>>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>>        KSP Object: (mg_levels_3_esteig_) 4 MPI processes
>>>>          type: gmres
>>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>            happy breakdown tolerance 1e-30
>>>>          maximum iterations=10, initial guess is zero
>>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>          left preconditioning
>>>>          using PRECONDITIONED norm type for convergence test
>>>>        estimating eigenvalues using noisy right hand side
>>>>      maximum iterations=2, nonzero initial guess
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_levels_3_) 4 MPI processes
>>>>      type: sor
>>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=6726, cols=6726, bs=6
>>>>        total: nonzeros=941796, allocated nonzeros=941796
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          using nonscalable MatPtAP() implementation
>>>>          using I-node (on process 0) routines: found 552 nodes, limit used is 5
>>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>  Down solver (pre-smoother) on level 4 -------------------------------
>>>>    KSP Object: (mg_levels_4_) 4 MPI processes
>>>>      type: chebyshev
>>>>        eigenvalue estimates used:  min = 0.163283, max = 1.79611
>>>>        eigenvalues estimate via gmres min 0.0350306, max 1.63283
>>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>>        KSP Object: (mg_levels_4_esteig_) 4 MPI processes
>>>>          type: gmres
>>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>            happy breakdown tolerance 1e-30
>>>>          maximum iterations=10, initial guess is zero
>>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>          left preconditioning
>>>>          using PRECONDITIONED norm type for convergence test
>>>>        estimating eigenvalues using noisy right hand side
>>>>      maximum iterations=2, nonzero initial guess
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_levels_4_) 4 MPI processes
>>>>      type: sor
>>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=41022, cols=41022, bs=6
>>>>        total: nonzeros=2852316, allocated nonzeros=2852316
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          using nonscalable MatPtAP() implementation
>>>>          using I-node (on process 0) routines: found 3432 nodes, limit used is 5
>>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>  Down solver (pre-smoother) on level 5 -------------------------------
>>>>    KSP Object: (mg_levels_5_) 4 MPI processes
>>>>      type: chebyshev
>>>>        eigenvalue estimates used:  min = 0.157236, max = 1.7296
>>>>        eigenvalues estimate via gmres min 0.0317897, max 1.57236
>>>>        eigenvalues estimated using gmres with translations  [0. 0.1; 0. 1.1]
>>>>        KSP Object: (mg_levels_5_esteig_) 4 MPI processes
>>>>          type: gmres
>>>>            restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>>>            happy breakdown tolerance 1e-30
>>>>          maximum iterations=10, initial guess is zero
>>>>          tolerances:  relative=1e-12, absolute=1e-50, divergence=10000.
>>>>          left preconditioning
>>>>          using PRECONDITIONED norm type for convergence test
>>>>        estimating eigenvalues using noisy right hand side
>>>>      maximum iterations=2, nonzero initial guess
>>>>      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>>>      left preconditioning
>>>>      using NONE norm type for convergence test
>>>>    PC Object: (mg_levels_5_) 4 MPI processes
>>>>      type: sor
>>>>        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
>>>>      linear system matrix = precond matrix:
>>>>      Mat Object: () 4 MPI processes
>>>>        type: mpiaij
>>>>        rows=543606, cols=543606, bs=6
>>>>        total: nonzeros=29224836, allocated nonzeros=29302596
>>>>        total number of mallocs used during MatSetValues calls =0
>>>>          has attached near null space
>>>>          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
>>>>  Up solver (post-smoother) same as down solver (pre-smoother)
>>>>  linear system matrix = precond matrix:
>>>>  Mat Object: () 4 MPI processes
>>>>    type: mpiaij
>>>>    rows=543606, cols=543606, bs=6
>>>>    total: nonzeros=29224836, allocated nonzeros=29302596
>>>>    total number of mallocs used during MatSetValues calls =0
>>>>      has attached near null space
>>>>      using I-node (on process 0) routines: found 45644 nodes, limit used is 5
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/61980556/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Oct 29 11:25:10 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 11:25:10 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
Message-ID: <69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>

Some more results on iterations and CPU time: 


1) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 9 KSP iterations, 66 secs
2) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4: 9 KSP iterations, 66 secs
3) -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 12 KSP iterations, 29 secs
4) -pc_mg_levels 2 -mg_levels_ksp_max_it 4: 12 KSP iterations, 28 secs

So, it appears that ?-pc_gamg_square_graph 0? leads to a significant increase in CPU time. 

What would be a more reasonable value for ?-pc_gamg_square_graph? ? 

Regards,
Manav



1) 
-pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04

Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.5896e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 

------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0 1.0610e-04 4.6 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BuildTwoSidedF        69 1.0 3.3554e-01 2.0 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
VecMDot               34 1.0 6.3575e-02 1.4 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  2907
VecNorm               41 1.0 1.3934e-02 1.6 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3199
VecScale              38 1.0 5.5904e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3695
VecCopy               44 1.0 1.1493e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               129 1.0 1.9488e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                3 1.0 1.0059e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3243
VecAYPX              120 1.0 4.7344e-02 1.2 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1607
VecAXPBYCZ            80 1.0 3.9797e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5464
VecWAXPY               1 1.0 6.1512e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   884
VecMAXPY              37 1.0 5.3887e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4015
VecAssemblyBegin      60 1.0 1.1819e-01 9.8 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
VecAssemblyEnd        60 1.0 1.5020e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      11 1.0 5.6922e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1050
VecScatterBegin      226 1.0 1.8359e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
VecScatterEnd        226 1.0 4.3040e+01369.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 49  0  0  0  0  49  0  0  0  0     0
VecSetRandom           1 1.0 1.7331e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecReduceArith         3 1.0 4.8590e-04 1.0 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6713
VecReduceComm          1 1.0 1.5974e-05 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          38 1.0 1.9025e-02 1.4 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  3257
MatMult              129 1.0 1.9964e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3742
MatMultAdd            10 1.0 4.2991e+011654.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 49  0  1  4  0  49  0  1  4  0     5
MatMultTranspose      10 1.0 6.5614e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3472
MatSolve              10 0.0 1.0351e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2112
MatSOR               111 1.0 3.3396e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  1933
MatLUFactorSym         1 1.0 2.3057e+00193417.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatLUFactorNum         1 1.0 3.9508e+018285439.5 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2879
MatScale               3 1.0 1.2049e-02 1.2 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  2026
MatResidual           10 1.0 1.6667e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3507
MatAssemblyBegin      25 1.0 2.2748e-01 3.0 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
MatAssemblyEnd        25 1.0 2.6301e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
MatGetRow         593372 1.0 5.4903e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 0.0 1.3105e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCreateSubMat        2 1.0 6.8462e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
MatGetOrdering         1 0.0 5.0859e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             1 1.0 1.8659e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
MatZeroEntries         9 1.0 1.5140e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView                5 1.7 1.4541e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
MatAXPY                1 1.0 5.6004e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             1 1.0 3.1570e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1101
MatMatMultSym          1 1.0 2.4168e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
MatMatMultNum          1 1.0 7.3501e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  4729
MatPtAP                1 1.0 1.3500e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  1706
MatPtAPSymbolic        1 1.0 9.7516e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
MatPtAPNumeric         1 1.0 3.7405e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6157
MatGetLocalMat         3 1.0 5.3542e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          3 1.0 2.5439e-03 1.1 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
SNESSolve              1 1.0 5.9280e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2256
SNESFunctionEval       2 1.0 4.4243e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
SNESJacobianEval       1 1.0 3.1546e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  5  0  3  4  4   5  0  3  4  4     0
SNESLineSearch         1 1.0 2.2030e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  3  0 14  4 16   3  0 14  4 17     2
SFSetGraph             1 1.0 2.1458e-06 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetUp                1 1.0 1.5092e-04 2.4 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin           4 1.0 4.3154e-05 1.0 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
SFBcastEnd             4 1.0 3.8099e-0432.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        29 1.0 1.0221e-01 1.2 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  3298
KSPSetUp               6 1.0 9.9370e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 5.1697e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 78100 65 76 52  78100 65 76 53  2587
PCGAMGGraph_AGG        1 1.0 4.4754e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
PCGAMGCoarse_AGG       1 1.0 2.5029e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
PCGAMGProl_AGG         1 1.0 1.0502e-01 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
PCGAMGPOpt_AGG         1 1.0 6.2535e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  1767
GAMG: createProl       1 1.0 1.1819e+00 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30   936
  Graph                2 1.0 4.4748e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
  MIS/Agg              1 1.0 1.8790e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
  SA: col data         1 1.0 5.0070e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
  SA: frmProl0         1 1.0 5.1879e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
  SA: smooth           1 1.0 3.9527e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  1  0  2  1  3   1  0  2  1  3   937
GAMG: partLevel        1 1.0 2.1293e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1082
  repartition          1 1.0 3.4499e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Invert-Sort          1 1.0 1.0570e-0221.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Move A               1 1.0 4.4678e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
  Move P               1 1.0 2.8244e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
PCSetUp                2 1.0 4.5188e+0113.6 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 21 88 25 64 44  21 88 25 64 45  2593
PCSetUpOnBlocks       10 1.0 4.1865e+01570107.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2717
PCApply               10 1.0 4.8200e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 73 97 37 12  4  73 97 37 12  4  2690
------------------------------------------------------------------------------------------------------------------------


2) 
-pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4

Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 6.5760e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 

------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0 1.9002e-04 8.7 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BuildTwoSidedF        69 1.0 2.6071e-01 4.8 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
VecMDot               34 1.0 4.7955e-02 1.3 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  3854
VecNorm               41 1.0 7.8864e-03 1.4 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  5652
VecScale              38 1.0 4.7059e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4390
VecCopy               44 1.0 1.0849e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               129 1.0 1.9698e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                3 1.0 1.0378e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3143
VecAYPX              120 1.0 4.6406e-02 1.1 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1640
VecAXPBYCZ            80 1.0 4.0414e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5380
VecWAXPY               1 1.0 7.6008e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   715
VecMAXPY              37 1.0 4.8892e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4425
VecAssemblyBegin      60 1.0 5.2444e-02 2.0 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
VecAssemblyEnd        60 1.0 1.1539e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      11 1.0 6.2540e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   956
VecScatterBegin      226 1.0 1.7562e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
VecScatterEnd        226 1.0 4.3872e+01319.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 50  0  0  0  0  50  0  0  0  0     0
VecSetRandom           1 1.0 1.5719e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecReduceArith         3 1.0 5.7292e-04 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5693
VecReduceComm          1 1.0 9.0122e-0515.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          38 1.0 1.2057e-02 1.2 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  5140
MatMult              129 1.0 1.9545e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3822
MatMultAdd            10 1.0 4.3806e+011689.9 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 50  0  1  4  0  50  0  1  4  0     5
MatMultTranspose      10 1.0 6.7213e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3390
MatSolve              10 0.0 1.0379e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2106
MatSOR               111 1.0 3.1935e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  2022
MatLUFactorSym         1 1.0 2.3361e+00288179.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatLUFactorNum         1 1.0 4.0301e+0110564673.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2823
MatScale               3 1.0 1.4505e-02 1.0 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1683
MatResidual           10 1.0 1.6052e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3641
MatAssemblyBegin      25 1.0 2.4469e-01 7.5 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
MatAssemblyEnd        25 1.0 2.2773e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
MatGetRow         593372 1.0 4.2183e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 0.0 9.4678e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCreateSubMat        2 1.0 6.3013e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
MatGetOrdering         1 0.0 4.1698e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             1 1.0 1.4472e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
MatZeroEntries         9 1.0 1.5332e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView                5 1.7 1.6737e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
MatAXPY                1 1.0 3.6167e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             1 1.0 2.6676e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1303
MatMatMultSym          1 1.0 2.0918e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
MatMatMultNum          1 1.0 5.7073e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  6090
MatPtAP                1 1.0 1.0664e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  2160
MatPtAPSymbolic        1 1.0 7.2993e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
MatPtAPNumeric         1 1.0 3.3595e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6855
MatGetLocalMat         3 1.0 5.2164e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          3 1.0 2.5141e-03 1.2 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
SNESSolve              1 1.0 5.8984e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2267
SNESFunctionEval       2 1.0 4.4220e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
SNESJacobianEval       1 1.0 2.7677e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  4  0  3  4  4   4  0  3  4  4     0
SNESLineSearch         1 1.0 2.3403e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  4  0 14  4 16   4  0 14  4 17     2
SFSetGraph             1 1.0 1.9073e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetUp                1 1.0 2.3508e-04 3.9 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin           4 1.0 3.2902e-05 1.1 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
SFBcastEnd             4 1.0 1.2112e-0416.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        29 1.0 8.2165e-02 1.1 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  4102
KSPSetUp               6 1.0 8.9242e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 5.1788e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 79100 65 76 52  79100 65 76 53  2582
PCGAMGGraph_AGG        1 1.0 3.8242e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
PCGAMGCoarse_AGG       1 1.0 2.0711e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
PCGAMGProl_AGG         1 1.0 7.1434e-02 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
PCGAMGPOpt_AGG         1 1.0 5.3496e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  2065
GAMG: createProl       1 1.0 9.9368e-01 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30  1114
  Graph                2 1.0 3.8235e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
  MIS/Agg              1 1.0 1.4579e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
  SA: col data         1 1.0 3.0082e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
  SA: frmProl0         1 1.0 3.8682e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
  SA: smooth           1 1.0 3.2894e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  0  0  2  1  3   0  0  2  1  3  1126
GAMG: partLevel        1 1.0 1.7867e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1289
  repartition          1 1.0 2.9397e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Invert-Sort          1 1.0 7.5300e-0316.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Move A               1 1.0 4.5402e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
  Move P               1 1.0 2.2056e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
PCSetUp                2 1.0 4.5469e+0116.3 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 20 88 25 64 44  20 88 25 64 45  2577
PCSetUpOnBlocks       10 1.0 4.2679e+01701992.3 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2665
PCApply               10 1.0 4.8823e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 74 97 37 12  4  74 97 37 12  4  2656
------------------------------------------------------------------------------------------------------------------------


3) 
-pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04

Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.9122e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 

------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0 5.1999e-0423.5 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BuildTwoSidedF        74 1.0 1.6531e-01 2.2 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  0  0  2  5  0   0  0  2  5  0     0
VecMDot               37 1.0 7.6608e-02 1.3 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  2881
VecNorm               44 1.0 1.9918e-02 2.0 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  2402
VecScale              41 1.0 5.9566e-03 1.1 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3742
VecCopy               53 1.0 1.3126e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               151 1.0 2.0747e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                3 1.0 1.2219e-03 1.6 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2669
VecAYPX              156 1.0 6.1843e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1600
VecAXPBYCZ           104 1.0 5.3596e-02 1.0 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5274
VecWAXPY               1 1.0 6.4707e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   840
VecMAXPY              40 1.0 6.1941e-02 1.1 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4125
VecAssemblyBegin      64 1.0 7.2242e-02 1.3 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
VecAssemblyEnd        64 1.0 1.4734e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      11 1.0 6.0835e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   983
VecScatterBegin      270 1.0 1.7434e-02 1.1 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
VecScatterEnd        270 1.0 5.8946e+0035.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
VecSetRandom           1 1.0 2.9080e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecReduceArith         3 1.0 6.4564e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5052
VecReduceComm          1 1.0 1.6594e-0427.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          41 1.0 2.5299e-02 1.6 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  2643
MatMult              162 1.0 2.5775e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3640
MatMultAdd            13 1.0 5.7682e+00227.3 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    37
MatMultTranspose      13 1.0 6.4702e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3268
MatSolve              13 0.0 3.4193e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1850
MatSOR               141 1.0 4.2887e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1912
MatLUFactorSym         1 1.0 5.2129e-0139753.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 4.8021e+00805666.4 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2767
MatScale               3 1.0 1.0527e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1699
MatResidual           13 1.0 2.1939e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3463
MatAssemblyBegin      33 1.0 1.1140e-01 5.0 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
MatAssemblyEnd        33 1.0 1.9615e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
MatGetRow         593372 1.0 4.5789e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 0.0 4.0052e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCreateSubMat        2 1.0 2.7033e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
MatGetOrdering         1 0.0 1.7007e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             1 1.0 2.4688e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
MatZeroEntries        10 1.0 1.4357e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatView                5 1.7 1.7419e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
MatAXPY                1 1.0 3.4560e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             1 1.0 2.6894e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1292
MatMatMultSym          1 1.0 2.1331e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
MatMatMultNum          1 1.0 5.5709e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  6239
MatPtAP                1 1.0 6.6371e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  2039
MatPtAPSymbolic        1 1.0 4.6951e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
MatPtAPNumeric         1 1.0 1.9424e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6967
MatTrnMatMult          1 1.0 5.6805e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   292
MatTrnMatMultSym       1 1.0 3.4907e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
MatTrnMatMultNum       1 1.0 2.2120e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   751
MatGetLocalMat         4 1.0 3.8992e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          3 1.0 2.6469e-03 1.2 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
SNESSolve              1 1.0 2.2497e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 77100 97 79 89  77100 97 79 90  1536
SNESFunctionEval       2 1.0 4.5633e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
SNESJacobianEval       1 1.0 3.0567e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
SNESLineSearch         1 1.0 2.3701e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
SFSetGraph             1 1.0 3.0994e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetUp                1 1.0 5.7101e-04 6.6 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin           4 1.0 5.6982e-05 1.2 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
SFBcastEnd             4 1.0 1.6999e-04 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        32 1.0 1.2112e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3375
KSPSetUp               6 1.0 9.6612e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 1.4872e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2323
PCGAMGGraph_AGG        1 1.0 4.4312e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
PCGAMGCoarse_AGG       1 1.0 6.4611e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   257
PCGAMGProl_AGG         1 1.0 9.3410e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
PCGAMGPOpt_AGG         1 1.0 5.4947e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  1999
GAMG: createProl       1 1.0 1.1555e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32   966
  Graph                2 1.0 4.4305e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
  MIS/Agg              1 1.0 2.4810e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
  SA: col data         1 1.0 4.5491e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
  SA: frmProl0         1 1.0 4.5125e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
  SA: smooth           1 1.0 3.2692e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1113
GAMG: partLevel        1 1.0 9.6929e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1396
  repartition          1 1.0 1.0109e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Invert-Sort          1 1.0 1.2600e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Move A               1 1.0 1.2418e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
  Move P               1 1.0 1.6575e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
PCSetUp                2 1.0 7.4755e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2108
PCSetUpOnBlocks       13 1.0 5.3405e+0062049.0 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2488
PCApply               13 1.0 1.2494e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2496
------------------------------------------------------------------------------------------------------------------------



4) 
-pc_mg_levels 2 -mg_levels_ksp_max_it 4

Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
 0:      Main Stage: 2.8086e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 

------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------
Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
------------------------------------------------------------------------------------------------------------------------

--- Event Stage 0: Main Stage

BuildTwoSided          1 1.0 9.6083e-05 5.3 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
BuildTwoSidedF        74 1.0 2.0508e-01 2.9 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  1  0  2  5  0   1  0  2  5  0     0
VecMDot               37 1.0 6.2755e-02 1.2 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  3517
VecNorm               44 1.0 1.2301e-02 1.5 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  3889
VecScale              41 1.0 5.2664e-03 1.0 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4232
VecCopy               53 1.0 1.3689e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecSet               151 1.0 1.9269e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecAXPY                3 1.0 1.1799e-03 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2764
VecAYPX              156 1.0 6.0851e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1626
VecAXPBYCZ           104 1.0 5.3286e-02 1.1 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5305
VecWAXPY               1 1.0 7.0786e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   768
VecMAXPY              40 1.0 6.2898e-02 1.0 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4062
VecAssemblyBegin      64 1.0 1.0848e-01 2.7 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
VecAssemblyEnd        64 1.0 1.1230e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecPointwiseMult      11 1.0 6.5417e-03 1.1 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   914
VecScatterBegin      270 1.0 1.6606e-02 1.0 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
VecScatterEnd        270 1.0 5.5767e+0044.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
VecSetRandom           1 1.0 1.5750e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecReduceArith         3 1.0 6.3324e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5151
VecReduceComm          1 1.0 1.5712e-0431.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
VecNormalize          41 1.0 1.7057e-02 1.3 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3920
MatMult              162 1.0 2.5049e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3745
MatMultAdd            13 1.0 5.5041e+00222.4 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    38
MatMultTranspose      13 1.0 6.2563e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3379
MatSolve              13 0.0 3.4633e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1827
MatSOR               141 1.0 4.1983e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1953
MatLUFactorSym         1 1.0 4.5545e-0135375.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatLUFactorNum         1 1.0 4.6021e+00772110.8 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2887
MatScale               3 1.0 1.1058e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1618
MatResidual           13 1.0 2.1592e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3519
MatAssemblyBegin      33 1.0 1.1083e-01 3.3 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
MatAssemblyEnd        33 1.0 1.7722e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
MatGetRow         593372 1.0 4.5285e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetRowIJ            1 0.0 3.5741e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCreateSubMat        2 1.0 2.5588e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
MatGetOrdering         1 0.0 1.6127e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatCoarsen             1 1.0 1.4122e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
MatZeroEntries        10 1.0 1.5053e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
MatView                5 1.7 1.6868e-03 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
MatAXPY                1 1.0 4.3107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatMatMult             1 1.0 2.5280e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1375
MatMatMultSym          1 1.0 1.9445e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
MatMatMultNum          1 1.0 5.8333e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  5959
MatPtAP                1 1.0 6.7796e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  1996
MatPtAPSymbolic        1 1.0 4.8317e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
MatPtAPNumeric         1 1.0 1.9495e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6941
MatTrnMatMult          1 1.0 4.1985e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   396
MatTrnMatMultSym       1 1.0 2.3958e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
MatTrnMatMultNum       1 1.0 1.8097e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   918
MatGetLocalMat         4 1.0 4.4538e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
MatGetBrAoCol          3 1.0 3.8249e-03 1.6 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
SNESSolve              1 1.0 2.1417e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 76100 97 79 89  76100 97 79 90  1614
SNESFunctionEval       2 1.0 4.4048e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
SNESJacobianEval       1 1.0 2.6810e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
SNESLineSearch         1 1.0 2.2640e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
SFSetGraph             1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFSetUp                1 1.0 1.4687e-04 2.3 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
SFBcastBegin           4 1.0 3.1948e-05 1.1 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
SFBcastEnd             4 1.0 4.2200e-05 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPGMRESOrthog        32 1.0 1.0959e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3730
KSPSetUp               6 1.0 9.1093e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
KSPSolve               1 1.0 1.4326e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2412
PCGAMGGraph_AGG        1 1.0 3.8501e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
PCGAMGCoarse_AGG       1 1.0 4.7253e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   352
PCGAMGProl_AGG         1 1.0 6.9987e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
PCGAMGPOpt_AGG         1 1.0 5.4272e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  2024
GAMG: createProl       1 1.0 1.0504e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32  1063
  Graph                2 1.0 3.8496e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
  MIS/Agg              1 1.0 1.4219e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
  SA: col data         1 1.0 3.1489e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
  SA: frmProl0         1 1.0 3.5651e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
  SA: smooth           1 1.0 3.2070e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1135
GAMG: partLevel        1 1.0 9.6892e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1397
  repartition          1 1.0 1.1492e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Invert-Sort          1 1.0 1.3120e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
  Move A               1 1.0 1.2847e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
  Move P               1 1.0 1.4728e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
PCSetUp                2 1.0 7.1033e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2218
PCSetUpOnBlocks       13 1.0 5.0738e+0063336.2 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2618
PCApply               13 1.0 1.2059e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2586
------------------------------------------------------------------------------------------------------------------------


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/6963dea0/attachment-0001.html>

From bhatiamanav at gmail.com  Mon Oct 29 11:43:15 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 11:43:15 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
Message-ID: <2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>

Hi, 

  I am trying to understand the meaning of ?pc_gamg_square_graph?. I did not find a specific reference to ?square? in the tutorial here: http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf <http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf> . 

  What would be a good place to look for some background on this? 

Regards,
Manav

> On Oct 29, 2018, at 11:25 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Some more results on iterations and CPU time: 
> 
> 
> 1) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 9 KSP iterations, 66 secs
> 2) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4: 9 KSP iterations, 66 secs
> 3) -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 12 KSP iterations, 29 secs
> 4) -pc_mg_levels 2 -mg_levels_ksp_max_it 4: 12 KSP iterations, 28 secs
> 
> So, it appears that ?-pc_gamg_square_graph 0? leads to a significant increase in CPU time. 
> 
> What would be a more reasonable value for ?-pc_gamg_square_graph? ? 
> 
> Regards,
> Manav
> 
> 
> 
> 1) 
> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
> 
> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>  0:      Main Stage: 6.5896e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
> 
> ------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
> 
> --- Event Stage 0: Main Stage
> 
> BuildTwoSided          1 1.0 1.0610e-04 4.6 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> BuildTwoSidedF        69 1.0 3.3554e-01 2.0 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
> VecMDot               34 1.0 6.3575e-02 1.4 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  2907
> VecNorm               41 1.0 1.3934e-02 1.6 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3199
> VecScale              38 1.0 5.5904e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3695
> VecCopy               44 1.0 1.1493e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet               129 1.0 1.9488e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecAXPY                3 1.0 1.0059e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3243
> VecAYPX              120 1.0 4.7344e-02 1.2 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1607
> VecAXPBYCZ            80 1.0 3.9797e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5464
> VecWAXPY               1 1.0 6.1512e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   884
> VecMAXPY              37 1.0 5.3887e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4015
> VecAssemblyBegin      60 1.0 1.1819e-01 9.8 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> VecAssemblyEnd        60 1.0 1.5020e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecPointwiseMult      11 1.0 5.6922e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1050
> VecScatterBegin      226 1.0 1.8359e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
> VecScatterEnd        226 1.0 4.3040e+01369.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 49  0  0  0  0  49  0  0  0  0     0
> VecSetRandom           1 1.0 1.7331e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecReduceArith         3 1.0 4.8590e-04 1.0 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6713
> VecReduceComm          1 1.0 1.5974e-05 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecNormalize          38 1.0 1.9025e-02 1.4 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  3257
> MatMult              129 1.0 1.9964e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3742
> MatMultAdd            10 1.0 4.2991e+011654.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 49  0  1  4  0  49  0  1  4  0     5
> MatMultTranspose      10 1.0 6.5614e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3472
> MatSolve              10 0.0 1.0351e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2112
> MatSOR               111 1.0 3.3396e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  1933
> MatLUFactorSym         1 1.0 2.3057e+00193417.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> MatLUFactorNum         1 1.0 3.9508e+018285439.5 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2879
> MatScale               3 1.0 1.2049e-02 1.2 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  2026
> MatResidual           10 1.0 1.6667e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3507
> MatAssemblyBegin      25 1.0 2.2748e-01 3.0 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
> MatAssemblyEnd        25 1.0 2.6301e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
> MatGetRow         593372 1.0 5.4903e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetRowIJ            1 0.0 1.3105e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCreateSubMat        2 1.0 6.8462e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
> MatGetOrdering         1 0.0 5.0859e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCoarsen             1 1.0 1.8659e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> MatZeroEntries         9 1.0 1.5140e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatView                5 1.7 1.4541e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
> MatAXPY                1 1.0 5.6004e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMatMult             1 1.0 3.1570e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1101
> MatMatMultSym          1 1.0 2.4168e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
> MatMatMultNum          1 1.0 7.3501e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  4729
> MatPtAP                1 1.0 1.3500e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  1706
> MatPtAPSymbolic        1 1.0 9.7516e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
> MatPtAPNumeric         1 1.0 3.7405e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6157
> MatGetLocalMat         3 1.0 5.3542e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetBrAoCol          3 1.0 2.5439e-03 1.1 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
> SNESSolve              1 1.0 5.9280e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2256
> SNESFunctionEval       2 1.0 4.4243e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
> SNESJacobianEval       1 1.0 3.1546e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  5  0  3  4  4   5  0  3  4  4     0
> SNESLineSearch         1 1.0 2.2030e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  3  0 14  4 16   3  0 14  4 17     2
> SFSetGraph             1 1.0 2.1458e-06 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFSetUp                1 1.0 1.5092e-04 2.4 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFBcastBegin           4 1.0 4.3154e-05 1.0 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> SFBcastEnd             4 1.0 3.8099e-0432.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPGMRESOrthog        29 1.0 1.0221e-01 1.2 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  3298
> KSPSetUp               6 1.0 9.9370e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 5.1697e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 78100 65 76 52  78100 65 76 53  2587
> PCGAMGGraph_AGG        1 1.0 4.4754e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
> PCGAMGCoarse_AGG       1 1.0 2.5029e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> PCGAMGProl_AGG         1 1.0 1.0502e-01 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
> PCGAMGPOpt_AGG         1 1.0 6.2535e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  1767
> GAMG: createProl       1 1.0 1.1819e+00 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30   936
>   Graph                2 1.0 4.4748e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>   MIS/Agg              1 1.0 1.8790e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>   SA: col data         1 1.0 5.0070e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>   SA: frmProl0         1 1.0 5.1879e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>   SA: smooth           1 1.0 3.9527e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  1  0  2  1  3   1  0  2  1  3   937
> GAMG: partLevel        1 1.0 2.1293e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1082
>   repartition          1 1.0 3.4499e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Invert-Sort          1 1.0 1.0570e-0221.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Move A               1 1.0 4.4678e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>   Move P               1 1.0 2.8244e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
> PCSetUp                2 1.0 4.5188e+0113.6 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 21 88 25 64 44  21 88 25 64 45  2593
> PCSetUpOnBlocks       10 1.0 4.1865e+01570107.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2717
> PCApply               10 1.0 4.8200e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 73 97 37 12  4  73 97 37 12  4  2690
> ------------------------------------------------------------------------------------------------------------------------
> 
> 
> 2) 
> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4
> 
> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>  0:      Main Stage: 6.5760e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
> 
> ------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
> 
> --- Event Stage 0: Main Stage
> 
> BuildTwoSided          1 1.0 1.9002e-04 8.7 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> BuildTwoSidedF        69 1.0 2.6071e-01 4.8 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
> VecMDot               34 1.0 4.7955e-02 1.3 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  3854
> VecNorm               41 1.0 7.8864e-03 1.4 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  5652
> VecScale              38 1.0 4.7059e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4390
> VecCopy               44 1.0 1.0849e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet               129 1.0 1.9698e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecAXPY                3 1.0 1.0378e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3143
> VecAYPX              120 1.0 4.6406e-02 1.1 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1640
> VecAXPBYCZ            80 1.0 4.0414e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5380
> VecWAXPY               1 1.0 7.6008e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   715
> VecMAXPY              37 1.0 4.8892e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4425
> VecAssemblyBegin      60 1.0 5.2444e-02 2.0 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> VecAssemblyEnd        60 1.0 1.1539e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecPointwiseMult      11 1.0 6.2540e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   956
> VecScatterBegin      226 1.0 1.7562e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
> VecScatterEnd        226 1.0 4.3872e+01319.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 50  0  0  0  0  50  0  0  0  0     0
> VecSetRandom           1 1.0 1.5719e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecReduceArith         3 1.0 5.7292e-04 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5693
> VecReduceComm          1 1.0 9.0122e-0515.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecNormalize          38 1.0 1.2057e-02 1.2 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  5140
> MatMult              129 1.0 1.9545e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3822
> MatMultAdd            10 1.0 4.3806e+011689.9 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 50  0  1  4  0  50  0  1  4  0     5
> MatMultTranspose      10 1.0 6.7213e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3390
> MatSolve              10 0.0 1.0379e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2106
> MatSOR               111 1.0 3.1935e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  2022
> MatLUFactorSym         1 1.0 2.3361e+00288179.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> MatLUFactorNum         1 1.0 4.0301e+0110564673.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2823
> MatScale               3 1.0 1.4505e-02 1.0 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1683
> MatResidual           10 1.0 1.6052e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3641
> MatAssemblyBegin      25 1.0 2.4469e-01 7.5 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
> MatAssemblyEnd        25 1.0 2.2773e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
> MatGetRow         593372 1.0 4.2183e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetRowIJ            1 0.0 9.4678e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCreateSubMat        2 1.0 6.3013e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
> MatGetOrdering         1 0.0 4.1698e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCoarsen             1 1.0 1.4472e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> MatZeroEntries         9 1.0 1.5332e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatView                5 1.7 1.6737e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
> MatAXPY                1 1.0 3.6167e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMatMult             1 1.0 2.6676e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1303
> MatMatMultSym          1 1.0 2.0918e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
> MatMatMultNum          1 1.0 5.7073e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  6090
> MatPtAP                1 1.0 1.0664e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  2160
> MatPtAPSymbolic        1 1.0 7.2993e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
> MatPtAPNumeric         1 1.0 3.3595e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6855
> MatGetLocalMat         3 1.0 5.2164e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetBrAoCol          3 1.0 2.5141e-03 1.2 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
> SNESSolve              1 1.0 5.8984e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2267
> SNESFunctionEval       2 1.0 4.4220e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
> SNESJacobianEval       1 1.0 2.7677e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  4  0  3  4  4   4  0  3  4  4     0
> SNESLineSearch         1 1.0 2.3403e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  4  0 14  4 16   4  0 14  4 17     2
> SFSetGraph             1 1.0 1.9073e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFSetUp                1 1.0 2.3508e-04 3.9 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFBcastBegin           4 1.0 3.2902e-05 1.1 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> SFBcastEnd             4 1.0 1.2112e-0416.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPGMRESOrthog        29 1.0 8.2165e-02 1.1 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  4102
> KSPSetUp               6 1.0 8.9242e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 5.1788e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 79100 65 76 52  79100 65 76 53  2582
> PCGAMGGraph_AGG        1 1.0 3.8242e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
> PCGAMGCoarse_AGG       1 1.0 2.0711e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> PCGAMGProl_AGG         1 1.0 7.1434e-02 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
> PCGAMGPOpt_AGG         1 1.0 5.3496e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  2065
> GAMG: createProl       1 1.0 9.9368e-01 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30  1114
>   Graph                2 1.0 3.8235e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>   MIS/Agg              1 1.0 1.4579e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>   SA: col data         1 1.0 3.0082e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>   SA: frmProl0         1 1.0 3.8682e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>   SA: smooth           1 1.0 3.2894e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  0  0  2  1  3   0  0  2  1  3  1126
> GAMG: partLevel        1 1.0 1.7867e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1289
>   repartition          1 1.0 2.9397e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Invert-Sort          1 1.0 7.5300e-0316.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Move A               1 1.0 4.5402e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>   Move P               1 1.0 2.2056e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
> PCSetUp                2 1.0 4.5469e+0116.3 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 20 88 25 64 44  20 88 25 64 45  2577
> PCSetUpOnBlocks       10 1.0 4.2679e+01701992.3 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2665
> PCApply               10 1.0 4.8823e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 74 97 37 12  4  74 97 37 12  4  2656
> ------------------------------------------------------------------------------------------------------------------------
> 
> 
> 3) 
> -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
> 
> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>  0:      Main Stage: 2.9122e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
> 
> ------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
> 
> --- Event Stage 0: Main Stage
> 
> BuildTwoSided          1 1.0 5.1999e-0423.5 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> BuildTwoSidedF        74 1.0 1.6531e-01 2.2 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  0  0  2  5  0   0  0  2  5  0     0
> VecMDot               37 1.0 7.6608e-02 1.3 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  2881
> VecNorm               44 1.0 1.9918e-02 2.0 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  2402
> VecScale              41 1.0 5.9566e-03 1.1 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3742
> VecCopy               53 1.0 1.3126e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet               151 1.0 2.0747e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecAXPY                3 1.0 1.2219e-03 1.6 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2669
> VecAYPX              156 1.0 6.1843e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1600
> VecAXPBYCZ           104 1.0 5.3596e-02 1.0 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5274
> VecWAXPY               1 1.0 6.4707e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   840
> VecMAXPY              40 1.0 6.1941e-02 1.1 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4125
> VecAssemblyBegin      64 1.0 7.2242e-02 1.3 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> VecAssemblyEnd        64 1.0 1.4734e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecPointwiseMult      11 1.0 6.0835e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   983
> VecScatterBegin      270 1.0 1.7434e-02 1.1 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
> VecScatterEnd        270 1.0 5.8946e+0035.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
> VecSetRandom           1 1.0 2.9080e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecReduceArith         3 1.0 6.4564e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5052
> VecReduceComm          1 1.0 1.6594e-0427.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecNormalize          41 1.0 2.5299e-02 1.6 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  2643
> MatMult              162 1.0 2.5775e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3640
> MatMultAdd            13 1.0 5.7682e+00227.3 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    37
> MatMultTranspose      13 1.0 6.4702e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3268
> MatSolve              13 0.0 3.4193e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1850
> MatSOR               141 1.0 4.2887e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1912
> MatLUFactorSym         1 1.0 5.2129e-0139753.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatLUFactorNum         1 1.0 4.8021e+00805666.4 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2767
> MatScale               3 1.0 1.0527e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1699
> MatResidual           13 1.0 2.1939e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3463
> MatAssemblyBegin      33 1.0 1.1140e-01 5.0 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
> MatAssemblyEnd        33 1.0 1.9615e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
> MatGetRow         593372 1.0 4.5789e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetRowIJ            1 0.0 4.0052e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCreateSubMat        2 1.0 2.7033e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
> MatGetOrdering         1 0.0 1.7007e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCoarsen             1 1.0 2.4688e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> MatZeroEntries        10 1.0 1.4357e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatView                5 1.7 1.7419e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
> MatAXPY                1 1.0 3.4560e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMatMult             1 1.0 2.6894e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1292
> MatMatMultSym          1 1.0 2.1331e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
> MatMatMultNum          1 1.0 5.5709e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  6239
> MatPtAP                1 1.0 6.6371e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  2039
> MatPtAPSymbolic        1 1.0 4.6951e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
> MatPtAPNumeric         1 1.0 1.9424e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6967
> MatTrnMatMult          1 1.0 5.6805e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   292
> MatTrnMatMultSym       1 1.0 3.4907e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
> MatTrnMatMultNum       1 1.0 2.2120e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   751
> MatGetLocalMat         4 1.0 3.8992e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetBrAoCol          3 1.0 2.6469e-03 1.2 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
> SNESSolve              1 1.0 2.2497e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 77100 97 79 89  77100 97 79 90  1536
> SNESFunctionEval       2 1.0 4.5633e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
> SNESJacobianEval       1 1.0 3.0567e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
> SNESLineSearch         1 1.0 2.3701e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
> SFSetGraph             1 1.0 3.0994e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFSetUp                1 1.0 5.7101e-04 6.6 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFBcastBegin           4 1.0 5.6982e-05 1.2 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> SFBcastEnd             4 1.0 1.6999e-04 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPGMRESOrthog        32 1.0 1.2112e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3375
> KSPSetUp               6 1.0 9.6612e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 1.4872e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2323
> PCGAMGGraph_AGG        1 1.0 4.4312e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
> PCGAMGCoarse_AGG       1 1.0 6.4611e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   257
> PCGAMGProl_AGG         1 1.0 9.3410e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
> PCGAMGPOpt_AGG         1 1.0 5.4947e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  1999
> GAMG: createProl       1 1.0 1.1555e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32   966
>   Graph                2 1.0 4.4305e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
>   MIS/Agg              1 1.0 2.4810e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>   SA: col data         1 1.0 4.5491e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>   SA: frmProl0         1 1.0 4.5125e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>   SA: smooth           1 1.0 3.2692e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1113
> GAMG: partLevel        1 1.0 9.6929e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1396
>   repartition          1 1.0 1.0109e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Invert-Sort          1 1.0 1.2600e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Move A               1 1.0 1.2418e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>   Move P               1 1.0 1.6575e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
> PCSetUp                2 1.0 7.4755e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2108
> PCSetUpOnBlocks       13 1.0 5.3405e+0062049.0 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2488
> PCApply               13 1.0 1.2494e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2496
> ------------------------------------------------------------------------------------------------------------------------
> 
> 
> 
> 4) 
> -pc_mg_levels 2 -mg_levels_ksp_max_it 4
> 
> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>  0:      Main Stage: 2.8086e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
> 
> ------------------------------------------------------------------------------------------------------------------------
> ------------------------------------------------------------------------------------------------------------------------
> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
> ------------------------------------------------------------------------------------------------------------------------
> 
> --- Event Stage 0: Main Stage
> 
> BuildTwoSided          1 1.0 9.6083e-05 5.3 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> BuildTwoSidedF        74 1.0 2.0508e-01 2.9 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  1  0  2  5  0   1  0  2  5  0     0
> VecMDot               37 1.0 6.2755e-02 1.2 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  3517
> VecNorm               44 1.0 1.2301e-02 1.5 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  3889
> VecScale              41 1.0 5.2664e-03 1.0 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4232
> VecCopy               53 1.0 1.3689e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecSet               151 1.0 1.9269e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecAXPY                3 1.0 1.1799e-03 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2764
> VecAYPX              156 1.0 6.0851e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1626
> VecAXPBYCZ           104 1.0 5.3286e-02 1.1 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5305
> VecWAXPY               1 1.0 7.0786e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   768
> VecMAXPY              40 1.0 6.2898e-02 1.0 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4062
> VecAssemblyBegin      64 1.0 1.0848e-01 2.7 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> VecAssemblyEnd        64 1.0 1.1230e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecPointwiseMult      11 1.0 6.5417e-03 1.1 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   914
> VecScatterBegin      270 1.0 1.6606e-02 1.0 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
> VecScatterEnd        270 1.0 5.5767e+0044.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
> VecSetRandom           1 1.0 1.5750e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecReduceArith         3 1.0 6.3324e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5151
> VecReduceComm          1 1.0 1.5712e-0431.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
> VecNormalize          41 1.0 1.7057e-02 1.3 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3920
> MatMult              162 1.0 2.5049e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3745
> MatMultAdd            13 1.0 5.5041e+00222.4 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    38
> MatMultTranspose      13 1.0 6.2563e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3379
> MatSolve              13 0.0 3.4633e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1827
> MatSOR               141 1.0 4.1983e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1953
> MatLUFactorSym         1 1.0 4.5545e-0135375.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatLUFactorNum         1 1.0 4.6021e+00772110.8 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2887
> MatScale               3 1.0 1.1058e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1618
> MatResidual           13 1.0 2.1592e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3519
> MatAssemblyBegin      33 1.0 1.1083e-01 3.3 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
> MatAssemblyEnd        33 1.0 1.7722e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
> MatGetRow         593372 1.0 4.5285e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetRowIJ            1 0.0 3.5741e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCreateSubMat        2 1.0 2.5588e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
> MatGetOrdering         1 0.0 1.6127e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatCoarsen             1 1.0 1.4122e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
> MatZeroEntries        10 1.0 1.5053e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
> MatView                5 1.7 1.6868e-03 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
> MatAXPY                1 1.0 4.3107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatMatMult             1 1.0 2.5280e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1375
> MatMatMultSym          1 1.0 1.9445e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
> MatMatMultNum          1 1.0 5.8333e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  5959
> MatPtAP                1 1.0 6.7796e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  1996
> MatPtAPSymbolic        1 1.0 4.8317e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
> MatPtAPNumeric         1 1.0 1.9495e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6941
> MatTrnMatMult          1 1.0 4.1985e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   396
> MatTrnMatMultSym       1 1.0 2.3958e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
> MatTrnMatMultNum       1 1.0 1.8097e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   918
> MatGetLocalMat         4 1.0 4.4538e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> MatGetBrAoCol          3 1.0 3.8249e-03 1.6 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
> SNESSolve              1 1.0 2.1417e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 76100 97 79 89  76100 97 79 90  1614
> SNESFunctionEval       2 1.0 4.4048e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
> SNESJacobianEval       1 1.0 2.6810e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
> SNESLineSearch         1 1.0 2.2640e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
> SFSetGraph             1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFSetUp                1 1.0 1.4687e-04 2.3 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> SFBcastBegin           4 1.0 3.1948e-05 1.1 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
> SFBcastEnd             4 1.0 4.2200e-05 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPGMRESOrthog        32 1.0 1.0959e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3730
> KSPSetUp               6 1.0 9.1093e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
> KSPSolve               1 1.0 1.4326e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2412
> PCGAMGGraph_AGG        1 1.0 3.8501e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
> PCGAMGCoarse_AGG       1 1.0 4.7253e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   352
> PCGAMGProl_AGG         1 1.0 6.9987e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
> PCGAMGPOpt_AGG         1 1.0 5.4272e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  2024
> GAMG: createProl       1 1.0 1.0504e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32  1063
>   Graph                2 1.0 3.8496e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
>   MIS/Agg              1 1.0 1.4219e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>   SA: col data         1 1.0 3.1489e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>   SA: frmProl0         1 1.0 3.5651e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>   SA: smooth           1 1.0 3.2070e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1135
> GAMG: partLevel        1 1.0 9.6892e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1397
>   repartition          1 1.0 1.1492e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Invert-Sort          1 1.0 1.3120e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>   Move A               1 1.0 1.2847e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>   Move P               1 1.0 1.4728e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
> PCSetUp                2 1.0 7.1033e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2218
> PCSetUpOnBlocks       13 1.0 5.0738e+0063336.2 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2618
> PCApply               13 1.0 1.2059e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2586
> ------------------------------------------------------------------------------------------------------------------------
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/9e3ed536/attachment-0001.html>

From jed at jedbrown.org  Mon Oct 29 12:21:17 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 29 Oct 2018 11:21:17 -0600
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
Message-ID: <875zxkfz3m.fsf@jedbrown.org>

https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html

Squaring the graph speeds up coarsening at the expense of convergence
rate.

Manav Bhatia <bhatiamanav at gmail.com> writes:

> Hi, 
>
>   I am trying to understand the meaning of ?pc_gamg_square_graph?. I did not find a specific reference to ?square? in the tutorial here: http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf <http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf> . 
>
>   What would be a good place to look for some background on this? 
>
> Regards,
> Manav
>
>> On Oct 29, 2018, at 11:25 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Some more results on iterations and CPU time: 
>> 
>> 
>> 1) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 9 KSP iterations, 66 secs
>> 2) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4: 9 KSP iterations, 66 secs
>> 3) -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 12 KSP iterations, 29 secs
>> 4) -pc_mg_levels 2 -mg_levels_ksp_max_it 4: 12 KSP iterations, 28 secs

To reduce costs further, you can try -mg_levels_ksp_type richardson
and/or reducing the number of iterations, as in -mg_levels_ksp_max_it 2.

>> So, it appears that ?-pc_gamg_square_graph 0? leads to a significant increase in CPU time. 
>> 
>> What would be a more reasonable value for ?-pc_gamg_square_graph? ? 

It is 1 by default.  Not squaring the graph leads to a first coarsening
that is still pretty large (and often sees some stencil growth) so the
grid complexity can be high.

>> Regards,
>> Manav
>> 
>> 
>> 
>> 1) 
>> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
>> 
>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>  0:      Main Stage: 6.5896e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
>> 
>> ------------------------------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------------------------------
>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> --- Event Stage 0: Main Stage
>> 
>> BuildTwoSided          1 1.0 1.0610e-04 4.6 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> BuildTwoSidedF        69 1.0 3.3554e-01 2.0 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
>> VecMDot               34 1.0 6.3575e-02 1.4 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  2907
>> VecNorm               41 1.0 1.3934e-02 1.6 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3199
>> VecScale              38 1.0 5.5904e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3695
>> VecCopy               44 1.0 1.1493e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecSet               129 1.0 1.9488e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecAXPY                3 1.0 1.0059e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3243
>> VecAYPX              120 1.0 4.7344e-02 1.2 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1607
>> VecAXPBYCZ            80 1.0 3.9797e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5464
>> VecWAXPY               1 1.0 6.1512e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   884
>> VecMAXPY              37 1.0 5.3887e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4015
>> VecAssemblyBegin      60 1.0 1.1819e-01 9.8 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> VecAssemblyEnd        60 1.0 1.5020e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecPointwiseMult      11 1.0 5.6922e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1050
>> VecScatterBegin      226 1.0 1.8359e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
>> VecScatterEnd        226 1.0 4.3040e+01369.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 49  0  0  0  0  49  0  0  0  0     0
>> VecSetRandom           1 1.0 1.7331e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecReduceArith         3 1.0 4.8590e-04 1.0 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6713
>> VecReduceComm          1 1.0 1.5974e-05 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecNormalize          38 1.0 1.9025e-02 1.4 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  3257
>> MatMult              129 1.0 1.9964e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3742
>> MatMultAdd            10 1.0 4.2991e+011654.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 49  0  1  4  0  49  0  1  4  0     5
>> MatMultTranspose      10 1.0 6.5614e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3472
>> MatSolve              10 0.0 1.0351e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2112
>> MatSOR               111 1.0 3.3396e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  1933
>> MatLUFactorSym         1 1.0 2.3057e+00193417.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>> MatLUFactorNum         1 1.0 3.9508e+018285439.5 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2879
>> MatScale               3 1.0 1.2049e-02 1.2 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  2026
>> MatResidual           10 1.0 1.6667e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3507
>> MatAssemblyBegin      25 1.0 2.2748e-01 3.0 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
>> MatAssemblyEnd        25 1.0 2.6301e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
>> MatGetRow         593372 1.0 5.4903e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetRowIJ            1 0.0 1.3105e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCreateSubMat        2 1.0 6.8462e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
>> MatGetOrdering         1 0.0 5.0859e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCoarsen             1 1.0 1.8659e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> MatZeroEntries         9 1.0 1.5140e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatView                5 1.7 1.4541e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>> MatAXPY                1 1.0 5.6004e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatMatMult             1 1.0 3.1570e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1101
>> MatMatMultSym          1 1.0 2.4168e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
>> MatMatMultNum          1 1.0 7.3501e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  4729
>> MatPtAP                1 1.0 1.3500e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  1706
>> MatPtAPSymbolic        1 1.0 9.7516e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
>> MatPtAPNumeric         1 1.0 3.7405e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6157
>> MatGetLocalMat         3 1.0 5.3542e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetBrAoCol          3 1.0 2.5439e-03 1.1 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
>> SNESSolve              1 1.0 5.9280e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2256
>> SNESFunctionEval       2 1.0 4.4243e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
>> SNESJacobianEval       1 1.0 3.1546e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  5  0  3  4  4   5  0  3  4  4     0
>> SNESLineSearch         1 1.0 2.2030e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  3  0 14  4 16   3  0 14  4 17     2
>> SFSetGraph             1 1.0 2.1458e-06 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFSetUp                1 1.0 1.5092e-04 2.4 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFBcastBegin           4 1.0 4.3154e-05 1.0 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> SFBcastEnd             4 1.0 3.8099e-0432.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPGMRESOrthog        29 1.0 1.0221e-01 1.2 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  3298
>> KSPSetUp               6 1.0 9.9370e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPSolve               1 1.0 5.1697e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 78100 65 76 52  78100 65 76 53  2587
>> PCGAMGGraph_AGG        1 1.0 4.4754e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>> PCGAMGCoarse_AGG       1 1.0 2.5029e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> PCGAMGProl_AGG         1 1.0 1.0502e-01 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
>> PCGAMGPOpt_AGG         1 1.0 6.2535e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  1767
>> GAMG: createProl       1 1.0 1.1819e+00 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30   936
>>   Graph                2 1.0 4.4748e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>   MIS/Agg              1 1.0 1.8790e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>   SA: col data         1 1.0 5.0070e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>>   SA: frmProl0         1 1.0 5.1879e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>>   SA: smooth           1 1.0 3.9527e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  1  0  2  1  3   1  0  2  1  3   937
>> GAMG: partLevel        1 1.0 2.1293e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1082
>>   repartition          1 1.0 3.4499e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Invert-Sort          1 1.0 1.0570e-0221.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Move A               1 1.0 4.4678e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>>   Move P               1 1.0 2.8244e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
>> PCSetUp                2 1.0 4.5188e+0113.6 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 21 88 25 64 44  21 88 25 64 45  2593
>> PCSetUpOnBlocks       10 1.0 4.1865e+01570107.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2717
>> PCApply               10 1.0 4.8200e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 73 97 37 12  4  73 97 37 12  4  2690
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> 
>> 2) 
>> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4
>> 
>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>  0:      Main Stage: 6.5760e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
>> 
>> ------------------------------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------------------------------
>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> --- Event Stage 0: Main Stage
>> 
>> BuildTwoSided          1 1.0 1.9002e-04 8.7 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> BuildTwoSidedF        69 1.0 2.6071e-01 4.8 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
>> VecMDot               34 1.0 4.7955e-02 1.3 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  3854
>> VecNorm               41 1.0 7.8864e-03 1.4 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  5652
>> VecScale              38 1.0 4.7059e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4390
>> VecCopy               44 1.0 1.0849e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecSet               129 1.0 1.9698e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecAXPY                3 1.0 1.0378e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3143
>> VecAYPX              120 1.0 4.6406e-02 1.1 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1640
>> VecAXPBYCZ            80 1.0 4.0414e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5380
>> VecWAXPY               1 1.0 7.6008e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   715
>> VecMAXPY              37 1.0 4.8892e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4425
>> VecAssemblyBegin      60 1.0 5.2444e-02 2.0 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> VecAssemblyEnd        60 1.0 1.1539e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecPointwiseMult      11 1.0 6.2540e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   956
>> VecScatterBegin      226 1.0 1.7562e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
>> VecScatterEnd        226 1.0 4.3872e+01319.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 50  0  0  0  0  50  0  0  0  0     0
>> VecSetRandom           1 1.0 1.5719e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecReduceArith         3 1.0 5.7292e-04 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5693
>> VecReduceComm          1 1.0 9.0122e-0515.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecNormalize          38 1.0 1.2057e-02 1.2 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  5140
>> MatMult              129 1.0 1.9545e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3822
>> MatMultAdd            10 1.0 4.3806e+011689.9 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 50  0  1  4  0  50  0  1  4  0     5
>> MatMultTranspose      10 1.0 6.7213e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3390
>> MatSolve              10 0.0 1.0379e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2106
>> MatSOR               111 1.0 3.1935e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  2022
>> MatLUFactorSym         1 1.0 2.3361e+00288179.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>> MatLUFactorNum         1 1.0 4.0301e+0110564673.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2823
>> MatScale               3 1.0 1.4505e-02 1.0 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1683
>> MatResidual           10 1.0 1.6052e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3641
>> MatAssemblyBegin      25 1.0 2.4469e-01 7.5 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
>> MatAssemblyEnd        25 1.0 2.2773e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
>> MatGetRow         593372 1.0 4.2183e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetRowIJ            1 0.0 9.4678e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCreateSubMat        2 1.0 6.3013e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
>> MatGetOrdering         1 0.0 4.1698e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCoarsen             1 1.0 1.4472e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> MatZeroEntries         9 1.0 1.5332e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatView                5 1.7 1.6737e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>> MatAXPY                1 1.0 3.6167e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatMatMult             1 1.0 2.6676e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1303
>> MatMatMultSym          1 1.0 2.0918e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
>> MatMatMultNum          1 1.0 5.7073e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  6090
>> MatPtAP                1 1.0 1.0664e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  2160
>> MatPtAPSymbolic        1 1.0 7.2993e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
>> MatPtAPNumeric         1 1.0 3.3595e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6855
>> MatGetLocalMat         3 1.0 5.2164e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetBrAoCol          3 1.0 2.5141e-03 1.2 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
>> SNESSolve              1 1.0 5.8984e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2267
>> SNESFunctionEval       2 1.0 4.4220e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
>> SNESJacobianEval       1 1.0 2.7677e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  4  0  3  4  4   4  0  3  4  4     0
>> SNESLineSearch         1 1.0 2.3403e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  4  0 14  4 16   4  0 14  4 17     2
>> SFSetGraph             1 1.0 1.9073e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFSetUp                1 1.0 2.3508e-04 3.9 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFBcastBegin           4 1.0 3.2902e-05 1.1 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> SFBcastEnd             4 1.0 1.2112e-0416.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPGMRESOrthog        29 1.0 8.2165e-02 1.1 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  4102
>> KSPSetUp               6 1.0 8.9242e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPSolve               1 1.0 5.1788e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 79100 65 76 52  79100 65 76 53  2582
>> PCGAMGGraph_AGG        1 1.0 3.8242e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>> PCGAMGCoarse_AGG       1 1.0 2.0711e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> PCGAMGProl_AGG         1 1.0 7.1434e-02 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
>> PCGAMGPOpt_AGG         1 1.0 5.3496e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  2065
>> GAMG: createProl       1 1.0 9.9368e-01 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30  1114
>>   Graph                2 1.0 3.8235e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>   MIS/Agg              1 1.0 1.4579e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>   SA: col data         1 1.0 3.0082e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>>   SA: frmProl0         1 1.0 3.8682e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>>   SA: smooth           1 1.0 3.2894e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  0  0  2  1  3   0  0  2  1  3  1126
>> GAMG: partLevel        1 1.0 1.7867e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1289
>>   repartition          1 1.0 2.9397e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Invert-Sort          1 1.0 7.5300e-0316.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Move A               1 1.0 4.5402e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>>   Move P               1 1.0 2.2056e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
>> PCSetUp                2 1.0 4.5469e+0116.3 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 20 88 25 64 44  20 88 25 64 45  2577
>> PCSetUpOnBlocks       10 1.0 4.2679e+01701992.3 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2665
>> PCApply               10 1.0 4.8823e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 74 97 37 12  4  74 97 37 12  4  2656
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> 
>> 3) 
>> -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
>> 
>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>  0:      Main Stage: 2.9122e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
>> 
>> ------------------------------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------------------------------
>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> --- Event Stage 0: Main Stage
>> 
>> BuildTwoSided          1 1.0 5.1999e-0423.5 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> BuildTwoSidedF        74 1.0 1.6531e-01 2.2 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  0  0  2  5  0   0  0  2  5  0     0
>> VecMDot               37 1.0 7.6608e-02 1.3 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  2881
>> VecNorm               44 1.0 1.9918e-02 2.0 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  2402
>> VecScale              41 1.0 5.9566e-03 1.1 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3742
>> VecCopy               53 1.0 1.3126e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecSet               151 1.0 2.0747e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecAXPY                3 1.0 1.2219e-03 1.6 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2669
>> VecAYPX              156 1.0 6.1843e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1600
>> VecAXPBYCZ           104 1.0 5.3596e-02 1.0 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5274
>> VecWAXPY               1 1.0 6.4707e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   840
>> VecMAXPY              40 1.0 6.1941e-02 1.1 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4125
>> VecAssemblyBegin      64 1.0 7.2242e-02 1.3 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> VecAssemblyEnd        64 1.0 1.4734e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecPointwiseMult      11 1.0 6.0835e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   983
>> VecScatterBegin      270 1.0 1.7434e-02 1.1 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
>> VecScatterEnd        270 1.0 5.8946e+0035.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
>> VecSetRandom           1 1.0 2.9080e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecReduceArith         3 1.0 6.4564e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5052
>> VecReduceComm          1 1.0 1.6594e-0427.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecNormalize          41 1.0 2.5299e-02 1.6 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  2643
>> MatMult              162 1.0 2.5775e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3640
>> MatMultAdd            13 1.0 5.7682e+00227.3 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    37
>> MatMultTranspose      13 1.0 6.4702e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3268
>> MatSolve              13 0.0 3.4193e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1850
>> MatSOR               141 1.0 4.2887e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1912
>> MatLUFactorSym         1 1.0 5.2129e-0139753.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatLUFactorNum         1 1.0 4.8021e+00805666.4 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2767
>> MatScale               3 1.0 1.0527e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1699
>> MatResidual           13 1.0 2.1939e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3463
>> MatAssemblyBegin      33 1.0 1.1140e-01 5.0 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
>> MatAssemblyEnd        33 1.0 1.9615e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
>> MatGetRow         593372 1.0 4.5789e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetRowIJ            1 0.0 4.0052e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCreateSubMat        2 1.0 2.7033e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
>> MatGetOrdering         1 0.0 1.7007e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCoarsen             1 1.0 2.4688e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> MatZeroEntries        10 1.0 1.4357e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatView                5 1.7 1.7419e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>> MatAXPY                1 1.0 3.4560e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatMatMult             1 1.0 2.6894e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1292
>> MatMatMultSym          1 1.0 2.1331e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
>> MatMatMultNum          1 1.0 5.5709e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  6239
>> MatPtAP                1 1.0 6.6371e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  2039
>> MatPtAPSymbolic        1 1.0 4.6951e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
>> MatPtAPNumeric         1 1.0 1.9424e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6967
>> MatTrnMatMult          1 1.0 5.6805e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   292
>> MatTrnMatMultSym       1 1.0 3.4907e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
>> MatTrnMatMultNum       1 1.0 2.2120e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   751
>> MatGetLocalMat         4 1.0 3.8992e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetBrAoCol          3 1.0 2.6469e-03 1.2 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
>> SNESSolve              1 1.0 2.2497e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 77100 97 79 89  77100 97 79 90  1536
>> SNESFunctionEval       2 1.0 4.5633e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
>> SNESJacobianEval       1 1.0 3.0567e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
>> SNESLineSearch         1 1.0 2.3701e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
>> SFSetGraph             1 1.0 3.0994e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFSetUp                1 1.0 5.7101e-04 6.6 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFBcastBegin           4 1.0 5.6982e-05 1.2 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> SFBcastEnd             4 1.0 1.6999e-04 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPGMRESOrthog        32 1.0 1.2112e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3375
>> KSPSetUp               6 1.0 9.6612e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPSolve               1 1.0 1.4872e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2323
>> PCGAMGGraph_AGG        1 1.0 4.4312e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
>> PCGAMGCoarse_AGG       1 1.0 6.4611e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   257
>> PCGAMGProl_AGG         1 1.0 9.3410e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
>> PCGAMGPOpt_AGG         1 1.0 5.4947e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  1999
>> GAMG: createProl       1 1.0 1.1555e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32   966
>>   Graph                2 1.0 4.4305e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
>>   MIS/Agg              1 1.0 2.4810e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>   SA: col data         1 1.0 4.5491e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>>   SA: frmProl0         1 1.0 4.5125e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>   SA: smooth           1 1.0 3.2692e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1113
>> GAMG: partLevel        1 1.0 9.6929e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1396
>>   repartition          1 1.0 1.0109e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Invert-Sort          1 1.0 1.2600e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Move A               1 1.0 1.2418e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>>   Move P               1 1.0 1.6575e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
>> PCSetUp                2 1.0 7.4755e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2108
>> PCSetUpOnBlocks       13 1.0 5.3405e+0062049.0 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2488
>> PCApply               13 1.0 1.2494e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2496
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> 
>> 
>> 4) 
>> -pc_mg_levels 2 -mg_levels_ksp_max_it 4
>> 
>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>                         Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>  0:      Main Stage: 2.8086e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
>> 
>> ------------------------------------------------------------------------------------------------------------------------
>> ------------------------------------------------------------------------------------------------------------------------
>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>                    Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> --- Event Stage 0: Main Stage
>> 
>> BuildTwoSided          1 1.0 9.6083e-05 5.3 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> BuildTwoSidedF        74 1.0 2.0508e-01 2.9 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  1  0  2  5  0   1  0  2  5  0     0
>> VecMDot               37 1.0 6.2755e-02 1.2 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  3517
>> VecNorm               44 1.0 1.2301e-02 1.5 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  3889
>> VecScale              41 1.0 5.2664e-03 1.0 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4232
>> VecCopy               53 1.0 1.3689e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecSet               151 1.0 1.9269e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecAXPY                3 1.0 1.1799e-03 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2764
>> VecAYPX              156 1.0 6.0851e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1626
>> VecAXPBYCZ           104 1.0 5.3286e-02 1.1 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5305
>> VecWAXPY               1 1.0 7.0786e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   768
>> VecMAXPY              40 1.0 6.2898e-02 1.0 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4062
>> VecAssemblyBegin      64 1.0 1.0848e-01 2.7 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> VecAssemblyEnd        64 1.0 1.1230e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecPointwiseMult      11 1.0 6.5417e-03 1.1 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   914
>> VecScatterBegin      270 1.0 1.6606e-02 1.0 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
>> VecScatterEnd        270 1.0 5.5767e+0044.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
>> VecSetRandom           1 1.0 1.5750e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecReduceArith         3 1.0 6.3324e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5151
>> VecReduceComm          1 1.0 1.5712e-0431.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> VecNormalize          41 1.0 1.7057e-02 1.3 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3920
>> MatMult              162 1.0 2.5049e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3745
>> MatMultAdd            13 1.0 5.5041e+00222.4 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    38
>> MatMultTranspose      13 1.0 6.2563e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3379
>> MatSolve              13 0.0 3.4633e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1827
>> MatSOR               141 1.0 4.1983e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1953
>> MatLUFactorSym         1 1.0 4.5545e-0135375.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatLUFactorNum         1 1.0 4.6021e+00772110.8 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2887
>> MatScale               3 1.0 1.1058e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1618
>> MatResidual           13 1.0 2.1592e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3519
>> MatAssemblyBegin      33 1.0 1.1083e-01 3.3 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
>> MatAssemblyEnd        33 1.0 1.7722e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
>> MatGetRow         593372 1.0 4.5285e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetRowIJ            1 0.0 3.5741e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCreateSubMat        2 1.0 2.5588e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
>> MatGetOrdering         1 0.0 1.6127e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatCoarsen             1 1.0 1.4122e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>> MatZeroEntries        10 1.0 1.5053e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>> MatView                5 1.7 1.6868e-03 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>> MatAXPY                1 1.0 4.3107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatMatMult             1 1.0 2.5280e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1375
>> MatMatMultSym          1 1.0 1.9445e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
>> MatMatMultNum          1 1.0 5.8333e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  5959
>> MatPtAP                1 1.0 6.7796e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  1996
>> MatPtAPSymbolic        1 1.0 4.8317e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
>> MatPtAPNumeric         1 1.0 1.9495e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6941
>> MatTrnMatMult          1 1.0 4.1985e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   396
>> MatTrnMatMultSym       1 1.0 2.3958e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
>> MatTrnMatMultNum       1 1.0 1.8097e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   918
>> MatGetLocalMat         4 1.0 4.4538e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> MatGetBrAoCol          3 1.0 3.8249e-03 1.6 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
>> SNESSolve              1 1.0 2.1417e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 76100 97 79 89  76100 97 79 90  1614
>> SNESFunctionEval       2 1.0 4.4048e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
>> SNESJacobianEval       1 1.0 2.6810e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
>> SNESLineSearch         1 1.0 2.2640e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
>> SFSetGraph             1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFSetUp                1 1.0 1.4687e-04 2.3 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> SFBcastBegin           4 1.0 3.1948e-05 1.1 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>> SFBcastEnd             4 1.0 4.2200e-05 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPGMRESOrthog        32 1.0 1.0959e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3730
>> KSPSetUp               6 1.0 9.1093e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>> KSPSolve               1 1.0 1.4326e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2412
>> PCGAMGGraph_AGG        1 1.0 3.8501e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
>> PCGAMGCoarse_AGG       1 1.0 4.7253e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   352
>> PCGAMGProl_AGG         1 1.0 6.9987e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
>> PCGAMGPOpt_AGG         1 1.0 5.4272e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  2024
>> GAMG: createProl       1 1.0 1.0504e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32  1063
>>   Graph                2 1.0 3.8496e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
>>   MIS/Agg              1 1.0 1.4219e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>   SA: col data         1 1.0 3.1489e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>>   SA: frmProl0         1 1.0 3.5651e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>   SA: smooth           1 1.0 3.2070e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1135
>> GAMG: partLevel        1 1.0 9.6892e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1397
>>   repartition          1 1.0 1.1492e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Invert-Sort          1 1.0 1.3120e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>   Move A               1 1.0 1.2847e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>>   Move P               1 1.0 1.4728e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
>> PCSetUp                2 1.0 7.1033e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2218
>> PCSetUpOnBlocks       13 1.0 5.0738e+0063336.2 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2618
>> PCApply               13 1.0 1.2059e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2586
>> ------------------------------------------------------------------------------------------------------------------------
>> 
>> 

From knepley at gmail.com  Mon Oct 29 12:29:01 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 29 Oct 2018 13:29:01 -0400
Subject: [petsc-users] [petsc-maint] How to impose boundary conditions
 using DMDA
In-Reply-To: <1e4a8501e44d4fdf8727d7ef7a217fff@mek.dtu.dk>
References: <265c311721234abea1563b41642a13de@mek.dtu.dk>
	<CAMYG4G=7mFcwg8uOxir9S9NYoOjmBnFdEoSNA7NRbkty9GzDjg@mail.gmail.com>
	<516B1429-4E9E-4665-9B13-CC6F3CA86111@anl.gov>
	<CAMYG4Gmpb5KdkiB4fAy7UF7yUZQ3eDmirKPCpLTxXkCP7zMqeA@mail.gmail.com>
	<e3accb27d537466a871687e1e1fb5d78@mek.dtu.dk>
	<CAMYG4GmWNhyo3cuU7koHqJcjBcMdniRc9=wFcCXKdKwKPHuViA@mail.gmail.com>
	<2BEEBED7-C1CC-4DE4-BA4F-7CCF4A26C78E@gmail.com>
	<CAMYG4GmRt8x8hVopoeDKMyNaXE1SotFOZ1xwRCHzj+rH+wbt-A@mail.gmail.com>
	<ED2A5E73-4A90-47A5-8820-6511CF5174F7@gmx.li>
	<1e4a8501e44d4fdf8727d7ef7a217fff@mek.dtu.dk>
Message-ID: <CAMYG4G=kMR=knwSXw4p_ukkbeUyYkvBy4iOZ3YpdM=Zc4T+2YQ@mail.gmail.com>

On Mon, Oct 29, 2018 at 9:37 AM Fengwen Wang <fwan at mek.dtu.dk> wrote:

> Yes.  I would like to a rotational BCs(Twist bcs).
>
You can do that with some work. I have implemented a Mobius strip (see
DM_BOUNDARY_TWIST). However,
I have no idea how to put a function space down on a non-orientable
manifold.

  Thanks,

     Matt


> Such a boundary is needed to reduce the computational domain.
>
>
> Thanks lot
>
>
> Best regards
>
> Fengwen
>
>
>
>
> ____________________________________________________
> Senior Researcher
> Department of Mechanical Engineering, DTU
> Nils Koppels All?
> Building 404
> 2800  Kgs. Lyngby
> fwan at mek.dtu.dk
> ------------------------------
> *From:* Lawrence Mitchell <wence at gmx.li>
> *Sent:* Monday, October 29, 2018 1:17:34 PM
> *To:* Matt Knepley
> *Cc:* Fengwen Wang; PETSc; PETSc
> *Subject:* Re: [petsc-users] [petsc-maint] How to impose boundary
> conditions using DMDA
>
>
>
> > On 29 Oct 2018, at 10:56, Matthew Knepley <knepley at gmail.com> wrote:
> >
> > You can certainly map the right edge to the top edge (its topologically
> a square again), but that
> > mapping is not smooth, and I do not know how you would make a global
> basis for the approximation space.
>
> Yeah, I thought this was topologically just a cone, but that would have:
>
> u(x=1) = -v(y=1)
>
> and
>
> v(x=1) = u(y=1) (note matching sign).
>
> So if you wanted the v(x=1) = -u(y=1) there would be a twist somehow, and
> then a discontinuity in the mapping.
>
> Lawrence
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/51561a7f/attachment.html>

From bhatiamanav at gmail.com  Mon Oct 29 12:50:35 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 12:50:35 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <875zxkfz3m.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
Message-ID: <72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>

Thanks, Jed. 

The description says: ? Square the graph, ie. compute A'*A before aggregating it"

What is A here? 

What is the impact of setting this to 0, which led to a very significant increase in the CPU time in my case? 

-Manav

> On Oct 29, 2018, at 12:21 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetSquareGraph.html
> 
> Squaring the graph speeds up coarsening at the expense of convergence
> rate.
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> Hi, 
>> 
>>  I am trying to understand the meaning of ?pc_gamg_square_graph?. I did not find a specific reference to ?square? in the tutorial here: http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf <http://user.it.uu.se/~maya/Projects/3phase/AMG_parallel_Falgout.pdf> . 
>> 
>>  What would be a good place to look for some background on this? 
>> 
>> Regards,
>> Manav
>> 
>>> On Oct 29, 2018, at 11:25 AM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>> 
>>> Some more results on iterations and CPU time: 
>>> 
>>> 
>>> 1) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 9 KSP iterations, 66 secs
>>> 2) -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4: 9 KSP iterations, 66 secs
>>> 3) -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04: 12 KSP iterations, 29 secs
>>> 4) -pc_mg_levels 2 -mg_levels_ksp_max_it 4: 12 KSP iterations, 28 secs
> 
> To reduce costs further, you can try -mg_levels_ksp_type richardson
> and/or reducing the number of iterations, as in -mg_levels_ksp_max_it 2.
> 
>>> So, it appears that ?-pc_gamg_square_graph 0? leads to a significant increase in CPU time. 
>>> 
>>> What would be a more reasonable value for ?-pc_gamg_square_graph? ? 
> 
> It is 1 by default.  Not squaring the graph leads to a first coarsening
> that is still pretty large (and often sees some stencil growth) so the
> grid complexity can be high.
> 
>>> Regards,
>>> Manav
>>> 
>>> 
>>> 
>>> 1) 
>>> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
>>> 
>>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>>                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>> 0:      Main Stage: 6.5896e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
>>> 
>>> ------------------------------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------------------------------
>>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> --- Event Stage 0: Main Stage
>>> 
>>> BuildTwoSided          1 1.0 1.0610e-04 4.6 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> BuildTwoSidedF        69 1.0 3.3554e-01 2.0 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
>>> VecMDot               34 1.0 6.3575e-02 1.4 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  2907
>>> VecNorm               41 1.0 1.3934e-02 1.6 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3199
>>> VecScale              38 1.0 5.5904e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3695
>>> VecCopy               44 1.0 1.1493e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecSet               129 1.0 1.9488e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecAXPY                3 1.0 1.0059e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3243
>>> VecAYPX              120 1.0 4.7344e-02 1.2 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1607
>>> VecAXPBYCZ            80 1.0 3.9797e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5464
>>> VecWAXPY               1 1.0 6.1512e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   884
>>> VecMAXPY              37 1.0 5.3887e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4015
>>> VecAssemblyBegin      60 1.0 1.1819e-01 9.8 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> VecAssemblyEnd        60 1.0 1.5020e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecPointwiseMult      11 1.0 5.6922e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1050
>>> VecScatterBegin      226 1.0 1.8359e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
>>> VecScatterEnd        226 1.0 4.3040e+01369.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 49  0  0  0  0  49  0  0  0  0     0
>>> VecSetRandom           1 1.0 1.7331e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecReduceArith         3 1.0 4.8590e-04 1.0 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  6713
>>> VecReduceComm          1 1.0 1.5974e-05 3.9 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecNormalize          38 1.0 1.9025e-02 1.4 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  3257
>>> MatMult              129 1.0 1.9964e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3742
>>> MatMultAdd            10 1.0 4.2991e+011654.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 49  0  1  4  0  49  0  1  4  0     5
>>> MatMultTranspose      10 1.0 6.5614e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3472
>>> MatSolve              10 0.0 1.0351e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2112
>>> MatSOR               111 1.0 3.3396e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  1933
>>> MatLUFactorSym         1 1.0 2.3057e+00193417.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>>> MatLUFactorNum         1 1.0 3.9508e+018285439.5 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2879
>>> MatScale               3 1.0 1.2049e-02 1.2 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  2026
>>> MatResidual           10 1.0 1.6667e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3507
>>> MatAssemblyBegin      25 1.0 2.2748e-01 3.0 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
>>> MatAssemblyEnd        25 1.0 2.6301e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
>>> MatGetRow         593372 1.0 5.4903e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetRowIJ            1 0.0 1.3105e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCreateSubMat        2 1.0 6.8462e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
>>> MatGetOrdering         1 0.0 5.0859e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCoarsen             1 1.0 1.8659e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> MatZeroEntries         9 1.0 1.5140e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatView                5 1.7 1.4541e-03 3.2 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>> MatAXPY                1 1.0 5.6004e-02 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatMatMult             1 1.0 3.1570e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1101
>>> MatMatMultSym          1 1.0 2.4168e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
>>> MatMatMultNum          1 1.0 7.3501e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  4729
>>> MatPtAP                1 1.0 1.3500e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  1706
>>> MatPtAPSymbolic        1 1.0 9.7516e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
>>> MatPtAPNumeric         1 1.0 3.7405e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6157
>>> MatGetLocalMat         3 1.0 5.3542e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetBrAoCol          3 1.0 2.5439e-03 1.1 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
>>> SNESSolve              1 1.0 5.9280e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2256
>>> SNESFunctionEval       2 1.0 4.4243e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
>>> SNESJacobianEval       1 1.0 3.1546e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  5  0  3  4  4   5  0  3  4  4     0
>>> SNESLineSearch         1 1.0 2.2030e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  3  0 14  4 16   3  0 14  4 17     2
>>> SFSetGraph             1 1.0 2.1458e-06 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFSetUp                1 1.0 1.5092e-04 2.4 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFBcastBegin           4 1.0 4.3154e-05 1.0 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> SFBcastEnd             4 1.0 3.8099e-0432.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPGMRESOrthog        29 1.0 1.0221e-01 1.2 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  3298
>>> KSPSetUp               6 1.0 9.9370e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPSolve               1 1.0 5.1697e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 78100 65 76 52  78100 65 76 53  2587
>>> PCGAMGGraph_AGG        1 1.0 4.4754e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>> PCGAMGCoarse_AGG       1 1.0 2.5029e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> PCGAMGProl_AGG         1 1.0 1.0502e-01 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
>>> PCGAMGPOpt_AGG         1 1.0 6.2535e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  1767
>>> GAMG: createProl       1 1.0 1.1819e+00 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30   936
>>>  Graph                2 1.0 4.4748e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>>  MIS/Agg              1 1.0 1.8790e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>>  SA: col data         1 1.0 5.0070e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>>>  SA: frmProl0         1 1.0 5.1879e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>>>  SA: smooth           1 1.0 3.9527e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  1  0  2  1  3   1  0  2  1  3   937
>>> GAMG: partLevel        1 1.0 2.1293e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1082
>>>  repartition          1 1.0 3.4499e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Invert-Sort          1 1.0 1.0570e-0221.2 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Move A               1 1.0 4.4678e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>>>  Move P               1 1.0 2.8244e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
>>> PCSetUp                2 1.0 4.5188e+0113.6 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 21 88 25 64 44  21 88 25 64 45  2593
>>> PCSetUpOnBlocks       10 1.0 4.1865e+01570107.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2717
>>> PCApply               10 1.0 4.8200e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 73 97 37 12  4  73 97 37 12  4  2690
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> 
>>> 2) 
>>> -pc_mg_levels 2 -pc_gamg_square_graph 0 -mg_levels_ksp_max_it 4
>>> 
>>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>>                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>> 0:      Main Stage: 6.5760e+01 100.0%  1.3377e+11 100.0%  3.138e+03 100.0%  5.425e+04      100.0%  5.120e+02  98.7% 
>>> 
>>> ------------------------------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------------------------------
>>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> --- Event Stage 0: Main Stage
>>> 
>>> BuildTwoSided          1 1.0 1.9002e-04 8.7 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> BuildTwoSidedF        69 1.0 2.6071e-01 4.8 0.00e+00 0.0 5.5e+01 1.1e+05 0.0e+00  0  0  2  4  0   0  0  2  4  0     0
>>> VecMDot               34 1.0 4.7955e-02 1.3 4.66e+07 1.0 0.0e+00 0.0e+00 3.4e+01  0  0  0  0  7   0  0  0  0  7  3854
>>> VecNorm               41 1.0 7.8864e-03 1.4 1.12e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  5652
>>> VecScale              38 1.0 4.7059e-03 1.0 5.20e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4390
>>> VecCopy               44 1.0 1.0849e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecSet               129 1.0 1.9698e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecAXPY                3 1.0 1.0378e-03 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3143
>>> VecAYPX              120 1.0 4.6406e-02 1.1 1.92e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1640
>>> VecAXPBYCZ            80 1.0 4.0414e-02 1.0 5.48e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5380
>>> VecWAXPY               1 1.0 7.6008e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   715
>>> VecMAXPY              37 1.0 4.8892e-02 1.0 5.45e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4425
>>> VecAssemblyBegin      60 1.0 5.2444e-02 2.0 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> VecAssemblyEnd        60 1.0 1.1539e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecPointwiseMult      11 1.0 6.2540e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   956
>>> VecScatterBegin      226 1.0 1.7562e-02 1.1 0.00e+00 0.0 2.1e+03 2.0e+04 0.0e+00  0  0 66 25  0   0  0 66 25  0     0
>>> VecScatterEnd        226 1.0 4.3872e+01319.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 50  0  0  0  0  50  0  0  0  0     0
>>> VecSetRandom           1 1.0 1.5719e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecReduceArith         3 1.0 5.7292e-04 1.2 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5693
>>> VecReduceComm          1 1.0 9.0122e-0515.1 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecNormalize          38 1.0 1.2057e-02 1.2 1.56e+07 1.0 0.0e+00 0.0e+00 3.8e+01  0  0  0  0  7   0  0  0  0  7  5140
>>> MatMult              129 1.0 1.9545e+00 1.0 1.88e+09 1.0 1.3e+03 6.9e+03 0.0e+00  3  6 41  5  0   3  6 41  5  0  3822
>>> MatMultAdd            10 1.0 4.3806e+011689.9 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00 50  0  1  4  0  50  0  1  4  0     5
>>> MatMultTranspose      10 1.0 6.7213e-02 1.1 5.73e+07 1.0 3.0e+01 2.1e+05 0.0e+00  0  0  1  4  0   0  0  1  4  0  3390
>>> MatSolve              10 0.0 1.0379e+00 0.0 2.19e+09 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  2106
>>> MatSOR               111 1.0 3.1935e+00 1.0 1.63e+09 1.0 0.0e+00 0.0e+00 0.0e+00  5  5  0  0  0   5  5  0  0  0  2022
>>> MatLUFactorSym         1 1.0 2.3361e+00288179.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>>> MatLUFactorNum         1 1.0 4.0301e+0110564673.9 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 15 85  0  0  0  15 85  0  0  0  2823
>>> MatScale               3 1.0 1.4505e-02 1.0 6.14e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1683
>>> MatResidual           10 1.0 1.6052e-01 1.1 1.47e+08 1.0 1.0e+02 6.9e+03 0.0e+00  0  0  3  0  0   0  0  3  0  0  3641
>>> MatAssemblyBegin      25 1.0 2.4469e-01 7.5 0.00e+00 0.0 3.0e+01 2.0e+05 0.0e+00  0  0  1  3  0   0  0  1  3  0     0
>>> MatAssemblyEnd        25 1.0 2.2773e-01 1.0 0.00e+00 0.0 1.2e+02 3.4e+03 6.4e+01  0  0  4  0 12   0  0  4  0 12     0
>>> MatGetRow         593372 1.0 4.2183e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetRowIJ            1 0.0 9.4678e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCreateSubMat        2 1.0 6.3013e-01 1.0 0.00e+00 0.0 2.1e+01 4.5e+06 3.2e+01  1  0  1 56  6   1  0  1 56  6     0
>>> MatGetOrdering         1 0.0 4.1698e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCoarsen             1 1.0 1.4472e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> MatZeroEntries         9 1.0 1.5332e-01 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatView                5 1.7 1.6737e-03 3.4 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>> MatAXPY                1 1.0 3.6167e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatMatMult             1 1.0 2.6676e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.3e+01  0  0  2  1  3   0  0  2  1  3  1303
>>> MatMatMultSym          1 1.0 2.0918e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  0  0  2  0  2   0  0  2  0  2     0
>>> MatMatMultNum          1 1.0 5.7073e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  0  0  0  0   0  0  0  0  0  6090
>>> MatPtAP                1 1.0 1.0664e+00 1.0 5.79e+08 1.0 9.5e+01 7.0e+04 1.6e+01  2  2  3  4  3   2  2  3  4  3  2160
>>> MatPtAPSymbolic        1 1.0 7.2993e-01 1.0 0.00e+00 0.0 6.0e+01 5.5e+04 7.0e+00  1  0  2  2  1   1  0  2  2  1     0
>>> MatPtAPNumeric         1 1.0 3.3595e-01 1.0 5.79e+08 1.0 3.5e+01 9.7e+04 8.0e+00  1  2  1  2  2   1  2  1  2  2  6855
>>> MatGetLocalMat         3 1.0 5.2164e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetBrAoCol          3 1.0 2.5141e-03 1.2 0.00e+00 0.0 7.0e+01 5.0e+04 0.0e+00  0  0  2  2  0   0  0  2  2  0     0
>>> SNESSolve              1 1.0 5.8984e+01 1.0 1.20e+1127.3 3.0e+03 4.9e+04 4.6e+02 90100 97 88 89  90100 97 88 90  2267
>>> SNESFunctionEval       2 1.0 4.4220e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02  7  0 29  8 32   7  0 29  8 33     0
>>> SNESJacobianEval       1 1.0 2.7677e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01  4  0  3  4  4   4  0  3  4  4     0
>>> SNESLineSearch         1 1.0 2.3403e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  4  0 14  4 16   4  0 14  4 17     2
>>> SFSetGraph             1 1.0 1.9073e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFSetUp                1 1.0 2.3508e-04 3.9 0.00e+00 0.0 1.5e+01 3.8e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFBcastBegin           4 1.0 3.2902e-05 1.1 0.00e+00 0.0 4.0e+01 5.8e+02 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> SFBcastEnd             4 1.0 1.2112e-0416.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPGMRESOrthog        29 1.0 8.2165e-02 1.1 8.49e+07 1.0 0.0e+00 0.0e+00 2.9e+01  0  0  0  0  6   0  0  0  0  6  4102
>>> KSPSetUp               6 1.0 8.9242e-03 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPSolve               1 1.0 5.1788e+01 1.0 1.20e+1127.3 2.0e+03 6.4e+04 2.7e+02 79100 65 76 52  79100 65 76 53  2582
>>> PCGAMGGraph_AGG        1 1.0 3.8242e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>> PCGAMGCoarse_AGG       1 1.0 2.0711e-03 1.0 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> PCGAMGProl_AGG         1 1.0 7.1434e-02 1.0 0.00e+00 0.0 3.9e+02 1.3e+03 8.6e+01  0  0 12  0 17   0  0 12  0 17     0
>>> PCGAMGPOpt_AGG         1 1.0 5.3496e-01 1.0 2.78e+08 1.0 1.6e+02 1.1e+04 4.2e+01  1  1  5  1  8   1  1  5  1  8  2065
>>> GAMG: createProl       1 1.0 9.9368e-01 1.0 2.79e+08 1.0 6.5e+02 3.6e+03 1.5e+02  2  1 21  1 30   2  1 21  1 30  1114
>>>  Graph                2 1.0 3.8235e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  2  0  5   1  0  2  0  5     4
>>>  MIS/Agg              1 1.0 1.4579e-03 1.1 0.00e+00 0.0 5.5e+01 5.2e+02 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>>  SA: col data         1 1.0 3.0082e-02 1.0 0.00e+00 0.0 3.7e+02 1.2e+03 7.4e+01  0  0 12  0 14   0  0 12  0 14     0
>>>  SA: frmProl0         1 1.0 3.8682e-02 1.0 0.00e+00 0.0 1.8e+01 5.4e+03 8.0e+00  0  0  1  0  2   0  0  1  0  2     0
>>>  SA: smooth           1 1.0 3.2894e-01 1.0 9.33e+07 1.0 6.0e+01 1.8e+04 1.5e+01  0  0  2  1  3   0  0  2  1  3  1126
>>> GAMG: partLevel        1 1.0 1.7867e+00 1.0 5.79e+08 1.0 1.2e+02 8.6e+05 6.7e+01  3  2  4 63 13   3  2  4 63 13  1289
>>>  repartition          1 1.0 2.9397e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Invert-Sort          1 1.0 7.5300e-0316.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Move A               1 1.0 4.5402e-01 1.0 0.00e+00 0.0 1.5e+01 6.3e+06 1.7e+01  1  0  0 56  3   1  0  0 56  3     0
>>>  Move P               1 1.0 2.2056e-01 1.0 0.00e+00 0.0 6.0e+00 5.2e+04 1.7e+01  0  0  0  0  3   0  0  0  0  3     0
>>> PCSetUp                2 1.0 4.5469e+0116.3 1.15e+11135.8 7.8e+02 1.4e+05 2.3e+02 20 88 25 64 44  20 88 25 64 45  2577
>>> PCSetUpOnBlocks       10 1.0 4.2679e+01701992.3 1.14e+11 0.0 0.0e+00 0.0e+00 0.0e+00 16 85  0  0  0  16 85  0  0  0  2665
>>> PCApply               10 1.0 4.8823e+01 1.0 1.19e+1135.1 1.2e+03 1.7e+04 2.1e+01 74 97 37 12  4  74 97 37 12  4  2656
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> 
>>> 3) 
>>> -pc_mg_levels 2 -mg_levels_ksp_max_it 4 -pc_gamg_threshold 0.04
>>> 
>>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>>                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>> 0:      Main Stage: 2.9122e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
>>> 
>>> ------------------------------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------------------------------
>>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> --- Event Stage 0: Main Stage
>>> 
>>> BuildTwoSided          1 1.0 5.1999e-0423.5 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> BuildTwoSidedF        74 1.0 1.6531e-01 2.2 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  0  0  2  5  0   0  0  2  5  0     0
>>> VecMDot               37 1.0 7.6608e-02 1.3 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  2881
>>> VecNorm               44 1.0 1.9918e-02 2.0 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  2402
>>> VecScale              41 1.0 5.9566e-03 1.1 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  3742
>>> VecCopy               53 1.0 1.3126e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecSet               151 1.0 2.0747e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecAXPY                3 1.0 1.2219e-03 1.6 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2669
>>> VecAYPX              156 1.0 6.1843e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1600
>>> VecAXPBYCZ           104 1.0 5.3596e-02 1.0 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5274
>>> VecWAXPY               1 1.0 6.4707e-04 1.3 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   840
>>> VecMAXPY              40 1.0 6.1941e-02 1.1 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4125
>>> VecAssemblyBegin      64 1.0 7.2242e-02 1.3 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> VecAssemblyEnd        64 1.0 1.4734e-04 1.3 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecPointwiseMult      11 1.0 6.0835e-03 1.2 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   983
>>> VecScatterBegin      270 1.0 1.7434e-02 1.1 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
>>> VecScatterEnd        270 1.0 5.8946e+0035.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
>>> VecSetRandom           1 1.0 2.9080e-03 1.8 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecReduceArith         3 1.0 6.4564e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5052
>>> VecReduceComm          1 1.0 1.6594e-0427.8 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecNormalize          41 1.0 2.5299e-02 1.6 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  2643
>>> MatMult              162 1.0 2.5775e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3640
>>> MatMultAdd            13 1.0 5.7682e+00227.3 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    37
>>> MatMultTranspose      13 1.0 6.4702e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3268
>>> MatSolve              13 0.0 3.4193e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1850
>>> MatSOR               141 1.0 4.2887e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1912
>>> MatLUFactorSym         1 1.0 5.2129e-0139753.9 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatLUFactorNum         1 1.0 4.8021e+00805666.4 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2767
>>> MatScale               3 1.0 1.0527e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1699
>>> MatResidual           13 1.0 2.1939e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3463
>>> MatAssemblyBegin      33 1.0 1.1140e-01 5.0 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
>>> MatAssemblyEnd        33 1.0 1.9615e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
>>> MatGetRow         593372 1.0 4.5789e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetRowIJ            1 0.0 4.0052e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCreateSubMat        2 1.0 2.7033e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
>>> MatGetOrdering         1 0.0 1.7007e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCoarsen             1 1.0 2.4688e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> MatZeroEntries        10 1.0 1.4357e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatView                5 1.7 1.7419e-03 3.8 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>> MatAXPY                1 1.0 3.4560e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatMatMult             1 1.0 2.6894e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1292
>>> MatMatMultSym          1 1.0 2.1331e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
>>> MatMatMultNum          1 1.0 5.5709e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  6239
>>> MatPtAP                1 1.0 6.6371e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  2039
>>> MatPtAPSymbolic        1 1.0 4.6951e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
>>> MatPtAPNumeric         1 1.0 1.9424e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6967
>>> MatTrnMatMult          1 1.0 5.6805e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   292
>>> MatTrnMatMultSym       1 1.0 3.4907e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>> MatTrnMatMultNum       1 1.0 2.2120e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   751
>>> MatGetLocalMat         4 1.0 3.8992e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetBrAoCol          3 1.0 2.6469e-03 1.2 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
>>> SNESSolve              1 1.0 2.2497e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 77100 97 79 89  77100 97 79 90  1536
>>> SNESFunctionEval       2 1.0 4.5633e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
>>> SNESJacobianEval       1 1.0 3.0567e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
>>> SNESLineSearch         1 1.0 2.3701e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
>>> SFSetGraph             1 1.0 3.0994e-06 1.6 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFSetUp                1 1.0 5.7101e-04 6.6 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFBcastBegin           4 1.0 5.6982e-05 1.2 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> SFBcastEnd             4 1.0 1.6999e-04 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPGMRESOrthog        32 1.0 1.2112e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3375
>>> KSPSetUp               6 1.0 9.6612e-03 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPSolve               1 1.0 1.4872e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2323
>>> PCGAMGGraph_AGG        1 1.0 4.4312e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
>>> PCGAMGCoarse_AGG       1 1.0 6.4611e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   257
>>> PCGAMGProl_AGG         1 1.0 9.3410e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
>>> PCGAMGPOpt_AGG         1 1.0 5.4947e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  1999
>>> GAMG: createProl       1 1.0 1.1555e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32   966
>>>  Graph                2 1.0 4.4305e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  2  0  1  0  4   2  0  1  0  4     4
>>>  MIS/Agg              1 1.0 2.4810e-03 1.2 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>>  SA: col data         1 1.0 4.5491e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>>>  SA: frmProl0         1 1.0 4.5125e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>>  SA: smooth           1 1.0 3.2692e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1113
>>> GAMG: partLevel        1 1.0 9.6929e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1396
>>>  repartition          1 1.0 1.0109e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Invert-Sort          1 1.0 1.2600e-03 5.7 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Move A               1 1.0 1.2418e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>>>  Move P               1 1.0 1.6575e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
>>> PCSetUp                2 1.0 7.4755e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2108
>>> PCSetUpOnBlocks       13 1.0 5.3405e+0062049.0 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2488
>>> PCApply               13 1.0 1.2494e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2496
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> 
>>> 
>>> 4) 
>>> -pc_mg_levels 2 -mg_levels_ksp_max_it 4
>>> 
>>> Summary of Stages:   ----- Time ------  ----- Flop -----  --- Messages ---  -- Message Lengths --  -- Reductions --
>>>                        Avg     %Total     Avg     %Total   counts   %Total     Avg         %Total   counts   %Total 
>>> 0:      Main Stage: 2.8086e+01 100.0%  3.4599e+10 100.0%  3.618e+03 100.0%  2.554e+04      100.0%  5.350e+02  98.7% 
>>> 
>>> ------------------------------------------------------------------------------------------------------------------------
>>> ------------------------------------------------------------------------------------------------------------------------
>>> Event                Count      Time (sec)     Flop                             --- Global ---  --- Stage ---   Total
>>>                   Max Ratio  Max     Ratio   Max  Ratio  Mess   Avg len Reduct  %T %F %M %L %R  %T %F %M %L %R Mflop/s
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> --- Event Stage 0: Main Stage
>>> 
>>> BuildTwoSided          1 1.0 9.6083e-05 5.3 0.00e+00 0.0 5.0e+00 4.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> BuildTwoSidedF        74 1.0 2.0508e-01 2.9 0.00e+00 0.0 7.8e+01 5.7e+04 0.0e+00  1  0  2  5  0   1  0  2  5  0     0
>>> VecMDot               37 1.0 6.2755e-02 1.2 5.56e+07 1.0 0.0e+00 0.0e+00 3.7e+01  0  1  0  0  7   0  1  0  0  7  3517
>>> VecNorm               44 1.0 1.2301e-02 1.5 1.21e+07 1.0 0.0e+00 0.0e+00 4.4e+01  0  0  0  0  8   0  0  0  0  8  3889
>>> VecScale              41 1.0 5.2664e-03 1.0 5.61e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  4232
>>> VecCopy               53 1.0 1.3689e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecSet               151 1.0 1.9269e-02 1.4 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecAXPY                3 1.0 1.1799e-03 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  2764
>>> VecAYPX              156 1.0 6.0851e-02 1.2 2.49e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  1626
>>> VecAXPBYCZ           104 1.0 5.3286e-02 1.1 7.12e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  5305
>>> VecWAXPY               1 1.0 7.0786e-04 1.2 1.37e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   768
>>> VecMAXPY              40 1.0 6.2898e-02 1.0 6.44e+07 1.0 0.0e+00 0.0e+00 0.0e+00  0  1  0  0  0   0  1  0  0  0  4062
>>> VecAssemblyBegin      64 1.0 1.0848e-01 2.7 0.00e+00 0.0 2.5e+01 4.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> VecAssemblyEnd        64 1.0 1.1230e-04 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecPointwiseMult      11 1.0 6.5417e-03 1.1 1.51e+06 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0   914
>>> VecScatterBegin      270 1.0 1.6606e-02 1.0 0.00e+00 0.0 2.5e+03 1.5e+04 0.0e+00  0  0 69 40  0   0  0 69 40  0     0
>>> VecScatterEnd        270 1.0 5.5767e+0044.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 15  0  0  0  0  15  0  0  0  0     0
>>> VecSetRandom           1 1.0 1.5750e-03 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecReduceArith         3 1.0 6.3324e-04 1.3 8.22e+05 1.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0  5151
>>> VecReduceComm          1 1.0 1.5712e-0431.4 0.00e+00 0.0 0.0e+00 0.0e+00 1.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> VecNormalize          41 1.0 1.7057e-02 1.3 1.68e+07 1.0 0.0e+00 0.0e+00 4.1e+01  0  0  0  0  8   0  0  0  0  8  3920
>>> MatMult              162 1.0 2.5049e+00 1.0 2.36e+09 1.0 1.6e+03 6.9e+03 0.0e+00  9 27 45 12  0   9 27 45 12  0  3745
>>> MatMultAdd            13 1.0 5.5041e+00222.4 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00 15  1  1  4  0  15  1  1  4  0    38
>>> MatMultTranspose      13 1.0 6.2563e-02 1.1 5.35e+07 1.0 3.9e+01 8.5e+04 0.0e+00  0  1  1  4  0   0  1  1  4  0  3379
>>> MatSolve              13 0.0 3.4633e-01 0.0 6.33e+08 0.0 0.0e+00 0.0e+00 0.0e+00  0  2  0  0  0   0  2  0  0  0  1827
>>> MatSOR               141 1.0 4.1983e+00 1.0 2.07e+09 1.0 0.0e+00 0.0e+00 0.0e+00 15 24  0  0  0  15 24  0  0  0  1953
>>> MatLUFactorSym         1 1.0 4.5545e-0135375.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatLUFactorNum         1 1.0 4.6021e+00772110.8 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  4 38  0  0  0   4 38  0  0  0  2887
>>> MatScale               3 1.0 1.1058e-02 1.0 4.52e+06 1.0 1.0e+01 1.2e+03 0.0e+00  0  0  0  0  0   0  0  0  0  0  1618
>>> MatResidual           13 1.0 2.1592e-01 1.1 1.91e+08 1.0 1.3e+02 6.9e+03 0.0e+00  1  2  4  1  0   1  2  4  1  0  3519
>>> MatAssemblyBegin      33 1.0 1.1083e-01 3.3 0.00e+00 0.0 5.2e+01 8.2e+04 0.0e+00  0  0  1  5  0   0  0  1  5  0     0
>>> MatAssemblyEnd        33 1.0 1.7722e-01 1.0 0.00e+00 0.0 1.5e+02 1.4e+03 7.2e+01  1  0  4  0 13   1  0  4  0 13     0
>>> MatGetRow         593372 1.0 4.5285e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetRowIJ            1 0.0 3.5741e-03 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCreateSubMat        2 1.0 2.5588e-01 1.0 0.00e+00 0.0 2.1e+01 1.2e+06 3.2e+01  1  0  1 28  6   1  0  1 28  6     0
>>> MatGetOrdering         1 0.0 1.6127e-02 0.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatCoarsen             1 1.0 1.4122e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>> MatZeroEntries        10 1.0 1.5053e-01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  1  0  0  0  0   1  0  0  0  0     0
>>> MatView                5 1.7 1.6868e-03 3.3 0.00e+00 0.0 0.0e+00 0.0e+00 3.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>> MatAXPY                1 1.0 4.3107e-02 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatMatMult             1 1.0 2.5280e-01 1.0 8.75e+07 1.0 6.0e+01 1.8e+04 1.2e+01  1  1  2  1  2   1  1  2  1  2  1375
>>> MatMatMultSym          1 1.0 1.9445e-01 1.0 0.00e+00 0.0 5.0e+01 1.3e+04 1.2e+01  1  0  1  1  2   1  0  1  1  2     0
>>> MatMatMultNum          1 1.0 5.8333e-02 1.0 8.75e+07 1.0 1.0e+01 4.1e+04 0.0e+00  0  1  0  0  0   0  1  0  0  0  5959
>>> MatPtAP                1 1.0 6.7796e-01 1.0 3.42e+08 1.0 9.5e+01 3.5e+04 1.5e+01  2  4  3  4  3   2  4  3  4  3  1996
>>> MatPtAPSymbolic        1 1.0 4.8317e-01 1.0 0.00e+00 0.0 6.0e+01 3.4e+04 7.0e+00  2  0  2  2  1   2  0  2  2  1     0
>>> MatPtAPNumeric         1 1.0 1.9495e-01 1.0 3.42e+08 1.0 3.5e+01 3.6e+04 8.0e+00  1  4  1  1  1   1  4  1  1  1  6941
>>> MatTrnMatMult          1 1.0 4.1985e-02 1.0 4.18e+06 1.0 6.5e+01 6.6e+03 1.5e+01  0  0  2  0  3   0  0  2  0  3   396
>>> MatTrnMatMultSym       1 1.0 2.3958e-02 1.0 0.00e+00 0.0 3.0e+01 3.1e+03 7.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>> MatTrnMatMultNum       1 1.0 1.8097e-02 1.0 4.18e+06 1.0 3.5e+01 9.7e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1   918
>>> MatGetLocalMat         4 1.0 4.4538e-02 1.1 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> MatGetBrAoCol          3 1.0 3.8249e-03 1.6 0.00e+00 0.0 7.0e+01 4.0e+04 0.0e+00  0  0  2  3  0   0  0  2  3  0     0
>>> SNESSolve              1 1.0 2.1417e+01 1.0 1.91e+10 3.7 3.5e+03 2.1e+04 4.8e+02 76100 97 79 89  76100 97 79 90  1614
>>> SNESFunctionEval       2 1.0 4.4048e+00 1.0 5.48e+05 1.0 9.0e+02 1.5e+04 1.7e+02 16  0 25 14 31  16  0 25 14 31     0
>>> SNESJacobianEval       1 1.0 2.6810e+00 1.0 0.00e+00 0.0 1.1e+02 6.9e+04 1.9e+01 10  0  3  8  4  10  0  3  8  4     0
>>> SNESLineSearch         1 1.0 2.2640e+00 1.0 1.23e+06 1.0 4.5e+02 1.5e+04 8.5e+01  8  0 12  7 16   8  0 12  7 16     2
>>> SFSetGraph             1 1.0 2.8610e-06 1.5 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFSetUp                1 1.0 1.4687e-04 2.3 0.00e+00 0.0 1.5e+01 7.7e+02 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> SFBcastBegin           4 1.0 3.1948e-05 1.1 0.00e+00 0.0 4.0e+01 1.2e+03 0.0e+00  0  0  1  0  0   0  0  1  0  0     0
>>> SFBcastEnd             4 1.0 4.2200e-05 4.7 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPGMRESOrthog        32 1.0 1.0959e-01 1.1 1.03e+08 1.0 0.0e+00 0.0e+00 3.2e+01  0  1  0  0  6   0  1  0  0  6  3730
>>> KSPSetUp               6 1.0 9.1093e-03 1.2 0.00e+00 0.0 0.0e+00 0.0e+00 2.0e+00  0  0  0  0  0   0  0  0  0  0     0
>>> KSPSolve               1 1.0 1.4326e+01 1.0 1.91e+10 3.7 2.5e+03 2.1e+04 2.9e+02 51100 69 56 54  51100 69 56 55  2412
>>> PCGAMGGraph_AGG        1 1.0 3.8501e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
>>> PCGAMGCoarse_AGG       1 1.0 4.7253e-02 1.0 4.18e+06 1.0 1.7e+02 3.5e+03 2.1e+01  0  0  5  1  4   0  0  5  1  4   352
>>> PCGAMGProl_AGG         1 1.0 6.9987e-02 1.0 0.00e+00 0.0 4.0e+02 2.8e+03 8.6e+01  0  0 11  1 16   0  0 11  1 16     0
>>> PCGAMGPOpt_AGG         1 1.0 5.4272e-01 1.0 2.77e+08 1.0 1.6e+02 1.1e+04 4.1e+01  2  3  4  2  8   2  3  4  2  8  2024
>>> GAMG: createProl       1 1.0 1.0504e+00 1.0 2.81e+08 1.0 7.8e+02 4.5e+03 1.7e+02  4  3 22  4 32   4  3 22  4 32  1063
>>>  Graph                2 1.0 3.8496e-01 1.0 4.09e+05 1.0 5.0e+01 4.6e+02 2.4e+01  1  0  1  0  4   1  0  1  0  4     4
>>>  MIS/Agg              1 1.0 1.4219e-03 1.1 0.00e+00 0.0 5.5e+01 1.0e+03 2.0e+00  0  0  2  0  0   0  0  2  0  0     0
>>>  SA: col data         1 1.0 3.1489e-02 1.0 0.00e+00 0.0 3.7e+02 2.3e+03 7.4e+01  0  0 10  1 14   0  0 10  1 14     0
>>>  SA: frmProl0         1 1.0 3.5651e-02 1.0 0.00e+00 0.0 3.5e+01 7.6e+03 8.0e+00  0  0  1  0  1   0  0  1  0  1     0
>>>  SA: smooth           1 1.0 3.2070e-01 1.0 9.16e+07 1.0 6.0e+01 1.8e+04 1.4e+01  1  1  2  1  3   1  1  2  1  3  1135
>>> GAMG: partLevel        1 1.0 9.6892e-01 1.0 3.42e+08 1.0 1.2e+02 2.5e+05 6.6e+01  3  4  3 34 12   3  4  3 34 12  1397
>>>  repartition          1 1.0 1.1492e-04 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 6.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Invert-Sort          1 1.0 1.3120e-03 6.0 0.00e+00 0.0 0.0e+00 0.0e+00 4.0e+00  0  0  0  0  1   0  0  0  0  1     0
>>>  Move A               1 1.0 1.2847e-01 1.0 0.00e+00 0.0 1.5e+01 1.7e+06 1.7e+01  0  0  0 28  3   0  0  0 28  3     0
>>>  Move P               1 1.0 1.4728e-01 1.0 0.00e+00 0.0 6.0e+00 2.1e+04 1.7e+01  1  0  0  0  3   1  0  0  0  3     0
>>> PCSetUp                2 1.0 7.1033e+00 3.5 1.39e+1022.7 9.1e+02 3.8e+04 2.5e+02 12 46 25 38 45  12 46 25 38 46  2218
>>> PCSetUpOnBlocks       13 1.0 5.0738e+0063336.2 1.33e+10 0.0 0.0e+00 0.0e+00 0.0e+00  5 38  0  0  0   5 38  0  0  0  2618
>>> PCApply               13 1.0 1.2059e+01 1.0 1.83e+10 4.3 1.5e+03 1.1e+04 2.1e+01 43 90 41 18  4  43 90 41 18  4  2586
>>> ------------------------------------------------------------------------------------------------------------------------
>>> 
>>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/6c722205/attachment-0001.html>

From jed at jedbrown.org  Mon Oct 29 12:53:21 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 29 Oct 2018 11:53:21 -0600
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
Message-ID: <87woq0ej1q.fsf@jedbrown.org>

Manav Bhatia <bhatiamanav at gmail.com> writes:

> Thanks, Jed. 
>
> The description says: ? Square the graph, ie. compute A'*A before aggregating it"
>
> What is A here? 

The original matrix, or its "graph" (your 6x6 blocks condensed to scalars).

> What is the impact of setting this to 0, which led to a very significant increase in the CPU time in my case? 

The aggregates are formed on the connectivity of your original matrix,
so root nodes are aggregated only with their first neighbors, resulting
in slower coarsening.

From bhatiamanav at gmail.com  Mon Oct 29 12:59:39 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 12:59:39 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <87woq0ej1q.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
Message-ID: <548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>

Thanks for the clarification. 

I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 

-Manav

> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> Thanks, Jed. 
>> 
>> The description says: ? Square the graph, ie. compute A'*A before aggregating it"
>> 
>> What is A here? 
> 
> The original matrix, or its "graph" (your 6x6 blocks condensed to scalars).
> 
>> What is the impact of setting this to 0, which led to a very significant increase in the CPU time in my case? 
> 
> The aggregates are formed on the connectivity of your original matrix,
> so root nodes are aggregated only with their first neighbors, resulting
> in slower coarsening.


From jed at jedbrown.org  Mon Oct 29 13:03:27 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 29 Oct 2018 12:03:27 -0600
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
Message-ID: <87tvl4eikw.fsf@jedbrown.org>

Manav Bhatia <bhatiamanav at gmail.com> writes:

> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 

Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
behaving well, this makes little difference because the coarse
approximations are no worse than the approximations made in the first
coarsening (from fine level to first coarsened level).  But there are
many stubborn problems for which it does hurt, as you have seen.

From bhatiamanav at gmail.com  Mon Oct 29 13:12:38 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 13:12:38 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <87tvl4eikw.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
Message-ID: <A337EC13-C70B-47D7-9D3E-1DDC74045814@gmail.com>

Thanks! 

> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
> 
> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
> behaving well, this makes little difference because the coarse
> approximations are no worse than the approximations made in the first
> coarsening (from fine level to first coarsened level).  But there are
> many stubborn problems for which it does hurt, as you have seen.


From bhatiamanav at gmail.com  Mon Oct 29 13:44:02 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 13:44:02 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <87tvl4eikw.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
Message-ID: <A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>

I am experimenting with the threshold parameter. 

As described in the manual:  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html>


? Conversely reducing the threshold increases the rate of coarsening (aggressive coarsening) and thereby reduces the complexity of the coarse grids, and generally results in slower solver converge rates."


I did some experimentation for my problem: 

1) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0   : 22 KSP iterations, 39 seconds
2) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.5 : Did not converge in over 1000 KSP iterations

According to the manual reducing the threshold value would result in slower convergence rates. However, I am observing opposite behavior. 

Is this problem specific? Or maybe I am misunderstanding something here. 

Also, I am assuming that the threshold refers to some parameter in the connectivity graph, which is independent of the entries in the matrix. So, the value of threshold is independent of the magnitudes of the entries in the matrix (?). 

I would appreciate some guidance here. 

Regards,
Manav

> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
> 
> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
> behaving well, this makes little difference because the coarse
> approximations are no worse than the approximations made in the first
> coarsening (from fine level to first coarsened level).  But there are
> many stubborn problems for which it does hurt, as you have seen.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/2d1985ad/attachment.html>

From jed at jedbrown.org  Mon Oct 29 13:57:30 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 29 Oct 2018 12:57:30 -0600
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
	<A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>
Message-ID: <87ftwoeg2t.fsf@jedbrown.org>

Manav Bhatia <bhatiamanav at gmail.com> writes:

> I am experimenting with the threshold parameter. 
>
> As described in the manual:  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html>
>
>
> ? Conversely reducing the threshold increases the rate of coarsening (aggressive coarsening) and thereby reduces the complexity of the coarse grids, and generally results in slower solver converge rates."
>
>
> I did some experimentation for my problem: 
>
> 1) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0   : 22 KSP iterations, 39 seconds
> 2) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.5 : Did not converge in over 1000 KSP iterations

0.5 is enormous and likely results in "weird" aggregates.  Try 0.001 to
0.05 for example.  You can see the coarsening rate and grid complexity
in -ksp_view output.

> According to the manual reducing the threshold value would result in slower convergence rates. However, I am observing opposite behavior. 
>
> Is this problem specific? Or maybe I am misunderstanding something here. 
>
> Also, I am assuming that the threshold refers to some parameter in the connectivity graph, which is independent of the entries in the matrix. So, the value of threshold is independent of the magnitudes of the entries in the matrix (?). 

Yeah, the standard strength measure is

  |a_{ij}| / sqrt(|a_{ii}| |a_{jj}|)

> I would appreciate some guidance here. 
>
> Regards,
> Manav
>
>> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>> 
>>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
>> 
>> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
>> behaving well, this makes little difference because the coarse
>> approximations are no worse than the approximations made in the first
>> coarsening (from fine level to first coarsened level).  But there are
>> many stubborn problems for which it does hurt, as you have seen.

From bhatiamanav at gmail.com  Mon Oct 29 14:03:33 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 14:03:33 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <87ftwoeg2t.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
	<A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>
	<87ftwoeg2t.fsf@jedbrown.org>
Message-ID: <7578DD46-E883-465E-88DF-5F750AC3EE93@gmail.com>

This is really helpful, Jed! 

Thanks,
Manav

> On Oct 29, 2018, at 1:57 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> I am experimenting with the threshold parameter. 
>> 
>> As described in the manual:  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html>
>> 
>> 
>> ? Conversely reducing the threshold increases the rate of coarsening (aggressive coarsening) and thereby reduces the complexity of the coarse grids, and generally results in slower solver converge rates."
>> 
>> 
>> I did some experimentation for my problem: 
>> 
>> 1) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0   : 22 KSP iterations, 39 seconds
>> 2) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.5 : Did not converge in over 1000 KSP iterations
> 
> 0.5 is enormous and likely results in "weird" aggregates.  Try 0.001 to
> 0.05 for example.  You can see the coarsening rate and grid complexity
> in -ksp_view output.
> 
>> According to the manual reducing the threshold value would result in slower convergence rates. However, I am observing opposite behavior. 
>> 
>> Is this problem specific? Or maybe I am misunderstanding something here. 
>> 
>> Also, I am assuming that the threshold refers to some parameter in the connectivity graph, which is independent of the entries in the matrix. So, the value of threshold is independent of the magnitudes of the entries in the matrix (?). 
> 
> Yeah, the standard strength measure is
> 
>  |a_{ij}| / sqrt(|a_{ii}| |a_{jj}|)
> 
>> I would appreciate some guidance here. 
>> 
>> Regards,
>> Manav
>> 
>>> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>> 
>>>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
>>> 
>>> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
>>> behaving well, this makes little difference because the coarse
>>> approximations are no worse than the approximations made in the first
>>> coarsening (from fine level to first coarsened level).  But there are
>>> many stubborn problems for which it does hurt, as you have seen.


From bsmith at mcs.anl.gov  Mon Oct 29 14:06:18 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Mon, 29 Oct 2018 19:06:18 +0000
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
Message-ID: <6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>


  Exactly how much does it increase with number of levels? Send a chart number of levels and number of iterations. With say -mg_levels_ksp_maxit 4

   Thanks

   Barry




> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
> 
> Thanks for the clarification. 
> 
> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
> 
> -Manav
> 
>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>> 
>>> Thanks, Jed. 
>>> 
>>> The description says: ? Square the graph, ie. compute A'*A before aggregating it"
>>> 
>>> What is A here? 
>> 
>> The original matrix, or its "graph" (your 6x6 blocks condensed to scalars).
>> 
>>> What is the impact of setting this to 0, which led to a very significant increase in the CPU time in my case? 
>> 
>> The aggregates are formed on the connectivity of your original matrix,
>> so root nodes are aggregated only with their first neighbors, resulting
>> in slower coarsening.
> 


From bhatiamanav at gmail.com  Mon Oct 29 14:19:21 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 14:19:21 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <87ftwoeg2t.fsf@jedbrown.org>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
	<A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>
	<87ftwoeg2t.fsf@jedbrown.org>
Message-ID: <97A4B5FA-6E3A-493D-8366-071D587AE904@gmail.com>

I am trying to follow the meaning of this note from the manual: 

? GAMG removes small values from the graph with this threshold, and thus reducing the coupling in the graph and a different (perhaps better) coarser set of points"

So, I understand that increasing the value of threshold will result in larger number of connectivity edges to be dropped in the coarsening step. 

However, conceptually I think of the connectivity in the Jacobian matrix as the off-diagonal terms, which are critical to defining the coupled nature of the dofs in the problem. If I understand the manual correctly (which I am likely not) AMG prefers that in the coarsening step more of these couplings be dropped to enhance convergence (?). 

I can?t quite reconcile this with the potentially disastrous prospect of dropping couplings in the original matrix (without MG). But, somehow it works favorably for the coarsening step in a MG (?). 

I would appreciate some guidance. 

Regards,
Manav

> On Oct 29, 2018, at 1:57 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Manav Bhatia <bhatiamanav at gmail.com> writes:
> 
>> I am experimenting with the threshold parameter. 
>> 
>> As described in the manual:  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html>
>> 
>> 
>> ? Conversely reducing the threshold increases the rate of coarsening (aggressive coarsening) and thereby reduces the complexity of the coarse grids, and generally results in slower solver converge rates."
>> 
>> 
>> I did some experimentation for my problem: 
>> 
>> 1) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0   : 22 KSP iterations, 39 seconds
>> 2) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.5 : Did not converge in over 1000 KSP iterations
> 
> 0.5 is enormous and likely results in "weird" aggregates.  Try 0.001 to
> 0.05 for example.  You can see the coarsening rate and grid complexity
> in -ksp_view output.
> 
>> According to the manual reducing the threshold value would result in slower convergence rates. However, I am observing opposite behavior. 
>> 
>> Is this problem specific? Or maybe I am misunderstanding something here. 
>> 
>> Also, I am assuming that the threshold refers to some parameter in the connectivity graph, which is independent of the entries in the matrix. So, the value of threshold is independent of the magnitudes of the entries in the matrix (?). 
> 
> Yeah, the standard strength measure is
> 
>  |a_{ij}| / sqrt(|a_{ii}| |a_{jj}|)
> 
>> I would appreciate some guidance here. 
>> 
>> Regards,
>> Manav
>> 
>>> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>> 
>>>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
>>> 
>>> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
>>> behaving well, this makes little difference because the coarse
>>> approximations are no worse than the approximations made in the first
>>> coarsening (from fine level to first coarsened level).  But there are
>>> many stubborn problems for which it does hurt, as you have seen.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/6bb8f744/attachment.html>

From bhatiamanav at gmail.com  Mon Oct 29 14:27:46 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Mon, 29 Oct 2018 14:27:46 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
Message-ID: <129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>

Barry, 

   Here are some quick numbers with the following options on 4 CPUs and 543,606 dofs: 

-mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.

 #levels   |    #KSP Iters
???????????
     2        |       18
     3        |       18
     4        |       40
     5        |       59

-Manav


> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
> 
>  Exactly how much does it increase with number of levels? Send a chart number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
> 
>   Thanks
> 
>   Barry
> 
> 
> 
> 
>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>> 
>> Thanks for the clarification. 
>> 
>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
>> 
>> -Manav
>> 
>>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>> 
>>>> Thanks, Jed. 
>>>> 
>>>> The description says: ? Square the graph, ie. compute A'*A before aggregating it"
>>>> 
>>>> What is A here? 
>>> 
>>> The original matrix, or its "graph" (your 6x6 blocks condensed to scalars).
>>> 
>>>> What is the impact of setting this to 0, which led to a very significant increase in the CPU time in my case? 
>>> 
>>> The aggregates are formed on the connectivity of your original matrix,
>>> so root nodes are aggregated only with their first neighbors, resulting
>>> in slower coarsening.
>> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/fdd990d4/attachment.html>

From jed at jedbrown.org  Mon Oct 29 14:30:25 2018
From: jed at jedbrown.org (Jed Brown)
Date: Mon, 29 Oct 2018 13:30:25 -0600
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <97A4B5FA-6E3A-493D-8366-071D587AE904@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<87tvl4eikw.fsf@jedbrown.org>
	<A39F0AE9-020D-4C84-B74C-E59845763A45@gmail.com>
	<87ftwoeg2t.fsf@jedbrown.org>
	<97A4B5FA-6E3A-493D-8366-071D587AE904@gmail.com>
Message-ID: <877ei0eejy.fsf@jedbrown.org>

This is about removing edges when creating aggregates.  Consider
diffusion with an anisotropic conductivity tensor.  If you drop
connections (do not aggregate across) in the small-conductivity
direction, then your aggregates will be aligned with the high
conductivity, yielding what is called semi-coarsening in the MG
literature.  If you don't drop any connections in the strength graph,
then you get larger isotropic aggregates and thus your coarse space will
be inadequate.

Manav Bhatia <bhatiamanav at gmail.com> writes:

> I am trying to follow the meaning of this note from the manual: 
>
> ? GAMG removes small values from the graph with this threshold, and thus reducing the coupling in the graph and a different (perhaps better) coarser set of points"
>
> So, I understand that increasing the value of threshold will result in larger number of connectivity edges to be dropped in the coarsening step. 
>
> However, conceptually I think of the connectivity in the Jacobian matrix as the off-diagonal terms, which are critical to defining the coupled nature of the dofs in the problem. If I understand the manual correctly (which I am likely not) AMG prefers that in the coarsening step more of these couplings be dropped to enhance convergence (?). 
>
> I can?t quite reconcile this with the potentially disastrous prospect of dropping couplings in the original matrix (without MG). But, somehow it works favorably for the coarsening step in a MG (?). 
>
> I would appreciate some guidance. 
>
> Regards,
> Manav
>
>> On Oct 29, 2018, at 1:57 PM, Jed Brown <jed at jedbrown.org> wrote:
>> 
>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>> 
>>> I am experimenting with the threshold parameter. 
>>> 
>>> As described in the manual:  https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html <https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCGAMGSetThreshold.html>
>>> 
>>> 
>>> ? Conversely reducing the threshold increases the rate of coarsening (aggressive coarsening) and thereby reduces the complexity of the coarse grids, and generally results in slower solver converge rates."
>>> 
>>> 
>>> I did some experimentation for my problem: 
>>> 
>>> 1) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0   : 22 KSP iterations, 39 seconds
>>> 2) -pc_mg_levels 3 -mg_levels_ksp_max_it 3 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.5 : Did not converge in over 1000 KSP iterations
>> 
>> 0.5 is enormous and likely results in "weird" aggregates.  Try 0.001 to
>> 0.05 for example.  You can see the coarsening rate and grid complexity
>> in -ksp_view output.
>> 
>>> According to the manual reducing the threshold value would result in slower convergence rates. However, I am observing opposite behavior. 
>>> 
>>> Is this problem specific? Or maybe I am misunderstanding something here. 
>>> 
>>> Also, I am assuming that the threshold refers to some parameter in the connectivity graph, which is independent of the entries in the matrix. So, the value of threshold is independent of the magnitudes of the entries in the matrix (?). 
>> 
>> Yeah, the standard strength measure is
>> 
>>  |a_{ij}| / sqrt(|a_{ii}| |a_{jj}|)
>> 
>>> I would appreciate some guidance here. 
>>> 
>>> Regards,
>>> Manav
>>> 
>>>> On Oct 29, 2018, at 1:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>> 
>>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>>> 
>>>>> I also observed that the number of KSP iterations increases with an increase in the levels of AMG. Is this true, in general, for all/most applications? 
>>>> 
>>>> Adding levels replaces a direct solve with a deeper V-cycle.  When MG is
>>>> behaving well, this makes little difference because the coarse
>>>> approximations are no worse than the approximations made in the first
>>>> coarsening (from fine level to first coarsened level).  But there are
>>>> many stubborn problems for which it does hurt, as you have seen.

From mfadams at lbl.gov  Mon Oct 29 16:07:45 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 29 Oct 2018 17:07:45 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
Message-ID: <CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>

* the two level results tell us that MG is not doing well on the coarse
grids. So the coarse grids are the problem.

* Do not worry about timing now. Get the math correct. The two level solve
is not meant to be a solution just a diagnostic so don't try to optimize it
by squaring the graph. Use -pc_gamg_square_graph 0.

* It looks like you don't need 4 smoothing steps but lets keep it and we
can dial it back later.

* This table is interesting. First, you had about 12 iterations earlier and
I think your rtol was tighter than the default (so the iteration could
should go down not up). Do you know what change here?

Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
work. It is syntactic sugar to just add it to all levels like you did
manually.

Anyway, these number look reasonable. It is interesting that 3 levels ran
well but the 4th level ran poorly. This implies we want to slow down
coarsening on these levels, but ...

First can you please rerun this experiment with -pc_gamg_square_graph 0.

Also, please run with -info. This is very noisy but you can grep on "GAMG"
and send that output to us (about 15 lines).

Thanks,
Mark



On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Barry,
>
>    Here are some quick numbers with the following options on 4 CPUs and
> 543,606 dofs:
>
> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>
>  #levels   |    #KSP Iters
> ???????????
>      2        |       18
>      3        |       18
>      4        |       40
>      5        |       59
>
> -Manav
>
>
> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>
>
>  Exactly how much does it increase with number of levels? Send a chart
> number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
>
>   Thanks
>
>   Barry
>
>
>
>
> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>
> Thanks for the clarification.
>
> I also observed that the number of KSP iterations increases with an
> increase in the levels of AMG. Is this true, in general, for all/most
> applications?
>
> -Manav
>
> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>
> Manav Bhatia <bhatiamanav at gmail.com> writes:
>
> Thanks, Jed.
>
> The description says: ? Square the graph, ie. compute A'*A before
> aggregating it"
>
> What is A here?
>
>
> The original matrix, or its "graph" (your 6x6 blocks condensed to scalars).
>
> What is the impact of setting this to 0, which led to a very significant
> increase in the CPU time in my case?
>
>
> The aggregates are formed on the connectivity of your original matrix,
> so root nodes are aggregated only with their first neighbors, resulting
> in slower coarsening.
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181029/9b94571b/attachment.html>

From Peterspy at outlook.com  Mon Oct 29 21:50:32 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Tue, 30 Oct 2018 02:50:32 +0000
Subject: [petsc-users] Questions about writing makefile
Message-ID: <HK0PR04MB2547A1BDB4970A8F1E930EE1D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>

Hi all
I am wondering how to write a makefile using petsc package where it route is not in petsc-dir.
I am trying to start compiling like ??mpiifort ?CO3 -I?? -L?? ??.
Thanks in advance.

----------------------------------------------------------------
SONG Pengyang ??????????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86)15762264643

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/d9ef9a43/attachment.html>

From balay at mcs.anl.gov  Mon Oct 29 22:14:45 2018
From: balay at mcs.anl.gov (Balay, Satish)
Date: Tue, 30 Oct 2018 03:14:45 +0000
Subject: [petsc-users] Questions about writing makefile
In-Reply-To: <HK0PR04MB2547A1BDB4970A8F1E930EE1D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>
References: <HK0PR04MB2547A1BDB4970A8F1E930EE1D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>
Message-ID: <alpine.LFD.2.21.1810292213310.18407@asterix>

You'll have to give more details on the problem you are having.

Do you already have a makefile that partially works?

And what change do you need to do?

You can start off with a makefile from PETSc examples dir - and modify as needed..

Satish

On Tue, 30 Oct 2018, SONG Pengyang via petsc-users wrote:

> Hi all
> I am wondering how to write a makefile using petsc package where it route is not in petsc-dir.
> I am trying to start compiling like ??mpiifort ?CO3 -I?? -L?? ??.
> Thanks in advance.
> 
> ----------------------------------------------------------------
> SONG Pengyang ??????????
> MSc candidate in Physical Oceanography
> College of Oceanic and Atmospheric Sciences
> Ocean University of China
> No. 238 Songling Road, Qingdao 266100, Shandong, China
> TEL: (+86)15762264643
> 
> 

From Peterspy at outlook.com  Tue Oct 30 02:10:57 2018
From: Peterspy at outlook.com (SONG Pengyang)
Date: Tue, 30 Oct 2018 07:10:57 +0000
Subject: [petsc-users] Questions about writing makefile
Message-ID: <HK0PR04MB254732685636E5072D15F524D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>

Thank you, Satish.
I apologize for my undetailed question. The makefile problem usually happens due to different environment variables, so I find the administrator of the cluster and solve the problem.
Another question is about function DMDACreate3d, const PetscInt lx[] needs to be defined if I want to define the size of each cell (processor), how can it be used in fortran? I have tried "integer, parameter" in fortran but it does not work.
Regards,
Pengyang

----------------------------------------------------------------
SONG Pengyang ??????????
MSc candidate in Physical Oceanography
College of Oceanic and Atmospheric Sciences
Ocean University of China
No. 238 Songling Road, Qingdao 266100, Shandong, China
TEL: (+86)15762264643

-----????????-----
??????: Balay, Satish <balay at mcs.anl.gov> 
????????: 2018??10??30?? 11:15
??????: SONG Pengyang <Peterspy at outlook.com>
????: petsc-users at mcs.anl.gov
????: Re: [petsc-users] Questions about writing makefile

You'll have to give more details on the problem you are having.

Do you already have a makefile that partially works?

And what change do you need to do?

You can start off with a makefile from PETSc examples dir - and modify as needed..

Satish

On Tue, 30 Oct 2018, SONG Pengyang via petsc-users wrote:

> Hi all
> I am wondering how to write a makefile using petsc package where it route is not in petsc-dir.
> I am trying to start compiling like ??mpiifort ?CO3 -I?? -L?? ??.
> Thanks in advance.
> 
> ----------------------------------------------------------------
> SONG Pengyang ??????????
> MSc candidate in Physical Oceanography College of Oceanic and 
> Atmospheric Sciences Ocean University of China No. 238 Songling Road, 
> Qingdao 266100, Shandong, China
> TEL: (+86)15762264643
> 
> 

From niko.karin at gmail.com  Tue Oct 30 04:22:39 2018
From: niko.karin at gmail.com (Karin&NiKo)
Date: Tue, 30 Oct 2018 10:22:39 +0100
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
Message-ID: <CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>

Manav,

How are interpolated the rotational degrees of freedom? AFAIK, when using
smoothed aggregation, the interpolation process tries to mimic linear
interpolation, which can be OK for the displacement DOF but is not for the
rotational DOF using some plate and shell formulations.
This can explain poor convergence of a multilevel approach, which needs to
restrict and extrapolate the unknowns. In order to check this hypothesis,
you can try a test case with zero rotations.

Nicolas

Le lun. 29 oct. 2018 ? 22:13, Mark Adams via petsc-users <
petsc-users at mcs.anl.gov> a ?crit :

> * the two level results tell us that MG is not doing well on the coarse
> grids. So the coarse grids are the problem.
>
> * Do not worry about timing now. Get the math correct. The two level solve
> is not meant to be a solution just a diagnostic so don't try to optimize it
> by squaring the graph. Use -pc_gamg_square_graph 0.
>
> * It looks like you don't need 4 smoothing steps but lets keep it and we
> can dial it back later.
>
> * This table is interesting. First, you had about 12 iterations earlier
> and I think your rtol was tighter than the default (so the iteration could
> should go down not up). Do you know what change here?
>
> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
> work. It is syntactic sugar to just add it to all levels like you did
> manually.
>
> Anyway, these number look reasonable. It is interesting that 3 levels ran
> well but the 4th level ran poorly. This implies we want to slow down
> coarsening on these levels, but ...
>
> First can you please rerun this experiment with -pc_gamg_square_graph 0.
>
> Also, please run with -info. This is very noisy but you can grep on "GAMG"
> and send that output to us (about 15 lines).
>
> Thanks,
> Mark
>
>
>
> On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com>
> wrote:
>
>> Barry,
>>
>>    Here are some quick numbers with the following options on 4 CPUs and
>> 543,606 dofs:
>>
>> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>>
>>  #levels   |    #KSP Iters
>> ???????????
>>      2        |       18
>>      3        |       18
>>      4        |       40
>>      5        |       59
>>
>> -Manav
>>
>>
>> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>
>>
>>  Exactly how much does it increase with number of levels? Send a chart
>> number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
>>
>>   Thanks
>>
>>   Barry
>>
>>
>>
>>
>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com> wrote:
>>
>> Thanks for the clarification.
>>
>> I also observed that the number of KSP iterations increases with an
>> increase in the levels of AMG. Is this true, in general, for all/most
>> applications?
>>
>> -Manav
>>
>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>
>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>
>> Thanks, Jed.
>>
>> The description says: ? Square the graph, ie. compute A'*A before
>> aggregating it"
>>
>> What is A here?
>>
>>
>> The original matrix, or its "graph" (your 6x6 blocks condensed to
>> scalars).
>>
>> What is the impact of setting this to 0, which led to a very significant
>> increase in the CPU time in my case?
>>
>>
>> The aggregates are formed on the connectivity of your original matrix,
>> so root nodes are aggregated only with their first neighbors, resulting
>> in slower coarsening.
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/f2f530ca/attachment.html>

From knepley at gmail.com  Tue Oct 30 06:25:07 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 30 Oct 2018 07:25:07 -0400
Subject: [petsc-users] Questions about writing makefile
In-Reply-To: <HK0PR04MB254732685636E5072D15F524D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>
References: <HK0PR04MB254732685636E5072D15F524D0CC0@HK0PR04MB2547.apcprd04.prod.outlook.com>
Message-ID: <CAMYG4GmCrRifYvmg+uYoyrf+rjd6ZXdq=Kj9fsQhdBpVnPgMRA@mail.gmail.com>

On Tue, Oct 30, 2018 at 3:16 AM SONG Pengyang via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Thank you, Satish.
> I apologize for my undetailed question. The makefile problem usually
> happens due to different environment variables, so I find the administrator
> of the cluster and solve the problem.
> Another question is about function DMDACreate3d, const PetscInt lx[] needs
> to be defined if I want to define the size of each cell (processor), how
> can it be used in fortran? I have tried "integer, parameter" in fortran but
> it does not work.
>

PETSC_NULL_INTEGER

  Thanks

    Matt


> Regards,
> Pengyang
>
> ----------------------------------------------------------------
> SONG Pengyang ?????
> MSc candidate in Physical Oceanography
> College of Oceanic and Atmospheric Sciences
> Ocean University of China
> No. 238 Songling Road, Qingdao 266100, Shandong, China
> TEL: (+86)15762264643
>
> -----????-----
> ???: Balay, Satish <balay at mcs.anl.gov>
> ????: 2018?10?30? 11:15
> ???: SONG Pengyang <Peterspy at outlook.com>
> ??: petsc-users at mcs.anl.gov
> ??: Re: [petsc-users] Questions about writing makefile
>
> You'll have to give more details on the problem you are having.
>
> Do you already have a makefile that partially works?
>
> And what change do you need to do?
>
> You can start off with a makefile from PETSc examples dir - and modify as
> needed..
>
> Satish
>
> On Tue, 30 Oct 2018, SONG Pengyang via petsc-users wrote:
>
> > Hi all
> > I am wondering how to write a makefile using petsc package where it
> route is not in petsc-dir.
> > I am trying to start compiling like ?mpiifort ?O3 -I? -L? ?.
> > Thanks in advance.
> >
> > ----------------------------------------------------------------
> > SONG Pengyang ?????
> > MSc candidate in Physical Oceanography College of Oceanic and
> > Atmospheric Sciences Ocean University of China No. 238 Songling Road,
> > Qingdao 266100, Shandong, China
> > TEL: (+86)15762264643
> >
> >
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/19c1ce7c/attachment.html>

From mfadams at lbl.gov  Tue Oct 30 07:55:37 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Oct 2018 08:55:37 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
Message-ID: <CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>

Nicolas,

Smoothed aggregation is fine with shells. see the original SA paper (
https://link.springer.com/article/10.1007/BF02238511).

The rotational modes, which are the non-trivial modes that must be
supplied, are used in the interpolation.

Mark

On Tue, Oct 30, 2018 at 5:22 AM Karin&NiKo <niko.karin at gmail.com> wrote:

> Manav,
>
> How are interpolated the rotational degrees of freedom? AFAIK, when using
> smoothed aggregation, the interpolation process tries to mimic linear
> interpolation, which can be OK for the displacement DOF but is not for the
> rotational DOF using some plate and shell formulations.
> This can explain poor convergence of a multilevel approach, which needs to
> restrict and extrapolate the unknowns. In order to check this hypothesis,
> you can try a test case with zero rotations.
>
> Nicolas
>
> Le lun. 29 oct. 2018 ? 22:13, Mark Adams via petsc-users <
> petsc-users at mcs.anl.gov> a ?crit :
>
>> * the two level results tell us that MG is not doing well on the coarse
>> grids. So the coarse grids are the problem.
>>
>> * Do not worry about timing now. Get the math correct. The two level
>> solve is not meant to be a solution just a diagnostic so don't try to
>> optimize it by squaring the graph. Use -pc_gamg_square_graph 0.
>>
>> * It looks like you don't need 4 smoothing steps but lets keep it and we
>> can dial it back later.
>>
>> * This table is interesting. First, you had about 12 iterations earlier
>> and I think your rtol was tighter than the default (so the iteration could
>> should go down not up). Do you know what change here?
>>
>> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
>> work. It is syntactic sugar to just add it to all levels like you did
>> manually.
>>
>> Anyway, these number look reasonable. It is interesting that 3 levels ran
>> well but the 4th level ran poorly. This implies we want to slow down
>> coarsening on these levels, but ...
>>
>> First can you please rerun this experiment with -pc_gamg_square_graph 0.
>>
>> Also, please run with -info. This is very noisy but you can grep on
>> "GAMG" and send that output to us (about 15 lines).
>>
>> Thanks,
>> Mark
>>
>>
>>
>> On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com>
>> wrote:
>>
>>> Barry,
>>>
>>>    Here are some quick numbers with the following options on 4 CPUs and
>>> 543,606 dofs:
>>>
>>> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>>>
>>>  #levels   |    #KSP Iters
>>> ???????????
>>>      2        |       18
>>>      3        |       18
>>>      4        |       40
>>>      5        |       59
>>>
>>> -Manav
>>>
>>>
>>> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
>>>
>>>
>>>  Exactly how much does it increase with number of levels? Send a chart
>>> number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
>>>
>>>   Thanks
>>>
>>>   Barry
>>>
>>>
>>>
>>>
>>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com>
>>> wrote:
>>>
>>> Thanks for the clarification.
>>>
>>> I also observed that the number of KSP iterations increases with an
>>> increase in the levels of AMG. Is this true, in general, for all/most
>>> applications?
>>>
>>> -Manav
>>>
>>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>
>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>>
>>> Thanks, Jed.
>>>
>>> The description says: ? Square the graph, ie. compute A'*A before
>>> aggregating it"
>>>
>>> What is A here?
>>>
>>>
>>> The original matrix, or its "graph" (your 6x6 blocks condensed to
>>> scalars).
>>>
>>> What is the impact of setting this to 0, which led to a very significant
>>> increase in the CPU time in my case?
>>>
>>>
>>> The aggregates are formed on the connectivity of your original matrix,
>>> so root nodes are aggregated only with their first neighbors, resulting
>>> in slower coarsening.
>>>
>>>
>>>
>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/74050b05/attachment-0001.html>

From niko.karin at gmail.com  Tue Oct 30 11:38:49 2018
From: niko.karin at gmail.com (Karin&NiKo)
Date: Tue, 30 Oct 2018 17:38:49 +0100
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
	<CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
Message-ID: <CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>

Ok, sorry for my misunderstanding  and thank you for the clarification.

Le mar. 30 oct. 2018 13:55, Mark Adams <mfadams at lbl.gov> a ?crit :

> Nicolas,
>
> Smoothed aggregation is fine with shells. see the original SA paper (
> https://link.springer.com/article/10.1007/BF02238511).
>
> The rotational modes, which are the non-trivial modes that must be
> supplied, are used in the interpolation.
>
> Mark
>
> On Tue, Oct 30, 2018 at 5:22 AM Karin&NiKo <niko.karin at gmail.com> wrote:
>
>> Manav,
>>
>> How are interpolated the rotational degrees of freedom? AFAIK, when using
>> smoothed aggregation, the interpolation process tries to mimic linear
>> interpolation, which can be OK for the displacement DOF but is not for the
>> rotational DOF using some plate and shell formulations.
>> This can explain poor convergence of a multilevel approach, which needs
>> to restrict and extrapolate the unknowns. In order to check this
>> hypothesis, you can try a test case with zero rotations.
>>
>> Nicolas
>>
>> Le lun. 29 oct. 2018 ? 22:13, Mark Adams via petsc-users <
>> petsc-users at mcs.anl.gov> a ?crit :
>>
>>> * the two level results tell us that MG is not doing well on the coarse
>>> grids. So the coarse grids are the problem.
>>>
>>> * Do not worry about timing now. Get the math correct. The two level
>>> solve is not meant to be a solution just a diagnostic so don't try to
>>> optimize it by squaring the graph. Use -pc_gamg_square_graph 0.
>>>
>>> * It looks like you don't need 4 smoothing steps but lets keep it and we
>>> can dial it back later.
>>>
>>> * This table is interesting. First, you had about 12 iterations earlier
>>> and I think your rtol was tighter than the default (so the iteration could
>>> should go down not up). Do you know what change here?
>>>
>>> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
>>> work. It is syntactic sugar to just add it to all levels like you did
>>> manually.
>>>
>>> Anyway, these number look reasonable. It is interesting that 3 levels
>>> ran well but the 4th level ran poorly. This implies we want to slow down
>>> coarsening on these levels, but ...
>>>
>>> First can you please rerun this experiment with -pc_gamg_square_graph 0.
>>>
>>> Also, please run with -info. This is very noisy but you can grep on
>>> "GAMG" and send that output to us (about 15 lines).
>>>
>>> Thanks,
>>> Mark
>>>
>>>
>>>
>>> On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com>
>>> wrote:
>>>
>>>> Barry,
>>>>
>>>>    Here are some quick numbers with the following options on 4 CPUs and
>>>> 543,606 dofs:
>>>>
>>>> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>>>>
>>>>  #levels   |    #KSP Iters
>>>> ???????????
>>>>      2        |       18
>>>>      3        |       18
>>>>      4        |       40
>>>>      5        |       59
>>>>
>>>> -Manav
>>>>
>>>>
>>>> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>
>>>>  Exactly how much does it increase with number of levels? Send a chart
>>>> number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
>>>>
>>>>   Thanks
>>>>
>>>>   Barry
>>>>
>>>>
>>>>
>>>>
>>>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com>
>>>> wrote:
>>>>
>>>> Thanks for the clarification.
>>>>
>>>> I also observed that the number of KSP iterations increases with an
>>>> increase in the levels of AMG. Is this true, in general, for all/most
>>>> applications?
>>>>
>>>> -Manav
>>>>
>>>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>
>>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>>>
>>>> Thanks, Jed.
>>>>
>>>> The description says: ? Square the graph, ie. compute A'*A before
>>>> aggregating it"
>>>>
>>>> What is A here?
>>>>
>>>>
>>>> The original matrix, or its "graph" (your 6x6 blocks condensed to
>>>> scalars).
>>>>
>>>> What is the impact of setting this to 0, which led to a very
>>>> significant increase in the CPU time in my case?
>>>>
>>>>
>>>> The aggregates are formed on the connectivity of your original matrix,
>>>> so root nodes are aggregated only with their first neighbors, resulting
>>>> in slower coarsening.
>>>>
>>>>
>>>>
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/86cec5b5/attachment.html>

From bhatiamanav at gmail.com  Tue Oct 30 10:32:06 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Tue, 30 Oct 2018 10:32:06 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
Message-ID: <D481EE5D-98AE-45DB-9C2B-DC18A25D4DED@gmail.com>



> On Oct 29, 2018, at 4:07 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> * the two level results tell us that MG is not doing well on the coarse grids. So the coarse grids are the problem.
> 
> * Do not worry about timing now. Get the math correct. The two level solve is not meant to be a solution just a diagnostic so don't try to optimize it by squaring the graph. Use -pc_gamg_square_graph 0.
> 
> * It looks like you don't need 4 smoothing steps but lets keep it and we can dial it back later.
> 
> * This table is interesting. First, you had about 12 iterations earlier and I think your rtol was tighter than the default (so the iteration could should go down not up). Do you know what change here?

I maintained the same tolerance using -ksp_rtol 1.e-8 (did not copy this into the parameter list in the previous email). 

The 2 level iterations was a typo and should have been 12.  I have corrected the table here: 
-mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-8

Case#  |  #levels   |    #KSP Iters
??????????????
     1      |     2         |       12
     2      |     3         |       18
     3      |     4         |       40
     4      |     5         |       59


> 
> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does work. It is syntactic sugar to just add it to all levels like you did manually.
> 
> Anyway, these number look reasonable. It is interesting that 3 levels ran well but the 4th level ran poorly. This implies we want to slow down coarsening on these levels, but ...
> 
> First can you please rerun this experiment with -pc_gamg_square_graph 0.
> 

Following is the summary of the results with -pc_gamg_square_graph 0

-mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -ksp_rtol 1.e-8

Case#  |  #levels   |    #KSP Iters
??????????????
     5      |     2        |         9
     6      |     3        |       13
     7      |     4        |       31
     8      |     5        |       50


> Also, please run with -info. This is very noisy but you can grep on "GAMG" and send that output to us (about 15 lines).

I have included information about all 8 cases below. 

Thanks for your help! 

-Manav

> 
> Thanks,
> Mark
> 
> 


Case 1)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 2 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 2.980759132912e+01 
    1 KSP Residual norm 1.268404746671e+00 
    2 KSP Residual norm 1.420311012425e-01 
    3 KSP Residual norm 3.026536678757e-02 
    4 KSP Residual norm 6.511170312990e-03 
    5 KSP Residual norm 1.539620841789e-03 
    6 KSP Residual norm 3.655528499924e-04 
    7 KSP Residual norm 8.111524453985e-05 
    8 KSP Residual norm 1.995956470679e-05 
    9 KSP Residual norm 4.397662980825e-06 
   10 KSP Residual norm 9.636956929325e-07 
   11 KSP Residual norm 3.013384202118e-07 
   12 KSP Residual norm 1.867699579372e-07 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 6837 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 10296 with simple aggregation
[0] PCSetUp_GAMG(): 1) N=41022, n data cols=6, nnz/row (ave)=69, 1 active pes
[0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.0976


Case 2)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 3 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 1.721859579323e+01 
    1 KSP Residual norm 2.372150715400e+00 
    2 KSP Residual norm 4.332444773735e-01 
    3 KSP Residual norm 1.828644450393e-01 
    4 KSP Residual norm 6.508336322670e-02 
    5 KSP Residual norm 2.352085101499e-02 
    6 KSP Residual norm 9.165114885090e-03 
    7 KSP Residual norm 3.674781085969e-03 
    8 KSP Residual norm 1.331249870878e-03 
    9 KSP Residual norm 4.566260302483e-04 
   10 KSP Residual norm 1.795556967223e-04 
   11 KSP Residual norm 6.793314859559e-05 
   12 KSP Residual norm 2.504121594312e-05 
   13 KSP Residual norm 9.167163291488e-06 
   14 KSP Residual norm 3.283556007932e-06 
   15 KSP Residual norm 1.277586362108e-06 
   16 KSP Residual norm 4.852580652992e-07 
   17 KSP Residual norm 1.901791545937e-07 
   18 KSP Residual norm 7.974015462515e-08 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 6837 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=41022, n data cols=6, nnz/row (ave)=69, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 11.6521 nnz ave. (N=6837)
[0] PCGAMGProlongator_AGG(): New grid 1121 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.785527e+00 min=6.042089e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 1656 with simple aggregation
[0] PCSetUp_GAMG(): 2) N=6726, n data cols=6, nnz/row (ave)=140, 1 active pes
[0] PCSetUp_GAMG(): 3 levels, grid complexity = 1.12982


Case 3)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 4 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 5.715310219090e+00 
    1 KSP Residual norm 1.702306568500e+00 
    2 KSP Residual norm 1.105404714191e+00 
    3 KSP Residual norm 7.521070321363e-01 
    4 KSP Residual norm 5.151770148497e-01 
    5 KSP Residual norm 3.407231098045e-01 
    6 KSP Residual norm 2.154066019733e-01 
    7 KSP Residual norm 1.320975463963e-01 
    8 KSP Residual norm 8.464181469022e-02 
    9 KSP Residual norm 5.404783722947e-02 
   10 KSP Residual norm 3.571728349786e-02 
   11 KSP Residual norm 2.363141413294e-02 
   12 KSP Residual norm 1.530000070717e-02 
   13 KSP Residual norm 9.398576202667e-03 
   14 KSP Residual norm 5.389891497192e-03 
   15 KSP Residual norm 3.308851504727e-03 
   16 KSP Residual norm 2.233599324934e-03 
   17 KSP Residual norm 1.471510415641e-03 
   18 KSP Residual norm 9.214853138234e-04 
   19 KSP Residual norm 6.090808616289e-04 
   20 KSP Residual norm 4.205193238717e-04 
   21 KSP Residual norm 2.728384192436e-04 
   22 KSP Residual norm 1.647370935175e-04 
   23 KSP Residual norm 1.007397471037e-04 
   24 KSP Residual norm 6.133965622058e-05 
   25 KSP Residual norm 3.650073757918e-05 
   26 KSP Residual norm 2.220817401390e-05 
   27 KSP Residual norm 1.423846811011e-05 
   28 KSP Residual norm 9.147106610671e-06 
   29 KSP Residual norm 5.622102212398e-06 
   30 KSP Residual norm 3.427704849314e-06 
   31 KSP Residual norm 2.562333599882e-06 
   32 KSP Residual norm 1.780393486817e-06 
   33 KSP Residual norm 1.175933486658e-06 
   34 KSP Residual norm 7.633724873183e-07 
   35 KSP Residual norm 4.635656121553e-07 
   36 KSP Residual norm 2.809545347534e-07 
   37 KSP Residual norm 1.736631971561e-07 
   38 KSP Residual norm 1.074049251439e-07 
   39 KSP Residual norm 6.374646169363e-08 
   40 KSP Residual norm 4.140669801911e-08 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 6837 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=41022, n data cols=6, nnz/row (ave)=69, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 11.6521 nnz ave. (N=6837)
[0] PCGAMGProlongator_AGG(): New grid 1121 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.785527e+00 min=6.042089e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=6726, n data cols=6, nnz/row (ave)=140, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 23.3696 nnz ave. (N=1121)
[0] PCGAMGProlongator_AGG(): New grid 107 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=3.003329e+00 min=8.421562e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 144 with simple aggregation
[0] PCSetUp_GAMG(): 3) N=642, n data cols=6, nnz/row (ave)=154, 1 active pes
[0] PCSetUp_GAMG(): 4 levels, grid complexity = 1.13323


Case 4)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 5 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 1.796966525877e+00 
    1 KSP Residual norm 1.272395001001e+00 
    2 KSP Residual norm 9.587765774342e-01 
    3 KSP Residual norm 7.503468727302e-01 
    4 KSP Residual norm 5.929828008661e-01 
    5 KSP Residual norm 4.667907393525e-01 
    6 KSP Residual norm 3.582573847190e-01 
    7 KSP Residual norm 2.641549535092e-01 
    8 KSP Residual norm 1.897005575664e-01 
    9 KSP Residual norm 1.290185919854e-01 
   10 KSP Residual norm 8.674970372217e-02 
   11 KSP Residual norm 5.719457030929e-02 
   12 KSP Residual norm 3.670199440851e-02 
   13 KSP Residual norm 2.279854011416e-02 
   14 KSP Residual norm 1.413541409138e-02 
   15 KSP Residual norm 9.904419724774e-03 
   16 KSP Residual norm 7.479600408120e-03 
   17 KSP Residual norm 5.653470278882e-03 
   18 KSP Residual norm 4.357609222840e-03 
   19 KSP Residual norm 3.501603455533e-03 
   20 KSP Residual norm 2.740008551458e-03 
   21 KSP Residual norm 2.000305455177e-03 
   22 KSP Residual norm 1.465665578257e-03 
   23 KSP Residual norm 1.063641050628e-03 
   24 KSP Residual norm 7.320397918812e-04 
   25 KSP Residual norm 4.987179366244e-04 
   26 KSP Residual norm 3.531130935050e-04 
   27 KSP Residual norm 2.534997938742e-04 
   28 KSP Residual norm 1.815505756854e-04 
   29 KSP Residual norm 1.303047003006e-04 
   30 KSP Residual norm 9.159509138706e-05 
   31 KSP Residual norm 7.324246273743e-05 
   32 KSP Residual norm 5.465350505655e-05 
   33 KSP Residual norm 3.876661576537e-05 
   34 KSP Residual norm 2.792720150402e-05 
   35 KSP Residual norm 2.108207616943e-05 
   36 KSP Residual norm 1.611370701356e-05 
   37 KSP Residual norm 1.208049584171e-05 
   38 KSP Residual norm 9.075511243294e-06 
   39 KSP Residual norm 7.091514153440e-06 
   40 KSP Residual norm 5.623979353049e-06 
   41 KSP Residual norm 4.346769151236e-06 
   42 KSP Residual norm 3.197496840488e-06 
   43 KSP Residual norm 2.265919711506e-06 
   44 KSP Residual norm 1.600975191547e-06 
   45 KSP Residual norm 1.132335469737e-06 
   46 KSP Residual norm 7.710870825137e-07 
   47 KSP Residual norm 5.064951347280e-07 
   48 KSP Residual norm 3.448475289519e-07 
   49 KSP Residual norm 2.450120965032e-07 
   50 KSP Residual norm 1.826746152120e-07 
   51 KSP Residual norm 1.445084170865e-07 
   52 KSP Residual norm 1.200408904185e-07 
   53 KSP Residual norm 9.882182031280e-08 
   54 KSP Residual norm 7.914655725555e-08 
   55 KSP Residual norm 6.110632924313e-08 
   56 KSP Residual norm 4.632304250505e-08 
   57 KSP Residual norm 3.242693236988e-08 
   58 KSP Residual norm 2.195784301351e-08 
   59 KSP Residual norm 1.571496587971e-08 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGCoarsen_AGG(): Square Graph on level 1 of 1 to square
[0] PCGAMGProlongator_AGG(): New grid 6837 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=41022, n data cols=6, nnz/row (ave)=69, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 11.6521 nnz ave. (N=6837)
[0] PCGAMGProlongator_AGG(): New grid 1121 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.785527e+00 min=6.042089e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=6726, n data cols=6, nnz/row (ave)=140, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 23.3696 nnz ave. (N=1121)
[0] PCGAMGProlongator_AGG(): New grid 107 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=3.003329e+00 min=8.421562e-02 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=642, n data cols=6, nnz/row (ave)=154, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 24.5 nnz ave. (N=107)
[0] PCGAMGProlongator_AGG(): New grid 9 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.863578e+00 min=9.275407e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 6 with simple aggregation
[0] PCSetUp_GAMG(): 4) N=54, n data cols=6, nnz/row (ave)=54, 1 active pes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.13333


Case 5)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 2 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 4.658748806834e+01 
    1 KSP Residual norm 6.825811660437e-01 
    2 KSP Residual norm 3.642264109013e-02 
    3 KSP Residual norm 2.013016230540e-03 
    4 KSP Residual norm 2.013432241615e-04 
    5 KSP Residual norm 2.105457376227e-05 
    6 KSP Residual norm 3.003346327786e-06 
    7 KSP Residual norm 1.445664726245e-06 
    8 KSP Residual norm 6.952678680043e-07 
    9 KSP Residual norm 3.813304720986e-07 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 25260 with simple aggregation
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=103, 1 active pes
[0] PCSetUp_GAMG(): 2 levels, grid complexity = 1.35717


Case 6)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 3 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 2.700027956285e+01 
    1 KSP Residual norm 2.081487493276e+00 
    2 KSP Residual norm 2.504352351366e-01 
    3 KSP Residual norm 7.222719417818e-02 
    4 KSP Residual norm 1.769085428189e-02 
    5 KSP Residual norm 4.200829546873e-03 
    6 KSP Residual norm 1.100496697039e-03 
    7 KSP Residual norm 2.880192907717e-04 
    8 KSP Residual norm 7.621296303148e-05 
    9 KSP Residual norm 1.923264828609e-05 
   10 KSP Residual norm 4.962521087771e-06 
   11 KSP Residual norm 1.223640145242e-06 
   12 KSP Residual norm 3.668999985201e-07 
   13 KSP Residual norm 1.828878545694e-07 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=103, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 17.2399 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 2021 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.704298e+00 min=6.370030e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 3006 with simple aggregation
[0] PCSetUp_GAMG(): 2) N=12126, n data cols=6, nnz/row (ave)=172, 1 active pes
[0] PCSetUp_GAMG(): 3 levels, grid complexity = 1.42875


Case 7)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 4 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 9.908487889752e+00 
    1 KSP Residual norm 2.252812343792e+00 
    2 KSP Residual norm 1.165408110334e+00 
    3 KSP Residual norm 6.434735893158e-01 
    4 KSP Residual norm 3.696466541726e-01 
    5 KSP Residual norm 2.135677187677e-01 
    6 KSP Residual norm 1.257757438010e-01 
    7 KSP Residual norm 7.128420785165e-02 
    8 KSP Residual norm 3.976090525850e-02 
    9 KSP Residual norm 2.279000353245e-02 
   10 KSP Residual norm 1.300701287311e-02 
   11 KSP Residual norm 7.498254128331e-03 
   12 KSP Residual norm 4.288501173737e-03 
   13 KSP Residual norm 2.316089581521e-03 
   14 KSP Residual norm 1.263802234905e-03 
   15 KSP Residual norm 7.296829650365e-04 
   16 KSP Residual norm 4.261675876945e-04 
   17 KSP Residual norm 2.440548986676e-04 
   18 KSP Residual norm 1.436965272125e-04 
   19 KSP Residual norm 8.010925146489e-05 
   20 KSP Residual norm 4.339850994759e-05 
   21 KSP Residual norm 2.411926753762e-05 
   22 KSP Residual norm 1.306205068099e-05 
   23 KSP Residual norm 7.542931204731e-06 
   24 KSP Residual norm 4.283178968440e-06 
   25 KSP Residual norm 2.483127922661e-06 
   26 KSP Residual norm 1.517894242402e-06 
   27 KSP Residual norm 8.801147250084e-07 
   28 KSP Residual norm 5.088326120723e-07 
   29 KSP Residual norm 2.732311721999e-07 
   30 KSP Residual norm 1.450174505696e-07 
   31 KSP Residual norm 9.861763824710e-08 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=103, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 17.2399 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 2021 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.704298e+00 min=6.370030e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=12126, n data cols=6, nnz/row (ave)=172, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 28.7325 nnz ave. (N=2021)
[0] PCGAMGProlongator_AGG(): New grid 161 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.212396e+00 min=1.149492e-01 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 204 with simple aggregation
[0] PCSetUp_GAMG(): 3) N=966, n data cols=6, nnz/row (ave)=176, 1 active pes
[0] PCSetUp_GAMG(): 4 levels, grid complexity = 1.43458


Case 8)  -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -ksp_rtol 1.e-8 -pc_mg_levels 5 
Dhcp-90-164:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 2.218294876305e+00 
    1 KSP Residual norm 1.632572879485e+00 
    2 KSP Residual norm 1.182643328612e+00 
    3 KSP Residual norm 8.718741753444e-01 
    4 KSP Residual norm 6.533291909118e-01 
    5 KSP Residual norm 4.866483224152e-01 
    6 KSP Residual norm 3.484903886854e-01 
    7 KSP Residual norm 2.348475670124e-01 
    8 KSP Residual norm 1.537187403224e-01 
    9 KSP Residual norm 1.026536155589e-01 
   10 KSP Residual norm 6.824513463193e-02 
   11 KSP Residual norm 4.627422220135e-02 
   12 KSP Residual norm 3.117616798825e-02 
   13 KSP Residual norm 2.081871084553e-02 
   14 KSP Residual norm 1.460827781562e-02 
   15 KSP Residual norm 1.037979487427e-02 
   16 KSP Residual norm 6.886117836686e-03 
   17 KSP Residual norm 4.634120402061e-03 
   18 KSP Residual norm 2.822150912459e-03 
   19 KSP Residual norm 1.699261862470e-03 
   20 KSP Residual norm 1.127636890762e-03 
   21 KSP Residual norm 7.233233068369e-04 
   22 KSP Residual norm 4.545631018082e-04 
   23 KSP Residual norm 2.995811344995e-04 
   24 KSP Residual norm 1.984480794186e-04 
   25 KSP Residual norm 1.339576789380e-04 
   26 KSP Residual norm 8.964509794502e-05 
   27 KSP Residual norm 6.063175592350e-05 
   28 KSP Residual norm 4.271774591608e-05 
   29 KSP Residual norm 3.026766800790e-05 
   30 KSP Residual norm 2.191056439926e-05 
   31 KSP Residual norm 1.813101162702e-05 
   32 KSP Residual norm 1.410197450206e-05 
   33 KSP Residual norm 9.889952949347e-06 
   34 KSP Residual norm 6.488236898017e-06 
   35 KSP Residual norm 4.232064596117e-06 
   36 KSP Residual norm 2.736230796810e-06 
   37 KSP Residual norm 1.829705002417e-06 
   38 KSP Residual norm 1.272435477754e-06 
   39 KSP Residual norm 8.374839157107e-07 
   40 KSP Residual norm 5.423071251782e-07 
   41 KSP Residual norm 3.638843185090e-07 
   42 KSP Residual norm 2.318570275176e-07 
   43 KSP Residual norm 1.494263195512e-07 
   44 KSP Residual norm 1.014479883907e-07 
   45 KSP Residual norm 7.010792145263e-08 
   46 KSP Residual norm 5.302212871376e-08 
   47 KSP Residual norm 4.129998076695e-08 
   48 KSP Residual norm 3.174687615554e-08 
   49 KSP Residual norm 2.425058914303e-08 
   50 KSP Residual norm 1.810773091084e-08 
Dhcp-90-164:tmp manav$ cat t | grep GAMG
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.748254e+00 min=6.352623e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=103, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 17.2399 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 2021 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.704298e+00 min=6.370030e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=12126, n data cols=6, nnz/row (ave)=172, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 28.7325 nnz ave. (N=2021)
[0] PCGAMGProlongator_AGG(): New grid 161 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.212396e+00 min=1.149492e-01 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=966, n data cols=6, nnz/row (ave)=176, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 28.2353 nnz ave. (N=161)
[0] PCGAMGProlongator_AGG(): New grid 14 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.257007e+00 min=1.166837e-01 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 18 with simple aggregation
[0] PCSetUp_GAMG(): 4) N=84, n data cols=6, nnz/row (ave)=82, 1 active pes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.43482


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/f21e131b/attachment-0001.html>

From repepo at gmail.com  Tue Oct 30 12:10:59 2018
From: repepo at gmail.com (Santiago Andres Triana)
Date: Tue, 30 Oct 2018 18:10:59 +0100
Subject: [petsc-users] Segmentation violation
Message-ID: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>

Hi petsc-users,

I am solving a generalized eigenvalue problem using ex7 in
$SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
The program runs fine, with correct solutions on 12-core node and also on a
mac laptop.

However, on a 16-core workstation running Debian testing (fresh install)
and also a fresh install of petsc and slepc I get the following error:

 $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4
-eps_target -2e-3+1.01i

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
[1]PETSC ERROR:
------------------------------------------------------------------------
[1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
probably memory access out of range
[1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
[1]PETSC ERROR: or see
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X
to find memory corruption errors
[1]PETSC ERROR: likely location of problem given in stack below
[1]PETSC ERROR: ---------------------  Stack Frames
------------------------------------
[1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
[1]PETSC ERROR:       INSTEAD the line number of the start of the function
[1]PETSC ERROR:       is given.
[1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205
/home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
[1]PETSC ERROR: [1] MatLUFactorNumeric line 3054
/home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
[1]PETSC ERROR: [1] PCSetUp_LU line 59
/home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
[1]PETSC ERROR: [1] PCSetUp line 894
/home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
[1]PETSC ERROR: [1] KSPSetUp line 304
/home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
[1]PETSC ERROR: [1] STSetUp_Sinvert line 96
/home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
[1]PETSC ERROR: [1] STSetUp line 233
/home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
[1]PETSC ERROR: [1] EPSSetUp line 104
/home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
[1]PETSC ERROR: [1] EPSSolve line 129
/home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
[1]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[1]PETSC ERROR: Signal received
[1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
[1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2
Tue Oct 30 17:40:51 2018
[1]PETSC ERROR: Configure options --download-mpich
-with-scalar-type=complex --download-mumps --download-parmetis
--download-metis --download-scalapack --download-fblaslapack
--with-debugging=1 --download-superlu_dist --download-ptscotch
[1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1



the expected output is the following (on a compute node running petsc-3.9.2
and also on a mac laptop running petsc-3.10.2):

$ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4
-eps_target -2e-3+1.01i

Generalized eigenproblem stored in file.

 Reading COMPLEX matrices from binary files...
 Number of iterations of the method: 2
 Number of linear iterations of the method: 27
 Solution method: krylovschur

 Number of requested eigenvalues: 4
 Stopping condition: tol=1e-08, maxit=63157
 Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL;
iterations 2
 ---------------------- --------------------
            k             ||Ax-kBx||/||kx||
 ---------------------- --------------------
  -0.002806+1.009827i       2.00821e-19
  -0.002980+1.008417i       8.08359e-17
  -0.002676+1.011755i       9.49342e-18
  -0.003201+1.007367i       1.50869e-16
 ---------------------- --------------------


Just in case, the matrices can be downloaded from here if any one wants to
give them a try
https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0


I tried different petsc/slepc versions to no avail, including an OS
reinstall. So any help would be highly appreciated. Thanks in advance!

Santiago
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/7d27b599/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 30 12:42:57 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Tue, 30 Oct 2018 17:42:57 +0000
Subject: [petsc-users] Segmentation violation
In-Reply-To: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
References: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
Message-ID: <AAC2E358-F818-477F-8668-20DDC50330A7@anl.gov>


   Can you run the code on the "failing" machine using valgrind? https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind

  Barry


> On Oct 30, 2018, at 12:10 PM, Santiago Andres Triana via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Hi petsc-users,
> 
> I am solving a generalized eigenvalue problem using ex7 in $SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
> The program runs fine, with correct solutions on 12-core node and also on a mac laptop.
> 
> However, on a 16-core workstation running Debian testing (fresh install) and also a fresh install of petsc and slepc I get the following error:
> 
>  $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> 
> Generalized eigenproblem stored in file.
> 
>  Reading COMPLEX matrices from binary files...
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205 /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054 /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> [1]PETSC ERROR: [1] PCSetUp line 894 /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 304 /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: [1] STSetUp_Sinvert line 96 /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> [1]PETSC ERROR: [1] STSetUp line 233 /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> [1]PETSC ERROR: [1] EPSSetUp line 104 /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> [1]PETSC ERROR: [1] EPSSolve line 129 /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018 
> [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2 Tue Oct 30 17:40:51 2018
> [1]PETSC ERROR: Configure options --download-mpich -with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack --download-fblaslapack --with-debugging=1 --download-superlu_dist --download-ptscotch
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> 
> 
> 
> the expected output is the following (on a compute node running petsc-3.9.2 and also on a mac laptop running petsc-3.10.2):
> 
> $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> 
> Generalized eigenproblem stored in file.
> 
>  Reading COMPLEX matrices from binary files...
>  Number of iterations of the method: 2
>  Number of linear iterations of the method: 27
>  Solution method: krylovschur
> 
>  Number of requested eigenvalues: 4
>  Stopping condition: tol=1e-08, maxit=63157
>  Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL; iterations 2
>  ---------------------- --------------------
>             k             ||Ax-kBx||/||kx||
>  ---------------------- --------------------
>   -0.002806+1.009827i       2.00821e-19
>   -0.002980+1.008417i       8.08359e-17
>   -0.002676+1.011755i       9.49342e-18
>   -0.003201+1.007367i       1.50869e-16
>  ---------------------- --------------------
> 
> 
> Just in case, the matrices can be downloaded from here if any one wants to give them a try
> https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
> https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0
> 
> 
> I tried different petsc/slepc versions to no avail, including an OS reinstall. So any help would be highly appreciated. Thanks in advance!
> 
> Santiago


From knepley at gmail.com  Tue Oct 30 12:58:57 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 30 Oct 2018 13:58:57 -0400
Subject: [petsc-users] Segmentation violation
In-Reply-To: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
References: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
Message-ID: <CAMYG4Gncy51iUiz4bmAmRM2DoJ==7KEhOOZV8PSSKfD2EKpV5A@mail.gmail.com>

On Tue, Oct 30, 2018 at 1:18 PM Santiago Andres Triana via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi petsc-users,
>
> I am solving a generalized eigenvalue problem using ex7 in
> $SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
> The program runs fine, with correct solutions on 12-core node and also on
> a mac laptop.
>
> However, on a 16-core workstation running Debian testing (fresh install)
> and also a fresh install of petsc and slepc I get the following error:
>
>  $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4
> -eps_target -2e-3+1.01i
>
> Generalized eigenproblem stored in file.
>
>  Reading COMPLEX matrices from binary files...
> [1]PETSC ERROR:
> ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
> probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see
> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS
> X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames
> ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
> available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205
> /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054
> /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> [1]PETSC ERROR: [1] PCSetUp_LU line 59
> /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> [1]PETSC ERROR: [1] PCSetUp line 894
> /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 304
> /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: [1] STSetUp_Sinvert line 96
> /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> [1]PETSC ERROR: [1] STSetUp line 233
> /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> [1]PETSC ERROR: [1] EPSSetUp line 104
> /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> [1]PETSC ERROR: [1] EPSSolve line 129
> /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> [1]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2
> Tue Oct 30 17:40:51 2018
> [1]PETSC ERROR: Configure options --download-mpich
> -with-scalar-type=complex --download-mumps --download-parmetis
> --download-metis --download-scalapack --download-fblaslapack
> --with-debugging=1 --download-superlu_dist --download-ptscotch
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
>
>
>
> the expected output is the following (on a compute node running
> petsc-3.9.2 and also on a mac laptop running petsc-3.10.2):
>
> $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4
> -eps_target -2e-3+1.01i
>
> Generalized eigenproblem stored in file.
>
>  Reading COMPLEX matrices from binary files...
>  Number of iterations of the method: 2
>  Number of linear iterations of the method: 27
>  Solution method: krylovschur
>
>  Number of requested eigenvalues: 4
>  Stopping condition: tol=1e-08, maxit=63157
>  Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL;
> iterations 2
>  ---------------------- --------------------
>             k             ||Ax-kBx||/||kx||
>  ---------------------- --------------------
>   -0.002806+1.009827i       2.00821e-19
>   -0.002980+1.008417i       8.08359e-17
>   -0.002676+1.011755i       9.49342e-18
>   -0.003201+1.007367i       1.50869e-16
>  ---------------------- --------------------
>
>
> Just in case, the matrices can be downloaded from here if any one wants to
> give them a try
> https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
> https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0
>

Its not the matrices. Would you be willing to reconfigure and test with
SuperLU_dist? Since it consistently fails
in the MUMPS factorization, it seems like it is either a bug in MUMPS or a
bug in our interface to MUMPS. Valgrind
should show us which one (as Barry suggests), but running with SuperLU_dist
should get you going in the mean time.

   Matt


> I tried different petsc/slepc versions to no avail, including an OS
> reinstall. So any help would be highly appreciated. Thanks in advance!
>
> Santiago
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/7efa5b64/attachment-0001.html>

From mfadams at lbl.gov  Tue Oct 30 13:51:47 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Oct 2018 14:51:47 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
	<CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
	<CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>
Message-ID: <CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>

The last two cases were cut off but that is OK.

Lets just do -pc_gamg_square_graph 0 and please add/do -ksp_view -
mg_levels_ksp_type richardson -pc_mg_levels 5 -gamg_est_ksp_type cg

Send the grep GAMG again, and the ksp_view output.

And then do this again with: -pc_gamg_agg_nsmooths

I think the eigen estimators are messed up on the coarse grids.


On Tue, Oct 30, 2018 at 12:39 PM Karin&NiKo <niko.karin at gmail.com> wrote:

> Ok, sorry for my misunderstanding  and thank you for the clarification.
>
> Le mar. 30 oct. 2018 13:55, Mark Adams <mfadams at lbl.gov> a ?crit :
>
>> Nicolas,
>>
>> Smoothed aggregation is fine with shells. see the original SA paper (
>> https://link.springer.com/article/10.1007/BF02238511).
>>
>> The rotational modes, which are the non-trivial modes that must be
>> supplied, are used in the interpolation.
>>
>> Mark
>>
>> On Tue, Oct 30, 2018 at 5:22 AM Karin&NiKo <niko.karin at gmail.com> wrote:
>>
>>> Manav,
>>>
>>> How are interpolated the rotational degrees of freedom? AFAIK, when
>>> using smoothed aggregation, the interpolation process tries to mimic linear
>>> interpolation, which can be OK for the displacement DOF but is not for the
>>> rotational DOF using some plate and shell formulations.
>>> This can explain poor convergence of a multilevel approach, which needs
>>> to restrict and extrapolate the unknowns. In order to check this
>>> hypothesis, you can try a test case with zero rotations.
>>>
>>> Nicolas
>>>
>>> Le lun. 29 oct. 2018 ? 22:13, Mark Adams via petsc-users <
>>> petsc-users at mcs.anl.gov> a ?crit :
>>>
>>>> * the two level results tell us that MG is not doing well on the coarse
>>>> grids. So the coarse grids are the problem.
>>>>
>>>> * Do not worry about timing now. Get the math correct. The two level
>>>> solve is not meant to be a solution just a diagnostic so don't try to
>>>> optimize it by squaring the graph. Use -pc_gamg_square_graph 0.
>>>>
>>>> * It looks like you don't need 4 smoothing steps but lets keep it and
>>>> we can dial it back later.
>>>>
>>>> * This table is interesting. First, you had about 12 iterations earlier
>>>> and I think your rtol was tighter than the default (so the iteration could
>>>> should go down not up). Do you know what change here?
>>>>
>>>> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
>>>> work. It is syntactic sugar to just add it to all levels like you did
>>>> manually.
>>>>
>>>> Anyway, these number look reasonable. It is interesting that 3 levels
>>>> ran well but the 4th level ran poorly. This implies we want to slow down
>>>> coarsening on these levels, but ...
>>>>
>>>> First can you please rerun this experiment with -pc_gamg_square_graph 0.
>>>>
>>>> Also, please run with -info. This is very noisy but you can grep on
>>>> "GAMG" and send that output to us (about 15 lines).
>>>>
>>>> Thanks,
>>>> Mark
>>>>
>>>>
>>>>
>>>> On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com>
>>>> wrote:
>>>>
>>>>> Barry,
>>>>>
>>>>>    Here are some quick numbers with the following options on 4 CPUs
>>>>> and 543,606 dofs:
>>>>>
>>>>> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>>>>>
>>>>>  #levels   |    #KSP Iters
>>>>> ???????????
>>>>>      2        |       18
>>>>>      3        |       18
>>>>>      4        |       40
>>>>>      5        |       59
>>>>>
>>>>> -Manav
>>>>>
>>>>>
>>>>> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>
>>>>>  Exactly how much does it increase with number of levels? Send a chart
>>>>> number of levels and number of iterations. With say -mg_levels_ksp_maxit 4
>>>>>
>>>>>   Thanks
>>>>>
>>>>>   Barry
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com>
>>>>> wrote:
>>>>>
>>>>> Thanks for the clarification.
>>>>>
>>>>> I also observed that the number of KSP iterations increases with an
>>>>> increase in the levels of AMG. Is this true, in general, for all/most
>>>>> applications?
>>>>>
>>>>> -Manav
>>>>>
>>>>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>
>>>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>>>>
>>>>> Thanks, Jed.
>>>>>
>>>>> The description says: ? Square the graph, ie. compute A'*A before
>>>>> aggregating it"
>>>>>
>>>>> What is A here?
>>>>>
>>>>>
>>>>> The original matrix, or its "graph" (your 6x6 blocks condensed to
>>>>> scalars).
>>>>>
>>>>> What is the impact of setting this to 0, which led to a very
>>>>> significant increase in the CPU time in my case?
>>>>>
>>>>>
>>>>> The aggregates are formed on the connectivity of your original matrix,
>>>>> so root nodes are aggregated only with their first neighbors, resulting
>>>>> in slower coarsening.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/51a40669/attachment.html>

From mfadams at lbl.gov  Tue Oct 30 14:17:03 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Tue, 30 Oct 2018 15:17:03 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
	<CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
	<CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>
	<CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>
Message-ID: <CADOhEh5oPcnHYueZPHMpyv+ZW8ALJQYYaq1yD=y2LJsbC12B6A@mail.gmail.com>

And you can also always use -mg_levels_esteig_ksp_type cg

On Tue, Oct 30, 2018 at 2:51 PM Mark Adams <mfadams at lbl.gov> wrote:

> The last two cases were cut off but that is OK.
>
> Lets just do -pc_gamg_square_graph 0 and please add/do -ksp_view -
> mg_levels_ksp_type richardson -pc_mg_levels 5 -gamg_est_ksp_type cg
>
> Send the grep GAMG again, and the ksp_view output.
>
> And then do this again with: -pc_gamg_agg_nsmooths
>
> I think the eigen estimators are messed up on the coarse grids.
>
>
> On Tue, Oct 30, 2018 at 12:39 PM Karin&NiKo <niko.karin at gmail.com> wrote:
>
>> Ok, sorry for my misunderstanding  and thank you for the clarification.
>>
>> Le mar. 30 oct. 2018 13:55, Mark Adams <mfadams at lbl.gov> a ?crit :
>>
>>> Nicolas,
>>>
>>> Smoothed aggregation is fine with shells. see the original SA paper (
>>> https://link.springer.com/article/10.1007/BF02238511).
>>>
>>> The rotational modes, which are the non-trivial modes that must be
>>> supplied, are used in the interpolation.
>>>
>>> Mark
>>>
>>> On Tue, Oct 30, 2018 at 5:22 AM Karin&NiKo <niko.karin at gmail.com> wrote:
>>>
>>>> Manav,
>>>>
>>>> How are interpolated the rotational degrees of freedom? AFAIK, when
>>>> using smoothed aggregation, the interpolation process tries to mimic linear
>>>> interpolation, which can be OK for the displacement DOF but is not for the
>>>> rotational DOF using some plate and shell formulations.
>>>> This can explain poor convergence of a multilevel approach, which needs
>>>> to restrict and extrapolate the unknowns. In order to check this
>>>> hypothesis, you can try a test case with zero rotations.
>>>>
>>>> Nicolas
>>>>
>>>> Le lun. 29 oct. 2018 ? 22:13, Mark Adams via petsc-users <
>>>> petsc-users at mcs.anl.gov> a ?crit :
>>>>
>>>>> * the two level results tell us that MG is not doing well on the
>>>>> coarse grids. So the coarse grids are the problem.
>>>>>
>>>>> * Do not worry about timing now. Get the math correct. The two level
>>>>> solve is not meant to be a solution just a diagnostic so don't try to
>>>>> optimize it by squaring the graph. Use -pc_gamg_square_graph 0.
>>>>>
>>>>> * It looks like you don't need 4 smoothing steps but lets keep it and
>>>>> we can dial it back later.
>>>>>
>>>>> * This table is interesting. First, you had about 12 iterations
>>>>> earlier and I think your rtol was tighter than the default (so the
>>>>> iteration could should go down not up). Do you know what change here?
>>>>>
>>>>> Note, even though -mg_levels_ksp_max_it is not in the ksp_view it does
>>>>> work. It is syntactic sugar to just add it to all levels like you did
>>>>> manually.
>>>>>
>>>>> Anyway, these number look reasonable. It is interesting that 3 levels
>>>>> ran well but the 4th level ran poorly. This implies we want to slow down
>>>>> coarsening on these levels, but ...
>>>>>
>>>>> First can you please rerun this experiment with -pc_gamg_square_graph
>>>>> 0.
>>>>>
>>>>> Also, please run with -info. This is very noisy but you can grep on
>>>>> "GAMG" and send that output to us (about 15 lines).
>>>>>
>>>>> Thanks,
>>>>> Mark
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 29, 2018 at 3:34 PM Manav Bhatia <bhatiamanav at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Barry,
>>>>>>
>>>>>>    Here are some quick numbers with the following options on 4 CPUs
>>>>>> and 543,606 dofs:
>>>>>>
>>>>>> -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 1 -pc_gamg_threshold 0.
>>>>>>
>>>>>>  #levels   |    #KSP Iters
>>>>>> ???????????
>>>>>>      2        |       18
>>>>>>      3        |       18
>>>>>>      4        |       40
>>>>>>      5        |       59
>>>>>>
>>>>>> -Manav
>>>>>>
>>>>>>
>>>>>> On Oct 29, 2018, at 2:06 PM, Smith, Barry F. <bsmith at mcs.anl.gov>
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>  Exactly how much does it increase with number of levels? Send a
>>>>>> chart number of levels and number of iterations. With say
>>>>>> -mg_levels_ksp_maxit 4
>>>>>>
>>>>>>   Thanks
>>>>>>
>>>>>>   Barry
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Oct 29, 2018, at 12:59 PM, Manav Bhatia <bhatiamanav at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>> Thanks for the clarification.
>>>>>>
>>>>>> I also observed that the number of KSP iterations increases with an
>>>>>> increase in the levels of AMG. Is this true, in general, for all/most
>>>>>> applications?
>>>>>>
>>>>>> -Manav
>>>>>>
>>>>>> On Oct 29, 2018, at 12:53 PM, Jed Brown <jed at jedbrown.org> wrote:
>>>>>>
>>>>>> Manav Bhatia <bhatiamanav at gmail.com> writes:
>>>>>>
>>>>>> Thanks, Jed.
>>>>>>
>>>>>> The description says: ? Square the graph, ie. compute A'*A before
>>>>>> aggregating it"
>>>>>>
>>>>>> What is A here?
>>>>>>
>>>>>>
>>>>>> The original matrix, or its "graph" (your 6x6 blocks condensed to
>>>>>> scalars).
>>>>>>
>>>>>> What is the impact of setting this to 0, which led to a very
>>>>>> significant increase in the CPU time in my case?
>>>>>>
>>>>>>
>>>>>> The aggregates are formed on the connectivity of your original matrix,
>>>>>> so root nodes are aggregated only with their first neighbors,
>>>>>> resulting
>>>>>> in slower coarsening.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/242fdba8/attachment.html>

From t.appel17 at imperial.ac.uk  Tue Oct 30 16:14:56 2018
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Tue, 30 Oct 2018 21:14:56 +0000
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
Message-ID: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>

Dear users,

Following a suggestion from Matthew Knepley I?ve been trying to apply fieldsplit/gamg for my set of PDEs but I?m still encountering issues despite various tests. pc_gamg simply won?t start.
Note that direct solvers always yield the correct, physical result.
Removing the fieldsplit to focus on the gamg bit and trying to solve the linear system on a modest size problem still gives, with

'-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres -pc_type gamg'

[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Petsc has generated inconsistent data
[3]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.

And since then, after adding '-pc_gamg_sym_graph true' I have been getting
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration

-ksp_chebyshev_esteig_noisy 0/1 does not change anything

Knowing that Chebyshev eigen estimator needs a positive spectrum I tried ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.

It seems that I have indeed eigenvalues of rather high magnitude in the spectrum of my operator without being able to determine the reason.
The eigenvectors look like small artifacts at the wall-inflow or wall-outflow corners with zero anywhere else but I do not know how to interpret this.
Equations are time-harmonic linearized Navier-Stokes to which a forcing is applied, there?s no time-marching.

Matrix is formed with a MPIAIJ type. The formulation is incompressible, in complex arithmetic and the 2D physical domain is mapped to a logically rectangular, regular collocated grid with a high-order finite difference method.
I determine the ownership of the rows/degrees of freedom of the matrix with PetscSplitOwnership and I?m not using DMDA.

The Fortran application code is memory-leak free and has undergone a strict verification/validation procedure for different variations of the PDEs.

If there?s any problem with the matrix what could help for the diagnostic? At this point I?m running out of ideas so I would really appreciate additional suggestions and discussions.

Thanks for your continued support,


Thibaut

From knepley at gmail.com  Tue Oct 30 18:12:35 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 30 Oct 2018 19:12:35 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
Message-ID: <CAMYG4GnRdMDs=3eaw589o7hYvi60wShHrO_EzY9q6pmHFiOpgQ@mail.gmail.com>

On Tue, Oct 30, 2018 at 5:21 PM Appel, Thibaut via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear users,
>
> Following a suggestion from Matthew Knepley I?ve been trying to apply
> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
> despite various tests. pc_gamg simply won?t start.
> Note that direct solvers always yield the correct, physical result.
> Removing the fieldsplit to focus on the gamg bit


This is a mistake I think because you want a simpler operator to try GAMG
on.

Can you go back to splitting the system, and try using GMRES as the
smoother. Its important to see
whether the smoother makes no progress, or the coarse correction stinks.

  Thanks,

     Matt


> and trying to solve the linear system on a modest size problem still
> gives, with
>
> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
> -pc_type gamg'
>
> [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3]PETSC ERROR: Petsc has generated inconsistent data
> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>
> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>
> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>
> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>
> It seems that I have indeed eigenvalues of rather high magnitude in the
> spectrum of my operator without being able to determine the reason.
> The eigenvectors look like small artifacts at the wall-inflow or
> wall-outflow corners with zero anywhere else but I do not know how to
> interpret this.
> Equations are time-harmonic linearized Navier-Stokes to which a forcing is
> applied, there?s no time-marching.
>
> Matrix is formed with a MPIAIJ type. The formulation is incompressible, in
> complex arithmetic and the 2D physical domain is mapped to a logically
> rectangular, regular collocated grid with a high-order finite difference
> method.
> I determine the ownership of the rows/degrees of freedom of the matrix
> with PetscSplitOwnership and I?m not using DMDA.
>
> The Fortran application code is memory-leak free and has undergone a
> strict verification/validation procedure for different variations of the
> PDEs.
>
> If there?s any problem with the matrix what could help for the diagnostic?
> At this point I?m running out of ideas so I would really appreciate
> additional suggestions and discussions.
>
> Thanks for your continued support,
>
>
> Thibaut



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181030/4b53d7e4/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 30 19:48:29 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 31 Oct 2018 00:48:29 +0000
Subject: [petsc-users] Segmentation violation
In-Reply-To: <CAOrd_SeEvxaGc9WCyi-Myu9bWfqgtvOCGttKyMvPinED8hEifQ@mail.gmail.com>
References: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
	<AAC2E358-F818-477F-8668-20DDC50330A7@anl.gov>
	<CAOrd_SeEvxaGc9WCyi-Myu9bWfqgtvOCGttKyMvPinED8hEifQ@mail.gmail.com>
Message-ID: <D898239D-337E-4825-A202-571BED1E0C3F@mcs.anl.gov>


  Yeah this doesn't look good for MUMPS but isn't for sure the problem either. 

   The valgrind output should be sent to the MUMPS developers.

   Hong, 

         Can you send this to the MUMPS developers and see what they say?

    Thanks

   Barry


> On Oct 30, 2018, at 2:04 PM, Santiago Andres Triana <repepo at gmail.com> wrote:
> 
> This is the output of
> mpiexec -n 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc -eps_nev 4 -eps_target -2e-3+1.01i -st_type sinvert
> 
> Generalized eigenproblem stored in file.
> 
>  Reading COMPLEX matrices from binary files...
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205 /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054 /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> [1]PETSC ERROR: [1] PCSetUp line 894 /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 304 /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: [1] STSetUp_Sinvert line 96 /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> [1]PETSC ERROR: [1] STSetUp line 233 /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> [1]PETSC ERROR: [1] EPSSetUp line 104 /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> [1]PETSC ERROR: [1] EPSSolve line 129 /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018 
> [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2 Tue Oct 30 19:42:18 2018
> [1]PETSC ERROR: Configure options --download-mpich -with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack --download-fblaslapack --with-debugging=1 --download-superlu_dist --download-ptscotch
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> 
> 
> 
> and one of the two valgrind logs (the other was empty):
> 
> ==63004== Use of uninitialised value of size 8
> ==63004==    at 0x694F8FF: zmumps_redistribution_ (zfac_distrib_distentry.F:367)
> ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
> ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
> ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
> ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
> ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
> ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
> ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
> ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
> ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
> ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
> ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
> ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
> ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
> ==63004==    by 0x10B6FD: main (ex7.c:134)
> ==63004== 
> ==63004== Invalid read of size 4
> ==63004==    at 0x694F8FF: zmumps_redistribution_ (zfac_distrib_distentry.F:367)
> ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
> ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
> ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
> ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
> ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
> ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
> ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
> ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
> ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
> ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
> ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
> ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
> ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
> ==63004==    by 0x10B6FD: main (ex7.c:134)
> ==63004==  Address 0xe5ffda3101296ca0 is not stack'd, malloc'd or (recently) free'd
> ==63004== 
> 
> 
> Hope it gives enough info. Thanks!
> 
> 
> 
> On Tue, Oct 30, 2018 at 6:50 PM Smith, Barry F. <bsmith at mcs.anl.gov> wrote:
> 
>    Can you run the code on the "failing" machine using valgrind? https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> 
>   Barry
> 
> 
> > On Oct 30, 2018, at 12:10 PM, Santiago Andres Triana via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > Hi petsc-users,
> > 
> > I am solving a generalized eigenvalue problem using ex7 in $SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
> > The program runs fine, with correct solutions on 12-core node and also on a mac laptop.
> > 
> > However, on a 16-core workstation running Debian testing (fresh install) and also a fresh install of petsc and slepc I get the following error:
> > 
> >  $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> > 
> > Generalized eigenproblem stored in file.
> > 
> >  Reading COMPLEX matrices from binary files...
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [1]PETSC ERROR: likely location of problem given in stack below
> > [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> > [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> > [1]PETSC ERROR:       is given.
> > [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205 /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> > [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054 /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> > [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> > [1]PETSC ERROR: [1] PCSetUp line 894 /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> > [1]PETSC ERROR: [1] KSPSetUp line 304 /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> > [1]PETSC ERROR: [1] STSetUp_Sinvert line 96 /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> > [1]PETSC ERROR: [1] STSetUp line 233 /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> > [1]PETSC ERROR: [1] EPSSetUp line 104 /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> > [1]PETSC ERROR: [1] EPSSolve line 129 /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> > [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [1]PETSC ERROR: Signal received
> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018 
> > [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2 Tue Oct 30 17:40:51 2018
> > [1]PETSC ERROR: Configure options --download-mpich -with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack --download-fblaslapack --with-debugging=1 --download-superlu_dist --download-ptscotch
> > [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> > 
> > 
> > 
> > the expected output is the following (on a compute node running petsc-3.9.2 and also on a mac laptop running petsc-3.10.2):
> > 
> > $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> > 
> > Generalized eigenproblem stored in file.
> > 
> >  Reading COMPLEX matrices from binary files...
> >  Number of iterations of the method: 2
> >  Number of linear iterations of the method: 27
> >  Solution method: krylovschur
> > 
> >  Number of requested eigenvalues: 4
> >  Stopping condition: tol=1e-08, maxit=63157
> >  Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL; iterations 2
> >  ---------------------- --------------------
> >             k             ||Ax-kBx||/||kx||
> >  ---------------------- --------------------
> >   -0.002806+1.009827i       2.00821e-19
> >   -0.002980+1.008417i       8.08359e-17
> >   -0.002676+1.011755i       9.49342e-18
> >   -0.003201+1.007367i       1.50869e-16
> >  ---------------------- --------------------
> > 
> > 
> > Just in case, the matrices can be downloaded from here if any one wants to give them a try
> > https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
> > https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0
> > 
> > 
> > I tried different petsc/slepc versions to no avail, including an OS reinstall. So any help would be highly appreciated. Thanks in advance!
> > 
> > Santiago
> 


From hzhang at mcs.anl.gov  Tue Oct 30 20:41:06 2018
From: hzhang at mcs.anl.gov (Zhang, Hong)
Date: Wed, 31 Oct 2018 01:41:06 +0000
Subject: [petsc-users] Segmentation violation
In-Reply-To: <D898239D-337E-4825-A202-571BED1E0C3F@mcs.anl.gov>
References: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
	<AAC2E358-F818-477F-8668-20DDC50330A7@anl.gov>
	<CAOrd_SeEvxaGc9WCyi-Myu9bWfqgtvOCGttKyMvPinED8hEifQ@mail.gmail.com>
	<D898239D-337E-4825-A202-571BED1E0C3F@mcs.anl.gov>
Message-ID: <CAGCphBt+uXkhKqH31xNA=qd8z1JQ7KdKKFmO0RTm3KdwHDTG1w@mail.gmail.com>

Santiago,
The shift '-eps_target -2e-3+1.01i' is very close to the eigenvalues. What happens if you pick a target little away from your eigenvalues?
I suspect mumps encounters a zero pivot during numerical factorization. There are options to handle it, but I need matrices A and B to investigate.
I am not sure if the problem comes from memory bug.
Anyway, I'm cc'ing mumps developers here.

Hong

On Tue, Oct 30, 2018 at 8:09 PM Smith, Barry F. via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:

  Yeah this doesn't look good for MUMPS but isn't for sure the problem either.

   The valgrind output should be sent to the MUMPS developers.

   Hong,

         Can you send this to the MUMPS developers and see what they say?

    Thanks

   Barry


> On Oct 30, 2018, at 2:04 PM, Santiago Andres Triana <repepo at gmail.com<mailto:repepo at gmail.com>> wrote:
>
> This is the output of
> mpiexec -n 2 valgrind --tool=memcheck -q --num-callers=20 --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc -eps_nev 4 -eps_target -2e-3+1.01i -st_type sinvert
>
> Generalized eigenproblem stored in file.
>
>  Reading COMPLEX matrices from binary files...
> [1]PETSC ERROR: ------------------------------------------------------------------------
> [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> [1]PETSC ERROR: likely location of problem given in stack below
> [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> [1]PETSC ERROR:       is given.
> [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205 /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054 /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> [1]PETSC ERROR: [1] PCSetUp line 894 /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> [1]PETSC ERROR: [1] KSPSetUp line 304 /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> [1]PETSC ERROR: [1] STSetUp_Sinvert line 96 /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> [1]PETSC ERROR: [1] STSetUp line 233 /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> [1]PETSC ERROR: [1] EPSSetUp line 104 /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> [1]PETSC ERROR: [1] EPSSolve line 129 /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [1]PETSC ERROR: Signal received
> [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2 Tue Oct 30 19:42:18 2018
> [1]PETSC ERROR: Configure options --download-mpich -with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack --download-fblaslapack --with-debugging=1 --download-superlu_dist --download-ptscotch
> [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
>
>
>
> and one of the two valgrind logs (the other was empty):
>
> ==63004== Use of uninitialised value of size 8
> ==63004==    at 0x694F8FF: zmumps_redistribution_ (zfac_distrib_distentry.F:367)
> ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
> ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
> ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
> ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
> ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
> ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
> ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
> ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
> ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
> ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
> ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
> ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
> ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
> ==63004==    by 0x10B6FD: main (ex7.c:134)
> ==63004==
> ==63004== Invalid read of size 4
> ==63004==    at 0x694F8FF: zmumps_redistribution_ (zfac_distrib_distentry.F:367)
> ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
> ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
> ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
> ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
> ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
> ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
> ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
> ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
> ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
> ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
> ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
> ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
> ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
> ==63004==    by 0x10B6FD: main (ex7.c:134)
> ==63004==  Address 0xe5ffda3101296ca0 is not stack'd, malloc'd or (recently) free'd
> ==63004==
>
>
> Hope it gives enough info. Thanks!
>
>
>
> On Tue, Oct 30, 2018 at 6:50 PM Smith, Barry F. <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>> wrote:
>
>    Can you run the code on the "failing" machine using valgrind? https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>
>   Barry
>
>
> > On Oct 30, 2018, at 12:10 PM, Santiago Andres Triana via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
> >
> > Hi petsc-users,
> >
> > I am solving a generalized eigenvalue problem using ex7 in $SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
> > The program runs fine, with correct solutions on 12-core node and also on a mac laptop.
> >
> > However, on a 16-core workstation running Debian testing (fresh install) and also a fresh install of petsc and slepc I get the following error:
> >
> >  $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> >
> > Generalized eigenproblem stored in file.
> >
> >  Reading COMPLEX matrices from binary files...
> > [1]PETSC ERROR: ------------------------------------------------------------------------
> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range
> > [1]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger
> > [1]PETSC ERROR: or see http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac OS X to find memory corruption errors
> > [1]PETSC ERROR: likely location of problem given in stack below
> > [1]PETSC ERROR: ---------------------  Stack Frames ------------------------------------
> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not available,
> > [1]PETSC ERROR:       INSTEAD the line number of the start of the function
> > [1]PETSC ERROR:       is given.
> > [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205 /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
> > [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054 /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
> > [1]PETSC ERROR: [1] PCSetUp_LU line 59 /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
> > [1]PETSC ERROR: [1] PCSetUp line 894 /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
> > [1]PETSC ERROR: [1] KSPSetUp line 304 /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
> > [1]PETSC ERROR: [1] STSetUp_Sinvert line 96 /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
> > [1]PETSC ERROR: [1] STSetUp line 233 /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
> > [1]PETSC ERROR: [1] EPSSetUp line 104 /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
> > [1]PETSC ERROR: [1] EPSSolve line 129 /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
> > [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> > [1]PETSC ERROR: Signal received
> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> > [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
> > [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by spin2 Tue Oct 30 17:40:51 2018
> > [1]PETSC ERROR: Configure options --download-mpich -with-scalar-type=complex --download-mumps --download-parmetis --download-metis --download-scalapack --download-fblaslapack --with-debugging=1 --download-superlu_dist --download-ptscotch
> > [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
> >
> >
> >
> > the expected output is the following (on a compute node running petsc-3.9.2 and also on a mac laptop running petsc-3.10.2):
> >
> > $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert -eps_nev 4 -eps_target -2e-3+1.01i
> >
> > Generalized eigenproblem stored in file.
> >
> >  Reading COMPLEX matrices from binary files...
> >  Number of iterations of the method: 2
> >  Number of linear iterations of the method: 27
> >  Solution method: krylovschur
> >
> >  Number of requested eigenvalues: 4
> >  Stopping condition: tol=1e-08, maxit=63157
> >  Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL; iterations 2
> >  ---------------------- --------------------
> >             k             ||Ax-kBx||/||kx||
> >  ---------------------- --------------------
> >   -0.002806+1.009827i       2.00821e-19
> >   -0.002980+1.008417i       8.08359e-17
> >   -0.002676+1.011755i       9.49342e-18
> >   -0.003201+1.007367i       1.50869e-16
> >  ---------------------- --------------------
> >
> >
> > Just in case, the matrices can be downloaded from here if any one wants to give them a try
> > https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
> > https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0
> >
> >
> > I tried different petsc/slepc versions to no avail, including an OS reinstall. So any help would be highly appreciated. Thanks in advance!
> >
> > Santiago
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/149427fb/attachment-0001.html>

From repepo at gmail.com  Wed Oct 31 04:05:31 2018
From: repepo at gmail.com (Santiago Andres Triana)
Date: Wed, 31 Oct 2018 10:05:31 +0100
Subject: [petsc-users] Segmentation violation
In-Reply-To: <CAGCphBt+uXkhKqH31xNA=qd8z1JQ7KdKKFmO0RTm3KdwHDTG1w@mail.gmail.com>
References: <CAOrd_SdiG-+GVXQ-n_DrssX1LDreceiov3u_HT9DvMkP9-jaQA@mail.gmail.com>
	<AAC2E358-F818-477F-8668-20DDC50330A7@anl.gov>
	<CAOrd_SeEvxaGc9WCyi-Myu9bWfqgtvOCGttKyMvPinED8hEifQ@mail.gmail.com>
	<D898239D-337E-4825-A202-571BED1E0C3F@mcs.anl.gov>
	<CAGCphBt+uXkhKqH31xNA=qd8z1JQ7KdKKFmO0RTm3KdwHDTG1w@mail.gmail.com>
Message-ID: <CAOrd_ScEffZ9tCg9Kx+j3PEehVsA5=5Rj9E41BzH4orEcVwozQ@mail.gmail.com>

Hi Hong,

You can find the matrices here:
https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0

Changing the target value leads to the same error. What is strange is that
this works without a problem on two other machines. But in my main
workstation (the one I use for developing and testing) it fails :(

Thanks so much for your help!
Santiago



On Wed, Oct 31, 2018 at 2:48 AM Zhang, Hong <hzhang at mcs.anl.gov> wrote:

> Santiago,
> The shift '-eps_target -2e-3+1.01i' is very close to the eigenvalues. What
> happens if you pick a target little away from your eigenvalues?
> I suspect mumps encounters a zero pivot during numerical factorization.
> There are options to handle it, but I need matrices A and B to investigate.
> I am not sure if the problem comes from memory bug.
> Anyway, I'm cc'ing mumps developers here.
>
> Hong
>
> On Tue, Oct 30, 2018 at 8:09 PM Smith, Barry F. via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>>
>>   Yeah this doesn't look good for MUMPS but isn't for sure the problem
>> either.
>>
>>    The valgrind output should be sent to the MUMPS developers.
>>
>>    Hong,
>>
>>          Can you send this to the MUMPS developers and see what they say?
>>
>>     Thanks
>>
>>    Barry
>>
>>
>> > On Oct 30, 2018, at 2:04 PM, Santiago Andres Triana <repepo at gmail.com>
>> wrote:
>> >
>> > This is the output of
>> > mpiexec -n 2 valgrind --tool=memcheck -q --num-callers=20
>> --log-file=valgrind.log.%p ./ex7 -malloc off -f1 A.petsc -f2 B.petsc
>> -eps_nev 4 -eps_target -2e-3+1.01i -st_type sinvert
>> >
>> > Generalized eigenproblem stored in file.
>> >
>> >  Reading COMPLEX matrices from binary files...
>> > [1]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> > [1]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > [1]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple Mac
>> OS X to find memory corruption errors
>> > [1]PETSC ERROR: likely location of problem given in stack below
>> > [1]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> > [1]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> > [1]PETSC ERROR:       is given.
>> > [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205
>> /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
>> > [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054
>> /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
>> > [1]PETSC ERROR: [1] PCSetUp_LU line 59
>> /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
>> > [1]PETSC ERROR: [1] PCSetUp line 894
>> /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
>> > [1]PETSC ERROR: [1] KSPSetUp line 304
>> /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>> > [1]PETSC ERROR: [1] STSetUp_Sinvert line 96
>> /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
>> > [1]PETSC ERROR: [1] STSetUp line 233
>> /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
>> > [1]PETSC ERROR: [1] EPSSetUp line 104
>> /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
>> > [1]PETSC ERROR: [1] EPSSolve line 129
>> /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
>> > [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > [1]PETSC ERROR: Signal received
>> > [1]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>> for trouble shooting.
>> > [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>> > [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by
>> spin2 Tue Oct 30 19:42:18 2018
>> > [1]PETSC ERROR: Configure options --download-mpich
>> -with-scalar-type=complex --download-mumps --download-parmetis
>> --download-metis --download-scalapack --download-fblaslapack
>> --with-debugging=1 --download-superlu_dist --download-ptscotch
>> > [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
>> >
>> >
>> >
>> > and one of the two valgrind logs (the other was empty):
>> >
>> > ==63004== Use of uninitialised value of size 8
>> > ==63004==    at 0x694F8FF: zmumps_redistribution_
>> (zfac_distrib_distentry.F:367)
>> > ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
>> > ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
>> > ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
>> > ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
>> > ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
>> > ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
>> > ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
>> > ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
>> > ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
>> > ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
>> > ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
>> > ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
>> > ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
>> > ==63004==    by 0x10B6FD: main (ex7.c:134)
>> > ==63004==
>> > ==63004== Invalid read of size 4
>> > ==63004==    at 0x694F8FF: zmumps_redistribution_
>> (zfac_distrib_distentry.F:367)
>> > ==63004==    by 0x68E1266: zmumps_fac_driver_ (zfac_driver.F:1777)
>> > ==63004==    by 0x6869F63: zmumps_ (zmumps_driver.F:1686)
>> > ==63004==    by 0x6861B64: zmumps_f77_ (zmumps_f77.F:267)
>> > ==63004==    by 0x685FB43: zmumps_c (mumps_c.c:417)
>> > ==63004==    by 0x5B741CD: MatFactorNumeric_MUMPS (mumps.c:1227)
>> > ==63004==    by 0x53C3DDB: MatLUFactorNumeric (matrix.c:3065)
>> > ==63004==    by 0x626E652: PCSetUp_LU (lu.c:131)
>> > ==63004==    by 0x6387B8D: PCSetUp (precon.c:932)
>> > ==63004==    by 0x649CD41: KSPSetUp (itfunc.c:391)
>> > ==63004==    by 0x4A0E3F7: STSetUp_Sinvert (sinvert.c:132)
>> > ==63004==    by 0x4A4033F: STSetUp (stsolve.c:271)
>> > ==63004==    by 0x4B586FB: EPSSetUp (epssetup.c:263)
>> > ==63004==    by 0x4B5D43A: EPSSolve (epssolve.c:135)
>> > ==63004==    by 0x10B6FD: main (ex7.c:134)
>> > ==63004==  Address 0xe5ffda3101296ca0 is not stack'd, malloc'd or
>> (recently) free'd
>> > ==63004==
>> >
>> >
>> > Hope it gives enough info. Thanks!
>> >
>> >
>> >
>> > On Tue, Oct 30, 2018 at 6:50 PM Smith, Barry F. <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> >    Can you run the code on the "failing" machine using valgrind?
>> https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> >
>> >   Barry
>> >
>> >
>> > > On Oct 30, 2018, at 12:10 PM, Santiago Andres Triana via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>> > >
>> > > Hi petsc-users,
>> > >
>> > > I am solving a generalized eigenvalue problem using ex7 in
>> $SLEPC_DIR/src/eps/examples/tutorials/. I provide the A and B matrices.
>> > > The program runs fine, with correct solutions on 12-core node and
>> also on a mac laptop.
>> > >
>> > > However, on a 16-core workstation running Debian testing (fresh
>> install) and also a fresh install of petsc and slepc I get the following
>> error:
>> > >
>> > >  $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert
>> -eps_nev 4 -eps_target -2e-3+1.01i
>> > >
>> > > Generalized eigenproblem stored in file.
>> > >
>> > >  Reading COMPLEX matrices from binary files...
>> > > [1]PETSC ERROR:
>> ------------------------------------------------------------------------
>> > > [1]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation,
>> probably memory access out of range
>> > > [1]PETSC ERROR: Try option -start_in_debugger or
>> -on_error_attach_debugger
>> > > [1]PETSC ERROR: or see
>> http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
>> > > [1]PETSC ERROR: or try http://valgrind.org on GNU/linux and Apple
>> Mac OS X to find memory corruption errors
>> > > [1]PETSC ERROR: likely location of problem given in stack below
>> > > [1]PETSC ERROR: ---------------------  Stack Frames
>> ------------------------------------
>> > > [1]PETSC ERROR: Note: The EXACT line numbers in the stack are not
>> available,
>> > > [1]PETSC ERROR:       INSTEAD the line number of the start of the
>> function
>> > > [1]PETSC ERROR:       is given.
>> > > [1]PETSC ERROR: [1] MatFactorNumeric_MUMPS line 1205
>> /home/spin2/petsc-3.10.2/src/mat/impls/aij/mpi/mumps/mumps.c
>> > > [1]PETSC ERROR: [1] MatLUFactorNumeric line 3054
>> /home/spin2/petsc-3.10.2/src/mat/interface/matrix.c
>> > > [1]PETSC ERROR: [1] PCSetUp_LU line 59
>> /home/spin2/petsc-3.10.2/src/ksp/pc/impls/factor/lu/lu.c
>> > > [1]PETSC ERROR: [1] PCSetUp line 894
>> /home/spin2/petsc-3.10.2/src/ksp/pc/interface/precon.c
>> > > [1]PETSC ERROR: [1] KSPSetUp line 304
>> /home/spin2/petsc-3.10.2/src/ksp/ksp/interface/itfunc.c
>> > > [1]PETSC ERROR: [1] STSetUp_Sinvert line 96
>> /home/spin2/slepc-3.10.1/src/sys/classes/st/impls/sinvert/sinvert.c
>> > > [1]PETSC ERROR: [1] STSetUp line 233
>> /home/spin2/slepc-3.10.1/src/sys/classes/st/interface/stsolve.c
>> > > [1]PETSC ERROR: [1] EPSSetUp line 104
>> /home/spin2/slepc-3.10.1/src/eps/interface/epssetup.c
>> > > [1]PETSC ERROR: [1] EPSSolve line 129
>> /home/spin2/slepc-3.10.1/src/eps/interface/epssolve.c
>> > > [1]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> > > [1]PETSC ERROR: Signal received
>> > > [1]PETSC ERROR: See
>> http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
>> > > [1]PETSC ERROR: Petsc Release Version 3.10.2, Oct, 09, 2018
>> > > [1]PETSC ERROR: ./ex7 on a arch-linux2-c-opt named wobble-wkst-as by
>> spin2 Tue Oct 30 17:40:51 2018
>> > > [1]PETSC ERROR: Configure options --download-mpich
>> -with-scalar-type=complex --download-mumps --download-parmetis
>> --download-metis --download-scalapack --download-fblaslapack
>> --with-debugging=1 --download-superlu_dist --download-ptscotch
>> > > [1]PETSC ERROR: #1 User provided function() line 0 in  unknown file
>> > > application called MPI_Abort(MPI_COMM_WORLD, 59) - process 1
>> > >
>> > >
>> > >
>> > > the expected output is the following (on a compute node running
>> petsc-3.9.2 and also on a mac laptop running petsc-3.10.2):
>> > >
>> > > $ mpiexec -n 2 ./ex7 -f1 A.petsc -f2 B.petsc -st_type sinvert
>> -eps_nev 4 -eps_target -2e-3+1.01i
>> > >
>> > > Generalized eigenproblem stored in file.
>> > >
>> > >  Reading COMPLEX matrices from binary files...
>> > >  Number of iterations of the method: 2
>> > >  Number of linear iterations of the method: 27
>> > >  Solution method: krylovschur
>> > >
>> > >  Number of requested eigenvalues: 4
>> > >  Stopping condition: tol=1e-08, maxit=63157
>> > >  Linear eigensolve converged (4 eigenpairs) due to CONVERGED_TOL;
>> iterations 2
>> > >  ---------------------- --------------------
>> > >             k             ||Ax-kBx||/||kx||
>> > >  ---------------------- --------------------
>> > >   -0.002806+1.009827i       2.00821e-19
>> > >   -0.002980+1.008417i       8.08359e-17
>> > >   -0.002676+1.011755i       9.49342e-18
>> > >   -0.003201+1.007367i       1.50869e-16
>> > >  ---------------------- --------------------
>> > >
>> > >
>> > > Just in case, the matrices can be downloaded from here if any one
>> wants to give them a try
>> > > https://www.dropbox.com/s/ejpa9owkv8tjnwi/A.petsc?dl=0
>> > > https://www.dropbox.com/s/urjtxaezl0cv3om/B.petsc?dl=0
>> > >
>> > >
>> > > I tried different petsc/slepc versions to no avail, including an OS
>> reinstall. So any help would be highly appreciated. Thanks in advance!
>> > >
>> > > Santiago
>> >
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/6db39c40/attachment.html>

From xing0029 at flinders.edu.au  Wed Oct 31 05:45:33 2018
From: xing0029 at flinders.edu.au (Wenjin Xing)
Date: Wed, 31 Oct 2018 10:45:33 +0000
Subject: [petsc-users] (no subject)
Message-ID: <SY3PR01MB15458011971A0417D8649F0084CD0@SY3PR01MB1545.ausprd01.prod.outlook.com>

My issue is summarized in the picture and posted in the link https://scicomp.stackexchange.com/questions/30458/what-does-the-error-this-matrix-type-does-not-have-a-find-zero-diagonals-define?noredirect=1#comment56074_30458

[cid:image001.png at 01D4715E.DAED5B40]

Kind regards
Wenjin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/1f4bf9a0/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 62890 bytes
Desc: image001.png
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/1f4bf9a0/attachment-0001.png>

From t.appel17 at imperial.ac.uk  Wed Oct 31 08:14:48 2018
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Wed, 31 Oct 2018 13:14:48 +0000
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <CAMYG4GnRdMDs=3eaw589o7hYvi60wShHrO_EzY9q6pmHFiOpgQ@mail.gmail.com>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CAMYG4GnRdMDs=3eaw589o7hYvi60wShHrO_EzY9q6pmHFiOpgQ@mail.gmail.com>
Message-ID: <7f6d9b45-3c9d-a7e8-7356-5585229c9f52@imperial.ac.uk>

Hi Matthew,

Which database option are you referring to?

I tried to add -fieldsplit_mg_levels_ksp_type gmres (and 
-fieldsplit_mg_levels_ksp_max_it 4 for another run) to my options (cf. 
below) which starts the iterations but it takes 1 hour for PETSc to do 
13 of them so it must be wrong.

Reminder: my baseline database options line reads

-ksp_view_pre -ksp_monitor -ksp_converged_reason \
-ksp_rtol 1.0e-8 -ksp_gmres_restart 300 \
-ksp_type fgmres \
-pc_type fieldsplit \
-pc_fieldsplit_type multiplicative \
-pc_fieldsplit_block_size 3 \
-pc_fieldsplit_0_fields 0?? \
-pc_fieldsplit_1_fields 1?? \
-pc_fieldsplit_2_fields 2?? \
-fieldsplit_pc_type gamg??? \
-fieldsplit_ksp_type gmres? \
-fieldsplit_ksp_rtol 1.0e-8

which gives

KSP Object: 1 MPI processes
 ? type: fgmres
 ??? restart=300, using Classical (unmodified) Gram-Schmidt 
Orthogonalization with no iterative refinement
 ??? happy breakdown tolerance 1e-30
 ? maximum iterations=10000, initial guess is zero
 ? tolerances:? relative=1e-08, absolute=1e-50, divergence=10000.
 ? left preconditioning
 ? using DEFAULT norm type for convergence test
PC Object: 1 MPI processes
 ? type: fieldsplit
 ? PC has not been set up so information may be incomplete
 ??? FieldSplit with MULTIPLICATIVE composition: total splits = 3, 
blocksize = 3
 ??? Solver info for each split is in the following KSP objects:
 ? Split number 0 Fields? 0
 ? KSP Object: (fieldsplit_0_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_0_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? Split number 1 Fields? 1
 ? KSP Object: (fieldsplit_1_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_1_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? Split number 2 Fields? 2
 ? KSP Object: (fieldsplit_2_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_2_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? linear system matrix = precond matrix:
 ? Mat Object: 1 MPI processes
 ??? type: seqaij
 ??? rows=52500, cols=52500
 ??? total: nonzeros=1127079, allocated nonzeros=1128624
 ??? total number of mallocs used during MatSetValues calls =0
 ????? not using I-node routines
 ? 0 KSP Residual norm 3.583290589961e+00
[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration 
0[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
[0]PETSC ERROR: Configure options --PETSC_ARCH=msi_cplx_debug 
--with-scalar-type=complex --with-precision=double --with-debugging=1 
--with-valgrind=1 --with-debugger=gdb --with-fortran-kernels=1 
--download-mpich --download-hwloc --download-fblaslapack 
--download-scalapack --download-metis --download-parmetis 
--download-ptscotch --download-mumps --download-slepc
[0]PETSC ERROR: #1 KSPSolve_Chebyshev() line 381 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/impls/cheby/cheby.c
[0]PETSC ERROR: #2 KSPSolve() line 780 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #3 PCMGMCycle_Private() line 20 in 
/home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #4 PCApply_MG() line 377 in 
/home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #5 PCApply() line 462 in 
/home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #6 KSP_PCApply() line 281 in 
/home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #7 KSPInitialResidual() line 67 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/interface/itres.c
[0]PETSC ERROR: #8 KSPSolve_GMRES() line 233 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/gmres.c
[0]PETSC ERROR: #9 KSPSolve() line 780 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #10 PCApply_FieldSplit() line 1107 in 
/home/thibaut/Packages/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
[0]PETSC ERROR: #11 PCApply() line 462 in 
/home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #12 KSP_PCApply() line 281 in 
/home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
[0]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: #15 KSPSolve() line 780 in 
/home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c


Thibaut



On 30/10/2018 23:12, Matthew Knepley wrote:
> On Tue, Oct 30, 2018 at 5:21 PM Appel, Thibaut via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Dear users,
>
>     Following a suggestion from Matthew Knepley I?ve been trying to
>     apply fieldsplit/gamg for my set of PDEs but I?m still
>     encountering issues despite various tests. pc_gamg simply won?t start.
>     Note that direct solvers always yield the correct, physical result.
>     Removing the fieldsplit to focus on the gamg bit 
>
>
> This is a mistake I think because you want a simpler operator to try 
> GAMG on.
>
> Can you go back to splitting the system, and try using GMRES as the 
> smoother. Its important to see
> whether the smoother makes no progress, or the coarse correction stinks.
>
> ? Thanks,
>
> ? ? ?Matt
>
>     and trying to solve the linear system on a modest size problem
>     still gives, with
>
>     '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type
>     gmres -pc_type gamg'
>
>     [3]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [3]PETSC ERROR: Petsc has generated inconsistent data
>     [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>     '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>     '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>
>     And since then, after adding '-pc_gamg_sym_graph true' I have been
>     getting
>     [0]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [0]PETSC ERROR: Petsc has generated inconsistent data
>     [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>
>     -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>
>     Knowing that Chebyshev eigen estimator needs a positive spectrum I
>     tried ?-mg_levels_ksp_type gmres? but iterations would just go on
>     endlessly.
>
>     It seems that I have indeed eigenvalues of rather high magnitude
>     in the spectrum of my operator without being able to determine the
>     reason.
>     The eigenvectors look like small artifacts at the wall-inflow or
>     wall-outflow corners with zero anywhere else but I do not know how
>     to interpret this.
>     Equations are time-harmonic linearized Navier-Stokes to which a
>     forcing is applied, there?s no time-marching.
>
>     Matrix is formed with a MPIAIJ type. The formulation is
>     incompressible, in complex arithmetic and the 2D physical domain
>     is mapped to a logically rectangular, regular collocated grid with
>     a high-order finite difference method.
>     I determine the ownership of the rows/degrees of freedom of the
>     matrix with PetscSplitOwnership and I?m not using DMDA.
>
>     The Fortran application code is memory-leak free and has undergone
>     a strict verification/validation procedure for different
>     variations of the PDEs.
>
>     If there?s any problem with the matrix what could help for the
>     diagnostic? At this point I?m running out of ideas so I would
>     really appreciate additional suggestions and discussions.
>
>     Thanks for your continued support,
>
>
>     Thibaut
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/ 
> <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/317e083d/attachment.html>

From mfadams at lbl.gov  Wed Oct 31 11:43:54 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 31 Oct 2018 12:43:54 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
Message-ID: <CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>

On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Dear users,
>
> Following a suggestion from Matthew Knepley I?ve been trying to apply
> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
> despite various tests. pc_gamg simply won?t start.
> Note that direct solvers always yield the correct, physical result.
> Removing the fieldsplit to focus on the gamg bit and trying to solve the
> linear system on a modest size problem still gives, with
>
> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
> -pc_type gamg'
>
> [3]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [3]PETSC ERROR: Petsc has generated inconsistent data
> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>
> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>
> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>
> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>

This is OK, but you need to use '-ksp_type *f*gmres' (this could be why it
is failing ...).

It looks like your matrix is 1) just the velocity field and 2) very
unsymmetric (eg, convection dominated). I would start with
?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.

I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths 0'


>
> It seems that I have indeed eigenvalues of rather high magnitude in the
> spectrum of my operator without being able to determine the reason.
> The eigenvectors look like small artifacts at the wall-inflow or
> wall-outflow corners with zero anywhere else but I do not know how to
> interpret this.
> Equations are time-harmonic linearized Navier-Stokes to which a forcing is
> applied, there?s no time-marching.
>

You mean you are in frequency domain?


>
> Matrix is formed with a MPIAIJ type. The formulation is incompressible, in
> complex arithmetic and the 2D physical domain is mapped to a logically
> rectangular,


This kind of messes up the null space that AMG depends on but AMG theory is
gone for NS anyway.


> regular collocated grid with a high-order finite difference method.
> I determine the ownership of the rows/degrees of freedom of the matrix
> with PetscSplitOwnership and I?m not using DMDA.
>

Our iterative solvers are probably not going to work well on this but you
should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). You need
to configure PETSc to download hypre.

Mark


>
> The Fortran application code is memory-leak free and has undergone a
> strict verification/validation procedure for different variations of the
> PDEs.
>
> If there?s any problem with the matrix what could help for the diagnostic?
> At this point I?m running out of ideas so I would really appreciate
> additional suggestions and discussions.
>
> Thanks for your continued support,
>
>
> Thibaut
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/eb6d3e78/attachment-0001.html>

From mfadams at lbl.gov  Wed Oct 31 11:50:24 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 31 Oct 2018 12:50:24 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <7f6d9b45-3c9d-a7e8-7356-5585229c9f52@imperial.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CAMYG4GnRdMDs=3eaw589o7hYvi60wShHrO_EzY9q6pmHFiOpgQ@mail.gmail.com>
	<7f6d9b45-3c9d-a7e8-7356-5585229c9f52@imperial.ac.uk>
Message-ID: <CADOhEh5E5Hz9p1A4vT3CkvWBjZvqyp15_Ewnt7az3-kuY=Av4Q@mail.gmail.com>

Again, you probably want to avoid Cheby. with  ?-mg_levels_ksp_type
richardson -mg_levels_pc_type sor? with the proper prefix.

I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one
block, and hence be a subpc. I'm not up on fieldsplit syntax.

On Wed, Oct 31, 2018 at 9:22 AM Thibaut Appel via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi Matthew,
>
> Which database option are you referring to?
>
> I tried to add -fieldsplit_mg_levels_ksp_type gmres (and
> -fieldsplit_mg_levels_ksp_max_it 4 for another run) to my options (cf.
> below) which starts the iterations but it takes 1 hour for PETSc to do 13
> of them so it must be wrong.
>
> Reminder: my baseline database options line reads
>
> -ksp_view_pre -ksp_monitor -ksp_converged_reason \
> -ksp_rtol 1.0e-8 -ksp_gmres_restart 300 \
> -ksp_type fgmres \
> -pc_type fieldsplit \
> -pc_fieldsplit_type multiplicative \
> -pc_fieldsplit_block_size 3 \
> -pc_fieldsplit_0_fields 0   \
> -pc_fieldsplit_1_fields 1   \
> -pc_fieldsplit_2_fields 2   \
> -fieldsplit_pc_type gamg    \
> -fieldsplit_ksp_type gmres  \
> -fieldsplit_ksp_rtol 1.0e-8
>
> which gives
>
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=300, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using DEFAULT norm type for convergence test
> PC Object: 1 MPI processes
>   type: fieldsplit
>   PC has not been set up so information may be incomplete
>     FieldSplit with MULTIPLICATIVE composition: total splits = 3,
> blocksize = 3
>     Solver info for each split is in the following KSP objects:
>   Split number 0 Fields  0
>   KSP Object: (fieldsplit_0_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_0_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 1 Fields  1
>   KSP Object: (fieldsplit_1_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_1_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 2 Fields  2
>   KSP Object: (fieldsplit_2_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_2_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=52500, cols=52500
>     total: nonzeros=1127079, allocated nonzeros=1128624
>     total number of mallocs used during MatSetValues calls =0
>       not using I-node routines
>   0 KSP Residual norm 3.583290589961e+00
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
> 0[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [0]PETSC ERROR: Configure options --PETSC_ARCH=msi_cplx_debug
> --with-scalar-type=complex --with-precision=double --with-debugging=1
> --with-valgrind=1 --with-debugger=gdb --with-fortran-kernels=1
> --download-mpich --download-hwloc --download-fblaslapack
> --download-scalapack --download-metis --download-parmetis
> --download-ptscotch --download-mumps --download-slepc
> [0]PETSC ERROR: #1 KSPSolve_Chebyshev() line 381 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #2 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #3 PCMGMCycle_Private() line 20 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #4 PCApply_MG() line 377 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #5 PCApply() line 462 in
> /home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #6 KSP_PCApply() line 281 in
> /home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #7 KSPInitialResidual() line 67 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 233 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #9 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #10 PCApply_FieldSplit() line 1107 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #11 PCApply() line 462 in
> /home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #12 KSP_PCApply() line 281 in
> /home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #15 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
>
>
> Thibaut
>
>
>
> On 30/10/2018 23:12, Matthew Knepley wrote:
>
> On Tue, Oct 30, 2018 at 5:21 PM Appel, Thibaut via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear users,
>>
>> Following a suggestion from Matthew Knepley I?ve been trying to apply
>> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
>> despite various tests. pc_gamg simply won?t start.
>> Note that direct solvers always yield the correct, physical result.
>> Removing the fieldsplit to focus on the gamg bit
>
>
> This is a mistake I think because you want a simpler operator to try GAMG
> on.
>
> Can you go back to splitting the system, and try using GMRES as the
> smoother. Its important to see
> whether the smoother makes no progress, or the coarse correction stinks.
>
>   Thanks,
>
>      Matt
>
>
>> and trying to solve the linear system on a modest size problem still
>> gives, with
>>
>> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
>> -pc_type gamg'
>>
>> [3]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [3]PETSC ERROR: Petsc has generated inconsistent data
>> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>>
>> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>>
>> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>>
>> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
>> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>>
>> It seems that I have indeed eigenvalues of rather high magnitude in the
>> spectrum of my operator without being able to determine the reason.
>> The eigenvectors look like small artifacts at the wall-inflow or
>> wall-outflow corners with zero anywhere else but I do not know how to
>> interpret this.
>> Equations are time-harmonic linearized Navier-Stokes to which a forcing
>> is applied, there?s no time-marching.
>>
>> Matrix is formed with a MPIAIJ type. The formulation is incompressible,
>> in complex arithmetic and the 2D physical domain is mapped to a logically
>> rectangular, regular collocated grid with a high-order finite difference
>> method.
>> I determine the ownership of the rows/degrees of freedom of the matrix
>> with PetscSplitOwnership and I?m not using DMDA.
>>
>> The Fortran application code is memory-leak free and has undergone a
>> strict verification/validation procedure for different variations of the
>> PDEs.
>>
>> If there?s any problem with the matrix what could help for the
>> diagnostic? At this point I?m running out of ideas so I would really
>> appreciate additional suggestions and discussions.
>>
>> Thanks for your continued support,
>>
>>
>> Thibaut
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/582e5751/attachment.html>

From knepley at gmail.com  Wed Oct 31 11:57:36 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 31 Oct 2018 12:57:36 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <7f6d9b45-3c9d-a7e8-7356-5585229c9f52@imperial.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CAMYG4GnRdMDs=3eaw589o7hYvi60wShHrO_EzY9q6pmHFiOpgQ@mail.gmail.com>
	<7f6d9b45-3c9d-a7e8-7356-5585229c9f52@imperial.ac.uk>
Message-ID: <CAMYG4GmAsWCT=1C-AaZOaZ5Vn6usZ0ds=e27zv_S+E901MOuhA@mail.gmail.com>

On Wed, Oct 31, 2018 at 9:14 AM Thibaut Appel <t.appel17 at imperial.ac.uk>
wrote:

> Hi Matthew,
>
> Which database option are you referring to?
>
> I tried to add -fieldsplit_mg_levels_ksp_type gmres (and
> -fieldsplit_mg_levels_ksp_max_it 4 for another run) to my options (cf.
> below) which starts the iterations but it takes 1 hour for PETSc to do 13
> of them so it must be wrong.
>
> Mark is correct. I wanted you to change the smoother. He shows how to
change it to Richardson (make sure you add the self-scale option), which is
probably the best choice.

  Thanks,

     Matt


> Reminder: my baseline database options line reads
>
> -ksp_view_pre -ksp_monitor -ksp_converged_reason \
> -ksp_rtol 1.0e-8 -ksp_gmres_restart 300 \
> -ksp_type fgmres \
> -pc_type fieldsplit \
> -pc_fieldsplit_type multiplicative \
> -pc_fieldsplit_block_size 3 \
> -pc_fieldsplit_0_fields 0   \
> -pc_fieldsplit_1_fields 1   \
> -pc_fieldsplit_2_fields 2   \
> -fieldsplit_pc_type gamg    \
> -fieldsplit_ksp_type gmres  \
> -fieldsplit_ksp_rtol 1.0e-8
>
> which gives
>
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=300, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using DEFAULT norm type for convergence test
> PC Object: 1 MPI processes
>   type: fieldsplit
>   PC has not been set up so information may be incomplete
>     FieldSplit with MULTIPLICATIVE composition: total splits = 3,
> blocksize = 3
>     Solver info for each split is in the following KSP objects:
>   Split number 0 Fields  0
>   KSP Object: (fieldsplit_0_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_0_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 1 Fields  1
>   KSP Object: (fieldsplit_1_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_1_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 2 Fields  2
>   KSP Object: (fieldsplit_2_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_2_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=52500, cols=52500
>     total: nonzeros=1127079, allocated nonzeros=1128624
>     total number of mallocs used during MatSetValues calls =0
>       not using I-node routines
>   0 KSP Residual norm 3.583290589961e+00
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Petsc has generated inconsistent data
> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
> 0[0]PETSC ERROR: Petsc Release Version 3.10.2, unknown
> [0]PETSC ERROR: Configure options --PETSC_ARCH=msi_cplx_debug
> --with-scalar-type=complex --with-precision=double --with-debugging=1
> --with-valgrind=1 --with-debugger=gdb --with-fortran-kernels=1
> --download-mpich --download-hwloc --download-fblaslapack
> --download-scalapack --download-metis --download-parmetis
> --download-ptscotch --download-mumps --download-slepc
> [0]PETSC ERROR: #1 KSPSolve_Chebyshev() line 381 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/cheby/cheby.c
> [0]PETSC ERROR: #2 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #3 PCMGMCycle_Private() line 20 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #4 PCApply_MG() line 377 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #5 PCApply() line 462 in
> /home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #6 KSP_PCApply() line 281 in
> /home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #7 KSPInitialResidual() line 67 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itres.c
> [0]PETSC ERROR: #8 KSPSolve_GMRES() line 233 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/gmres.c
> [0]PETSC ERROR: #9 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #10 PCApply_FieldSplit() line 1107 in
> /home/thibaut/Packages/petsc/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #11 PCApply() line 462 in
> /home/thibaut/Packages/petsc/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #12 KSP_PCApply() line 281 in
> /home/thibaut/Packages/petsc/include/petsc/private/kspimpl.h
> [0]PETSC ERROR: #13 KSPFGMRESCycle() line 166 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #14 KSPSolve_FGMRES() line 291 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #15 KSPSolve() line 780 in
> /home/thibaut/Packages/petsc/src/ksp/ksp/interface/itfunc.c
>
>
> Thibaut
>
>
>
> On 30/10/2018 23:12, Matthew Knepley wrote:
>
> On Tue, Oct 30, 2018 at 5:21 PM Appel, Thibaut via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear users,
>>
>> Following a suggestion from Matthew Knepley I?ve been trying to apply
>> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
>> despite various tests. pc_gamg simply won?t start.
>> Note that direct solvers always yield the correct, physical result.
>> Removing the fieldsplit to focus on the gamg bit
>
>
> This is a mistake I think because you want a simpler operator to try GAMG
> on.
>
> Can you go back to splitting the system, and try using GMRES as the
> smoother. Its important to see
> whether the smoother makes no progress, or the coarse correction stinks.
>
>   Thanks,
>
>      Matt
>
>
>> and trying to solve the linear system on a modest size problem still
>> gives, with
>>
>> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
>> -pc_type gamg'
>>
>> [3]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [3]PETSC ERROR: Petsc has generated inconsistent data
>> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>>
>> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>>
>> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>>
>> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
>> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>>
>> It seems that I have indeed eigenvalues of rather high magnitude in the
>> spectrum of my operator without being able to determine the reason.
>> The eigenvectors look like small artifacts at the wall-inflow or
>> wall-outflow corners with zero anywhere else but I do not know how to
>> interpret this.
>> Equations are time-harmonic linearized Navier-Stokes to which a forcing
>> is applied, there?s no time-marching.
>>
>> Matrix is formed with a MPIAIJ type. The formulation is incompressible,
>> in complex arithmetic and the 2D physical domain is mapped to a logically
>> rectangular, regular collocated grid with a high-order finite difference
>> method.
>> I determine the ownership of the rows/degrees of freedom of the matrix
>> with PetscSplitOwnership and I?m not using DMDA.
>>
>> The Fortran application code is memory-leak free and has undergone a
>> strict verification/validation procedure for different variations of the
>> PDEs.
>>
>> If there?s any problem with the matrix what could help for the
>> diagnostic? At this point I?m running out of ideas so I would really
>> appreciate additional suggestions and discussions.
>>
>> Thanks for your continued support,
>>
>>
>> Thibaut
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/dda1fa9b/attachment.html>

From gagiuntoli at gmail.com  Wed Oct 31 12:24:23 2018
From: gagiuntoli at gmail.com (Guido Giuntoli)
Date: Wed, 31 Oct 2018 18:24:23 +0100
Subject: [petsc-users] Two applications with PETSc
Message-ID: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>

Hi, I have two codes that use PETSc. The first one is parallel and uses MPI
and the other doesn't uses MPI (uses sequencial Mats and Vecs because the
problem is smaller). I need now to couple both codes and my question is how
do I deal with the PetscInitialize in the sequential code ? I know that
PetscInitialize calls MPI_Init so I think if the first code just called
MPI_Init before I will get an error or not ? Every process in the parallel
code needs to use the functions of the sequential code, so every process
will call PetscInitialize of the sequential code.

constrain : I would like to use the same compiled library of petsc to link
both codes.

Thank you, Guido.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/eaf406f6/attachment.html>

From t.appel17 at imperial.ac.uk  Wed Oct 31 13:13:01 2018
From: t.appel17 at imperial.ac.uk (Thibaut Appel)
Date: Wed, 31 Oct 2018 18:13:01 +0000
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
Message-ID: <991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>

Hi Mark, Matthew,

Thanks for taking the time.

1) You're not suggesting having -fieldsplit_X_ksp_type *f*gmres for each 
field, are you?

2) No, the matrix *has* pressure in one of the fields. Here it's a 2D 
problem (but we're also doing 3D), the unknowns are (p,u,v) and those 
are my 3 fields. We are dealing with subsonic/transsonic flows so it is 
convection dominated indeed.

3) We are in frequency domain with respect to time, i.e. 
\partial{phi}/\partial{t} = -i*omega*phi.

4) Hypre is unfortunately not an option since we are in complex arithmetic.


> I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one 
> block, and hence be a subpc. I'm not up on fieldsplit syntax.

According to the online manual page this syntax applies the suffix to 
all the defined fields?


> Mark is correct. I wanted you to change the smoother. He shows how to 
> change it to Richardson (make sure you add the self-scale option), 
> which is probably the best choice.
>
> ? Thanks,
>
> ? ? ?Matt

You did tell me to set it to GMRES if I'm not mistaken, that's why I 
tried "-fieldsplit_mg_levels_ksp_type gmres" (mentioned in the email). 
Also, it wasn't clear whether these should be applied to each block or 
the whole system, as the online manual pages + .pdf manual barely 
mention smoothers and how to manipulate MG objects with KSP/PC, this 
especially with PCFIELDSPLIT where examples are scarce.

 From what I can gather from your suggestions I tried (lines with X are 
repeated for X={0,1,2})

-ksp_view_pre -ksp_monitor -ksp_converged_reason \
-ksp_type fgmres -ksp_rtol 1.0e-8 \
-pc_type fieldsplit \
-pc_fieldsplit_type multiplicative \
-pc_fieldsplit_block_size 3 \
-pc_fieldsplit_0_fields 0 \
-pc_fieldsplit_1_fields 1 \
-pc_fieldsplit_2_fields 2 \
-fieldsplit_X_pc_type gamg \
-fieldsplit_X_ksp_type gmres \
-fieldsplit_X_ksp_rtol 1e-10 \
-fieldsplit_X_mg_levels_ksp_type richardson \
-fieldsplit_X_mg_levels_pc_type sor \
-fieldsplit_X_pc_gamg_agg_nsmooths 0 \
-fieldsplit_X_mg_levels_ksp_richardson_self_scale \
-log_view

which yields

KSP Object: 1 MPI processes
 ? type: fgmres
 ??? restart=30, using Classical (unmodified) Gram-Schmidt 
Orthogonalization with no iterative refinement
 ??? happy breakdown tolerance 1e-30
 ? maximum iterations=10000, initial guess is zero
 ? tolerances:? relative=1e-08, absolute=1e-50, divergence=10000.
 ? left preconditioning
 ? using DEFAULT norm type for convergence test
PC Object: 1 MPI processes
 ? type: fieldsplit
 ? PC has not been set up so information may be incomplete
 ??? FieldSplit with MULTIPLICATIVE composition: total splits = 3, 
blocksize = 3
 ??? Solver info for each split is in the following KSP objects:
 ? Split number 0 Fields? 0
 ? KSP Object: (fieldsplit_0_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_0_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? Split number 1 Fields? 1
 ? KSP Object: (fieldsplit_1_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_1_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? Split number 2 Fields? 2
 ? KSP Object: (fieldsplit_2_) 1 MPI processes
 ??? type: preonly
 ??? maximum iterations=10000, initial guess is zero
 ??? tolerances:? relative=1e-05, absolute=1e-50, divergence=10000.
 ??? left preconditioning
 ??? using DEFAULT norm type for convergence test
 ? PC Object: (fieldsplit_2_) 1 MPI processes
 ??? type not yet set
 ??? PC has not been set up so information may be incomplete
 ? linear system matrix = precond matrix:
 ? Mat Object: 1 MPI processes
 ??? type: seqaij
 ??? rows=52500, cols=52500
 ??? total: nonzeros=1127079, allocated nonzeros=1128624
 ??? total number of mallocs used during MatSetValues calls =0
 ????? not using I-node routines
 ? 0 KSP Residual norm 3.583290589961e+00
 ? 1 KSP Residual norm 0.000000000000e+00
Linear solve converged due to CONVERGED_ATOL iterations 1

so something must not be set correctly. The solution is identically zero 
everywhere.

Is that option list what you meant? If you could let me know what should 
be corrected.


Thanks for your support,


Thibaut


On 31/10/2018 16:43, Mark Adams wrote:
>
>
> On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users 
> <petsc-users at mcs.anl.gov <mailto:petsc-users at mcs.anl.gov>> wrote:
>
>     Dear users,
>
>     Following a suggestion from Matthew Knepley I?ve been trying to
>     apply fieldsplit/gamg for my set of PDEs but I?m still
>     encountering issues despite various tests. pc_gamg simply won?t start.
>     Note that direct solvers always yield the correct, physical result.
>     Removing the fieldsplit to focus on the gamg bit and trying to
>     solve the linear system on a modest size problem still gives, with
>
>     '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type
>     gmres -pc_type gamg'
>
>     [3]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [3]PETSC ERROR: Petsc has generated inconsistent data
>     [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>     '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>     '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>
>     And since then, after adding '-pc_gamg_sym_graph true' I have been
>     getting
>     [0]PETSC ERROR: --------------------- Error Message
>     --------------------------------------------------------------
>     [0]PETSC ERROR: Petsc has generated inconsistent data
>     [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>
>     -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>
>     Knowing that Chebyshev eigen estimator needs a positive spectrum I
>     tried ?-mg_levels_ksp_type gmres? but iterations would just go on
>     endlessly.
>
>
> This is OK, but you need to use '-ksp_type *f*gmres' (this could be 
> why it is failing ...).
>
> It looks like your matrix is 1) just the velocity field and 2) very 
> unsymmetric (eg, convection dominated). I would start with 
> ?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.
>
> I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths?0'
>
>
>     It seems that I have indeed eigenvalues of rather high magnitude
>     in the spectrum of my operator without being able to determine the
>     reason.
>     The eigenvectors look like small artifacts at the wall-inflow or
>     wall-outflow corners with zero anywhere else but I do not know how
>     to interpret this.
>     Equations are time-harmonic linearized Navier-Stokes to which a
>     forcing is applied, there?s no time-marching.
>
>
> You mean you are in frequency domain?
>
>
>     Matrix is formed with a MPIAIJ type. The formulation is
>     incompressible, in complex arithmetic and the 2D physical domain
>     is mapped to a logically rectangular, 
>
>
> This kind of messes up the null space that AMG depends on but AMG 
> theory is gone for NS anyway.
>
>     regular collocated grid with a high-order finite difference method.
>     I determine the ownership of the rows/degrees of freedom of the
>     matrix with PetscSplitOwnership and I?m not using DMDA.
>
>
> Our iterative solvers are probably not going to work well on this but 
> you should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). 
> You need to configure PETSc to download hypre.
>
> Mark
>
>
>     The Fortran application code is memory-leak free and has undergone
>     a strict verification/validation procedure for different
>     variations of the PDEs.
>
>     If there?s any problem with the matrix what could help for the
>     diagnostic? At this point I?m running out of ideas so I would
>     really appreciate additional suggestions and discussions.
>
>     Thanks for your continued support,
>
>
>     Thibaut
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/9773939a/attachment-0001.html>

From knepley at gmail.com  Wed Oct 31 13:38:00 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 31 Oct 2018 14:38:00 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
	<991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>
Message-ID: <CAMYG4Gmxu+QDvR4=pi2fv+jgvk+vVFCKieXC7Rj=+aAV4ruK0Q@mail.gmail.com>

On Wed, Oct 31, 2018 at 2:13 PM Thibaut Appel <t.appel17 at imperial.ac.uk>
wrote:

> Hi Mark, Matthew,
>
> Thanks for taking the time.
>
> 1) You're not suggesting having -fieldsplit_X_ksp_type *f*gmres for each
> field, are you?
>
> 2) No, the matrix *has* pressure in one of the fields. Here it's a 2D
> problem (but we're also doing 3D), the unknowns are (p,u,v) and those are
> my 3 fields. We are dealing with subsonic/transsonic flows so it is
> convection dominated indeed.
>
> 3) We are in frequency domain with respect to time, i.e.
> \partial{phi}/\partial{t} = -i*omega*phi.
>
> 4) Hypre is unfortunately not an option since we are in complex arithmetic.
>
>
> I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one
> block, and hence be a subpc. I'm not up on fieldsplit syntax.
>
> According to the online manual page this syntax applies the suffix to all
> the defined fields?
>
>
> Mark is correct. I wanted you to change the smoother. He shows how to
> change it to Richardson (make sure you add the self-scale option), which is
> probably the best choice.
>
>   Thanks,
>
>      Matt
>
> You did tell me to set it to GMRES if I'm not mistaken, that's why I tried
> "-fieldsplit_mg_levels_ksp_type gmres" (mentioned in the email). Also, it
> wasn't clear whether these should be applied to each block or the whole
> system, as the online manual pages + .pdf manual barely mention smoothers
> and how to manipulate MG objects with KSP/PC, this especially with
> PCFIELDSPLIT where examples are scarce.
>
> From what I can gather from your suggestions I tried (lines with X are
> repeated for X={0,1,2})
>
> This looks good. How can an identically zero vector produce a 0 residual?
You should always monitor with

  -ksp_monitor_true_residual.

   Thanks,

    Matt

> -ksp_view_pre -ksp_monitor -ksp_converged_reason \
> -ksp_type fgmres -ksp_rtol 1.0e-8 \
> -pc_type fieldsplit \
> -pc_fieldsplit_type multiplicative \
> -pc_fieldsplit_block_size 3 \
> -pc_fieldsplit_0_fields 0 \
> -pc_fieldsplit_1_fields 1 \
> -pc_fieldsplit_2_fields 2 \
> -fieldsplit_X_pc_type gamg \
> -fieldsplit_X_ksp_type gmres \
> -fieldsplit_X_ksp_rtol 1e-10 \
> -fieldsplit_X_mg_levels_ksp_type richardson \
> -fieldsplit_X_mg_levels_pc_type sor \
> -fieldsplit_X_pc_gamg_agg_nsmooths 0 \
> -fieldsplit_X_mg_levels_ksp_richardson_self_scale \
> -log_view
>
> which yields
>
> KSP Object: 1 MPI processes
>   type: fgmres
>     restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
>     happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using DEFAULT norm type for convergence test
> PC Object: 1 MPI processes
>   type: fieldsplit
>   PC has not been set up so information may be incomplete
>     FieldSplit with MULTIPLICATIVE composition: total splits = 3,
> blocksize = 3
>     Solver info for each split is in the following KSP objects:
>   Split number 0 Fields  0
>   KSP Object: (fieldsplit_0_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_0_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 1 Fields  1
>   KSP Object: (fieldsplit_1_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_1_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   Split number 2 Fields  2
>   KSP Object: (fieldsplit_2_) 1 MPI processes
>     type: preonly
>     maximum iterations=10000, initial guess is zero
>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>     left preconditioning
>     using DEFAULT norm type for convergence test
>   PC Object: (fieldsplit_2_) 1 MPI processes
>     type not yet set
>     PC has not been set up so information may be incomplete
>   linear system matrix = precond matrix:
>   Mat Object: 1 MPI processes
>     type: seqaij
>     rows=52500, cols=52500
>     total: nonzeros=1127079, allocated nonzeros=1128624
>     total number of mallocs used during MatSetValues calls =0
>       not using I-node routines
>   0 KSP Residual norm 3.583290589961e+00
>   1 KSP Residual norm 0.000000000000e+00
> Linear solve converged due to CONVERGED_ATOL iterations 1
>
> so something must not be set correctly. The solution is identically zero
> everywhere.
>
> Is that option list what you meant? If you could let me know what should
> be corrected.
>
>
> Thanks for your support,
>
>
> Thibaut
>
>
> On 31/10/2018 16:43, Mark Adams wrote:
>
>
>
> On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Dear users,
>>
>> Following a suggestion from Matthew Knepley I?ve been trying to apply
>> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
>> despite various tests. pc_gamg simply won?t start.
>> Note that direct solvers always yield the correct, physical result.
>> Removing the fieldsplit to focus on the gamg bit and trying to solve the
>> linear system on a modest size problem still gives, with
>>
>> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
>> -pc_type gamg'
>>
>> [3]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [3]PETSC ERROR: Petsc has generated inconsistent data
>> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>>
>> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
>> [0]PETSC ERROR: --------------------- Error Message
>> --------------------------------------------------------------
>> [0]PETSC ERROR: Petsc has generated inconsistent data
>> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>>
>> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>>
>> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
>> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>>
>
> This is OK, but you need to use '-ksp_type *f*gmres' (this could be why
> it is failing ...).
>
> It looks like your matrix is 1) just the velocity field and 2) very
> unsymmetric (eg, convection dominated). I would start with
> ?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.
>
> I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths 0'
>
>
>>
>> It seems that I have indeed eigenvalues of rather high magnitude in the
>> spectrum of my operator without being able to determine the reason.
>> The eigenvectors look like small artifacts at the wall-inflow or
>> wall-outflow corners with zero anywhere else but I do not know how to
>> interpret this.
>> Equations are time-harmonic linearized Navier-Stokes to which a forcing
>> is applied, there?s no time-marching.
>>
>
> You mean you are in frequency domain?
>
>
>>
>> Matrix is formed with a MPIAIJ type. The formulation is incompressible,
>> in complex arithmetic and the 2D physical domain is mapped to a logically
>> rectangular,
>
>
> This kind of messes up the null space that AMG depends on but AMG theory
> is gone for NS anyway.
>
>
>> regular collocated grid with a high-order finite difference method.
>> I determine the ownership of the rows/degrees of freedom of the matrix
>> with PetscSplitOwnership and I?m not using DMDA.
>>
>
> Our iterative solvers are probably not going to work well on this but you
> should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). You need
> to configure PETSc to download hypre.
>
> Mark
>
>
>>
>> The Fortran application code is memory-leak free and has undergone a
>> strict verification/validation procedure for different variations of the
>> PDEs.
>>
>> If there?s any problem with the matrix what could help for the
>> diagnostic? At this point I?m running out of ideas so I would really
>> appreciate additional suggestions and discussions.
>>
>> Thanks for your continued support,
>>
>>
>> Thibaut
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/a770f654/attachment.html>

From knepley at gmail.com  Wed Oct 31 13:53:35 2018
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 31 Oct 2018 14:53:35 -0400
Subject: [petsc-users] Two applications with PETSc
In-Reply-To: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>
References: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>
Message-ID: <CAMYG4GmZQgfWwM3gbQRPzNe1vaXjbYev+fYxoBY5uaT69w47fQ@mail.gmail.com>

On Wed, Oct 31, 2018 at 1:34 PM Guido Giuntoli via petsc-users <
petsc-users at mcs.anl.gov> wrote:

> Hi, I have two codes that use PETSc. The first one is parallel and uses
> MPI and the other doesn't uses MPI (uses sequencial Mats and Vecs because
> the problem is smaller). I need now to couple both codes and my question is
> how do I deal with the PetscInitialize in the sequential code ? I know that
> PetscInitialize calls MPI_Init so I think if the first code just called
> MPI_Init before I will get an error or not ? Every process in the parallel
> code needs to use the functions of the sequential code, so every process
> will call PetscInitialize of the sequential code.
>
> constrain : I would like to use the same compiled library of petsc to link
> both codes.
>

You should only call PetscInitialize() once (just like MPIInit()). You can
check whether it has been called using PetscInitialized().

  Thanks,

    Matt


> Thank you, Guido.
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.cse.buffalo.edu/~knepley/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/61663614/attachment.html>

From danyang.su at gmail.com  Wed Oct 31 13:54:06 2018
From: danyang.su at gmail.com (Danyang Su)
Date: Wed, 31 Oct 2018 11:54:06 -0700
Subject: [petsc-users] Two applications with PETSc
In-Reply-To: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>
References: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>
Message-ID: <3a1ebf7c-ed6e-f765-0f7e-5292d576644f@gmail.com>

Hi Guido,

You can try to use preprocessor flags in you code to separate sequential 
code and parallel code. I am not sure if PETSc has a better way to do 
this. Just a suggestion.

Regards,

Danyang

On 2018-10-31 10:24 a.m., Guido Giuntoli via petsc-users wrote:
> Hi, I have two codes that use PETSc. The first one is parallel and 
> uses MPI and the other doesn't uses MPI (uses sequencial Mats and Vecs 
> because the problem is smaller). I need now to couple both codes and 
> my question is how do I deal with the PetscInitialize in the 
> sequential code ? I know that PetscInitialize calls MPI_Init so I 
> think if the first code just called MPI_Init before I will get an 
> error or not ? Every process in the parallel code needs to use the 
> functions of the sequential code, so every process will call 
> PetscInitialize of the sequential code.
>
> constrain : I would like to use the same compiled library of petsc to 
> link both codes.
>
> Thank you, Guido.

From gagiuntoli at gmail.com  Wed Oct 31 15:23:55 2018
From: gagiuntoli at gmail.com (Guido Giuntoli)
Date: Wed, 31 Oct 2018 21:23:55 +0100
Subject: [petsc-users] Two applications with PETSc
In-Reply-To: <CAMYG4GmZQgfWwM3gbQRPzNe1vaXjbYev+fYxoBY5uaT69w47fQ@mail.gmail.com>
References: <CADOH_Gfshf+gJEJt4TSqFPYDj5fM6qV_+9vo1cWDMcQ6-mzCBQ@mail.gmail.com>
	<CAMYG4GmZQgfWwM3gbQRPzNe1vaXjbYev+fYxoBY5uaT69w47fQ@mail.gmail.com>
Message-ID: <CADOH_Gd5XWYmCcWsJgS1pHi9Zu95XZ2C8O9Naz9katwcPUDaPg@mail.gmail.com>

This is what I need ! Thank you Matt !

El El mi?, 31 oct 2018 a las 19:53, Matthew Knepley <knepley at gmail.com>
escribi?:

> On Wed, Oct 31, 2018 at 1:34 PM Guido Giuntoli via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
>> Hi, I have two codes that use PETSc. The first one is parallel and uses
>> MPI and the other doesn't uses MPI (uses sequencial Mats and Vecs because
>> the problem is smaller). I need now to couple both codes and my question is
>> how do I deal with the PetscInitialize in the sequential code ? I know that
>> PetscInitialize calls MPI_Init so I think if the first code just called
>> MPI_Init before I will get an error or not ? Every process in the parallel
>> code needs to use the functions of the sequential code, so every process
>> will call PetscInitialize of the sequential code.
>>
>> constrain : I would like to use the same compiled library of petsc to
>> link both codes.
>>
>
> You should only call PetscInitialize() once (just like MPIInit()). You can
> check whether it has been called using PetscInitialized().
>
>   Thanks,
>
>     Matt
>
>
>> Thank you, Guido.
>>
>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/86e2766e/attachment.html>

From bhatiamanav at gmail.com  Wed Oct 31 14:42:13 2018
From: bhatiamanav at gmail.com (Manav Bhatia)
Date: Wed, 31 Oct 2018 14:42:13 -0500
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
	<CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
	<CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>
	<CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>
Message-ID: <36A8CB28-134A-4F52-B265-A730F6D68A20@gmail.com>

Here are the updates. I did not find the options to make much difference in the results. 

I noticed this message in the GAMG output for cases 2, 3:  HARD stop of coarsening on level 3.  Grid too small: 1 block nodes

Is this implying that the mesh on level 3 could not be coarsened towards levels 4/5? 

In one of your earlier emails you had mentioned about reducing the rate of coarsening. What is the option to do that. 

Common options: 
-pc_mg_levels 5 -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -mg_levels_ksp_type richardson -gamg_est_ksp_type cg

Case#  |  #levels   |    #KSP Iters  |  Extra Options 
????????????????????????????
     1      |     5        |       67            |       
     2      |     5        |       67            |    -pc_gamg_agg_nsmooths 2 
     3      |     5        |       67            |    -pc_gamg_agg_nsmooths 2  -mg_levels_esteig_ksp_type cg 

Regards,
Manav


1)  -pc_mg_levels 5 -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -mg_levels_ksp_type richardson -gamg_est_ksp_type cg
Blackbird-3:tmp manav$ cat t | grep "KSP\ Res"
    0 KSP Residual norm 1.851603973086e+00 
    1 KSP Residual norm 1.236968440850e+00 
    2 KSP Residual norm 9.390455456383e-01 
    3 KSP Residual norm 6.995550854087e-01 
    4 KSP Residual norm 5.376014874545e-01 
    5 KSP Residual norm 4.224938459190e-01 
    6 KSP Residual norm 3.299526332446e-01 
    7 KSP Residual norm 2.598913867489e-01 
    8 KSP Residual norm 2.021899918256e-01 
    9 KSP Residual norm 1.569629696968e-01 
   10 KSP Residual norm 1.199340095037e-01 
   11 KSP Residual norm 8.839078696637e-02 
   12 KSP Residual norm 6.343737778919e-02 
   13 KSP Residual norm 4.618676240943e-02 
   14 KSP Residual norm 3.423870718445e-02 
   15 KSP Residual norm 2.476201782979e-02 
   16 KSP Residual norm 1.816680574952e-02 
   17 KSP Residual norm 1.313655583815e-02 
   18 KSP Residual norm 9.537218399283e-03 
   19 KSP Residual norm 7.142389309622e-03 
   20 KSP Residual norm 5.254424711175e-03 
   21 KSP Residual norm 3.884143778641e-03 
   22 KSP Residual norm 2.972078747664e-03 
   23 KSP Residual norm 2.291218731265e-03 
   24 KSP Residual norm 1.750029682894e-03 
   25 KSP Residual norm 1.336782384261e-03 
   26 KSP Residual norm 1.008407782667e-03 
   27 KSP Residual norm 7.491621913428e-04 
   28 KSP Residual norm 5.407480688723e-04 
   29 KSP Residual norm 3.885903074862e-04 
   30 KSP Residual norm 2.823392094831e-04 
   31 KSP Residual norm 2.334379417608e-04 
   32 KSP Residual norm 1.862232013173e-04 
   33 KSP Residual norm 1.436841687740e-04 
   34 KSP Residual norm 1.071549728624e-04 
   35 KSP Residual norm 8.090249549012e-05 
   36 KSP Residual norm 6.198056235196e-05 
   37 KSP Residual norm 4.807122983322e-05 
   38 KSP Residual norm 3.749190126381e-05 
   39 KSP Residual norm 2.856634096713e-05 
   40 KSP Residual norm 2.143362174581e-05 
   41 KSP Residual norm 1.620193111270e-05 
   42 KSP Residual norm 1.205751485823e-05 
   43 KSP Residual norm 8.472022598540e-06 
   44 KSP Residual norm 5.899591838872e-06 
   45 KSP Residual norm 4.023678186795e-06 
   46 KSP Residual norm 2.700561982264e-06 
   47 KSP Residual norm 1.868774325367e-06 
   48 KSP Residual norm 1.291496854620e-06 
   49 KSP Residual norm 9.216156351135e-07 
   50 KSP Residual norm 6.624151736445e-07 
   51 KSP Residual norm 4.847699010466e-07 
   52 KSP Residual norm 3.717034432412e-07 
   53 KSP Residual norm 2.944641488187e-07 
   54 KSP Residual norm 2.405160144034e-07 
   55 KSP Residual norm 1.930168301359e-07 
   56 KSP Residual norm 1.543411662956e-07 
   57 KSP Residual norm 1.219117666559e-07 
   58 KSP Residual norm 1.018761570699e-07 
   59 KSP Residual norm 8.612996296470e-08 
   60 KSP Residual norm 7.804919225099e-08 
   61 KSP Residual norm 7.130576810131e-08 
   62 KSP Residual norm 5.229870149728e-08 
   63 KSP Residual norm 4.015915040910e-08 
   64 KSP Residual norm 3.139220241645e-08 
   65 KSP Residual norm 2.447134574457e-08 
   66 KSP Residual norm 1.875885438105e-08 
   67 KSP Residual norm 1.422423827541e-08 
Blackbird-3:tmp manav$ cat t | grep "GAMG"
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.150898e+00 min=3.523136e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=103, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 17.2399 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 2021 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=1.875419e+00 min=3.291195e-02 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=12126, n data cols=6, nnz/row (ave)=172, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 28.7325 nnz ave. (N=2021)
[0] PCGAMGProlongator_AGG(): New grid 161 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.227706e+00 min=4.290859e-02 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=966, n data cols=6, nnz/row (ave)=176, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 28.2353 nnz ave. (N=161)
[0] PCGAMGProlongator_AGG(): New grid 14 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.199096e+00 min=7.296159e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 18 with simple aggregation
[0] PCSetUp_GAMG(): 4) N=84, n data cols=6, nnz/row (ave)=82, 1 active pes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.43482
KSP Object: 4 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.   0.   0.  
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
  [1] PetscCommDuplicate(): Using internal PETSc communicator 4504571456 140452555073200
[2] PetscCommDuplicate(): Using internal PETSc communicator 4504063552 140197931234784
[3] PetscCommDuplicate(): Using internal PETSc communicator 4542250560 140215134305744
        Symmetric graph false
          Number of levels to square graph 0
          Number smoothing steps 1
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 4 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 4 MPI processes
      type: bjacobi
        number of blocks = 4
        Local solve is same for all blocks, in the following KSP and PC objects:
[0] PetscCommDuplicate(): Using internal PETSc communicator 4530183744 140522297539408
      KSP Object: (mg_coarse_sub_) 1 MPI processes
        type: preonly
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (mg_coarse_sub_) 1 MPI processes
        type: lu
          out-of-place factorization
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
          matrix ordering: nd
          factor fill ratio given 5., needed 1.01042
            Factored matrix follows:
              Mat Object: 1 MPI processes
                type: seqaij
                rows=84, cols=84, bs=6
                package used to perform factorization: petsc
                total: nonzeros=6984, allocated nonzeros=6984
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 19 nodes, limit used is 5
        linear system matrix = precond matrix:
        Mat Object: 1 MPI processes
          type: seqaij
          rows=84, cols=84, bs=6
          total: nonzeros=6912, allocated nonzeros=6912
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 22 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=84, cols=84, bs=6
        total: nonzeros=6912, allocated nonzeros=6912
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 22 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=966, cols=966, bs=6
        total: nonzeros=170460, allocated nonzeros=170460
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 68 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=12126, cols=12126, bs=6
        total: nonzeros=2091780, allocated nonzeros=2091780
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 1000 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=100902, cols=100902, bs=6
        total: nonzeros=10438380, allocated nonzeros=10438380
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 8420 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: () 4 MPI processes
        type: mpiaij
        rows=543606, cols=543606, bs=6
        total: nonzeros=29224836, allocated nonzeros=29302596
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: () 4 MPI processes
    type: mpiaij
    rows=543606, cols=543606, bs=6
    total: nonzeros=29224836, allocated nonzeros=29302596
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      using I-node (on process 0) routines: found 45644 nodes, limit used is 5


2)  -pc_mg_levels 5 -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -mg_levels_ksp_type richardson -gamg_est_ksp_type cg -pc_gamg_agg_nsmooths 2 
Dhcp-90-164:tmp manav$ less t | grep "KSP\ Res"
    0 KSP Residual norm 2.267825771593e+00 
    1 KSP Residual norm 1.408716287389e+00 
    2 KSP Residual norm 1.051634554284e+00 
    3 KSP Residual norm 7.943327803314e-01 
    4 KSP Residual norm 5.985495977659e-01 
    5 KSP Residual norm 4.876074034827e-01 
    6 KSP Residual norm 4.032803415998e-01 
    7 KSP Residual norm 3.151459268079e-01 
    8 KSP Residual norm 2.287875309266e-01 
    9 KSP Residual norm 1.689560628134e-01 
   10 KSP Residual norm 1.265410177028e-01 
   11 KSP Residual norm 9.716400303328e-02 
   12 KSP Residual norm 7.305887324105e-02 
   13 KSP Residual norm 5.252262598629e-02 
   14 KSP Residual norm 3.665923907231e-02 
   15 KSP Residual norm 2.602482063395e-02 
   16 KSP Residual norm 1.874425155866e-02 
   17 KSP Residual norm 1.338002525296e-02 
   18 KSP Residual norm 9.461558101831e-03 
   19 KSP Residual norm 6.860781203061e-03 
   20 KSP Residual norm 5.293924529472e-03 
   21 KSP Residual norm 4.001605646043e-03 
   22 KSP Residual norm 2.932783008322e-03 
   23 KSP Residual norm 2.190592529117e-03 
   24 KSP Residual norm 1.651488369324e-03 
   25 KSP Residual norm 1.242272218904e-03 
   26 KSP Residual norm 9.109308493484e-04 
   27 KSP Residual norm 6.734018279380e-04 
   28 KSP Residual norm 5.045747272596e-04 
   29 KSP Residual norm 3.708717871933e-04 
   30 KSP Residual norm 2.700583924223e-04 
   31 KSP Residual norm 2.279903216897e-04 
   32 KSP Residual norm 1.860788922727e-04 
   33 KSP Residual norm 1.393442047104e-04 
   34 KSP Residual norm 1.056758942826e-04 
   35 KSP Residual norm 8.292722358648e-05 
   36 KSP Residual norm 6.249605418622e-05 
   37 KSP Residual norm 4.515823660374e-05 
   38 KSP Residual norm 3.341965987471e-05 
   39 KSP Residual norm 2.455167026138e-05 
   40 KSP Residual norm 1.799936419620e-05 
   41 KSP Residual norm 1.301185237440e-05 
   42 KSP Residual norm 9.185857856804e-06 
   43 KSP Residual norm 6.639180846148e-06 
   44 KSP Residual norm 4.796369056787e-06 
   45 KSP Residual norm 3.517639548008e-06 
   46 KSP Residual norm 2.533810036316e-06 
   47 KSP Residual norm 1.776841834240e-06 
   48 KSP Residual norm 1.216140909196e-06 
   49 KSP Residual norm 8.985883410485e-07 
   50 KSP Residual norm 7.081872318657e-07 
   51 KSP Residual norm 5.298199755216e-07 
   52 KSP Residual norm 3.780633027727e-07 
   53 KSP Residual norm 2.776467936978e-07 
   54 KSP Residual norm 2.145923065527e-07 
   55 KSP Residual norm 1.847249755132e-07 
   56 KSP Residual norm 1.602876793711e-07 
   57 KSP Residual norm 1.405461845847e-07 
   58 KSP Residual norm 1.210529511057e-07 
   59 KSP Residual norm 1.040909071752e-07 
   60 KSP Residual norm 8.799454707008e-08 
   61 KSP Residual norm 7.574693709284e-08 
   62 KSP Residual norm 5.902264183743e-08 
   63 KSP Residual norm 4.928450943916e-08 
   64 KSP Residual norm 4.103274222934e-08 
   65 KSP Residual norm 3.333493557786e-08 
   66 KSP Residual norm 2.697332570486e-08 
   67 KSP Residual norm 2.047460738099e-08 
Dhcp-90-164:tmp manav$ less t | grep "GAMG"
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.150898e+00 min=3.523136e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=207, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 34.5235 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 1041 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.387504e+00 min=1.113469e-01 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=6246, n data cols=6, nnz/row (ave)=398, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 65.9061 nnz ave. (N=1041)
[0] PCGAMGProlongator_AGG(): New grid 38 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.352516e+00 min=4.279834e-02 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=228, n data cols=6, nnz/row (ave)=207, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 33.875 nnz ave. (N=38)
[0] PCGAMGProlongator_AGG(): New grid 1 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.109888e+00 min=7.508213e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 0 with simple aggregation
[0] PCSetUp_GAMG(): 4) N=6, n data cols=6, nnz/row (ave)=6, 1 active pes
[0] PCSetUp_GAMG(): HARD stop of coarsening on level 3.  Grid too small: 1 block nodes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.80239
KSP Object: 4 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.   0.   0.  
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
          Symmetric graph false
          Number of levels to square graph 0
          Number smoothing steps 2
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 4 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 4 MPI processes
      type: bjacobi
        number of blocks = 4
        Local solve is same for all blocks, in the following KSP and PC objects:
[0] PetscCommDuplicate(): Using internal PETSc communicator 4570619456 140283319649920
[2] PetscCommDuplicate(): Using internal PETSc communicator 4506250816 140419971129472
      KSP Object: (mg_coarse_sub_) 1 MPI processes
        type: preonly
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (mg_coarse_sub_) 1 MPI processes
        type: lu
          out-of-place factorization
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
          matrix ordering: nd
          factor fill ratio given 5., needed 1.
            Factored matrix follows:
              Mat Object: 1 MPI processes
                type: seqaij
                rows=6, cols=6, bs=6
                package used to perform factorization: petsc
                total: nonzeros=36, allocated nonzeros=36
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 2 nodes, limit used is 5
        linear system matrix = precond matrix:
        Mat Object: 1 MPI processes
          type: seqaij
          rows=6, cols=6, bs=6
          total: nonzeros=36, allocated nonzeros=36
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 2 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6, cols=6, bs=6
        total: nonzeros=36, allocated nonzeros=36
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 2 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=228, cols=228, bs=6
        total: nonzeros=47232, allocated nonzeros=47232
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 10 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6246, cols=6246, bs=6
        total: nonzeros=2489868, allocated nonzeros=2489868
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 490 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=100902, cols=100902, bs=6
        total: nonzeros=20912652, allocated nonzeros=20912652
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 8420 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: () 4 MPI processes
        type: mpiaij
        rows=543606, cols=543606, bs=6
        total: nonzeros=29224836, allocated nonzeros=29302596
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: () 4 MPI processes
    type: mpiaij
    rows=543606, cols=543606, bs=6
    total: nonzeros=29224836, allocated nonzeros=29302596
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      using I-node (on process 0) routines: found 45644 nodes, limit used is 5


2)  -pc_mg_levels 5 -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0 -pc_gamg_threshold 0. -mg_levels_ksp_type richardson -gamg_est_ksp_type cg -pc_gamg_agg_nsmooths 2  -mg_levels_esteig_ksp_type cg
Dhcp-90-164:tmp manav$ less t | grep "KSP\ Res"
    0 KSP Residual norm 2.267825771593e+00 
    1 KSP Residual norm 1.408716287387e+00 
    2 KSP Residual norm 1.051634554280e+00 
    3 KSP Residual norm 7.943327803270e-01 
    4 KSP Residual norm 5.985495977612e-01 
    5 KSP Residual norm 4.876074034779e-01 
    6 KSP Residual norm 4.032803415953e-01 
    7 KSP Residual norm 3.151459268039e-01 
    8 KSP Residual norm 2.287875309236e-01 
    9 KSP Residual norm 1.689560628111e-01 
   10 KSP Residual norm 1.265410177010e-01 
   11 KSP Residual norm 9.716400303191e-02 
   12 KSP Residual norm 7.305887323998e-02 
   13 KSP Residual norm 5.252262598550e-02 
   14 KSP Residual norm 3.665923907174e-02 
   15 KSP Residual norm 2.602482063353e-02 
   16 KSP Residual norm 1.874425155833e-02 
   17 KSP Residual norm 1.338002525269e-02 
   18 KSP Residual norm 9.461558101624e-03 
   19 KSP Residual norm 6.860781202901e-03 
   20 KSP Residual norm 5.293924529345e-03 
   21 KSP Residual norm 4.001605645944e-03 
   22 KSP Residual norm 2.932783008245e-03 
   23 KSP Residual norm 2.190592529059e-03 
   24 KSP Residual norm 1.651488369281e-03 
   25 KSP Residual norm 1.242272218876e-03 
   26 KSP Residual norm 9.109308493356e-04 
   27 KSP Residual norm 6.734018279390e-04 
   28 KSP Residual norm 5.045747272693e-04 
   29 KSP Residual norm 3.708717872080e-04 
   30 KSP Residual norm 2.700584035265e-04 
   31 KSP Residual norm 2.279903320914e-04 
   32 KSP Residual norm 1.860789013071e-04 
   33 KSP Residual norm 1.393442086232e-04 
   34 KSP Residual norm 1.056758941655e-04 
   35 KSP Residual norm 8.292722294552e-05 
   36 KSP Residual norm 6.249605489796e-05 
   37 KSP Residual norm 4.515823692238e-05 
   38 KSP Residual norm 3.341965924406e-05 
   39 KSP Residual norm 2.455167014490e-05 
   40 KSP Residual norm 1.799936521170e-05 
   41 KSP Residual norm 1.301185398833e-05 
   42 KSP Residual norm 9.185859481763e-06 
   43 KSP Residual norm 6.639182448645e-06 
   44 KSP Residual norm 4.796370558780e-06 
   45 KSP Residual norm 3.517640894677e-06 
   46 KSP Residual norm 2.533811059265e-06 
   47 KSP Residual norm 1.776842623521e-06 
   48 KSP Residual norm 1.216141654930e-06 
   49 KSP Residual norm 8.985890975369e-07 
   50 KSP Residual norm 7.081880149660e-07 
   51 KSP Residual norm 5.298207972657e-07 
   52 KSP Residual norm 3.780642754235e-07 
   53 KSP Residual norm 2.776479493176e-07 
   54 KSP Residual norm 2.145935700364e-07 
   55 KSP Residual norm 1.847262262413e-07 
   56 KSP Residual norm 1.602888215732e-07 
   57 KSP Residual norm 1.405471647242e-07 
   58 KSP Residual norm 1.210537184709e-07 
   59 KSP Residual norm 1.040914613543e-07 
   60 KSP Residual norm 8.800584999158e-08 
   61 KSP Residual norm 7.575320990546e-08 
   62 KSP Residual norm 5.902566537130e-08 
   63 KSP Residual norm 4.928732270849e-08 
   64 KSP Residual norm 4.103605656966e-08 
   65 KSP Residual norm 3.333879521091e-08 
   66 KSP Residual norm 2.697680027664e-08 
   67 KSP Residual norm 2.047615127912e-08 
Dhcp-90-164:tmp manav$ less t | grep "GAMG"
[0] PCSetUp_GAMG(): level 0) N=543606, n data rows=6, n data cols=6, nnz/row (ave)=54, np=4
[0] PCGAMGFilterGraph(): 	 98.7979% nnz after filtering, with threshold 0., 8.95943 nnz ave. (N=90601)
[0] PCGAMGProlongator_AGG(): New grid 16817 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.150898e+00 min=3.523136e-02 PC=jacobi
[0] PCSetUp_GAMG(): 1) N=100902, n data cols=6, nnz/row (ave)=207, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 34.5235 nnz ave. (N=16817)
[0] PCGAMGProlongator_AGG(): New grid 1041 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.387504e+00 min=1.113469e-01 PC=jacobi
[0] PCSetUp_GAMG(): 2) N=6246, n data cols=6, nnz/row (ave)=398, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 65.9061 nnz ave. (N=1041)
[0] PCGAMGProlongator_AGG(): New grid 38 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.352516e+00 min=4.279834e-02 PC=jacobi
[0] PCSetUp_GAMG(): 3) N=228, n data cols=6, nnz/row (ave)=207, 4 active pes
[0] PCGAMGFilterGraph(): 	 100.% nnz after filtering, with threshold 0., 33.875 nnz ave. (N=38)
[0] PCGAMGProlongator_AGG(): New grid 1 nodes
[0] PCGAMGOptProlongator_AGG(): Smooth P0: max eigen=2.109888e+00 min=7.508213e-02 PC=jacobi
[0] PCGAMGCreateLevel_GAMG(): Number of equations (loc) 0 with simple aggregation
[0] PCSetUp_GAMG(): 4) N=6, n data cols=6, nnz/row (ave)=6, 1 active pes
[0] PCSetUp_GAMG(): HARD stop of coarsening on level 3.  Grid too small: 1 block nodes
[0] PCSetUp_GAMG(): 5 levels, grid complexity = 1.80239
KSP Object: 4 MPI processes
  type: gmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 4 MPI processes
  type: gamg
    type is MULTIPLICATIVE, levels=5 cycles=v
      Cycles per PCApply=1
      Using externally compute Galerkin coarse grid matrices
      GAMG specific options
        Threshold for dropping small values in graph on each level =   0.   0.   0.  
        Threshold scaling factor for each level not specified = 1.
        AGG specific options
          Symmetric graph false
          Number of levels to square graph 0
          Number smoothing steps 2
  Coarse grid solver -- level -------------------------------
    KSP Object: (mg_coarse_) 4 MPI processes
      type: preonly
      maximum iterations=10000, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_coarse_) 4 MPI processes
      type: bjacobi
        number of blocks = 4
        Local solve is same for all blocks, in the following KSP and PC objects:
[0] PetscCommDuplicate(): Using internal PETSc communicator 4358889024 140338640884000
      KSP Object: (mg_coarse_sub_) 1 MPI processes
        type: preonly
        maximum iterations=1, initial guess is zero
        tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
        left preconditioning
        using NONE norm type for convergence test
      PC Object: (mg_coarse_sub_) 1 MPI processes
        type: lu
          out-of-place factorization
          tolerance for zero pivot 2.22045e-14
          using diagonal shift on blocks to prevent zero pivot [INBLOCKS]
          matrix ordering: nd
          factor fill ratio given 5., needed 1.
            Factored matrix follows:
              Mat Object: 1 MPI processes
                type: seqaij
                rows=6, cols=6, bs=6
                package used to perform factorization: petsc
                total: nonzeros=36, allocated nonzeros=36
                total number of mallocs used during MatSetValues calls =0
                  using I-node routines: found 2 nodes, limit used is 5
        linear system matrix = precond matrix:
        Mat Object: 1 MPI processes
          type: seqaij
          rows=6, cols=6, bs=6
          total: nonzeros=36, allocated nonzeros=36
          total number of mallocs used during MatSetValues calls =0
            using I-node routines: found 2 nodes, limit used is 5
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6, cols=6, bs=6
        total: nonzeros=36, allocated nonzeros=36
        total number of mallocs used during MatSetValues calls =0
          using I-node (on process 0) routines: found 2 nodes, limit used is 5
  Down solver (pre-smoother) on level 1 -------------------------------
    KSP Object: (mg_levels_1_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_1_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=228, cols=228, bs=6
        total: nonzeros=47232, allocated nonzeros=47232
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 10 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 2 -------------------------------
    KSP Object: (mg_levels_2_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_2_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=6246, cols=6246, bs=6
        total: nonzeros=2489868, allocated nonzeros=2489868
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 490 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 3 -------------------------------
    KSP Object: (mg_levels_3_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_3_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: 4 MPI processes
        type: mpiaij
        rows=100902, cols=100902, bs=6
        total: nonzeros=20912652, allocated nonzeros=20912652
        total number of mallocs used during MatSetValues calls =0
          using nonscalable MatPtAP() implementation
          using I-node (on process 0) routines: found 8420 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  Down solver (pre-smoother) on level 4 -------------------------------
    KSP Object: (mg_levels_4_) 4 MPI processes
      type: richardson
        damping factor=1.
      maximum iterations=4, nonzero initial guess
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
      left preconditioning
      using NONE norm type for convergence test
    PC Object: (mg_levels_4_) 4 MPI processes
      type: sor
        type = local_symmetric, iterations = 1, local iterations = 1, omega = 1.
      linear system matrix = precond matrix:
      Mat Object: () 4 MPI processes
        type: mpiaij
        rows=543606, cols=543606, bs=6
        total: nonzeros=29224836, allocated nonzeros=29302596
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          using I-node (on process 0) routines: found 45644 nodes, limit used is 5
  Up solver (post-smoother) same as down solver (pre-smoother)
  linear system matrix = precond matrix:
  Mat Object: () 4 MPI processes
    type: mpiaij
    rows=543606, cols=543606, bs=6
    total: nonzeros=29224836, allocated nonzeros=29302596
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      using I-node (on process 0) routines: found 45644 nodes, limit used is 5




> On Oct 30, 2018, at 1:51 PM, Mark Adams <mfadams at lbl.gov> wrote:
> 
> The last two cases were cut off but that is OK. 
> 
> Lets just do -pc_gamg_square_graph 0 and please add/do -ksp_view -mg_levels_ksp_type richardson -pc_mg_levels 5 -gamg_est_ksp_type cg  
> 
> Send the grep GAMG again, and the ksp_view output.
> 
> And then do this again with: -pc_gamg_agg_nsmooths 
> 
> I think the eigen estimators are messed up on the coarse grids.
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/9aadee68/attachment-0001.html>

From mfadams at lbl.gov  Wed Oct 31 16:05:59 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 31 Oct 2018 17:05:59 -0400
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <CAMYG4Gmxu+QDvR4=pi2fv+jgvk+vVFCKieXC7Rj=+aAV4ruK0Q@mail.gmail.com>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
	<991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>
	<CAMYG4Gmxu+QDvR4=pi2fv+jgvk+vVFCKieXC7Rj=+aAV4ruK0Q@mail.gmail.com>
Message-ID: <CADOhEh7br2T=3vcb+J-uFwg1ph8yB7bx3CyvFA_71-GVFwjQQA@mail.gmail.com>

These are indefinite (bad) Helmholtz problems. Right?

On Wed, Oct 31, 2018 at 2:38 PM Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 31, 2018 at 2:13 PM Thibaut Appel <t.appel17 at imperial.ac.uk>
> wrote:
>
>> Hi Mark, Matthew,
>>
>> Thanks for taking the time.
>>
>> 1) You're not suggesting having -fieldsplit_X_ksp_type *f*gmres for each
>> field, are you?
>>
>> 2) No, the matrix *has* pressure in one of the fields. Here it's a 2D
>> problem (but we're also doing 3D), the unknowns are (p,u,v) and those are
>> my 3 fields. We are dealing with subsonic/transsonic flows so it is
>> convection dominated indeed.
>>
>> 3) We are in frequency domain with respect to time, i.e.
>> \partial{phi}/\partial{t} = -i*omega*phi.
>>
>> 4) Hypre is unfortunately not an option since we are in complex
>> arithmetic.
>>
>>
>> I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one
>> block, and hence be a subpc. I'm not up on fieldsplit syntax.
>>
>> According to the online manual page this syntax applies the suffix to all
>> the defined fields?
>>
>>
>> Mark is correct. I wanted you to change the smoother. He shows how to
>> change it to Richardson (make sure you add the self-scale option), which is
>> probably the best choice.
>>
>>   Thanks,
>>
>>      Matt
>>
>> You did tell me to set it to GMRES if I'm not mistaken, that's why I
>> tried "-fieldsplit_mg_levels_ksp_type gmres" (mentioned in the email).
>> Also, it wasn't clear whether these should be applied to each block or the
>> whole system, as the online manual pages + .pdf manual barely mention
>> smoothers and how to manipulate MG objects with KSP/PC, this especially
>> with PCFIELDSPLIT where examples are scarce.
>>
>> From what I can gather from your suggestions I tried (lines with X are
>> repeated for X={0,1,2})
>>
>> This looks good. How can an identically zero vector produce a 0 residual?
> You should always monitor with
>
>   -ksp_monitor_true_residual.
>
>    Thanks,
>
>     Matt
>
>> -ksp_view_pre -ksp_monitor -ksp_converged_reason \
>> -ksp_type fgmres -ksp_rtol 1.0e-8 \
>> -pc_type fieldsplit \
>> -pc_fieldsplit_type multiplicative \
>> -pc_fieldsplit_block_size 3 \
>> -pc_fieldsplit_0_fields 0 \
>> -pc_fieldsplit_1_fields 1 \
>> -pc_fieldsplit_2_fields 2 \
>> -fieldsplit_X_pc_type gamg \
>> -fieldsplit_X_ksp_type gmres \
>> -fieldsplit_X_ksp_rtol 1e-10 \
>> -fieldsplit_X_mg_levels_ksp_type richardson \
>> -fieldsplit_X_mg_levels_pc_type sor \
>> -fieldsplit_X_pc_gamg_agg_nsmooths 0 \
>> -fieldsplit_X_mg_levels_ksp_richardson_self_scale \
>> -log_view
>>
>> which yields
>>
>> KSP Object: 1 MPI processes
>>   type: fgmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=10000, initial guess is zero
>>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>>   left preconditioning
>>   using DEFAULT norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: fieldsplit
>>   PC has not been set up so information may be incomplete
>>     FieldSplit with MULTIPLICATIVE composition: total splits = 3,
>> blocksize = 3
>>     Solver info for each split is in the following KSP objects:
>>   Split number 0 Fields  0
>>   KSP Object: (fieldsplit_0_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_0_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   Split number 1 Fields  1
>>   KSP Object: (fieldsplit_1_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_1_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   Split number 2 Fields  2
>>   KSP Object: (fieldsplit_2_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_2_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   linear system matrix = precond matrix:
>>   Mat Object: 1 MPI processes
>>     type: seqaij
>>     rows=52500, cols=52500
>>     total: nonzeros=1127079, allocated nonzeros=1128624
>>     total number of mallocs used during MatSetValues calls =0
>>       not using I-node routines
>>   0 KSP Residual norm 3.583290589961e+00
>>   1 KSP Residual norm 0.000000000000e+00
>> Linear solve converged due to CONVERGED_ATOL iterations 1
>>
>> so something must not be set correctly. The solution is identically zero
>> everywhere.
>>
>> Is that option list what you meant? If you could let me know what should
>> be corrected.
>>
>>
>> Thanks for your support,
>>
>>
>> Thibaut
>>
>>
>> On 31/10/2018 16:43, Mark Adams wrote:
>>
>>
>>
>> On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users <
>> petsc-users at mcs.anl.gov> wrote:
>>
>>> Dear users,
>>>
>>> Following a suggestion from Matthew Knepley I?ve been trying to apply
>>> fieldsplit/gamg for my set of PDEs but I?m still encountering issues
>>> despite various tests. pc_gamg simply won?t start.
>>> Note that direct solvers always yield the correct, physical result.
>>> Removing the fieldsplit to focus on the gamg bit and trying to solve the
>>> linear system on a modest size problem still gives, with
>>>
>>> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres
>>> -pc_type gamg'
>>>
>>> [3]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [3]PETSC ERROR: Petsc has generated inconsistent data
>>> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use
>>> '-(null)pc_gamg_sym_graph true' to symetrize the graph or
>>> '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>>>
>>> And since then, after adding '-pc_gamg_sym_graph true' I have been
>>> getting
>>> [0]PETSC ERROR: --------------------- Error Message
>>> --------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>>>
>>> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>>>
>>> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried
>>> ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>>>
>>
>> This is OK, but you need to use '-ksp_type *f*gmres' (this could be why
>> it is failing ...).
>>
>> It looks like your matrix is 1) just the velocity field and 2) very
>> unsymmetric (eg, convection dominated). I would start with
>> ?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.
>>
>> I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths 0'
>>
>>
>>>
>>> It seems that I have indeed eigenvalues of rather high magnitude in the
>>> spectrum of my operator without being able to determine the reason.
>>> The eigenvectors look like small artifacts at the wall-inflow or
>>> wall-outflow corners with zero anywhere else but I do not know how to
>>> interpret this.
>>> Equations are time-harmonic linearized Navier-Stokes to which a forcing
>>> is applied, there?s no time-marching.
>>>
>>
>> You mean you are in frequency domain?
>>
>>
>>>
>>> Matrix is formed with a MPIAIJ type. The formulation is incompressible,
>>> in complex arithmetic and the 2D physical domain is mapped to a logically
>>> rectangular,
>>
>>
>> This kind of messes up the null space that AMG depends on but AMG theory
>> is gone for NS anyway.
>>
>>
>>> regular collocated grid with a high-order finite difference method.
>>> I determine the ownership of the rows/degrees of freedom of the matrix
>>> with PetscSplitOwnership and I?m not using DMDA.
>>>
>>
>> Our iterative solvers are probably not going to work well on this but you
>> should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). You need
>> to configure PETSc to download hypre.
>>
>> Mark
>>
>>
>>>
>>> The Fortran application code is memory-leak free and has undergone a
>>> strict verification/validation procedure for different variations of the
>>> PDEs.
>>>
>>> If there?s any problem with the matrix what could help for the
>>> diagnostic? At this point I?m running out of ideas so I would really
>>> appreciate additional suggestions and discussions.
>>>
>>> Thanks for your continued support,
>>>
>>>
>>> Thibaut
>>
>>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
> https://www.cse.buffalo.edu/~knepley/
> <http://www.cse.buffalo.edu/~knepley/>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/a3ea73ae/attachment.html>

From mfadams at lbl.gov  Wed Oct 31 16:32:04 2018
From: mfadams at lbl.gov (Mark Adams)
Date: Wed, 31 Oct 2018 17:32:04 -0400
Subject: [petsc-users] Convergence of AMG
In-Reply-To: <36A8CB28-134A-4F52-B265-A730F6D68A20@gmail.com>
References: <AC0CC2FA-BF71-4D82-A26D-49878512C13B@gmail.com>
	<6B8DB287-3014-4087-930C-FB4EC0B19616@anl.gov>
	<4A541373-C59B-4E43-81A6-C4D96409D9B4@gmail.com>
	<CADOhEh7EWuSHCBa9p2f0yufMyH2ikRgRYaLqKeuJE_Oea09tKg@mail.gmail.com>
	<C40F3A6C-7F79-4215-9977-B17DF7548F01@gmail.com>
	<771C5999-2383-4B0F-A505-72B4C3AC0A37@gmail.com>
	<69CDD9CC-8956-42BE-A4A7-1008B2E5DF07@gmail.com>
	<2BFD0C70-4880-4971-95BA-36263C77D842@gmail.com>
	<875zxkfz3m.fsf@jedbrown.org>
	<72ADF8D5-BB41-4E31-B667-55F86CD606EB@gmail.com>
	<87woq0ej1q.fsf@jedbrown.org>
	<548872DB-2E1A-491E-BA7B-DBBE87383AF3@gmail.com>
	<6A3D9909-BFF8-48B1-BB74-B5930C947EEF@anl.gov>
	<129F1369-F4C1-4D2C-9B03-A57EAD342F51@gmail.com>
	<CADOhEh6JChS6suZJBGt=YqLD2EEXiJ7pCDrXt8foGkSvwiLw-Q@mail.gmail.com>
	<CA+gX-L_wrfm-gKQdd+mcCxWQ_HWc1N8tu1fJAggKvPkFQu+2=w@mail.gmail.com>
	<CADOhEh7DYdiXD2RWipGM9MnOkTbwC8EXXktPF7re_Zrw+-K+jg@mail.gmail.com>
	<CA+gX-L-mW9ZZUXvwuz+W+qFbyPDekB7+YqhZj7ZKhJhXd_2doQ@mail.gmail.com>
	<CADOhEh4ieuVHua6mutYp0YrcVTuL=FqAHb_qNVnEj_rwcq=ncg@mail.gmail.com>
	<36A8CB28-134A-4F52-B265-A730F6D68A20@gmail.com>
Message-ID: <CADOhEh460cOnW8107su+s_f0aDgvM1EDHTcvn9U+EejUNp=88Q@mail.gmail.com>

On Wed, Oct 31, 2018 at 3:43 PM Manav Bhatia <bhatiamanav at gmail.com> wrote:

> Here are the updates. I did not find the options to make much difference
> in the results.
>
> I noticed this message in the GAMG output for cases 2, 3:  HARD stop of
> coarsening on level 3.  Grid too small: 1 block nodes
>

Yea, this is what it looks like but I can see that there are 5 levels below.


>
> Is this implying that the mesh on level 3 could not be coarsened towards
> levels 4/5?
>
> In one of your earlier emails you had mentioned about reducing the rate of
> coarsening. What is the option to do that.
>

The coarsening rate is pretty slow but you could try reducing it more with
a finite -pc_gamg_threshold val (you could try val=0.01, 0.02, 0.05).


>
> Common options:
> -pc_mg_levels 5 -mg_levels_ksp_max_it 4 -pc_gamg_square_graph 0
> -pc_gamg_threshold 0. -mg_levels_ksp_type richardson -gamg_est_ksp_type cg
>
> Case#  |  #levels   |    #KSP Iters  |  Extra Options
> ????????????????????????????
>      1      |     5        |       67            |
>      2      |     5        |       67            |
>  -pc_gamg_agg_nsmooths 2
>      3      |     5        |       67            |
>  -pc_gamg_agg_nsmooths 2  -mg_levels_esteig_ksp_type cg
>

I have done plates a long time ago and they worked fine, but there is
something going wrong here. Perhaps my plate formulations were different
but I don't think so (what we called Reissner-Mindlin).

You might try an easier test case if this is not a super simple test (eg,
well shaped elements).

Otherwise you can limit the number of levels that you use and use a
parallel direct solver for the (large) coarse grid. To do that use:
"-pc_gamg_use_parallel_coarse_grid_solver -mg_coarse_pc_type lu" and
configure your PETSc with a parallel coarse grid solver like MUMPS or
SuperLU.

Then set -pc_gamg_coarse_eq_limit N with N being large. Increase it and
minimize the solve time. You will find that parallel lu is faster than MG
for "small" problems (note, small could be pretty large here). So you will
want to run LU first (ie, no AMG) to get a baseline and then try GAMG.

Oh, and you could also try hypre (=pc_type hypre -pc_hypre_type boomeramg).
Hypre should not be great in theory but it is worth a try. Maybe there's
just something going wrong with GAMG.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/7c4c3625/attachment-0001.html>

From t.appel17 at imperial.ac.uk  Wed Oct 31 17:39:41 2018
From: t.appel17 at imperial.ac.uk (Appel, Thibaut)
Date: Wed, 31 Oct 2018 22:39:41 +0000
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <CADOhEh7br2T=3vcb+J-uFwg1ph8yB7bx3CyvFA_71-GVFwjQQA@mail.gmail.com>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
	<991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>
	<CAMYG4Gmxu+QDvR4=pi2fv+jgvk+vVFCKieXC7Rj=+aAV4ruK0Q@mail.gmail.com>
	<CADOhEh7br2T=3vcb+J-uFwg1ph8yB7bx3CyvFA_71-GVFwjQQA@mail.gmail.com>
Message-ID: <B524E65E-0F7D-48AC-9AB6-F66D36EDD32C@ic.ac.uk>

Well yes naturally for the residual but adding -ksp_true_residual just gives

  0 KSP unpreconditioned resid norm 3.583290589961e+00 true resid norm 3.583290589961e+00 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP unpreconditioned resid norm 0.000000000000e+00 true resid norm 3.583290589961e+00 ||r(i)||/||b|| 1.000000000000e+00
Linear solve converged due to CONVERGED_ATOL iterations 1

Mark - if that helps - a Poisson equation is used for the pressure so the Helmholtz is the same as for the velocity in the interior.

Thibaut

Le 31 oct. 2018 ? 21:05, Mark Adams <mfadams at lbl.gov<mailto:mfadams at lbl.gov>> a ?crit :

These are indefinite (bad) Helmholtz problems. Right?

On Wed, Oct 31, 2018 at 2:38 PM Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:
On Wed, Oct 31, 2018 at 2:13 PM Thibaut Appel <t.appel17 at imperial.ac.uk> wrote:
Hi Mark, Matthew,

Thanks for taking the time.

1) You're not suggesting having -fieldsplit_X_ksp_type fgmres for each field, are you?

2) No, the matrix has pressure in one of the fields. Here it's a 2D problem (but we're also doing 3D), the unknowns are (p,u,v) and those are my 3 fields. We are dealing with subsonic/transsonic flows so it is convection dominated indeed.

3) We are in frequency domain with respect to time, i.e. \partial{phi}/\partial{t} = -i*omega*phi.

4) Hypre is unfortunately not an option since we are in complex arithmetic.



I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one block, and hence be a subpc. I'm not up on fieldsplit syntax.
According to the online manual page this syntax applies the suffix to all the defined fields?



Mark is correct. I wanted you to change the smoother. He shows how to change it to Richardson (make sure you add the self-scale option), which is probably the best choice.

  Thanks,

     Matt

You did tell me to set it to GMRES if I'm not mistaken, that's why I tried "-fieldsplit_mg_levels_ksp_type gmres" (mentioned in the email). Also, it wasn't clear whether these should be applied to each block or the whole system, as the online manual pages + .pdf manual barely mention smoothers and how to manipulate MG objects with KSP/PC, this especially with PCFIELDSPLIT where examples are scarce.

From what I can gather from your suggestions I tried (lines with X are repeated for X={0,1,2})

This looks good. How can an identically zero vector produce a 0 residual? You should always monitor with

  -ksp_monitor_true_residual.

   Thanks,

    Matt
-ksp_view_pre -ksp_monitor -ksp_converged_reason \
-ksp_type fgmres -ksp_rtol 1.0e-8 \
-pc_type fieldsplit \
-pc_fieldsplit_type multiplicative \
-pc_fieldsplit_block_size 3 \
-pc_fieldsplit_0_fields 0 \
-pc_fieldsplit_1_fields 1 \
-pc_fieldsplit_2_fields 2 \
-fieldsplit_X_pc_type gamg \
-fieldsplit_X_ksp_type gmres \
-fieldsplit_X_ksp_rtol 1e-10 \
-fieldsplit_X_mg_levels_ksp_type richardson \
-fieldsplit_X_mg_levels_pc_type sor \
-fieldsplit_X_pc_gamg_agg_nsmooths 0 \
-fieldsplit_X_mg_levels_ksp_richardson_self_scale \
-log_view

which yields

KSP Object: 1 MPI processes
  type: fgmres
    restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
    happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
  left preconditioning
  using DEFAULT norm type for convergence test
PC Object: 1 MPI processes
  type: fieldsplit
  PC has not been set up so information may be incomplete
    FieldSplit with MULTIPLICATIVE composition: total splits = 3, blocksize = 3
    Solver info for each split is in the following KSP objects:
  Split number 0 Fields  0
  KSP Object: (fieldsplit_0_) 1 MPI processes
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
    left preconditioning
    using DEFAULT norm type for convergence test
  PC Object: (fieldsplit_0_) 1 MPI processes
    type not yet set
    PC has not been set up so information may be incomplete
  Split number 1 Fields  1
  KSP Object: (fieldsplit_1_) 1 MPI processes
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
    left preconditioning
    using DEFAULT norm type for convergence test
  PC Object: (fieldsplit_1_) 1 MPI processes
    type not yet set
    PC has not been set up so information may be incomplete
  Split number 2 Fields  2
  KSP Object: (fieldsplit_2_) 1 MPI processes
    type: preonly
    maximum iterations=10000, initial guess is zero
    tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
    left preconditioning
    using DEFAULT norm type for convergence test
  PC Object: (fieldsplit_2_) 1 MPI processes
    type not yet set
    PC has not been set up so information may be incomplete
  linear system matrix = precond matrix:
  Mat Object: 1 MPI processes
    type: seqaij
    rows=52500, cols=52500
    total: nonzeros=1127079, allocated nonzeros=1128624
    total number of mallocs used during MatSetValues calls =0
      not using I-node routines
  0 KSP Residual norm 3.583290589961e+00
  1 KSP Residual norm 0.000000000000e+00
Linear solve converged due to CONVERGED_ATOL iterations 1

so something must not be set correctly. The solution is identically zero everywhere.

Is that option list what you meant? If you could let me know what should be corrected.



Thanks for your support,



Thibaut



On 31/10/2018 16:43, Mark Adams wrote:


On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users <petsc-users at mcs.anl.gov> wrote:
Dear users,

Following a suggestion from Matthew Knepley I?ve been trying to apply fieldsplit/gamg for my set of PDEs but I?m still encountering issues despite various tests. pc_gamg simply won?t start.
Note that direct solvers always yield the correct, physical result.
Removing the fieldsplit to focus on the gamg bit and trying to solve the linear system on a modest size problem still gives, with

'-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres -pc_type gamg'

[3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[3]PETSC ERROR: Petsc has generated inconsistent data
[3]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.

And since then, after adding '-pc_gamg_sym_graph true' I have been getting
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Petsc has generated inconsistent data
[0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration

-ksp_chebyshev_esteig_noisy 0/1 does not change anything

Knowing that Chebyshev eigen estimator needs a positive spectrum I tried ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.

This is OK, but you need to use '-ksp_type fgmres' (this could be why it is failing ...).

It looks like your matrix is 1) just the velocity field and 2) very unsymmetric (eg, convection dominated). I would start with ?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.

I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths 0'


It seems that I have indeed eigenvalues of rather high magnitude in the spectrum of my operator without being able to determine the reason.
The eigenvectors look like small artifacts at the wall-inflow or wall-outflow corners with zero anywhere else but I do not know how to interpret this.
Equations are time-harmonic linearized Navier-Stokes to which a forcing is applied, there?s no time-marching.

You mean you are in frequency domain?


Matrix is formed with a MPIAIJ type. The formulation is incompressible, in complex arithmetic and the 2D physical domain is mapped to a logically rectangular,

This kind of messes up the null space that AMG depends on but AMG theory is gone for NS anyway.

regular collocated grid with a high-order finite difference method.
I determine the ownership of the rows/degrees of freedom of the matrix with PetscSplitOwnership and I?m not using DMDA.

Our iterative solvers are probably not going to work well on this but you should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). You need to configure PETSc to download hypre.

Mark


The Fortran application code is memory-leak free and has undergone a strict verification/validation procedure for different variations of the PDEs.

If there?s any problem with the matrix what could help for the diagnostic? At this point I?m running out of ideas so I would really appreciate additional suggestions and discussions.

Thanks for your continued support,


Thibaut


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20181031/dfdad98a/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct 31 17:58:28 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Wed, 31 Oct 2018 22:58:28 +0000
Subject: [petsc-users] (no subject)
In-Reply-To: <SY3PR01MB15458011971A0417D8649F0084CD0@SY3PR01MB1545.ausprd01.prod.outlook.com>
References: <SY3PR01MB15458011971A0417D8649F0084CD0@SY3PR01MB1545.ausprd01.prod.outlook.com>
Message-ID: <912E0438-B0D1-478E-A106-47DCECDE66F6@anl.gov>


   This option only works with AIJ matrices; you must be using either BAIJ or SBAIJ matrices? (or a shell matrix)

   Barry


> On Oct 31, 2018, at 5:45 AM, Wenjin Xing via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> My issue is summarized in the picture and posted in the link https://scicomp.stackexchange.com/questions/30458/what-does-the-error-this-matrix-type-does-not-have-a-find-zero-diagonals-define?noredirect=1#comment56074_30458
>  
> <image001.png>
>  
> Kind regards
> Wenjin


From bsmith at mcs.anl.gov  Wed Oct 31 19:04:44 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 1 Nov 2018 00:04:44 +0000
Subject: [petsc-users] DIVERGED_NANORING with PC GAMG
In-Reply-To: <B524E65E-0F7D-48AC-9AB6-F66D36EDD32C@ic.ac.uk>
References: <34D20379-DEAC-4E98-8FA8-E7EC2831BE6D@ic.ac.uk>
	<CADOhEh7+ZAaz_7VFRBbd0B83J9x3AVuwRc0k+cednFXOtwTYvA@mail.gmail.com>
	<991561e5-1b9a-dbb2-333d-501251967077@imperial.ac.uk>
	<CAMYG4Gmxu+QDvR4=pi2fv+jgvk+vVFCKieXC7Rj=+aAV4ruK0Q@mail.gmail.com>
	<CADOhEh7br2T=3vcb+J-uFwg1ph8yB7bx3CyvFA_71-GVFwjQQA@mail.gmail.com>
	<B524E65E-0F7D-48AC-9AB6-F66D36EDD32C@ic.ac.uk>
Message-ID: <DE6675DF-5EBA-42C2-81DA-C1387EBE266F@anl.gov>



> On Oct 31, 2018, at 5:39 PM, Appel, Thibaut via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Well yes naturally for the residual but adding -ksp_true_residual just gives
> 
>   0 KSP unpreconditioned resid norm 3.583290589961e+00 true resid norm 3.583290589961e+00 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP unpreconditioned resid norm 0.000000000000e+00 true resid norm 3.583290589961e+00 ||r(i)||/||b|| 1.000000000000e+00
> Linear solve converged due to CONVERGED_ATOL iterations 1

   Very bad stuff is happening in the preconditioner. The preconditioner must have a null space (which it shouldn't have to be a useful preconditioner).

> 
> Mark - if that helps - a Poisson equation is used for the pressure so the Helmholtz is the same as for the velocity in the interior.
> 
> Thibaut
> 
>> Le 31 oct. 2018 ? 21:05, Mark Adams <mfadams at lbl.gov> a ?crit :
>> 
>> These are indefinite (bad) Helmholtz problems. Right?
>> 
>> On Wed, Oct 31, 2018 at 2:38 PM Matthew Knepley <knepley at gmail.com> wrote:
>> On Wed, Oct 31, 2018 at 2:13 PM Thibaut Appel <t.appel17 at imperial.ac.uk> wrote:
>> Hi Mark, Matthew,
>> 
>> Thanks for taking the time.
>> 
>> 1) You're not suggesting having -fieldsplit_X_ksp_type fgmres for each field, are you?
>> 
>> 2) No, the matrix has pressure in one of the fields. Here it's a 2D problem (but we're also doing 3D), the unknowns are (p,u,v) and those are my 3 fields. We are dealing with subsonic/transsonic flows so it is convection dominated indeed.
>> 
>> 3) We are in frequency domain with respect to time, i.e. \partial{phi}/\partial{t} = -i*omega*phi.
>> 
>> 4) Hypre is unfortunately not an option since we are in complex arithmetic.
>> 
>> 
>> 
>>> I'm not sure about "-fieldsplit_pc_type gamg" GAMG should work on one block, and hence be a subpc. I'm not up on fieldsplit syntax.
>> According to the online manual page this syntax applies the suffix to all the defined fields?
>> 
>> 
>> 
>>> Mark is correct. I wanted you to change the smoother. He shows how to change it to Richardson (make sure you add the self-scale option), which is probably the best choice.
>>> 
>>>   Thanks,
>>> 
>>>      Matt
>> 
>> You did tell me to set it to GMRES if I'm not mistaken, that's why I tried "-fieldsplit_mg_levels_ksp_type gmres" (mentioned in the email). Also, it wasn't clear whether these should be applied to each block or the whole system, as the online manual pages + .pdf manual barely mention smoothers and how to manipulate MG objects with KSP/PC, this especially with PCFIELDSPLIT where examples are scarce.
>> 
>> From what I can gather from your suggestions I tried (lines with X are repeated for X={0,1,2}) 
>> 
>> This looks good. How can an identically zero vector produce a 0 residual? You should always monitor with
>> 
>>   -ksp_monitor_true_residual.
>> 
>>    Thanks,
>>  
>>     Matt 
>> -ksp_view_pre -ksp_monitor -ksp_converged_reason \
>> -ksp_type fgmres -ksp_rtol 1.0e-8 \
>> -pc_type fieldsplit \
>> -pc_fieldsplit_type multiplicative \
>> -pc_fieldsplit_block_size 3 \
>> -pc_fieldsplit_0_fields 0 \
>> -pc_fieldsplit_1_fields 1 \
>> -pc_fieldsplit_2_fields 2 \
>> -fieldsplit_X_pc_type gamg \
>> -fieldsplit_X_ksp_type gmres \
>> -fieldsplit_X_ksp_rtol 1e-10 \
>> -fieldsplit_X_mg_levels_ksp_type richardson \
>> -fieldsplit_X_mg_levels_pc_type sor \
>> -fieldsplit_X_pc_gamg_agg_nsmooths 0 \
>> -fieldsplit_X_mg_levels_ksp_richardson_self_scale \
>> -log_view
>> 
>> which yields 
>> 
>> KSP Object: 1 MPI processes
>>   type: fgmres
>>     restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>>     happy breakdown tolerance 1e-30
>>   maximum iterations=10000, initial guess is zero
>>   tolerances:  relative=1e-08, absolute=1e-50, divergence=10000.
>>   left preconditioning
>>   using DEFAULT norm type for convergence test
>> PC Object: 1 MPI processes
>>   type: fieldsplit
>>   PC has not been set up so information may be incomplete
>>     FieldSplit with MULTIPLICATIVE composition: total splits = 3, blocksize = 3
>>     Solver info for each split is in the following KSP objects:
>>   Split number 0 Fields  0
>>   KSP Object: (fieldsplit_0_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_0_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   Split number 1 Fields  1
>>   KSP Object: (fieldsplit_1_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_1_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   Split number 2 Fields  2
>>   KSP Object: (fieldsplit_2_) 1 MPI processes
>>     type: preonly
>>     maximum iterations=10000, initial guess is zero
>>     tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>>     left preconditioning
>>     using DEFAULT norm type for convergence test
>>   PC Object: (fieldsplit_2_) 1 MPI processes
>>     type not yet set
>>     PC has not been set up so information may be incomplete
>>   linear system matrix = precond matrix:
>>   Mat Object: 1 MPI processes
>>     type: seqaij
>>     rows=52500, cols=52500
>>     total: nonzeros=1127079, allocated nonzeros=1128624
>>     total number of mallocs used during MatSetValues calls =0
>>       not using I-node routines
>>   0 KSP Residual norm 3.583290589961e+00 
>>   1 KSP Residual norm 0.000000000000e+00 
>> Linear solve converged due to CONVERGED_ATOL iterations 1
>> 
>> so something must not be set correctly. The solution is identically zero everywhere.
>> 
>> Is that option list what you meant? If you could let me know what should be corrected.
>> 
>> 
>> 
>> Thanks for your support,
>> 
>> 
>> 
>> Thibaut
>> 
>> 
>> 
>> On 31/10/2018 16:43, Mark Adams wrote:
>>> 
>>> 
>>> On Tue, Oct 30, 2018 at 5:23 PM Appel, Thibaut via petsc-users <petsc-users at mcs.anl.gov> wrote:
>>> Dear users,
>>> 
>>> Following a suggestion from Matthew Knepley I?ve been trying to apply fieldsplit/gamg for my set of PDEs but I?m still encountering issues despite various tests. pc_gamg simply won?t start.
>>> Note that direct solvers always yield the correct, physical result.
>>> Removing the fieldsplit to focus on the gamg bit and trying to solve the linear system on a modest size problem still gives, with
>>> 
>>> '-ksp_monitor -ksp_rtol 1.0e-10 -ksp_gmres_restart 300 -ksp_type gmres -pc_type gamg'
>>> 
>>> [3]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [3]PETSC ERROR: Petsc has generated inconsistent data
>>> [3]PETSC ERROR: Have un-symmetric graph (apparently). Use '-(null)pc_gamg_sym_graph true' to symetrize the graph or '-(null)pc_gamg_threshold -1' if the matrix is structurally symmetric.
>>> 
>>> And since then, after adding '-pc_gamg_sym_graph true' I have been getting
>>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>>> [0]PETSC ERROR: Petsc has generated inconsistent data
>>> [0]PETSC ERROR: Eigen estimator failed: DIVERGED_NANORINF at iteration
>>> 
>>> -ksp_chebyshev_esteig_noisy 0/1 does not change anything
>>> 
>>> Knowing that Chebyshev eigen estimator needs a positive spectrum I tried ?-mg_levels_ksp_type gmres? but iterations would just go on endlessly.
>>> 
>>> This is OK, but you need to use '-ksp_type fgmres' (this could be why it is failing ...). 
>>> 
>>> It looks like your matrix is 1) just the velocity field and 2) very unsymmetric (eg, convection dominated). I would start with ?-mg_levels_ksp_type richardson -mg_levels_pc_type sor?.
>>> 
>>> I would also start with unsmoothed aggregation: '-pc_gamg_nsmooths 0' 
>>>  
>>> 
>>> It seems that I have indeed eigenvalues of rather high magnitude in the spectrum of my operator without being able to determine the reason.
>>> The eigenvectors look like small artifacts at the wall-inflow or wall-outflow corners with zero anywhere else but I do not know how to interpret this.
>>> Equations are time-harmonic linearized Navier-Stokes to which a forcing is applied, there?s no time-marching.
>>> 
>>> You mean you are in frequency domain?
>>>  
>>> 
>>> Matrix is formed with a MPIAIJ type. The formulation is incompressible, in complex arithmetic and the 2D physical domain is mapped to a logically rectangular,
>>> 
>>> This kind of messes up the null space that AMG depends on but AMG theory is gone for NS anyway.
>>>  
>>> regular collocated grid with a high-order finite difference method.
>>> I determine the ownership of the rows/degrees of freedom of the matrix with PetscSplitOwnership and I?m not using DMDA.
>>> 
>>> Our iterative solvers are probably not going to work well on this but you should test hypre also (-pc_type hypre -pc_hypre_type boomeramg). You need to configure PETSc to download hypre.
>>> 
>>> Mark
>>>  
>>> 
>>> The Fortran application code is memory-leak free and has undergone a strict verification/validation procedure for different variations of the PDEs.
>>> 
>>> If there?s any problem with the matrix what could help for the diagnostic? At this point I?m running out of ideas so I would really appreciate additional suggestions and discussions.
>>> 
>>> Thanks for your continued support,
>>> 
>>> 
>>> Thibaut
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
>> 
>> https://www.cse.buffalo.edu/~knepley/
> 


From bsmith at mcs.anl.gov  Wed Oct 31 21:43:46 2018
From: bsmith at mcs.anl.gov (Smith, Barry F.)
Date: Thu, 1 Nov 2018 02:43:46 +0000
Subject: [petsc-users] (no subject)
In-Reply-To: <MEXPR01MB1542E55C02F0C306336D169E84CE0@MEXPR01MB1542.ausprd01.prod.outlook.com>
References: <SY3PR01MB15458011971A0417D8649F0084CD0@SY3PR01MB1545.ausprd01.prod.outlook.com>
	<912E0438-B0D1-478E-A106-47DCECDE66F6@anl.gov>
	<MEXPR01MB1542E55C02F0C306336D169E84CE0@MEXPR01MB1542.ausprd01.prod.outlook.com>
Message-ID: <E749574A-80D1-4D0D-9EF8-0309103A0DB8@mcs.anl.gov>

https://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind


> On Oct 31, 2018, at 9:18 PM, Wenjin Xing <xing0029 at flinders.edu.au> wrote:
> 
> Hi Barry
>  
> As you said, I have set the mat option to Aij. (MATAIJ = "aij" - A matrix type to be used for sparse matrices. This matrix type is identical to MATSEQAIJ when constructed with a single process communicator.) However, a new error pops up.  By the way, I am using a single processor, not in parallel mode.
>  
>  
>  
> <image002.jpg>
>  
> Kind regards
> Wenjin
>  
>  
>  
>  
> -----Original Message-----
> From: Smith, Barry F. [mailto:bsmith at mcs.anl.gov] 
> Sent: Thursday, 1 November 2018 9:28 AM
> To: Wenjin Xing <xing0029 at flinders.edu.au>
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] (no subject)
>  
>  
>    This option only works with AIJ matrices; you must be using either BAIJ or SBAIJ matrices? (or a shell matrix)
>  
>    Barry
>  
>  
> > On Oct 31, 2018, at 5:45 AM, Wenjin Xing via petsc-users <petsc-users at mcs.anl.gov> wrote:
> > 
> > My issue is summarized in the picture and posted in the link https://scicomp.stackexchange.com/questions/30458/what-does-the-error-this-matrix-type-does-not-have-a-find-zero-diagonals-define?noredirect=1#comment56074_30458
> >  
> > <image001.png>
> >  
> > Kind regards
> > Wenjin