From E.Mueller at bath.ac.uk  Thu Oct  1 03:32:44 2015
From: E.Mueller at bath.ac.uk (Eike Mueller)
Date: Thu, 1 Oct 2015 08:32:44 +0000
Subject: [petsc-users] fieldsplit-preconditioner for block-system of
 linear equations
In-Reply-To: <CAMYG4GnSp__-G_g79S9rGYFaSoUUdYV9UC1R-qRt6OjW3Pw8Ow@mail.gmail.com>
References: <FF7FA0DF-D0EB-4DF0-B6E1-34687F8C1976@bath.ac.uk>,
	<CAMYG4GnSp__-G_g79S9rGYFaSoUUdYV9UC1R-qRt6OjW3Pw8Ow@mail.gmail.com>
Message-ID: <1443688364418.44801@bath.ac.uk>

Hi Matt,


thanks a lot for your answer. Assembling the matrices into one large matrix should be ok, since I have control over that.


But am I right in assuming that the vectors have to be stored as one large PETSc vector? So when X = (x^(1),x^(2),...,x^(k)), where x^(i) are the individual vectors of length n_i, then I have to store X as one PETSc vector of length n_1+n_2+...+n_k? In the data structure I use, storage for the vectors is already allocated, so currently I use

VecCreateSeqWithArray() to construct k separate PETSc vectors. Sounds like I can't do that, but have to create one big PETSc vector, and copy the data into that? That's ok, if it's the only option, since the overhead should not be too large, but I wanted to check before doing something unnecessary.


Thanks,


Eike


________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: 30 September 2015 15:27
To: Eike Mueller
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] fieldsplit-preconditioner for block-system of linear equations

On Wed, Sep 30, 2015 at 3:00 AM, Eike Mueller <E.Mueller at bath.ac.uk<mailto:E.Mueller at bath.ac.uk>> wrote:
Dear PETSc,

I am solving a linear system of equations

(1) A.X = B

where the vector X is divided into chunks, such that X= (x_1,x_2,...,x_k) and each of the vectors x_i has length n_i (same for the vector B=(b_1,b_2,...,b_k)). k=5 in my case, and n_i >> 1. This partitioning implies that the matrix has a block-structure, where the submatrices A_{ij} are of size n_i x n_j. So explicitly for k=3:

        A_{1,1}.x_1 + A_{1,2}.x_2 + A_{1,3}.x_3 = b_1
(2)    A_{2,1}.x_1 + A_{2,2}.x_2 + A_{2,3}.x_3 = b_2
        A_{3,1}.x_1 + A_{3,2}.x_2 + A_{3,3}.x_3 = b_3

I now want to solve this system with a Krylov-method (say, GMRES) and precondition it with a field-split preconditioner, such that the Schur-complement is formed in the 1-block. I know how to do this if I have assembled the big matrix A, and I store all x_i in one big vector X (I construct the index-sets corresponding to the vectors x_1 and (x_2,x_3), and then call PCFieldSplitSetIS()). This all works without any problems.

However, I now want to do this for an existing code which

1. Assembles the matrices A_{ij} separately (i.e. they are not stored as part of a big matrix A, but are stored in independent memory locations)
2. Stores the vectors x_i separately, i.e. they are not stored as parts of one big chunk of memory of size n_1+...+n_k

I can of course implement the matrix application via a matrix shell, but is there still an easy way of using the fieldsplit preconditioner?

>From your description, it sounds like you could use

  http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html

to assemble your separate matrices directly into a large matrix. Furthermore, if it really benefits you
to have separate memory, you can change the matrix type from MATIJ to MATNEST dynamically,
but none of your code would change. This would let you check things with LU, but still optimize when
necessary.

  Thanks,

      Matt

The naive way I can think of is to allocate a new big matrix A, and copy the A_{ij} into the corresponding blocks. Then allocate big vectors x and b, and also copy in/out the data before and after the solve. This, however, seems to be wasteful, so before I go down this route I wanted to double check if there is a way around it, since this seems to be a very common problem?

Thanks a lot,

Eike




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/9018312a/attachment.html>

From knepley at gmail.com  Thu Oct  1 05:41:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 1 Oct 2015 05:41:09 -0500
Subject: [petsc-users] fieldsplit-preconditioner for block-system of
 linear equations
In-Reply-To: <1443688364418.44801@bath.ac.uk>
References: <FF7FA0DF-D0EB-4DF0-B6E1-34687F8C1976@bath.ac.uk>
	<CAMYG4GnSp__-G_g79S9rGYFaSoUUdYV9UC1R-qRt6OjW3Pw8Ow@mail.gmail.com>
	<1443688364418.44801@bath.ac.uk>
Message-ID: <CAMYG4GmhPJ9G5bETL=W30pWnxNfVuQJ+Rt4NLcie0BsSEJcxGA@mail.gmail.com>

On Thu, Oct 1, 2015 at 3:32 AM, Eike Mueller <E.Mueller at bath.ac.uk> wrote:

> Hi Matt,
>
>
> thanks a lot for your answer. Assembling the matrices into one large
> matrix should be ok, since I have control over that.
>
>
> But am I right in assuming that the vectors have to be stored as one large
> PETSc vector? So when X = (x^(1),x^(2),...,x^(k)), where x^(i) are the
> individual vectors of length n_i, then I have to store X as one PETSc
> vector of length n_1+n_2+...+n_k? In the data structure I use, storage for
> the vectors is already allocated, so currently I use
>
> VecCreateSeqWithArray() to construct k separate PETSc vectors. Sounds like
> I can't do that, but have to create one big PETSc vector, and copy the data
> into that? That's ok, if it's the only option, since the overhead should
> not be too large, but I wanted to check before doing something unnecessary.
>
 You should be able to use
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Vec/VecGetSubVector.html

  Thanks,

     Matt

> Thanks,
>
>
> Eike
>
>
> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* 30 September 2015 15:27
> *To:* Eike Mueller
> *Cc:* petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] fieldsplit-preconditioner for block-system
> of linear equations
>
> On Wed, Sep 30, 2015 at 3:00 AM, Eike Mueller <E.Mueller at bath.ac.uk>
> wrote:
>
>> Dear PETSc,
>>
>> I am solving a linear system of equations
>>
>> (1) A.X = B
>>
>> where the vector X is divided into chunks, such that X= (x_1,x_2,?,x_k)
>> and each of the vectors x_i has length n_i (same for the vector
>> B=(b_1,b_2,?,b_k)). k=5 in my case, and n_i >> 1. This partitioning implies
>> that the matrix has a block-structure, where the submatrices A_{ij} are of
>> size n_i x n_j. So explicitly for k=3:
>>
>>         A_{1,1}.x_1 + A_{1,2}.x_2 + A_{1,3}.x_3 = b_1
>> (2)    A_{2,1}.x_1 + A_{2,2}.x_2 + A_{2,3}.x_3 = b_2
>>         A_{3,1}.x_1 + A_{3,2}.x_2 + A_{3,3}.x_3 = b_3
>>
>> I now want to solve this system with a Krylov-method (say, GMRES) and
>> precondition it with a field-split preconditioner, such that the
>> Schur-complement is formed in the 1-block. I know how to do this if I have
>> assembled the big matrix A, and I store all x_i in one big vector X (I
>> construct the index-sets corresponding to the vectors x_1 and (x_2,x_3),
>> and then call PCFieldSplitSetIS()). This all works without any problems.
>>
>> However, I now want to do this for an existing code which
>>
>> 1. Assembles the matrices A_{ij} separately (i.e. they are not stored as
>> part of a big matrix A, but are stored in independent memory locations)
>> 2. Stores the vectors x_i separately, i.e. they are not stored as parts
>> of one big chunk of memory of size n_1+?+n_k
>>
>> I can of course implement the matrix application via a matrix shell, but
>> is there still an easy way of using the fieldsplit preconditioner?
>>
>
> From your description, it sounds like you could use
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatGetLocalSubMatrix.html
>
> to assemble your separate matrices directly into a large matrix.
> Furthermore, if it really benefits you
> to have separate memory, you can change the matrix type from MATIJ to
> MATNEST dynamically,
> but none of your code would change. This would let you check things with
> LU, but still optimize when
> necessary.
>
>   Thanks,
>
>       Matt
>
>
>> The naive way I can think of is to allocate a new big matrix A, and copy
>> the A_{ij} into the corresponding blocks. Then allocate big vectors x and
>> b, and also copy in/out the data before and after the solve. This, however,
>> seems to be wasteful, so before I go down this route I wanted to double
>> check if there is a way around it, since this seems to be a very common
>> problem?
>>
>> Thanks a lot,
>>
>> Eike
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/9cbf5521/attachment-0001.html>

From juhaj at iki.fi  Thu Oct  1 09:24:14 2015
From: juhaj at iki.fi (Juha Jaykka)
Date: Thu, 01 Oct 2015 15:24:14 +0100
Subject: [petsc-users] petsc+chombo
Message-ID: <1700733.a0gsQvG38I@vega>

Hi list,

I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing. It 
seems to me the chombo build system is not told to link in libgfortran.so, but 
no matter how I try to convince PETSc to add that, it won't. How should this 
be used, please?

The full configure line is

./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries --
with-debugging=0 \
          --useThreads 0 --with-clanguage=C++ --with-c-support \
          --with-fortran-interfaces=1 \
          --with-mpi=1 --with-mpi-shared=1 \
	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
base/liblapack.so \
          --with-scalapack=1 --with-scalapack-include=/usr/include \
          --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
          --with-mumps=0 \
	  --with-suitesparse=0 \
          --with-spooles=1 --with-spooles-include=/usr/include/spooles \
          --with-spooles-lib=/usr/lib/libspooles.so \
          --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
          --with-fftw=1 --with-fftw-include=/usr/include \
          --with-fftw-lib=[/usr/lib/x86_64-linux-
gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
          --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi --with-
hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
lgfortran -g" --LIBS="-lgfortran" \
	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
	  --with-afterimage=1 --with-afterimage-
include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
linux-gnu/libAfterImage.so \
	  --with-boost=1 \
	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
lib=/usr/lib/libnetcdf.so \
	  --with-triangle=0 \
	  --with-numpy=1 \
	  --with-hwloc=1  --with-external-packages-
dir=/home/juhaj/progs+3.6/automatic-downloads \
	  --with-parmetis=0 --with-metis=0 \
	  --with-tetgen=0 \
	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
openmp=1 \
	  --with-64-bit-indices \
	  --download-chombo

and the final error in configure is

/bin/csh -f -c "mpicxx -g          
o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
opt/Chombo/lib/test/AMRTimeDependent/../.. -
lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG  -Wl,-rpath,/usr/lib/x86_64-linux-
gnu -L/usr/lib/x86_64-linux-gnu -lhdf5_openmpi     -Wl,-
rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o 
testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f 
/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
/usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.mpif90.DEBUG.a(CFLeastSquares.o): 
undefined reference to symbol '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
/usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO missing 
from command line
collect2: error: ld returned 1 exit status



Cheers,
Juha
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/bcfc1331/attachment.pgp>

From bsmith at mcs.anl.gov  Thu Oct  1 09:53:26 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 1 Oct 2015 09:53:26 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <1700733.a0gsQvG38I@vega>
References: <1700733.a0gsQvG38I@vega>
Message-ID: <81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>


  We absolutely NEED configure.log to help you.

  Barry

> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> 
> Hi list,
> 
> I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing. It 
> seems to me the chombo build system is not told to link in libgfortran.so, but 
> no matter how I try to convince PETSc to add that, it won't. How should this 
> be used, please?
> 
> The full configure line is
> 
> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries --
> with-debugging=0 \
>          --useThreads 0 --with-clanguage=C++ --with-c-support \
>          --with-fortran-interfaces=1 \
>          --with-mpi=1 --with-mpi-shared=1 \
> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
> base/liblapack.so \
>          --with-scalapack=1 --with-scalapack-include=/usr/include \
>          --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
>          --with-mumps=0 \
> 	  --with-suitesparse=0 \
>          --with-spooles=1 --with-spooles-include=/usr/include/spooles \
>          --with-spooles-lib=/usr/lib/libspooles.so \
>          --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
>          --with-fftw=1 --with-fftw-include=/usr/include \
>          --with-fftw-lib=[/usr/lib/x86_64-linux-
> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
>          --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi --with-
> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> lgfortran -g" --LIBS="-lgfortran" \
> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> 	  --with-afterimage=1 --with-afterimage-
> include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
> linux-gnu/libAfterImage.so \
> 	  --with-boost=1 \
> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> lib=/usr/lib/libnetcdf.so \
> 	  --with-triangle=0 \
> 	  --with-numpy=1 \
> 	  --with-hwloc=1  --with-external-packages-
> dir=/home/juhaj/progs+3.6/automatic-downloads \
> 	  --with-parmetis=0 --with-metis=0 \
> 	  --with-tetgen=0 \
> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> openmp=1 \
> 	  --with-64-bit-indices \
> 	  --download-chombo
> 
> and the final error in configure is
> 
> /bin/csh -f -c "mpicxx -g          
> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> opt/Chombo/lib/test/AMRTimeDependent/../.. -
> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG  -Wl,-rpath,/usr/lib/x86_64-linux-
> gnu -L/usr/lib/x86_64-linux-gnu -lhdf5_openmpi     -Wl,-
> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o 
> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f 
> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.mpif90.DEBUG.a(CFLeastSquares.o): 
> undefined reference to symbol '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO missing 
> from command line
> collect2: error: ld returned 1 exit status
> 
> 
> 
> Cheers,
> Juha


From balay at mcs.anl.gov  Thu Oct  1 10:44:12 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 1 Oct 2015 10:44:12 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
References: <1700733.a0gsQvG38I@vega>
	<81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
Message-ID: <alpine.LFD.2.20.1510011042451.20838@asterix>

Hm - we usually try to avoiding creating binaries in externalpackages
[if not needed] - esp the examples/tests.

So perhaps chombo shoul have the following change.

Satish

------
diff --git a/config/BuildSystem/config/packages/Chombo.py b/config/BuildSystem/config/packages/Chombo.py
index effbca6..7a13b0a 100644
--- a/config/BuildSystem/config/packages/Chombo.py
+++ b/config/BuildSystem/config/packages/Chombo.py
@@ -110,7 +110,7 @@ class Configure(config.package.Package):
           raise RuntimeError('Error running make on Chombo: config value not found')
         config_value=poutput.split('=')[1]
         self.logPrint('Chombo installed using config=%s\n'%config_value)
-        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make all', timeout=2500, log = self.log)
+        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make lib', timeout=2500, log = self.log)
         output,err,ret = config.package.Package.executeShellCommand('cd '+self.packageDir+self.installSudo+'&& cp -f lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+' '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''), timeout=2500, log = self.log)
       except RuntimeError, e:
         raise RuntimeError('Error running make on Chombo: '+str(e))

On Thu, 1 Oct 2015, Barry Smith wrote:

> 
>   We absolutely NEED configure.log to help you.
> 
>   Barry
> 
> > On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> > 
> > Hi list,
> > 
> > I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing. It 
> > seems to me the chombo build system is not told to link in libgfortran.so, but 
> > no matter how I try to convince PETSc to add that, it won't. How should this 
> > be used, please?
> > 
> > The full configure line is
> > 
> > ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries --
> > with-debugging=0 \
> >          --useThreads 0 --with-clanguage=C++ --with-c-support \
> >          --with-fortran-interfaces=1 \
> >          --with-mpi=1 --with-mpi-shared=1 \
> > 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> > lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
> > base/liblapack.so \
> >          --with-scalapack=1 --with-scalapack-include=/usr/include \
> >          --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
> >          --with-mumps=0 \
> > 	  --with-suitesparse=0 \
> >          --with-spooles=1 --with-spooles-include=/usr/include/spooles \
> >          --with-spooles-lib=/usr/lib/libspooles.so \
> >          --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> > lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
> >          --with-fftw=1 --with-fftw-include=/usr/include \
> >          --with-fftw-lib=[/usr/lib/x86_64-linux-
> > gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
> >          --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi --with-
> > hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> > 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> > lgfortran -g" --LIBS="-lgfortran" \
> > 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> > 	  --with-afterimage=1 --with-afterimage-
> > include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
> > linux-gnu/libAfterImage.so \
> > 	  --with-boost=1 \
> > 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> > lib=/usr/lib/libnetcdf.so \
> > 	  --with-triangle=0 \
> > 	  --with-numpy=1 \
> > 	  --with-hwloc=1  --with-external-packages-
> > dir=/home/juhaj/progs+3.6/automatic-downloads \
> > 	  --with-parmetis=0 --with-metis=0 \
> > 	  --with-tetgen=0 \
> > 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> > openmp=1 \
> > 	  --with-64-bit-indices \
> > 	  --download-chombo
> > 
> > and the final error in configure is
> > 
> > /bin/csh -f -c "mpicxx -g          
> > o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> > L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../.. -
> > lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG  -Wl,-rpath,/usr/lib/x86_64-linux-
> > gnu -L/usr/lib/x86_64-linux-gnu -lhdf5_openmpi     -Wl,-
> > rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> > rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o 
> > testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f 
> > /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> > /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.mpif90.DEBUG.a(CFLeastSquares.o): 
> > undefined reference to symbol '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> > /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO missing 
> > from command line
> > collect2: error: ld returned 1 exit status
> > 
> > 
> > 
> > Cheers,
> > Juha
> 
> 


From bsmith at mcs.anl.gov  Thu Oct  1 13:39:34 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 1 Oct 2015 13:39:34 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <alpine.LFD.2.20.1510011042451.20838@asterix>
References: <1700733.a0gsQvG38I@vega>
	<81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
	<alpine.LFD.2.20.1510011042451.20838@asterix>
Message-ID: <C7553B2F-19F3-4E5F-BF6D-8021FB590040@mcs.anl.gov>


> On Oct 1, 2015, at 10:44 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> Hm - we usually try to avoiding creating binaries in externalpackages
> [if not needed] - esp the examples/tests.
> 
> So perhaps chombo shoul have the following change.

 Satish,

  Looks good to me, push if it works for you.

   Barry

> 
> Satish
> 
> ------
> diff --git a/config/BuildSystem/config/packages/Chombo.py b/config/BuildSystem/config/packages/Chombo.py
> index effbca6..7a13b0a 100644
> --- a/config/BuildSystem/config/packages/Chombo.py
> +++ b/config/BuildSystem/config/packages/Chombo.py
> @@ -110,7 +110,7 @@ class Configure(config.package.Package):
>           raise RuntimeError('Error running make on Chombo: config value not found')
>         config_value=poutput.split('=')[1]
>         self.logPrint('Chombo installed using config=%s\n'%config_value)
> -        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make all', timeout=2500, log = self.log)
> +        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make lib', timeout=2500, log = self.log)
>         output,err,ret = config.package.Package.executeShellCommand('cd '+self.packageDir+self.installSudo+'&& cp -f lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+' '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''), timeout=2500, log = self.log)
>       except RuntimeError, e:
>         raise RuntimeError('Error running make on Chombo: '+str(e))
> 
> On Thu, 1 Oct 2015, Barry Smith wrote:
> 
>> 
>>  We absolutely NEED configure.log to help you.
>> 
>>  Barry
>> 
>>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
>>> 
>>> Hi list,
>>> 
>>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing. It 
>>> seems to me the chombo build system is not told to link in libgfortran.so, but 
>>> no matter how I try to convince PETSc to add that, it won't. How should this 
>>> be used, please?
>>> 
>>> The full configure line is
>>> 
>>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries --
>>> with-debugging=0 \
>>>         --useThreads 0 --with-clanguage=C++ --with-c-support \
>>>         --with-fortran-interfaces=1 \
>>>         --with-mpi=1 --with-mpi-shared=1 \
>>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
>>> lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
>>> base/liblapack.so \
>>>         --with-scalapack=1 --with-scalapack-include=/usr/include \
>>>         --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
>>>         --with-mumps=0 \
>>> 	  --with-suitesparse=0 \
>>>         --with-spooles=1 --with-spooles-include=/usr/include/spooles \
>>>         --with-spooles-lib=/usr/lib/libspooles.so \
>>>         --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
>>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
>>>         --with-fftw=1 --with-fftw-include=/usr/include \
>>>         --with-fftw-lib=[/usr/lib/x86_64-linux-
>>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
>>>         --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi --with-
>>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
>>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
>>> lgfortran -g" --LIBS="-lgfortran" \
>>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
>>> 	  --with-afterimage=1 --with-afterimage-
>>> include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
>>> linux-gnu/libAfterImage.so \
>>> 	  --with-boost=1 \
>>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
>>> lib=/usr/lib/libnetcdf.so \
>>> 	  --with-triangle=0 \
>>> 	  --with-numpy=1 \
>>> 	  --with-hwloc=1  --with-external-packages-
>>> dir=/home/juhaj/progs+3.6/automatic-downloads \
>>> 	  --with-parmetis=0 --with-metis=0 \
>>> 	  --with-tetgen=0 \
>>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
>>> openmp=1 \
>>> 	  --with-64-bit-indices \
>>> 	  --download-chombo
>>> 
>>> and the final error in configure is
>>> 
>>> /bin/csh -f -c "mpicxx -g          
>>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
>>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
>>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG  -Wl,-rpath,/usr/lib/x86_64-linux-
>>> gnu -L/usr/lib/x86_64-linux-gnu -lhdf5_openmpi     -Wl,-
>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o 
>>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f 
>>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
>>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.mpif90.DEBUG.a(CFLeastSquares.o): 
>>> undefined reference to symbol '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
>>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO missing 
>>> from command line
>>> collect2: error: ld returned 1 exit status
>>> 
>>> 
>>> 
>>> Cheers,
>>> Juha
>> 
>> 
> 


From balay at mcs.anl.gov  Thu Oct  1 13:46:02 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 1 Oct 2015 13:46:02 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <C7553B2F-19F3-4E5F-BF6D-8021FB590040@mcs.anl.gov>
References: <1700733.a0gsQvG38I@vega>
	<81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
	<alpine.LFD.2.20.1510011042451.20838@asterix>
	<C7553B2F-19F3-4E5F-BF6D-8021FB590040@mcs.anl.gov>
Message-ID: <alpine.LFD.2.20.1510011345220.20838@asterix>

On Thu, 1 Oct 2015, Barry Smith wrote:

> 
> > On Oct 1, 2015, at 10:44 AM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > Hm - we usually try to avoiding creating binaries in externalpackages
> > [if not needed] - esp the examples/tests.
> > 
> > So perhaps chombo shoul have the following change.
> 
>  Satish,
> 
>   Looks good to me, push if it works for you.

pushed to maint [and merged to master]

Satish

> 
>    Barry
> 
> > 
> > Satish
> > 
> > ------
> > diff --git a/config/BuildSystem/config/packages/Chombo.py b/config/BuildSystem/config/packages/Chombo.py
> > index effbca6..7a13b0a 100644
> > --- a/config/BuildSystem/config/packages/Chombo.py
> > +++ b/config/BuildSystem/config/packages/Chombo.py
> > @@ -110,7 +110,7 @@ class Configure(config.package.Package):
> >           raise RuntimeError('Error running make on Chombo: config value not found')
> >         config_value=poutput.split('=')[1]
> >         self.logPrint('Chombo installed using config=%s\n'%config_value)
> > -        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make all', timeout=2500, log = self.log)
> > +        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make lib', timeout=2500, log = self.log)
> >         output,err,ret = config.package.Package.executeShellCommand('cd '+self.packageDir+self.installSudo+'&& cp -f lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+' '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''), timeout=2500, log = self.log)
> >       except RuntimeError, e:
> >         raise RuntimeError('Error running make on Chombo: '+str(e))
> > 
> > On Thu, 1 Oct 2015, Barry Smith wrote:
> > 
> >> 
> >>  We absolutely NEED configure.log to help you.
> >> 
> >>  Barry
> >> 
> >>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> >>> 
> >>> Hi list,
> >>> 
> >>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing. It 
> >>> seems to me the chombo build system is not told to link in libgfortran.so, but 
> >>> no matter how I try to convince PETSc to add that, it won't. How should this 
> >>> be used, please?
> >>> 
> >>> The full configure line is
> >>> 
> >>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries --
> >>> with-debugging=0 \
> >>>         --useThreads 0 --with-clanguage=C++ --with-c-support \
> >>>         --with-fortran-interfaces=1 \
> >>>         --with-mpi=1 --with-mpi-shared=1 \
> >>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> >>> lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
> >>> base/liblapack.so \
> >>>         --with-scalapack=1 --with-scalapack-include=/usr/include \
> >>>         --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
> >>>         --with-mumps=0 \
> >>> 	  --with-suitesparse=0 \
> >>>         --with-spooles=1 --with-spooles-include=/usr/include/spooles \
> >>>         --with-spooles-lib=/usr/lib/libspooles.so \
> >>>         --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> >>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
> >>>         --with-fftw=1 --with-fftw-include=/usr/include \
> >>>         --with-fftw-lib=[/usr/lib/x86_64-linux-
> >>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
> >>>         --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi --with-
> >>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> >>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> >>> lgfortran -g" --LIBS="-lgfortran" \
> >>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> >>> 	  --with-afterimage=1 --with-afterimage-
> >>> include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
> >>> linux-gnu/libAfterImage.so \
> >>> 	  --with-boost=1 \
> >>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> >>> lib=/usr/lib/libnetcdf.so \
> >>> 	  --with-triangle=0 \
> >>> 	  --with-numpy=1 \
> >>> 	  --with-hwloc=1  --with-external-packages-
> >>> dir=/home/juhaj/progs+3.6/automatic-downloads \
> >>> 	  --with-parmetis=0 --with-metis=0 \
> >>> 	  --with-tetgen=0 \
> >>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> >>> openmp=1 \
> >>> 	  --with-64-bit-indices \
> >>> 	  --download-chombo
> >>> 
> >>> and the final error in configure is
> >>> 
> >>> /bin/csh -f -c "mpicxx -g          
> >>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> >>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
> >>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG  -Wl,-rpath,/usr/lib/x86_64-linux-
> >>> gnu -L/usr/lib/x86_64-linux-gnu -lhdf5_openmpi     -Wl,-
> >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o 
> >>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f 
> >>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> >>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.mpif90.DEBUG.a(CFLeastSquares.o): 
> >>> undefined reference to symbol '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> >>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO missing 
> >>> from command line
> >>> collect2: error: ld returned 1 exit status
> >>> 
> >>> 
> >>> 
> >>> Cheers,
> >>> Juha
> >> 
> >> 
> > 
> 
> 


From gforsyth at gwu.edu  Thu Oct  1 14:10:59 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Thu, 1 Oct 2015 15:10:59 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k5h_wweUpUVbZW=V_c1X=XypUYwFD=YCU8o9XgbM5b1AA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<F8632F16-0A1F-41FA-9EE3-B1748BF4BA56@mcs.anl.gov>
	<CAHDu6k5h_wweUpUVbZW=V_c1X=XypUYwFD=YCU8o9XgbM5b1AA@mail.gmail.com>
Message-ID: <CAHDu6k6KwTij8v=4ntVERahm80RJ705D4bVu8sVqsLra-eaAdg@mail.gmail.com>

I think I found it.  I thought initially that the problem was in the shift
from KSPSetNullSpace to MatSetNullSpace, but in retrospect that doesn't
make much sense as they ostensibly are offering the exact same
functionality.  I discovered that our code returns the same
KSP_DIVERGED_INDEFINITE_PC error in commit e8f7834, which is the last
commit before KSPSetNullSpace was removed.

I've attached the bisection log between e8f7834 and 9fbf19a (v3.5.4) and
the problem seems to have been introduced in 25a145a ("fixed gamg coarse
grid to be general").

Thanks,
Gil Forsyth

On Wed, Sep 30, 2015 at 4:36 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> The exact same thing ran in serial without the indefinite problem, but it
> does crop up in all parallel runs.
>
> I've attached the failure run log and I'll start bisecting against 3.5.4
> to try to track down the change.
>
> Thanks!
> Gil Forsyth
>
> On Wed, Sep 30, 2015 at 4:24 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Did the exact same thing run in parallel without the indefinite problem?
>>
>>   Run that failure with -info and send all the output
>>
>>    You could use bisection to find out exactly what change in the library
>> breaks your example.
>>
>>   Barry
>>
>> > On Sep 30, 2015, at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> >
>> > Using PETSc master branch solved the problem in serial, but I'm still
>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>> runs to completion when I don't use GAMG.  Log is attached for the
>> following run.
>> >
>> > $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>> >
>> >
>> > Thanks again,
>> > Gil Forsyth
>> >
>> >
>> > On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > Ah, got it.  I'll checkout the master branch and see if the behavior
>> persists.
>> >
>> > Many thanks,
>> > Gil
>> >
>> > On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>> the behavior seems to have changed between versions.  The only difference
>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>> MatSetNullSpace.
>> >
>> > Mark made some GAMG changes which were later reversed because they had
>> unintended consequences like this.
>> > I think what Barry means is, "you should get the behavior you expect
>> using the master branch from PETSc development"
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> >   Update your PETSc
>> >
>> >
>> > > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > >
>> > > Hi Barry,
>> > >
>> > > We aren't explicitly setting GMRES anywhere in the code and I'm not
>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>> works with:
>> > >
>> > > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.5.4
>> > >
>> > > and shows that the coarse grid solver is of type:preonly
>> > >
>> > > running the newer version that uses MatSetNullSpace in its stead and
>> adding in -poisson_mg_coarse_ksp_type preonly
>> > >
>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.6.1
>> > >
>> > > still shows
>> > >
>> > > KSP Object:(poisson_) 1 MPI processes
>> > >   type: cg
>> > >   maximum iterations=10000
>> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> > >   left preconditioning
>> > >   using nonzero initial guess
>> > >   using PRECONDITIONED norm type for convergence test
>> > > PC Object:(poisson_) 1 MPI processes
>> > >   type: gamg
>> > >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>> > >       Cycles per PCApply=1
>> > >       Using Galerkin computed coarse grid matrices
>> > >       GAMG specific options
>> > >         Threshold for dropping small values from graph 0
>> > >         AGG specific options
>> > >           Symmetric graph false
>> > >   Coarse grid solver -- level -------------------------------
>> > >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> > >       type: gmres
>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> > >         GMRES: happy breakdown tolerance 1e-30
>> > >       maximum iterations=1, initial guess is zero
>> > >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> > >       left preconditioning
>> > >       using NONE norm type for convergence test
>> > >
>> > >
>> > > both logs are attached.
>> > >
>> > >
>> > > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > >
>> > >    This can't work. You can't use a GMRES inside a CG.   Try changing
>> to -poisson_mg_coarse_ksp_type preonly
>> > >
>> > > KSP Object:(poisson_) 1 MPI processes
>> > >   type: cg
>> > >
>> > > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> > >       type: gmres
>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> > >         GMRES: happy breakdown tolerance 1e-30
>> > >       maximum iterations=1, initial guess is zero
>> > >
>> > >
>> > > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > > >
>> > > >
>> > > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>> knepley at gmail.com> wrote:
>> > > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>> wrote:
>> > > > Hi all,
>> > > >
>> > > > I've been having some trouble with what should be a relatively
>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>> > > >
>> > > > I'm getting indefinite PC errors for a simple lid-driven cavity
>> test problem, 32x32 at Re 100
>> > > >
>> > > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>> set the null space.  This is for a 2D Poisson system with no immersed
>> boundary and so the null space is the constant vector.
>> > > >
>> > > >   MatNullSpace nsp;
>> > > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>> &nsp); CHKERRQ(ierr);
>> > > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>> > > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>> > > >
>> > > > Clearly this has to happen in the reverse order, since ksp2 would
>> not be created yet.
>> > > >
>> > > > For questions about solvers, we HAVE to see the complete output of
>> -ksp_view so we
>> > > > know what we are dealing with. Its also nice to have
>> -ksp_monitor_true_residual -ksp_converged_reason
>> > > >
>> > > >   Matt
>> > > >
>> > > > Yes -- sorry, those are both in inline files and are called in the
>> reverse order that I wrote them out.
>> > > >
>> > > > I've attached the output of
>> > > >
>> > > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>> -poisson_ksp_converged_reason > kspview.log
>> > > >
>> > > >
>> > > >
>> > > > And then setup the KSP with
>> > > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>> > > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>> > > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>> > > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>> > > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>> > > >
>> > > > The matrix QTBNQ does not change, only the rhs of the system is
>> updated.
>> > > >
>> > > > We run this with `-pc_type gamg -pc_gamg_type agg
>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>> > > >
>> > > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>> > > >
>> > > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>> > > >
>> > > > and the same code diverges after 1 timestep and returns a -8
>> KSP_DIVERGED_INDEFINITE_PC
>> > > >
>> > > > This is weird, especially because if we change nsmooths to 2, it
>> runs for 264 timesteps and the returns the same error.  But we have
>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>> right?
>> > > >
>> > > > Change nsmooths to 3 and it again diverges after 1 timestep.
>> > > >
>> > > > Change nsmooths to 4 and it runs to completion.
>> > > >
>> > > > It seems like either gamg's behavior has changed, or that
>> KSPSetNullSpace was doing something implicitly that we now need to do
>> explicitly in addition to MatSetNullSpace?
>> > > >
>> > > > Thanks,
>> > > > Gil Forsyth
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > > -- Norbert Wiener
>> > > >
>> > > > <kspview.log>
>> > >
>> > >
>> > > <kspview3.5.4><kspview3.6.1>
>> >
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> >
>> > <mpi_n2_indefinite_pc.log>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/64bddaaf/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: bisectlog
Type: application/octet-stream
Size: 2628 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/64bddaaf/attachment-0001.obj>

From mfadams at lbl.gov  Thu Oct  1 14:52:30 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 1 Oct 2015 14:52:30 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
Message-ID: <CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>

Can you please send a good log also, with the ksp_view.
Mark

On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Using PETSc master branch solved the problem in serial, but I'm still
> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
> runs to completion when I don't use GAMG.  Log is attached for the
> following run.
>
> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 $PETIBM_DIR/petibm-git/bin/petibm2d
> -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg
> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>
>
> Thanks again,
> Gil Forsyth
>
>
> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> Ah, got it.  I'll checkout the master branch and see if the behavior
>> persists.
>>
>> Many thanks,
>> Gil
>>
>> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>
>>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>>>> the behavior seems to have changed between versions.  The only difference
>>>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>>>> MatSetNullSpace.
>>>>
>>>
>>> Mark made some GAMG changes which were later reversed because they had
>>> unintended consequences like this.
>>> I think what Barry means is, "you should get the behavior you expect
>>> using the master branch from PETSc development"
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>> wrote:
>>>>
>>>>>
>>>>>   Update your PETSc
>>>>>
>>>>>
>>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>> >
>>>>> > Hi Barry,
>>>>> >
>>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>>> works with:
>>>>> >
>>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>> -poisson_ksp_converged_reason > kspview3.5.4
>>>>> >
>>>>> > and shows that the coarse grid solver is of type:preonly
>>>>> >
>>>>> > running the newer version that uses MatSetNullSpace in its stead and
>>>>> adding in -poisson_mg_coarse_ksp_type preonly
>>>>> >
>>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>>> kspview3.6.1
>>>>> >
>>>>> > still shows
>>>>> >
>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>> >   type: cg
>>>>> >   maximum iterations=10000
>>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>> >   left preconditioning
>>>>> >   using nonzero initial guess
>>>>> >   using PRECONDITIONED norm type for convergence test
>>>>> > PC Object:(poisson_) 1 MPI processes
>>>>> >   type: gamg
>>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>> >       Cycles per PCApply=1
>>>>> >       Using Galerkin computed coarse grid matrices
>>>>> >       GAMG specific options
>>>>> >         Threshold for dropping small values from graph 0
>>>>> >         AGG specific options
>>>>> >           Symmetric graph false
>>>>> >   Coarse grid solver -- level -------------------------------
>>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>> >       type: gmres
>>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>>> Orthogonalization with no iterative refinement
>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>> >       maximum iterations=1, initial guess is zero
>>>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>> >       left preconditioning
>>>>> >       using NONE norm type for convergence test
>>>>> >
>>>>> >
>>>>> > both logs are attached.
>>>>> >
>>>>> >
>>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>> >
>>>>> >    This can't work. You can't use a GMRES inside a CG.   Try
>>>>> changing to -poisson_mg_coarse_ksp_type preonly
>>>>> >
>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>> >   type: cg
>>>>> >
>>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>> >       type: gmres
>>>>> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>>>> Orthogonalization with no iterative refinement
>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>> >       maximum iterations=1, initial guess is zero
>>>>> >
>>>>> >
>>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>> wrote:
>>>>> > >
>>>>> > >
>>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>>> knepley at gmail.com> wrote:
>>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>> wrote:
>>>>> > > Hi all,
>>>>> > >
>>>>> > > I've been having some trouble with what should be a relatively
>>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>>> > >
>>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>>> test problem, 32x32 at Re 100
>>>>> > >
>>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>>>> set the null space.  This is for a 2D Poisson system with no immersed
>>>>> boundary and so the null space is the constant vector.
>>>>> > >
>>>>> > >   MatNullSpace nsp;
>>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>>>> &nsp); CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>>> > >
>>>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>>>> not be created yet.
>>>>> > >
>>>>> > > For questions about solvers, we HAVE to see the complete output of
>>>>> -ksp_view so we
>>>>> > > know what we are dealing with. Its also nice to have
>>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>>> > >
>>>>> > >   Matt
>>>>> > >
>>>>> > > Yes -- sorry, those are both in inline files and are called in the
>>>>> reverse order that I wrote them out.
>>>>> > >
>>>>> > > I've attached the output of
>>>>> > >
>>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>> -poisson_ksp_converged_reason > kspview.log
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > And then setup the KSP with
>>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>>>> CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>>>> CHKERRQ(ierr);
>>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>>> > >
>>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>>> updated.
>>>>> > >
>>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>>> > >
>>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>>> > >
>>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>>> > >
>>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>>> KSP_DIVERGED_INDEFINITE_PC
>>>>> > >
>>>>> > > This is weird, especially because if we change nsmooths to 2, it
>>>>> runs for 264 timesteps and the returns the same error.  But we have
>>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>>> right?
>>>>> > >
>>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>>> > >
>>>>> > > Change nsmooths to 4 and it runs to completion.
>>>>> > >
>>>>> > > It seems like either gamg's behavior has changed, or that
>>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>>> explicitly in addition to MatSetNullSpace?
>>>>> > >
>>>>> > > Thanks,
>>>>> > > Gil Forsyth
>>>>> > >
>>>>> > >
>>>>> > >
>>>>> > > --
>>>>> > > What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> > > -- Norbert Wiener
>>>>> > >
>>>>> > > <kspview.log>
>>>>> >
>>>>> >
>>>>> > <kspview3.5.4><kspview3.6.1>
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/04972487/attachment.html>

From gforsyth at gwu.edu  Thu Oct  1 15:21:24 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Thu, 1 Oct 2015 16:21:24 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
Message-ID: <CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>

I ran for one timestep against 3.5.4 with
#+BEGIN_SRC
petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
-poisson_pc_type gamg -poisson_pc_gamg_type agg
-poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
kspview_3.5.4.log
#+END_SRC

and then against 25a145a with the same inputs.  I notice that the poisson
multigrid solve in 25a145a is using GMRES again while 3.5.4 is using
preonly.

Logs from both runs are attached.


On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Can you please send a good log also, with the ksp_view.
> Mark
>
> On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> Using PETSc master branch solved the problem in serial, but I'm still
>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>> runs to completion when I don't use GAMG.  Log is attached for the
>> following run.
>>
>> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>>
>>
>> Thanks again,
>> Gil Forsyth
>>
>>
>> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>
>>> Ah, got it.  I'll checkout the master branch and see if the behavior
>>> persists.
>>>
>>> Many thanks,
>>> Gil
>>>
>>> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>
>>>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show
>>>>> that the behavior seems to have changed between versions.  The only
>>>>> difference in our code between 3.5.4 and 3.6.1 is the change from
>>>>> KSPSetNullSpace to MatSetNullSpace.
>>>>>
>>>>
>>>> Mark made some GAMG changes which were later reversed because they had
>>>> unintended consequences like this.
>>>> I think what Barry means is, "you should get the behavior you expect
>>>> using the master branch from PETSc development"
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>>   Update your PETSc
>>>>>>
>>>>>>
>>>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>>> >
>>>>>> > Hi Barry,
>>>>>> >
>>>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>>>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>>>> works with:
>>>>>> >
>>>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>> -poisson_ksp_converged_reason > kspview3.5.4
>>>>>> >
>>>>>> > and shows that the coarse grid solver is of type:preonly
>>>>>> >
>>>>>> > running the newer version that uses MatSetNullSpace in its stead
>>>>>> and adding in -poisson_mg_coarse_ksp_type preonly
>>>>>> >
>>>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>>>> kspview3.6.1
>>>>>> >
>>>>>> > still shows
>>>>>> >
>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>> >   type: cg
>>>>>> >   maximum iterations=10000
>>>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>> >   left preconditioning
>>>>>> >   using nonzero initial guess
>>>>>> >   using PRECONDITIONED norm type for convergence test
>>>>>> > PC Object:(poisson_) 1 MPI processes
>>>>>> >   type: gamg
>>>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>>> >       Cycles per PCApply=1
>>>>>> >       Using Galerkin computed coarse grid matrices
>>>>>> >       GAMG specific options
>>>>>> >         Threshold for dropping small values from graph 0
>>>>>> >         AGG specific options
>>>>>> >           Symmetric graph false
>>>>>> >   Coarse grid solver -- level -------------------------------
>>>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>> >       type: gmres
>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>> >       left preconditioning
>>>>>> >       using NONE norm type for convergence test
>>>>>> >
>>>>>> >
>>>>>> > both logs are attached.
>>>>>> >
>>>>>> >
>>>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>> wrote:
>>>>>> >
>>>>>> >    This can't work. You can't use a GMRES inside a CG.   Try
>>>>>> changing to -poisson_mg_coarse_ksp_type preonly
>>>>>> >
>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>> >   type: cg
>>>>>> >
>>>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>> >       type: gmres
>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>> >
>>>>>> >
>>>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>> wrote:
>>>>>> > >
>>>>>> > >
>>>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>>>> knepley at gmail.com> wrote:
>>>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>> wrote:
>>>>>> > > Hi all,
>>>>>> > >
>>>>>> > > I've been having some trouble with what should be a relatively
>>>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>>>> > >
>>>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>>>> test problem, 32x32 at Re 100
>>>>>> > >
>>>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>>>>> set the null space.  This is for a 2D Poisson system with no immersed
>>>>>> boundary and so the null space is the constant vector.
>>>>>> > >
>>>>>> > >   MatNullSpace nsp;
>>>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0,
>>>>>> NULL, &nsp); CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>>>> > >
>>>>>> > > Clearly this has to happen in the reverse order, since ksp2 would
>>>>>> not be created yet.
>>>>>> > >
>>>>>> > > For questions about solvers, we HAVE to see the complete output
>>>>>> of -ksp_view so we
>>>>>> > > know what we are dealing with. Its also nice to have
>>>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>>>> > >
>>>>>> > >   Matt
>>>>>> > >
>>>>>> > > Yes -- sorry, those are both in inline files and are called in
>>>>>> the reverse order that I wrote them out.
>>>>>> > >
>>>>>> > > I've attached the output of
>>>>>> > >
>>>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>> -poisson_ksp_converged_reason > kspview.log
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > And then setup the KSP with
>>>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>>>>> CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>>>>> CHKERRQ(ierr);
>>>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>>>> > >
>>>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>>>> updated.
>>>>>> > >
>>>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>>>> > >
>>>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>>>> > >
>>>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>>>> > >
>>>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>>>> KSP_DIVERGED_INDEFINITE_PC
>>>>>> > >
>>>>>> > > This is weird, especially because if we change nsmooths to 2, it
>>>>>> runs for 264 timesteps and the returns the same error.  But we have
>>>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>>>> right?
>>>>>> > >
>>>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>>>> > >
>>>>>> > > Change nsmooths to 4 and it runs to completion.
>>>>>> > >
>>>>>> > > It seems like either gamg's behavior has changed, or that
>>>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>>>> explicitly in addition to MatSetNullSpace?
>>>>>> > >
>>>>>> > > Thanks,
>>>>>> > > Gil Forsyth
>>>>>> > >
>>>>>> > >
>>>>>> > >
>>>>>> > > --
>>>>>> > > What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> > > -- Norbert Wiener
>>>>>> > >
>>>>>> > > <kspview.log>
>>>>>> >
>>>>>> >
>>>>>> > <kspview3.5.4><kspview3.6.1>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/69ce2e07/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview_3.5.4.log
Type: text/x-log
Size: 32521 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/69ce2e07/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview_25a145a.log
Type: text/x-log
Size: 34303 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/69ce2e07/attachment-0003.bin>

From gforsyth at gwu.edu  Thu Oct  1 15:27:23 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Thu, 1 Oct 2015 16:27:23 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
Message-ID: <CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>

Hi Mark,

I just noticed that in the previous commit 7743f89, it's also using GMRES
in the multigrid solve but doesn't complain until the 2nd timestep, so my
bisection criteria is off, since I was giving commits a PASS if they made
it through the one timestep without complaining about the indefinite PC. I
think I'm still close to the problem commit, but it's probably a little bit
before 25a145a.  Apologies for the goose chase.

Thanks,
Gil Forsyth

On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> I ran for one timestep against 3.5.4 with
> #+BEGIN_SRC
> petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
> -poisson_pc_type gamg -poisson_pc_gamg_type agg
> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
> kspview_3.5.4.log
> #+END_SRC
>
> and then against 25a145a with the same inputs.  I notice that the poisson
> multigrid solve in 25a145a is using GMRES again while 3.5.4 is using
> preonly.
>
> Logs from both runs are attached.
>
>
> On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
>
>> Can you please send a good log also, with the ksp_view.
>> Mark
>>
>> On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>
>>> Using PETSc master branch solved the problem in serial, but I'm still
>>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>>> runs to completion when I don't use GAMG.  Log is attached for the
>>> following run.
>>>
>>> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>>>
>>>
>>> Thanks again,
>>> Gil Forsyth
>>>
>>>
>>> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>
>>>> Ah, got it.  I'll checkout the master branch and see if the behavior
>>>> persists.
>>>>
>>>> Many thanks,
>>>> Gil
>>>>
>>>> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>> wrote:
>>>>>
>>>>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show
>>>>>> that the behavior seems to have changed between versions.  The only
>>>>>> difference in our code between 3.5.4 and 3.6.1 is the change from
>>>>>> KSPSetNullSpace to MatSetNullSpace.
>>>>>>
>>>>>
>>>>> Mark made some GAMG changes which were later reversed because they had
>>>>> unintended consequences like this.
>>>>> I think what Barry means is, "you should get the behavior you expect
>>>>> using the master branch from PETSc development"
>>>>>
>>>>>   Thanks,
>>>>>
>>>>>      Matt
>>>>>
>>>>>
>>>>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>   Update your PETSc
>>>>>>>
>>>>>>>
>>>>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi Barry,
>>>>>>> >
>>>>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm
>>>>>>> not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>>>>> works with:
>>>>>>> >
>>>>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>> -poisson_ksp_converged_reason > kspview3.5.4
>>>>>>> >
>>>>>>> > and shows that the coarse grid solver is of type:preonly
>>>>>>> >
>>>>>>> > running the newer version that uses MatSetNullSpace in its stead
>>>>>>> and adding in -poisson_mg_coarse_ksp_type preonly
>>>>>>> >
>>>>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>>>>> kspview3.6.1
>>>>>>> >
>>>>>>> > still shows
>>>>>>> >
>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>> >   type: cg
>>>>>>> >   maximum iterations=10000
>>>>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>> >   left preconditioning
>>>>>>> >   using nonzero initial guess
>>>>>>> >   using PRECONDITIONED norm type for convergence test
>>>>>>> > PC Object:(poisson_) 1 MPI processes
>>>>>>> >   type: gamg
>>>>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>>>> >       Cycles per PCApply=1
>>>>>>> >       Using Galerkin computed coarse grid matrices
>>>>>>> >       GAMG specific options
>>>>>>> >         Threshold for dropping small values from graph 0
>>>>>>> >         AGG specific options
>>>>>>> >           Symmetric graph false
>>>>>>> >   Coarse grid solver -- level -------------------------------
>>>>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>> >       type: gmres
>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>> >       left preconditioning
>>>>>>> >       using NONE norm type for convergence test
>>>>>>> >
>>>>>>> >
>>>>>>> > both logs are attached.
>>>>>>> >
>>>>>>> >
>>>>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> >    This can't work. You can't use a GMRES inside a CG.   Try
>>>>>>> changing to -poisson_mg_coarse_ksp_type preonly
>>>>>>> >
>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>> >   type: cg
>>>>>>> >
>>>>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>> >       type: gmres
>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>> >
>>>>>>> >
>>>>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>> wrote:
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>>>>> knepley at gmail.com> wrote:
>>>>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>> wrote:
>>>>>>> > > Hi all,
>>>>>>> > >
>>>>>>> > > I've been having some trouble with what should be a relatively
>>>>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>>>>> > >
>>>>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>>>>> test problem, 32x32 at Re 100
>>>>>>> > >
>>>>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>>>>>> set the null space.  This is for a 2D Poisson system with no immersed
>>>>>>> boundary and so the null space is the constant vector.
>>>>>>> > >
>>>>>>> > >   MatNullSpace nsp;
>>>>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0,
>>>>>>> NULL, &nsp); CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>>>>> > >
>>>>>>> > > Clearly this has to happen in the reverse order, since ksp2
>>>>>>> would not be created yet.
>>>>>>> > >
>>>>>>> > > For questions about solvers, we HAVE to see the complete output
>>>>>>> of -ksp_view so we
>>>>>>> > > know what we are dealing with. Its also nice to have
>>>>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>> > >
>>>>>>> > >   Matt
>>>>>>> > >
>>>>>>> > > Yes -- sorry, those are both in inline files and are called in
>>>>>>> the reverse order that I wrote them out.
>>>>>>> > >
>>>>>>> > > I've attached the output of
>>>>>>> > >
>>>>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>> -poisson_ksp_converged_reason > kspview.log
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > And then setup the KSP with
>>>>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>>>>>> CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>>>>>> CHKERRQ(ierr);
>>>>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>>>>> > >
>>>>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>>>>> updated.
>>>>>>> > >
>>>>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>>>>> > >
>>>>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>>>>> > >
>>>>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>>>>> > >
>>>>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>>>>> KSP_DIVERGED_INDEFINITE_PC
>>>>>>> > >
>>>>>>> > > This is weird, especially because if we change nsmooths to 2, it
>>>>>>> runs for 264 timesteps and the returns the same error.  But we have
>>>>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>>>>> right?
>>>>>>> > >
>>>>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>>>>> > >
>>>>>>> > > Change nsmooths to 4 and it runs to completion.
>>>>>>> > >
>>>>>>> > > It seems like either gamg's behavior has changed, or that
>>>>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>>>>> explicitly in addition to MatSetNullSpace?
>>>>>>> > >
>>>>>>> > > Thanks,
>>>>>>> > > Gil Forsyth
>>>>>>> > >
>>>>>>> > >
>>>>>>> > >
>>>>>>> > > --
>>>>>>> > > What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> > > -- Norbert Wiener
>>>>>>> > >
>>>>>>> > > <kspview.log>
>>>>>>> >
>>>>>>> >
>>>>>>> > <kspview3.5.4><kspview3.6.1>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> What most experimenters take for granted before they begin their
>>>>> experiments is infinitely more interesting than any results to which their
>>>>> experiments lead.
>>>>> -- Norbert Wiener
>>>>>
>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/a74c9263/attachment.html>

From matt.landreman at gmail.com  Thu Oct  1 19:04:42 2015
From: matt.landreman at gmail.com (Matt Landreman)
Date: Thu, 1 Oct 2015 20:04:42 -0400
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <FA18A1A8-D8D3-4B84-BBDA-6DDE76E84ABC@mcs.anl.gov>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
	<CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
	<0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>
	<FA18A1A8-D8D3-4B84-BBDA-6DDE76E84ABC@mcs.anl.gov>
Message-ID: <CADZ9mv=5p8i+UuCsJZk6M+OC1FoiVFLR9wReiPmAxkjf_cyf3w@mail.gmail.com>

Hi Barry,
Your suggestion of removing the "if (mumps->CleanUpMUMPS)" in mumps.c did
resolve the problem for me.
Thanks,
-Matt

On Wed, Sep 30, 2015 at 6:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>   Matt,
>
>    Please try the following: edit
>
> #undef __FUNCT__
> #define __FUNCT__ "MatDestroy_MUMPS"
> PetscErrorCode MatDestroy_MUMPS(Mat A)
> {
>   Mat_MUMPS      *mumps=(Mat_MUMPS*)A->spptr;
>   PetscErrorCode ierr;
>
>   PetscFunctionBegin;
>   if (mumps->CleanUpMUMPS) {
>
>   Remove this if () test and just always do the lines of clean up code
> after it. Let us know if this resolves the problem?
>
> Thanks
>
>    Barry
>
> This CleanUpMUMPS flag has always be goofy and definitely needs to be
> removed, the only question is if some other changes are needed when it is
> removed.
>
>
> > On Sep 30, 2015, at 4:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> >  Matt,
> >
> >   Yes, you must be right The MatDestroy() on the "partially factored"
> matrix should clean up everything properly but it sounds like it is not.
> I'll look at it right now but I only have a few minutes; if I can't resolve
> it really quickly it may take a day or two.
> >
> >
> >  Barry
> >
> >> On Sep 30, 2015, at 4:10 PM, Matt Landreman <matt.landreman at gmail.com>
> wrote:
> >>
> >> Hi Barry,
> >> I tried adding PetscMallocDump after SNESDestroy as you suggested. When
> mumps fails, PetscMallocDump shows a number of mallocs which are absent
> when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij()
> (line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total
> memory reported by PetscMallocDump after SNESDestroy is substantially
> (>20x) larger when mumps fails than when mumps succeeds, and this amount
> increases uniformly with each mumps failure.  So I think some of the
> mumps-related structures are not being deallocated by SNESDestroy if mumps
> generates an error.
> >> Thanks,
> >> -Matt
> >>
> >> On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >>
> >>> On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com>
> wrote:
> >>>
> >>> PETSc developers,
> >>>
> >>> I tried implementing a system for automatically increasing MUMPS
> ICNTL(14), along the lines described in this recent thread. If SNESSolve
> returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy,
> re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14),
> call SNESSolve again, and repeat as needed. The procedure works, but the
> peak memory required (as measured by the HPC system) is 50%-100% higher if
> the MUMPS solve has to be repeated compared to when MUMPS works on the 1st
> try (by starting with a large ICNTL(14)), even though SNESDestroy is called
> in between the attempts. Are there some PETSc or MUMPS structures which
> would not be deallocated immediately by SNESDestroy?  If so, how do I
> deallocate them?
> >>
> >>   They should be all destroyed automatically for you. You can use
> PetscMallocDump() after the SNES is destroyed to verify that all that
> memory is not properly freed.
> >>
> >>   My guess is that your new malloc() with the bigger workspace cannot
> "reuse" the space that was previously freed; so to the OS it looks like you
> are using a lot more space but in terms of physical memory you are not
> using more.
> >>
> >>  Barry
> >>
> >>>
> >>> Thanks,
> >>> Matt Landreman
> >>>
> >>>
> >>> On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> >>> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >>> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <
> david.knezevic at akselos.com> wrote:
> >>> In some cases, I get MUMPS error -9, i.e.:
> >>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization
> phase: INFO(1)=-9, INFO(2)=98927
> >>>
> >>> This is easily fixed by re-running the executable with
> -mat_mumps_icntl_14 on the commandline.
> >>>
> >>> However, I would like to update my code in order to do this
> automatically, i.e. detect the -9 error and re-run with the appropriate
> option. Is there a recommended way to do this? It seems to me that I could
> do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to
> call a function that sets the appropriate option and solves again, is that
> right? Are there any examples that illustrate this type of thing?
> >>>
> >>> I would not use the error handler. I would just check the ierr return
> code from the solver. I think you need the
> >>> INFO output, for which you can use MatMumpsGetInfo().
> >>>
> >>>
> >>> OK, that sounds good (and much simpler than what I had in mind),
> thanks for the help!
> >>>
> >>> David
> >>>
> >>>
> >>
> >>
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151001/74f1925c/attachment-0001.html>

From bsmith at mcs.anl.gov  Thu Oct  1 19:13:32 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 1 Oct 2015 19:13:32 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CADZ9mv=5p8i+UuCsJZk6M+OC1FoiVFLR9wReiPmAxkjf_cyf3w@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
	<CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
	<0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>
	<FA18A1A8-D8D3-4B84-BBDA-6DDE76E84ABC@mcs.anl.gov>
	<CADZ9mv=5p8i+UuCsJZk6M+OC1FoiVFLR9wReiPmAxkjf_cyf3w@mail.gmail.com>
Message-ID: <B0E2AFD6-C436-4D3D-A63A-2A28E273CF93@mcs.anl.gov>


  Excellent we'll make the change in PETSc and do our own testing. And eventually add the clean return from failed MUMPS functionality in PETSc.
> On Oct 1, 2015, at 7:04 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> 
> Hi Barry,
> Your suggestion of removing the "if (mumps->CleanUpMUMPS)" in mumps.c did resolve the problem for me.
> Thanks,
> -Matt
> 
> On Wed, Sep 30, 2015 at 6:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Matt,
> 
>    Please try the following: edit
> 
> #undef __FUNCT__
> #define __FUNCT__ "MatDestroy_MUMPS"
> PetscErrorCode MatDestroy_MUMPS(Mat A)
> {
>   Mat_MUMPS      *mumps=(Mat_MUMPS*)A->spptr;
>   PetscErrorCode ierr;
> 
>   PetscFunctionBegin;
>   if (mumps->CleanUpMUMPS) {
> 
>   Remove this if () test and just always do the lines of clean up code after it. Let us know if this resolves the problem?
> 
> Thanks
> 
>    Barry
> 
> This CleanUpMUMPS flag has always be goofy and definitely needs to be removed, the only question is if some other changes are needed when it is removed.
> 
> 
> > On Sep 30, 2015, at 4:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> >  Matt,
> >
> >   Yes, you must be right The MatDestroy() on the "partially factored" matrix should clean up everything properly but it sounds like it is not. I'll look at it right now but I only have a few minutes; if I can't resolve it really quickly it may take a day or two.
> >
> >
> >  Barry
> >
> >> On Sep 30, 2015, at 4:10 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> >>
> >> Hi Barry,
> >> I tried adding PetscMallocDump after SNESDestroy as you suggested. When mumps fails, PetscMallocDump shows a number of mallocs which are absent when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij() (line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total memory reported by PetscMallocDump after SNESDestroy is substantially (>20x) larger when mumps fails than when mumps succeeds, and this amount increases uniformly with each mumps failure.  So I think some of the mumps-related structures are not being deallocated by SNESDestroy if mumps generates an error.
> >> Thanks,
> >> -Matt
> >>
> >> On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>
> >>> On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> >>>
> >>> PETSc developers,
> >>>
> >>> I tried implementing a system for automatically increasing MUMPS ICNTL(14), along the lines described in this recent thread. If SNESSolve returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again, and repeat as needed. The procedure works, but the peak memory required (as measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be repeated compared to when MUMPS works on the 1st try (by starting with a large ICNTL(14)), even though SNESDestroy is called in between the attempts. Are there some PETSc or MUMPS structures which would not be deallocated immediately by SNESDestroy?  If so, how do I deallocate them?
> >>
> >>   They should be all destroyed automatically for you. You can use PetscMallocDump() after the SNES is destroyed to verify that all that memory is not properly freed.
> >>
> >>   My guess is that your new malloc() with the bigger workspace cannot "reuse" the space that was previously freed; so to the OS it looks like you are using a lot more space but in terms of physical memory you are not using more.
> >>
> >>  Barry
> >>
> >>>
> >>> Thanks,
> >>> Matt Landreman
> >>>
> >>>
> >>> On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> >>> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >>> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> >>> In some cases, I get MUMPS error -9, i.e.:
> >>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=98927
> >>>
> >>> This is easily fixed by re-running the executable with -mat_mumps_icntl_14 on the commandline.
> >>>
> >>> However, I would like to update my code in order to do this automatically, i.e. detect the -9 error and re-run with the appropriate option. Is there a recommended way to do this? It seems to me that I could do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to call a function that sets the appropriate option and solves again, is that right? Are there any examples that illustrate this type of thing?
> >>>
> >>> I would not use the error handler. I would just check the ierr return code from the solver. I think you need the
> >>> INFO output, for which you can use MatMumpsGetInfo().
> >>>
> >>>
> >>> OK, that sounds good (and much simpler than what I had in mind), thanks for the help!
> >>>
> >>> David
> >>>
> >>>
> >>
> >>
> >
> 
> 


From bsmith at mcs.anl.gov  Thu Oct  1 23:13:51 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 1 Oct 2015 23:13:51 -0500
Subject: [petsc-users] Automatically re-solving after MUMPS error
In-Reply-To: <CADZ9mv=5p8i+UuCsJZk6M+OC1FoiVFLR9wReiPmAxkjf_cyf3w@mail.gmail.com>
References: <CAJCWK9AwHDS0+eC5T+ZSF5kySqZmSywx7cHfnKHHLtCget29Dw@mail.gmail.com>
	<CAMYG4Gn0WOe6Y8w3k_1PnvDkexEuS7n9CKzodT7yvvr9Me6khg@mail.gmail.com>
	<CAJCWK9BihEkqFefFMPjQNBj5gzp6Zfh-r_gdh=Mh+SL5GJcD5A@mail.gmail.com>
	<CADZ9mvkNH3CvwZigy5cccX5q8oPeHOvDVxPn52q3J3wvn00xWQ@mail.gmail.com>
	<E589862D-F077-4233-A0DD-48F9257F32C3@mcs.anl.gov>
	<CADZ9mvkJM02+BsSkAtrsrfHXPD2WEKzkeqDR_1_WR40Mm3mfDA@mail.gmail.com>
	<0E0EDF20-41C6-4C5D-A32E-837F0FCE38D6@mcs.anl.gov>
	<FA18A1A8-D8D3-4B84-BBDA-6DDE76E84ABC@mcs.anl.gov>
	<CADZ9mv=5p8i+UuCsJZk6M+OC1FoiVFLR9wReiPmAxkjf_cyf3w@mail.gmail.com>
Message-ID: <02C401CE-45C9-4B89-9766-9A9E07D13597@mcs.anl.gov>


  Great, the error is fixed in barry/fix-mumps-destroy, merged into next for testing and will go into master soon.

  Barry

> On Oct 1, 2015, at 7:04 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> 
> Hi Barry,
> Your suggestion of removing the "if (mumps->CleanUpMUMPS)" in mumps.c did resolve the problem for me.
> Thanks,
> -Matt
> 
> On Wed, Sep 30, 2015 at 6:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Matt,
> 
>    Please try the following: edit
> 
> #undef __FUNCT__
> #define __FUNCT__ "MatDestroy_MUMPS"
> PetscErrorCode MatDestroy_MUMPS(Mat A)
> {
>   Mat_MUMPS      *mumps=(Mat_MUMPS*)A->spptr;
>   PetscErrorCode ierr;
> 
>   PetscFunctionBegin;
>   if (mumps->CleanUpMUMPS) {
> 
>   Remove this if () test and just always do the lines of clean up code after it. Let us know if this resolves the problem?
> 
> Thanks
> 
>    Barry
> 
> This CleanUpMUMPS flag has always be goofy and definitely needs to be removed, the only question is if some other changes are needed when it is removed.
> 
> 
> > On Sep 30, 2015, at 4:59 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >
> >  Matt,
> >
> >   Yes, you must be right The MatDestroy() on the "partially factored" matrix should clean up everything properly but it sounds like it is not. I'll look at it right now but I only have a few minutes; if I can't resolve it really quickly it may take a day or two.
> >
> >
> >  Barry
> >
> >> On Sep 30, 2015, at 4:10 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> >>
> >> Hi Barry,
> >> I tried adding PetscMallocDump after SNESDestroy as you suggested. When mumps fails, PetscMallocDump shows a number of mallocs which are absent when mumps succeeds, the largest being MatConvertToTriples_mpiaij_mpiaij() (line 638 in petsc-3.6.0/src/mat/impls/aij/mpi/mumps/mumps.c).  The total memory reported by PetscMallocDump after SNESDestroy is substantially (>20x) larger when mumps fails than when mumps succeeds, and this amount increases uniformly with each mumps failure.  So I think some of the mumps-related structures are not being deallocated by SNESDestroy if mumps generates an error.
> >> Thanks,
> >> -Matt
> >>
> >> On Wed, Sep 30, 2015 at 2:16 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >>
> >>> On Sep 30, 2015, at 1:06 PM, Matt Landreman <matt.landreman at gmail.com> wrote:
> >>>
> >>> PETSc developers,
> >>>
> >>> I tried implementing a system for automatically increasing MUMPS ICNTL(14), along the lines described in this recent thread. If SNESSolve returns ierr .ne. 0 due to MUMPS error -9, I call SNESDestroy, re-initialize SNES, call MatMumpsSetIcntl with a larger value of ICNTL(14), call SNESSolve again, and repeat as needed. The procedure works, but the peak memory required (as measured by the HPC system) is 50%-100% higher if the MUMPS solve has to be repeated compared to when MUMPS works on the 1st try (by starting with a large ICNTL(14)), even though SNESDestroy is called in between the attempts. Are there some PETSc or MUMPS structures which would not be deallocated immediately by SNESDestroy?  If so, how do I deallocate them?
> >>
> >>   They should be all destroyed automatically for you. You can use PetscMallocDump() after the SNES is destroyed to verify that all that memory is not properly freed.
> >>
> >>   My guess is that your new malloc() with the bigger workspace cannot "reuse" the space that was previously freed; so to the OS it looks like you are using a lot more space but in terms of physical memory you are not using more.
> >>
> >>  Barry
> >>
> >>>
> >>> Thanks,
> >>> Matt Landreman
> >>>
> >>>
> >>> On Tue, Sep 15, 2015 at 7:47 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> >>> On Tue, Sep 15, 2015 at 7:29 PM, Matthew Knepley <knepley at gmail.com> wrote:
> >>> On Tue, Sep 15, 2015 at 4:30 AM, David Knezevic <david.knezevic at akselos.com> wrote:
> >>> In some cases, I get MUMPS error -9, i.e.:
> >>> [2]PETSC ERROR: Error reported by MUMPS in numerical factorization phase: INFO(1)=-9, INFO(2)=98927
> >>>
> >>> This is easily fixed by re-running the executable with -mat_mumps_icntl_14 on the commandline.
> >>>
> >>> However, I would like to update my code in order to do this automatically, i.e. detect the -9 error and re-run with the appropriate option. Is there a recommended way to do this? It seems to me that I could do this with a PETSc error handler (e.g. PetscPushErrorHandler) in order to call a function that sets the appropriate option and solves again, is that right? Are there any examples that illustrate this type of thing?
> >>>
> >>> I would not use the error handler. I would just check the ierr return code from the solver. I think you need the
> >>> INFO output, for which you can use MatMumpsGetInfo().
> >>>
> >>>
> >>> OK, that sounds good (and much simpler than what I had in mind), thanks for the help!
> >>>
> >>> David
> >>>
> >>>
> >>
> >>
> >
> 
> 


From juhaj at iki.fi  Fri Oct  2 04:38:56 2015
From: juhaj at iki.fi (Juha Jaykka)
Date: Fri, 02 Oct 2015 10:38:56 +0100
Subject: [petsc-users] petsc+chombo
In-Reply-To: <81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
References: <1700733.a0gsQvG38I@vega>
	<81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
Message-ID: <2142492.ySKjvTbp0E@vega>

Dear Barry,

Please find the configure.log at http://www.iki.fi/~juhaj/configure.log.

On a more general level, I wonder if the chombo-petsc interoperability is 
documented anywhere? I grepped through the examples and came up blank.

Cheers,
Juha

On Thursday 01 Oct 2015 09:53:26 Barry Smith wrote:
>   We absolutely NEED configure.log to help you.
> 
>   Barry
> 
> > On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> > 
> > Hi list,
> > 
> > I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing.
> > It seems to me the chombo build system is not told to link in
> > libgfortran.so, but no matter how I try to convince PETSc to add that, it
> > won't. How should this be used, please?
> > 
> > The full configure line is
> > 
> > ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries
> > --
> > with-debugging=0 \
> > 
> >          --useThreads 0 --with-clanguage=C++ --with-c-support \
> >          --with-fortran-interfaces=1 \
> >          --with-mpi=1 --with-mpi-shared=1 \
> > 	  
> > 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> > 
> > lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
> > base/liblapack.so \
> > 
> >          --with-scalapack=1 --with-scalapack-include=/usr/include \
> >          --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
> >          --with-mumps=0 \
> > 	  
> > 	  --with-suitesparse=0 \
> > 	  
> >          --with-spooles=1 --with-spooles-include=/usr/include/spooles \
> >          --with-spooles-lib=/usr/lib/libspooles.so \
> >          --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> > 
> > lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
> > 
> >          --with-fftw=1 --with-fftw-include=/usr/include \
> >          --with-fftw-lib=[/usr/lib/x86_64-linux-
> > 
> > gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
> > 
> >          --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi
> >          --with-
> > 
> > hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> > 
> > 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> > 
> > lgfortran -g" --LIBS="-lgfortran" \
> > 
> > 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> > 	  --with-afterimage=1 --with-afterimage-
> > 
> > include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
> > linux-gnu/libAfterImage.so \
> > 
> > 	  --with-boost=1 \
> > 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> > 
> > lib=/usr/lib/libnetcdf.so \
> > 
> > 	  --with-triangle=0 \
> > 	  --with-numpy=1 \
> > 	  --with-hwloc=1  --with-external-packages-
> > 
> > dir=/home/juhaj/progs+3.6/automatic-downloads \
> > 
> > 	  --with-parmetis=0 --with-metis=0 \
> > 	  --with-tetgen=0 \
> > 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> > 
> > openmp=1 \
> > 
> > 	  --with-64-bit-indices \
> > 	  --download-chombo
> > 
> > and the final error in configure is
> > 
> > /bin/csh -f -c "mpicxx -g
> > o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> > L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../.. -
> > lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG 
> > -Wl,-rpath,/usr/lib/x86_64-linux- gnu -L/usr/lib/x86_64-linux-gnu
> > -lhdf5_openmpi     -Wl,-
> > rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> > rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o
> > testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f
> > /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> > /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.m
> > pif90.DEBUG.a(CFLeastSquares.o): undefined reference to symbol
> > '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> > /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO
> > missing from command line
> > collect2: error: ld returned 1 exit status
> > 
> > 
> > 
> > Cheers,
> > Juha
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: This is a digitally signed message part.
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151002/9662e594/attachment.pgp>

From bsmith at mcs.anl.gov  Fri Oct  2 13:23:40 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 2 Oct 2015 13:23:40 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <2142492.ySKjvTbp0E@vega>
References: <1700733.a0gsQvG38I@vega>
	<81440C76-646E-4A2E-BE6C-FAA8B1C16006@mcs.anl.gov>
	<2142492.ySKjvTbp0E@vega>
Message-ID: <F5FDAB99-6218-4429-8941-803783CC26DB@mcs.anl.gov>


  I think Satish already told you the fix. Please let us know if it does not work.

Hm - we usually try to avoiding creating binaries in externalpackages
[if not needed] - esp the examples/tests.

So perhaps chombo shoul have the following change.

Satish

------
diff --git a/config/BuildSystem/config/packages/Chombo.py b/config/BuildSystem/config/packages/Chombo.py
index effbca6..7a13b0a 100644
--- a/config/BuildSystem/config/packages/Chombo.py
+++ b/config/BuildSystem/config/packages/Chombo.py
@@ -110,7 +110,7 @@ class Configure(config.package.Package):
          raise RuntimeError('Error running make on Chombo: config value not found')
        config_value=poutput.split('=')[1]
        self.logPrint('Chombo installed using config=%s\n'%config_value)
-        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make all', timeout=2500, log = self.log)
+        output,err,ret = config.package.Package.executeShellCommand('cd '+os.path.join(self.packageDir,'lib') +' && make clean && make lib', timeout=2500, log = self.log)
        output,err,ret = config.package.Package.executeShellCommand('cd '+self.packageDir+self.installSudo+'&& cp -f lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+' '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''), timeout=2500, log = self.log)
      except RuntimeError, e:
        raise RuntimeError('Error running make on Chombo: '+str(e))


> On Oct 2, 2015, at 4:38 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> 
> Dear Barry,
> 
> Please find the configure.log at http://www.iki.fi/~juhaj/configure.log.
> 
> On a more general level, I wonder if the chombo-petsc interoperability is 
> documented anywhere? I grepped through the examples and came up blank.
> 
> Cheers,
> Juha
> 
> On Thursday 01 Oct 2015 09:53:26 Barry Smith wrote:
>>  We absolutely NEED configure.log to help you.
>> 
>>  Barry
>> 
>>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
>>> 
>>> Hi list,
>>> 
>>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep failing.
>>> It seems to me the chombo build system is not told to link in
>>> libgfortran.so, but no matter how I try to convince PETSc to add that, it
>>> won't. How should this be used, please?
>>> 
>>> The full configure line is
>>> 
>>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries
>>> --
>>> with-debugging=0 \
>>> 
>>>         --useThreads 0 --with-clanguage=C++ --with-c-support \
>>>         --with-fortran-interfaces=1 \
>>>         --with-mpi=1 --with-mpi-shared=1 \
>>> 	  
>>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
>>> 
>>> lib=/usr/lib/openblas-base/libblas.so --with-lapack-lib=/usr/lib/openblas-
>>> base/liblapack.so \
>>> 
>>>         --with-scalapack=1 --with-scalapack-include=/usr/include \
>>>         --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
>>>         --with-mumps=0 \
>>> 	  
>>> 	  --with-suitesparse=0 \
>>> 	  
>>>         --with-spooles=1 --with-spooles-include=/usr/include/spooles \
>>>         --with-spooles-lib=/usr/lib/libspooles.so \
>>>         --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
>>> 
>>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
>>> 
>>>         --with-fftw=1 --with-fftw-include=/usr/include \
>>>         --with-fftw-lib=[/usr/lib/x86_64-linux-
>>> 
>>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
>>> 
>>>         --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi
>>>         --with-
>>> 
>>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
>>> 
>>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
>>> 
>>> lgfortran -g" --LIBS="-lgfortran" \
>>> 
>>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
>>> 	  --with-afterimage=1 --with-afterimage-
>>> 
>>> include=/usr/include/libAfterImage --with-afterimage-lib=/usr/lib/x86_64-
>>> linux-gnu/libAfterImage.so \
>>> 
>>> 	  --with-boost=1 \
>>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
>>> 
>>> lib=/usr/lib/libnetcdf.so \
>>> 
>>> 	  --with-triangle=0 \
>>> 	  --with-numpy=1 \
>>> 	  --with-hwloc=1  --with-external-packages-
>>> 
>>> dir=/home/juhaj/progs+3.6/automatic-downloads \
>>> 
>>> 	  --with-parmetis=0 --with-metis=0 \
>>> 	  --with-tetgen=0 \
>>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
>>> 
>>> openmp=1 \
>>> 
>>> 	  --with-64-bit-indices \
>>> 	  --download-chombo
>>> 
>>> and the final error in configure is
>>> 
>>> /bin/csh -f -c "mpicxx -g
>>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
>>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
>>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG 
>>> -Wl,-rpath,/usr/lib/x86_64-linux- gnu -L/usr/lib/x86_64-linux-gnu
>>> -lhdf5_openmpi     -Wl,-
>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o
>>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f
>>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
>>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx.m
>>> pif90.DEBUG.a(CFLeastSquares.o): undefined reference to symbol
>>> '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
>>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO
>>> missing from command line
>>> collect2: error: ld returned 1 exit status
>>> 
>>> 
>>> 
>>> Cheers,
>>> Juha


From juhaj at iki.fi  Fri Oct  2 16:03:55 2015
From: juhaj at iki.fi (Juha =?ISO-8859-1?Q?J=E4ykk=E4?=)
Date: Fri, 02 Oct 2015 22:03:55 +0100
Subject: [petsc-users] petsc+chombo
In-Reply-To: <F5FDAB99-6218-4429-8941-803783CC26DB@mcs.anl.gov>
References: <1700733.a0gsQvG38I@vega> <2142492.ySKjvTbp0E@vega>
	<F5FDAB99-6218-4429-8941-803783CC26DB@mcs.anl.gov>
Message-ID: <6305086.fz3OUVolNh@rigel>

I'm sure that fixes it, but I would have hoped there is a proper way of just 
configuring and making PETSc with --download-chombo. I.e. without patching 
PETSc first. If there is no such way, I'm happy to apply the patch, but it is 
really a last resort for 3.6; for > 3.6 it would of course be fine as it could 
be released with the patch applied.

Cheers,
Juha

On Friday 02 Oct 2015 13:23:40 Barry Smith wrote:
>   I think Satish already told you the fix. Please let us know if it does not
> work.
> 
> Hm - we usually try to avoiding creating binaries in externalpackages
> [if not needed] - esp the examples/tests.
> 
> So perhaps chombo shoul have the following change.
> 
> Satish
> 
> ------
> diff --git a/config/BuildSystem/config/packages/Chombo.py
> b/config/BuildSystem/config/packages/Chombo.py index effbca6..7a13b0a
> 100644
> --- a/config/BuildSystem/config/packages/Chombo.py
> +++ b/config/BuildSystem/config/packages/Chombo.py
> @@ -110,7 +110,7 @@ class Configure(config.package.Package):
>           raise RuntimeError('Error running make on Chombo: config value not
> found') config_value=poutput.split('=')[1]
>         self.logPrint('Chombo installed using config=%s\n'%config_value)
> -        output,err,ret = config.package.Package.executeShellCommand('cd
> '+os.path.join(self.packageDir,'lib') +' && make clean && make all',
> timeout=2500, log = self.log) +        output,err,ret =
> config.package.Package.executeShellCommand('cd
> '+os.path.join(self.packageDir,'lib') +' && make clean && make lib',
> timeout=2500, log = self.log) output,err,ret =
> config.package.Package.executeShellCommand('cd
> '+self.packageDir+self.installSudo+'&& cp -f
> lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+'
> '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp
> -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''),
> timeout=2500, log = self.log) except RuntimeError, e:
>         raise RuntimeError('Error running make on Chombo: '+str(e))
> 
> > On Oct 2, 2015, at 4:38 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> > 
> > Dear Barry,
> > 
> > Please find the configure.log at http://www.iki.fi/~juhaj/configure.log.
> > 
> > On a more general level, I wonder if the chombo-petsc interoperability is
> > documented anywhere? I grepped through the examples and came up blank.
> > 
> > Cheers,
> > Juha
> > 
> > On Thursday 01 Oct 2015 09:53:26 Barry Smith wrote:
> >>  We absolutely NEED configure.log to help you.
> >>  
> >>  Barry
> >>  
> >>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> >>> 
> >>> Hi list,
> >>> 
> >>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep
> >>> failing.
> >>> It seems to me the chombo build system is not told to link in
> >>> libgfortran.so, but no matter how I try to convince PETSc to add that,
> >>> it
> >>> won't. How should this be used, please?
> >>> 
> >>> The full configure line is
> >>> 
> >>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries
> >>> --
> >>> with-debugging=0 \
> >>> 
> >>>         --useThreads 0 --with-clanguage=C++ --with-c-support \
> >>>         --with-fortran-interfaces=1 \
> >>>         --with-mpi=1 --with-mpi-shared=1 \
> >>> 	  
> >>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> >>> 
> >>> lib=/usr/lib/openblas-base/libblas.so
> >>> --with-lapack-lib=/usr/lib/openblas-
> >>> base/liblapack.so \
> >>> 
> >>>         --with-scalapack=1 --with-scalapack-include=/usr/include \
> >>>         --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
> >>>         --with-mumps=0 \
> >>> 	  
> >>> 	  --with-suitesparse=0 \
> >>> 	  
> >>>         --with-spooles=1 --with-spooles-include=/usr/include/spooles \
> >>>         --with-spooles-lib=/usr/lib/libspooles.so \
> >>>         --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> >>> 
> >>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
> >>> 
> >>>         --with-fftw=1 --with-fftw-include=/usr/include \
> >>>         --with-fftw-lib=[/usr/lib/x86_64-linux-
> >>> 
> >>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
> >>> 
> >>>         --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi
> >>>         --with-
> >>> 
> >>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> >>> 
> >>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> >>> 
> >>> lgfortran -g" --LIBS="-lgfortran" \
> >>> 
> >>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> >>> 	  --with-afterimage=1 --with-afterimage-
> >>> 
> >>> include=/usr/include/libAfterImage
> >>> --with-afterimage-lib=/usr/lib/x86_64-
> >>> linux-gnu/libAfterImage.so \
> >>> 
> >>> 	  --with-boost=1 \
> >>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> >>> 
> >>> lib=/usr/lib/libnetcdf.so \
> >>> 
> >>> 	  --with-triangle=0 \
> >>> 	  --with-numpy=1 \
> >>> 	  --with-hwloc=1  --with-external-packages-
> >>> 
> >>> dir=/home/juhaj/progs+3.6/automatic-downloads \
> >>> 
> >>> 	  --with-parmetis=0 --with-metis=0 \
> >>> 	  --with-tetgen=0 \
> >>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> >>> 
> >>> openmp=1 \
> >>> 
> >>> 	  --with-64-bit-indices \
> >>> 	  --download-chombo
> >>> 
> >>> and the final error in configure is
> >>> 
> >>> /bin/csh -f -c "mpicxx -g
> >>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> >>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
> >>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> >>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG
> >>> -Wl,-rpath,/usr/lib/x86_64-linux- gnu -L/usr/lib/x86_64-linux-gnu
> >>> -lhdf5_openmpi     -Wl,-
> >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o
> >>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f
> >>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> >>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> >>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx
> >>> .m
> >>> pif90.DEBUG.a(CFLeastSquares.o): undefined reference to symbol
> >>> '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> >>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO
> >>> missing from command line
> >>> collect2: error: ld returned 1 exit status
> >>> 
> >>> 
> >>> 
> >>> Cheers,
> >>> Juha

-- 
		 -----------------------------------------------
		| Juha J?ykk?, juhaj at iki.fi			|
		| http://koti.kapsi.fi/~juhaj/			|
		 -----------------------------------------------


From bsmith at mcs.anl.gov  Fri Oct  2 16:14:02 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 2 Oct 2015 16:14:02 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <6305086.fz3OUVolNh@rigel>
References: <1700733.a0gsQvG38I@vega> <2142492.ySKjvTbp0E@vega>
	<F5FDAB99-6218-4429-8941-803783CC26DB@mcs.anl.gov>
	<6305086.fz3OUVolNh@rigel>
Message-ID: <67056D81-7714-4944-9E69-49F3B5ECE85A@mcs.anl.gov>


   We have added it to our 3.6 repository so the next patch release will have the fix. It would be helpful for us to know that the patch worked for you since we have no way of testing it ourselves.

  Barry
> On Oct 2, 2015, at 4:03 PM, Juha J?ykk? <juhaj at iki.fi> wrote:
> 
> I'm sure that fixes it, but I would have hoped there is a proper way of just 
> configuring and making PETSc with --download-chombo. I.e. without patching 
> PETSc first. If there is no such way, I'm happy to apply the patch, but it is 
> really a last resort for 3.6; for > 3.6 it would of course be fine as it could 
> be released with the patch applied.
> 
> Cheers,
> Juha
> 
> On Friday 02 Oct 2015 13:23:40 Barry Smith wrote:
>>  I think Satish already told you the fix. Please let us know if it does not
>> work.
>> 
>> Hm - we usually try to avoiding creating binaries in externalpackages
>> [if not needed] - esp the examples/tests.
>> 
>> So perhaps chombo shoul have the following change.
>> 
>> Satish
>> 
>> ------
>> diff --git a/config/BuildSystem/config/packages/Chombo.py
>> b/config/BuildSystem/config/packages/Chombo.py index effbca6..7a13b0a
>> 100644
>> --- a/config/BuildSystem/config/packages/Chombo.py
>> +++ b/config/BuildSystem/config/packages/Chombo.py
>> @@ -110,7 +110,7 @@ class Configure(config.package.Package):
>>          raise RuntimeError('Error running make on Chombo: config value not
>> found') config_value=poutput.split('=')[1]
>>        self.logPrint('Chombo installed using config=%s\n'%config_value)
>> -        output,err,ret = config.package.Package.executeShellCommand('cd
>> '+os.path.join(self.packageDir,'lib') +' && make clean && make all',
>> timeout=2500, log = self.log) +        output,err,ret =
>> config.package.Package.executeShellCommand('cd
>> '+os.path.join(self.packageDir,'lib') +' && make clean && make lib',
>> timeout=2500, log = self.log) output,err,ret =
>> config.package.Package.executeShellCommand('cd
>> '+self.packageDir+self.installSudo+'&& cp -f
>> lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+'
>> '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp
>> -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''),
>> timeout=2500, log = self.log) except RuntimeError, e:
>>        raise RuntimeError('Error running make on Chombo: '+str(e))
>> 
>>> On Oct 2, 2015, at 4:38 AM, Juha Jaykka <juhaj at iki.fi> wrote:
>>> 
>>> Dear Barry,
>>> 
>>> Please find the configure.log at http://www.iki.fi/~juhaj/configure.log.
>>> 
>>> On a more general level, I wonder if the chombo-petsc interoperability is
>>> documented anywhere? I grepped through the examples and came up blank.
>>> 
>>> Cheers,
>>> Juha
>>> 
>>> On Thursday 01 Oct 2015 09:53:26 Barry Smith wrote:
>>>> We absolutely NEED configure.log to help you.
>>>> 
>>>> Barry
>>>> 
>>>>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
>>>>> 
>>>>> Hi list,
>>>>> 
>>>>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep
>>>>> failing.
>>>>> It seems to me the chombo build system is not told to link in
>>>>> libgfortran.so, but no matter how I try to convince PETSc to add that,
>>>>> it
>>>>> won't. How should this be used, please?
>>>>> 
>>>>> The full configure line is
>>>>> 
>>>>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries
>>>>> --
>>>>> with-debugging=0 \
>>>>> 
>>>>>        --useThreads 0 --with-clanguage=C++ --with-c-support \
>>>>>        --with-fortran-interfaces=1 \
>>>>>        --with-mpi=1 --with-mpi-shared=1 \
>>>>> 	  
>>>>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
>>>>> 
>>>>> lib=/usr/lib/openblas-base/libblas.so
>>>>> --with-lapack-lib=/usr/lib/openblas-
>>>>> base/liblapack.so \
>>>>> 
>>>>>        --with-scalapack=1 --with-scalapack-include=/usr/include \
>>>>>        --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
>>>>>        --with-mumps=0 \
>>>>> 	  
>>>>> 	  --with-suitesparse=0 \
>>>>> 	  
>>>>>        --with-spooles=1 --with-spooles-include=/usr/include/spooles \
>>>>>        --with-spooles-lib=/usr/lib/libspooles.so \
>>>>>        --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
>>>>> 
>>>>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
>>>>> 
>>>>>        --with-fftw=1 --with-fftw-include=/usr/include \
>>>>>        --with-fftw-lib=[/usr/lib/x86_64-linux-
>>>>> 
>>>>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
>>>>> 
>>>>>        --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi
>>>>>        --with-
>>>>> 
>>>>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
>>>>> 
>>>>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
>>>>> 
>>>>> lgfortran -g" --LIBS="-lgfortran" \
>>>>> 
>>>>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
>>>>> 	  --with-afterimage=1 --with-afterimage-
>>>>> 
>>>>> include=/usr/include/libAfterImage
>>>>> --with-afterimage-lib=/usr/lib/x86_64-
>>>>> linux-gnu/libAfterImage.so \
>>>>> 
>>>>> 	  --with-boost=1 \
>>>>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
>>>>> 
>>>>> lib=/usr/lib/libnetcdf.so \
>>>>> 
>>>>> 	  --with-triangle=0 \
>>>>> 	  --with-numpy=1 \
>>>>> 	  --with-hwloc=1  --with-external-packages-
>>>>> 
>>>>> dir=/home/juhaj/progs+3.6/automatic-downloads \
>>>>> 
>>>>> 	  --with-parmetis=0 --with-metis=0 \
>>>>> 	  --with-tetgen=0 \
>>>>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
>>>>> 
>>>>> openmp=1 \
>>>>> 
>>>>> 	  --with-64-bit-indices \
>>>>> 	  --download-chombo
>>>>> 
>>>>> and the final error in configure is
>>>>> 
>>>>> /bin/csh -f -c "mpicxx -g
>>>>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
>>>>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>>>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
>>>>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
>>>>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>>>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
>>>>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG
>>>>> -Wl,-rpath,/usr/lib/x86_64-linux- gnu -L/usr/lib/x86_64-linux-gnu
>>>>> -lhdf5_openmpi     -Wl,-
>>>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
>>>>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o
>>>>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f
>>>>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>>>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
>>>>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
>>>>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx
>>>>> .m
>>>>> pif90.DEBUG.a(CFLeastSquares.o): undefined reference to symbol
>>>>> '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
>>>>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO
>>>>> missing from command line
>>>>> collect2: error: ld returned 1 exit status
>>>>> 
>>>>> 
>>>>> 
>>>>> Cheers,
>>>>> Juha
> 
> -- 
> 		 -----------------------------------------------
> 		| Juha J?ykk?, juhaj at iki.fi			|
> 		| http://koti.kapsi.fi/~juhaj/			|
> 		 -----------------------------------------------
> 


From balay at mcs.anl.gov  Fri Oct  2 16:16:56 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Fri, 2 Oct 2015 16:16:56 -0500
Subject: [petsc-users] petsc+chombo
In-Reply-To: <6305086.fz3OUVolNh@rigel>
References: <1700733.a0gsQvG38I@vega> <2142492.ySKjvTbp0E@vega>
	<F5FDAB99-6218-4429-8941-803783CC26DB@mcs.anl.gov>
	<6305086.fz3OUVolNh@rigel>
Message-ID: <alpine.LFD.2.20.1510021612110.3151@asterix>

I'm not sure why you think a patch is 'really a last resort for 3.6'

This is how issues get fixed - and then you have the fixes available
in the next patch release 3.6.2 etc. You can apply a patch [stored in
a patchfile] with the command:

patch -Np1 < patchfile

Or use git to obtain petsc - and use 'maint' for the 3.6 release - and
corresponding fixes.  ttp://www.mcs.anl.gov/petsc/download/index.html

Or wait till petsc-3.6.2 gets released.

Satish


On Fri, 2 Oct 2015, Juha J?ykk? wrote:

> I'm sure that fixes it, but I would have hoped there is a proper way of just 
> configuring and making PETSc with --download-chombo. I.e. without patching 
> PETSc first. If there is no such way, I'm happy to apply the patch, but it is 
> really a last resort for 3.6; for > 3.6 it would of course be fine as it could 
> be released with the patch applied.
> 
> Cheers,
> Juha
> 
> On Friday 02 Oct 2015 13:23:40 Barry Smith wrote:
> >   I think Satish already told you the fix. Please let us know if it does not
> > work.
> > 
> > Hm - we usually try to avoiding creating binaries in externalpackages
> > [if not needed] - esp the examples/tests.
> > 
> > So perhaps chombo shoul have the following change.
> > 
> > Satish
> > 
> > ------
> > diff --git a/config/BuildSystem/config/packages/Chombo.py
> > b/config/BuildSystem/config/packages/Chombo.py index effbca6..7a13b0a
> > 100644
> > --- a/config/BuildSystem/config/packages/Chombo.py
> > +++ b/config/BuildSystem/config/packages/Chombo.py
> > @@ -110,7 +110,7 @@ class Configure(config.package.Package):
> >           raise RuntimeError('Error running make on Chombo: config value not
> > found') config_value=poutput.split('=')[1]
> >         self.logPrint('Chombo installed using config=%s\n'%config_value)
> > -        output,err,ret = config.package.Package.executeShellCommand('cd
> > '+os.path.join(self.packageDir,'lib') +' && make clean && make all',
> > timeout=2500, log = self.log) +        output,err,ret =
> > config.package.Package.executeShellCommand('cd
> > '+os.path.join(self.packageDir,'lib') +' && make clean && make lib',
> > timeout=2500, log = self.log) output,err,ret =
> > config.package.Package.executeShellCommand('cd
> > '+self.packageDir+self.installSudo+'&& cp -f
> > lib/lib*.'+self.setCompilers.AR_LIB_SUFFIX+'
> > '+os.path.join(self.installDir,self.libdir,'')+' &&  '+self.installSudo+'cp
> > -f lib/include/*.H '+os.path.join(self.installDir,self.includedir,''),
> > timeout=2500, log = self.log) except RuntimeError, e:
> >         raise RuntimeError('Error running make on Chombo: '+str(e))
> > 
> > > On Oct 2, 2015, at 4:38 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> > > 
> > > Dear Barry,
> > > 
> > > Please find the configure.log at http://www.iki.fi/~juhaj/configure.log.
> > > 
> > > On a more general level, I wonder if the chombo-petsc interoperability is
> > > documented anywhere? I grepped through the examples and came up blank.
> > > 
> > > Cheers,
> > > Juha
> > > 
> > > On Thursday 01 Oct 2015 09:53:26 Barry Smith wrote:
> > >>  We absolutely NEED configure.log to help you.
> > >>  
> > >>  Barry
> > >>  
> > >>> On Oct 1, 2015, at 9:24 AM, Juha Jaykka <juhaj at iki.fi> wrote:
> > >>> 
> > >>> Hi list,
> > >>> 
> > >>> I'm trying to compile PETSc 3.6 with --download-chombo, but keep
> > >>> failing.
> > >>> It seems to me the chombo build system is not told to link in
> > >>> libgfortran.so, but no matter how I try to convince PETSc to add that,
> > >>> it
> > >>> won't. How should this be used, please?
> > >>> 
> > >>> The full configure line is
> > >>> 
> > >>> ./configure --prefix=/home/juhaj/progs+3.6/petsc --with-shared-libraries
> > >>> --
> > >>> with-debugging=0 \
> > >>> 
> > >>>         --useThreads 0 --with-clanguage=C++ --with-c-support \
> > >>>         --with-fortran-interfaces=1 \
> > >>>         --with-mpi=1 --with-mpi-shared=1 \
> > >>> 	  
> > >>> 	  --with-blas-lapack-include=/usr/include/openblas --with-blas-
> > >>> 
> > >>> lib=/usr/lib/openblas-base/libblas.so
> > >>> --with-lapack-lib=/usr/lib/openblas-
> > >>> base/liblapack.so \
> > >>> 
> > >>>         --with-scalapack=1 --with-scalapack-include=/usr/include \
> > >>>         --with-scalapack-lib=/usr/lib/libscalapack-openmpi.so \
> > >>>         --with-mumps=0 \
> > >>> 	  
> > >>> 	  --with-suitesparse=0 \
> > >>> 	  
> > >>>         --with-spooles=1 --with-spooles-include=/usr/include/spooles \
> > >>>         --with-spooles-lib=/usr/lib/libspooles.so \
> > >>>         --with-hypre=1 --with-hypre-include=/usr/include --with-hypre-
> > >>> 
> > >>> lib=/usr/lib/libHYPRE_IJ_mv-2.8.0b.so \
> > >>> 
> > >>>         --with-fftw=1 --with-fftw-include=/usr/include \
> > >>>         --with-fftw-lib=[/usr/lib/x86_64-linux-
> > >>> 
> > >>> gnu/libfftw3.so,/usr/lib/x86_64-linux-gnu/libfftw3_mpi.so] \
> > >>> 
> > >>>         --with-hdf5=1 --with-hdf5-include=/usr/include/hdf5/openmpi
> > >>>         --with-
> > >>> 
> > >>> hdf5-lib=/usr/lib/x86_64-linux-gnu/libhdf5_openmpi.so \
> > >>> 
> > >>> 	  --CXX_LINKER_FLAGS="-Wl,--no-as-needed -lgfortran" --LDFLAGS="-
> > >>> 
> > >>> lgfortran -g" --LIBS="-lgfortran" \
> > >>> 
> > >>> 	  --with-clean=1 --with-memalign=64 --with-log=0 --with-valgrind=1 \
> > >>> 	  --with-afterimage=1 --with-afterimage-
> > >>> 
> > >>> include=/usr/include/libAfterImage
> > >>> --with-afterimage-lib=/usr/lib/x86_64-
> > >>> linux-gnu/libAfterImage.so \
> > >>> 
> > >>> 	  --with-boost=1 \
> > >>> 	  --with-netcdf=1 --with-netcdf-include=/usr/include/ --with-netcdf-
> > >>> 
> > >>> lib=/usr/lib/libnetcdf.so \
> > >>> 
> > >>> 	  --with-triangle=0 \
> > >>> 	  --with-numpy=1 \
> > >>> 	  --with-hwloc=1  --with-external-packages-
> > >>> 
> > >>> dir=/home/juhaj/progs+3.6/automatic-downloads \
> > >>> 
> > >>> 	  --with-parmetis=0 --with-metis=0 \
> > >>> 	  --with-tetgen=0 \
> > >>> 	  --with-scalar-type=real --with-pic=1 --with-gnu-compilers=1 --with-
> > >>> 
> > >>> openmp=1 \
> > >>> 
> > >>> 	  --with-64-bit-indices \
> > >>> 	  --download-chombo
> > >>> 
> > >>> and the final error in configure is
> > >>> 
> > >>> /bin/csh -f -c "mpicxx -g
> > >>> o/2d.Linux.64.mpicxx.mpif90.DEBUG/testFourthOrderFillPatch.o   -
> > >>> L/home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > >>> opt/Chombo/lib/test/AMRTimeDependent/../.. -
> > >>> lamrtimedependent2d.Linux.64.mpicxx.mpif90.DEBUG -
> > >>> lamrtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > >>> lboxtools2d.Linux.64.mpicxx.mpif90.DEBUG -
> > >>> lbasetools2d.Linux.64.mpicxx.mpif90.DEBUG
> > >>> -Wl,-rpath,/usr/lib/x86_64-linux- gnu -L/usr/lib/x86_64-linux-gnu
> > >>> -lhdf5_openmpi     -Wl,-
> > >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -llapack -Wl,-
> > >>> rpath,/usr/lib/openblas-base -L/usr/lib/openblas-base -lblas  -o
> > >>> testFourthOrderFillPatch2d.Linux.64.mpicxx.mpif90.DEBUG.ex |&  awk -f
> > >>> /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > >>> opt/Chombo/lib/test/AMRTimeDependent/../../mk/tempnam.awk"
> > >>> /usr/bin/ld: /home/juhaj/progs+3.6/automatic-downloads/arch-linux2-cxx-
> > >>> opt/Chombo/lib/test/AMRTimeDependent/../../libamrtools2d.Linux.64.mpicxx
> > >>> .m
> > >>> pif90.DEBUG.a(CFLeastSquares.o): undefined reference to symbol
> > >>> '_gfortran_transfer_integer_write@@GFORTRAN_1.4'
> > >>> /usr/lib/x86_64-linux-gnu/libgfortran.so.3: error adding symbols: DSO
> > >>> missing from command line
> > >>> collect2: error: ld returned 1 exit status
> > >>> 
> > >>> 
> > >>> 
> > >>> Cheers,
> > >>> Juha
> 
> 

From mfadams at lbl.gov  Sun Oct  4 07:57:22 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Sun, 4 Oct 2015 08:57:22 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
Message-ID: <CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>

I've lost this thread a bit, but you seem to be bisecting to find where a
problem started and you are noticing the gmres coarse grid solver.  We
fixed a bug where PETSc was resetting the coarse grid solver to GMRES when
it should not. So older versions have this, but the current version, and I
think this has been in place for all of v3.6, but it might have missed
v3.6.1, have the fix of not resetting the coarse grid solver type.  GAMG
sets the coarse grid solver type to preonly, but you can override it.  Let
me know if I'm missing something here.

I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This is the
number of smoothing steps of the prolongation operator and you should not
use more than 1.  In fact for CFD, you should use no smoothing (0),
probably.

Mark

On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Hi Mark,
>
> I just noticed that in the previous commit 7743f89, it's also using GMRES
> in the multigrid solve but doesn't complain until the 2nd timestep, so my
> bisection criteria is off, since I was giving commits a PASS if they made
> it through the one timestep without complaining about the indefinite PC. I
> think I'm still close to the problem commit, but it's probably a little bit
> before 25a145a.  Apologies for the goose chase.
>
> Thanks,
> Gil Forsyth
>
> On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> I ran for one timestep against 3.5.4 with
>> #+BEGIN_SRC
>> petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>> -poisson_pc_type gamg -poisson_pc_gamg_type agg
>> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
>> kspview_3.5.4.log
>> #+END_SRC
>>
>> and then against 25a145a with the same inputs.  I notice that the poisson
>> multigrid solve in 25a145a is using GMRES again while 3.5.4 is using
>> preonly.
>>
>> Logs from both runs are attached.
>>
>>
>> On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>
>>> Can you please send a good log also, with the ksp_view.
>>> Mark
>>>
>>> On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>
>>>> Using PETSc master branch solved the problem in serial, but I'm still
>>>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>>>> runs to completion when I don't use GAMG.  Log is attached for the
>>>> following run.
>>>>
>>>> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>>>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>>>>
>>>>
>>>> Thanks again,
>>>> Gil Forsyth
>>>>
>>>>
>>>> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>
>>>>> Ah, got it.  I'll checkout the master branch and see if the behavior
>>>>> persists.
>>>>>
>>>>> Many thanks,
>>>>> Gil
>>>>>
>>>>> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show
>>>>>>> that the behavior seems to have changed between versions.  The only
>>>>>>> difference in our code between 3.5.4 and 3.6.1 is the change from
>>>>>>> KSPSetNullSpace to MatSetNullSpace.
>>>>>>>
>>>>>>
>>>>>> Mark made some GAMG changes which were later reversed because they
>>>>>> had unintended consequences like this.
>>>>>> I think what Barry means is, "you should get the behavior you expect
>>>>>> using the master branch from PETSc development"
>>>>>>
>>>>>>   Thanks,
>>>>>>
>>>>>>      Matt
>>>>>>
>>>>>>
>>>>>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>   Update your PETSc
>>>>>>>>
>>>>>>>>
>>>>>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> > Hi Barry,
>>>>>>>> >
>>>>>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm
>>>>>>>> not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>>>>>> works with:
>>>>>>>> >
>>>>>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>>> -poisson_ksp_converged_reason > kspview3.5.4
>>>>>>>> >
>>>>>>>> > and shows that the coarse grid solver is of type:preonly
>>>>>>>> >
>>>>>>>> > running the newer version that uses MatSetNullSpace in its stead
>>>>>>>> and adding in -poisson_mg_coarse_ksp_type preonly
>>>>>>>> >
>>>>>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>>>>>> kspview3.6.1
>>>>>>>> >
>>>>>>>> > still shows
>>>>>>>> >
>>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>>> >   type: cg
>>>>>>>> >   maximum iterations=10000
>>>>>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>> >   left preconditioning
>>>>>>>> >   using nonzero initial guess
>>>>>>>> >   using PRECONDITIONED norm type for convergence test
>>>>>>>> > PC Object:(poisson_) 1 MPI processes
>>>>>>>> >   type: gamg
>>>>>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>>>>> >       Cycles per PCApply=1
>>>>>>>> >       Using Galerkin computed coarse grid matrices
>>>>>>>> >       GAMG specific options
>>>>>>>> >         Threshold for dropping small values from graph 0
>>>>>>>> >         AGG specific options
>>>>>>>> >           Symmetric graph false
>>>>>>>> >   Coarse grid solver -- level -------------------------------
>>>>>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>>> >       type: gmres
>>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>>> >       tolerances:  relative=1e-05, absolute=1e-50,
>>>>>>>> divergence=10000
>>>>>>>> >       left preconditioning
>>>>>>>> >       using NONE norm type for convergence test
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > both logs are attached.
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>>> wrote:
>>>>>>>> >
>>>>>>>> >    This can't work. You can't use a GMRES inside a CG.   Try
>>>>>>>> changing to -poisson_mg_coarse_ksp_type preonly
>>>>>>>> >
>>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>>> >   type: cg
>>>>>>>> >
>>>>>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>>> >       type: gmres
>>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>>> wrote:
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>>> wrote:
>>>>>>>> > > Hi all,
>>>>>>>> > >
>>>>>>>> > > I've been having some trouble with what should be a relatively
>>>>>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>>>>>> > >
>>>>>>>> > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>>>>>>> test problem, 32x32 at Re 100
>>>>>>>> > >
>>>>>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following
>>>>>>>> to set the null space.  This is for a 2D Poisson system with no immersed
>>>>>>>> boundary and so the null space is the constant vector.
>>>>>>>> > >
>>>>>>>> > >   MatNullSpace nsp;
>>>>>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0,
>>>>>>>> NULL, &nsp); CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>>>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>>>>>> > >
>>>>>>>> > > Clearly this has to happen in the reverse order, since ksp2
>>>>>>>> would not be created yet.
>>>>>>>> > >
>>>>>>>> > > For questions about solvers, we HAVE to see the complete output
>>>>>>>> of -ksp_view so we
>>>>>>>> > > know what we are dealing with. Its also nice to have
>>>>>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>>> > >
>>>>>>>> > >   Matt
>>>>>>>> > >
>>>>>>>> > > Yes -- sorry, those are both in inline files and are called in
>>>>>>>> the reverse order that I wrote them out.
>>>>>>>> > >
>>>>>>>> > > I've attached the output of
>>>>>>>> > >
>>>>>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory .
>>>>>>>> -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths
>>>>>>>> 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>>> -poisson_ksp_converged_reason > kspview.log
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > And then setup the KSP with
>>>>>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>>>>>>> CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>>>>>>> CHKERRQ(ierr);
>>>>>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>>>>>> > >
>>>>>>>> > > The matrix QTBNQ does not change, only the rhs of the system is
>>>>>>>> updated.
>>>>>>>> > >
>>>>>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>>>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>>>>>> > >
>>>>>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>>>>>> > >
>>>>>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>>>>>> > >
>>>>>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>>>>>> KSP_DIVERGED_INDEFINITE_PC
>>>>>>>> > >
>>>>>>>> > > This is weird, especially because if we change nsmooths to 2,
>>>>>>>> it runs for 264 timesteps and the returns the same error.  But we have
>>>>>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>>>>>> right?
>>>>>>>> > >
>>>>>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>>>>>> > >
>>>>>>>> > > Change nsmooths to 4 and it runs to completion.
>>>>>>>> > >
>>>>>>>> > > It seems like either gamg's behavior has changed, or that
>>>>>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>>>>>> explicitly in addition to MatSetNullSpace?
>>>>>>>> > >
>>>>>>>> > > Thanks,
>>>>>>>> > > Gil Forsyth
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > >
>>>>>>>> > > --
>>>>>>>> > > What most experimenters take for granted before they begin
>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>> their experiments lead.
>>>>>>>> > > -- Norbert Wiener
>>>>>>>> > >
>>>>>>>> > > <kspview.log>
>>>>>>>> >
>>>>>>>> >
>>>>>>>> > <kspview3.5.4><kspview3.6.1>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> What most experimenters take for granted before they begin their
>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>> experiments lead.
>>>>>> -- Norbert Wiener
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151004/9320524b/attachment.html>

From timothee.nicolas at gmail.com  Sun Oct  4 11:05:29 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Sun, 4 Oct 2015 18:05:29 +0200
Subject: [petsc-users] KSP step not accelerating despite good
 preconditioning in SNES
In-Reply-To: <CAGi1ndRNEdFHeRtrLL5i6eAxQoQJ4ygqhsg9ORKnXjVXKGVj+A@mail.gmail.com>
References: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>
	<CAMYG4Gkw--6We9=s8_fxLHoy1AuHudyyENu6xbg=18Gbsgmy5Q@mail.gmail.com>
	<CAGi1ndRNEdFHeRtrLL5i6eAxQoQJ4ygqhsg9ORKnXjVXKGVj+A@mail.gmail.com>
Message-ID: <CAGi1ndSf7pxH4V+eUHf2hXfVamepdL_TN35Fv_iuYs=c7hxb2w@mail.gmail.com>

Thank you. It seems to be working like a charm with PCSHELL, I have the
drastic reductions in KSP iterations ! (incidentally, my convergence
problem was probably mostly due to a mistake in the Schur matrix, but
putting things in the right place also helps !)

Now I have to move the KSP step in the preconditioner to multigrid method,
it should in principle be more efficient than the default KSP options for
my specific problem. However there is almost no example using multigrid in
the documentation. I found only ksp/ksp/examples/tutorials/ex42.c and
ksp/ksp/examples/tests/ex19.c, however in the first one it is not the
default and trying to use the option -stokes_pc_type pcmg produced errors
like
[0]PETSC ERROR: Unable to find requested PC type pcmg

Regarding the second one, it says at the top that it is the "bad way", so I
am not sure I should follow this example... In which sense is it a bad way
? Is it because there are only two levels and in principle we shoud have a
hierarchy until the lowest grid ?

In anycase if anyone has a nice multigrid example on a simple case like
Laplacian, in the "good way", I would be interested

Best

Timothee


2015-09-28 14:21 GMT+02:00 Timoth?e Nicolas <timothee.nicolas at gmail.com>:

> Thanks a lot, I expected something like that. I will look in this
> direction.
>
> Best
>
> Timoth?e
>
> 2015-09-28 21:17 GMT+09:00 Matthew Knepley <knepley at gmail.com>:
>
>> On Mon, Sep 28, 2015 at 2:53 AM, Timoth?e Nicolas <
>> timothee.nicolas at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I have something strange in my application and I don't know what could
>>> cause this. I am trying to do an implicit MHD problem and I thought I
>>> finally figured out the preconditioner step, but somehow I don't get the
>>> expected result, not even close.
>>>
>>> For the preconditioning, I am using an approximate Schur complement,
>>> which requires two relatively easy KSP inversions at each preconditioner
>>> application. I apply this algorithm directly to the result function at the
>>> end of the routine FormFunction. I have checked that the approximation to
>>> the inversion of the Jacobian is good, in the sense that when I multiply
>>> the preconditioned vector by the *total* Jacobian matrix, I indeed
>>> recover almost the initial unpreconditioned vector. Also, I know that my
>>> Jacobian matrix is correct, because (i) I have checked manually that F(X +
>>> dX) ~ F(X) + J * dX and (ii) when I don't use -snes_mf and use the provided
>>> Jacobian matrix the result is pretty much equivalent to using -snes_mf.
>>>
>>> In my understanding, this means that what I effectively feed to SNES at
>>> the end of my FormFunction routine is a good approximation to J^(-1) F. As
>>> a result, I naturally expect that the number of KSP iterations necessary to
>>> achieve one SNES iteration be drastically reduced. However, I observe
>>> virtually no change whatsoever in the number of iterations.
>>>
>>> Any thoughts about what I could be missing ? Maybe I forgot to set a
>>> SNES or KSP option somewhere ? I can send pieces of code if needs be.
>>>
>>
>> It sounds like you are putting this in the wrong place. If you have the
>> action of a good preconditioner for the Jacobian, then you
>> should use a PCSHELL and pass it to the KSP. It does not belong in the
>> FormFunction.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Best
>>>
>>> Timothee
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151004/dc7847ee/attachment-0001.html>

From dave.mayhem23 at gmail.com  Sun Oct  4 11:18:30 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sun, 4 Oct 2015 18:18:30 +0200
Subject: [petsc-users] KSP step not accelerating despite good
 preconditioning in SNES
In-Reply-To: <CAGi1ndSf7pxH4V+eUHf2hXfVamepdL_TN35Fv_iuYs=c7hxb2w@mail.gmail.com>
References: <CAGi1ndQHEUKmhiXLkJG3D9HAbCbqxisSLHWbgKwDJ5kfwd5TfQ@mail.gmail.com>
	<CAMYG4Gkw--6We9=s8_fxLHoy1AuHudyyENu6xbg=18Gbsgmy5Q@mail.gmail.com>
	<CAGi1ndRNEdFHeRtrLL5i6eAxQoQJ4ygqhsg9ORKnXjVXKGVj+A@mail.gmail.com>
	<CAGi1ndSf7pxH4V+eUHf2hXfVamepdL_TN35Fv_iuYs=c7hxb2w@mail.gmail.com>
Message-ID: <CAJ98EDrG_5HMWTWx152hLXYZuM98AzQ2E_y8DL-Zug8bB0XH9g@mail.gmail.com>

On Sunday, 4 October 2015, Timoth?e Nicolas <timothee.nicolas at gmail.com>
wrote:

> Thank you. It seems to be working like a charm with PCSHELL, I have the
> drastic reductions in KSP iterations ! (incidentally, my convergence
> problem was probably mostly due to a mistake in the Schur matrix, but
> putting things in the right place also helps !)
>
> Now I have to move the KSP step in the preconditioner to multigrid method,
> it should in principle be more efficient than the default KSP options for
> my specific problem. However there is almost no example using multigrid in
> the documentation. I found only ksp/ksp/examples/tutorials/ex42.c
>
and ksp/ksp/examples/tests/ex19.c, however in the first one it is not the
> default and trying to use the option -stokes_pc_type pcmg produced errors
> like
> [0]PETSC ERROR: Unable to find requested PC type pcmg
>

The option should be
-stokes_pc_type mg

Default will be a one level method so you need to set the number of levels
with an addition arguement. In ex42, levels are defined via coarsening.

I suggest using
-help | grep mg
 to learn what this option is called, and more generally to learn how
to configure the mg preconditioner.



> Regarding the second one, it says at the top that it is the "bad way", so
> I am not sure I should follow this example... In which sense is it a bad
> way ? Is it because there are only two levels and in principle we shoud
> have a hierarchy until the lowest grid ?
>
> In anycase if anyone has a nice multigrid example on a simple case like
> Laplacian, in the "good way", I would be interested
>
> Best
>
> Timothee
>
>
> 2015-09-28 14:21 GMT+02:00 Timoth?e Nicolas <timothee.nicolas at gmail.com
> <javascript:_e(%7B%7D,'cvml','timothee.nicolas at gmail.com');>>:
>
>> Thanks a lot, I expected something like that. I will look in this
>> direction.
>>
>> Best
>>
>> Timoth?e
>>
>> 2015-09-28 21:17 GMT+09:00 Matthew Knepley <knepley at gmail.com
>> <javascript:_e(%7B%7D,'cvml','knepley at gmail.com');>>:
>>
>>> On Mon, Sep 28, 2015 at 2:53 AM, Timoth?e Nicolas <
>>> timothee.nicolas at gmail.com
>>> <javascript:_e(%7B%7D,'cvml','timothee.nicolas at gmail.com');>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have something strange in my application and I don't know what could
>>>> cause this. I am trying to do an implicit MHD problem and I thought I
>>>> finally figured out the preconditioner step, but somehow I don't get the
>>>> expected result, not even close.
>>>>
>>>> For the preconditioning, I am using an approximate Schur complement,
>>>> which requires two relatively easy KSP inversions at each preconditioner
>>>> application. I apply this algorithm directly to the result function at the
>>>> end of the routine FormFunction. I have checked that the approximation to
>>>> the inversion of the Jacobian is good, in the sense that when I multiply
>>>> the preconditioned vector by the *total* Jacobian matrix, I indeed
>>>> recover almost the initial unpreconditioned vector. Also, I know that my
>>>> Jacobian matrix is correct, because (i) I have checked manually that F(X +
>>>> dX) ~ F(X) + J * dX and (ii) when I don't use -snes_mf and use the provided
>>>> Jacobian matrix the result is pretty much equivalent to using -snes_mf.
>>>>
>>>> In my understanding, this means that what I effectively feed to SNES at
>>>> the end of my FormFunction routine is a good approximation to J^(-1) F. As
>>>> a result, I naturally expect that the number of KSP iterations necessary to
>>>> achieve one SNES iteration be drastically reduced. However, I observe
>>>> virtually no change whatsoever in the number of iterations.
>>>>
>>>> Any thoughts about what I could be missing ? Maybe I forgot to set a
>>>> SNES or KSP option somewhere ? I can send pieces of code if needs be.
>>>>
>>>
>>> It sounds like you are putting this in the wrong place. If you have the
>>> action of a good preconditioner for the Jacobian, then you
>>> should use a PCSHELL and pass it to the KSP. It does not belong in the
>>> FormFunction.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Best
>>>>
>>>> Timothee
>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151004/3bbb11e1/attachment.html>

From zocca.marco at gmail.com  Mon Oct  5 00:50:29 2015
From: zocca.marco at gmail.com (Marco Zocca)
Date: Mon, 5 Oct 2015 07:50:29 +0200
Subject: [petsc-users] [ANN] PETSc Haskell bindings
Message-ID: <CAKE6T0Rn=+CSeiv9KHafDpmJ64GT_9A+Le-gaW_Dk8s_u33XxA@mail.gmail.com>

Dear all,

development of the Haskell bindings for the PETSc scientific
computation library is now public; the repository can be found at

https://github.com/ocramz/petsc-hs

The interface is still very much experimental and in a state of flow
but a number of basic functions are working.

I look forward to all feedback;
Kind regards,

Marco Zocca

From arne.morten.kvarving at sintef.no  Mon Oct  5 07:13:11 2015
From: arne.morten.kvarving at sintef.no (Arne Morten Kvarving)
Date: Mon, 5 Oct 2015 14:13:11 +0200
Subject: [petsc-users] Combining MPIUNI with duplicated comms
Message-ID: <56126957.4030707@sintef.no>

hi there.

first of all, i'm not 100% this is supposed to work. if it is not, feel 
free to tell me just that and i'll find a way to workaround.

in our code we use MPI_Comm_dup for various reasons. this works fine. 
except when i run in serial, i.e., using the MPUNI wrapper. if the 
PETSC_COMM_SELF communicator is duplicated, nastyness results when 
tearing down the KSP object. while this isn't really fatal as such, it 
is rather annoying.

please consider http://www.math.ntnu.no/~arnemort/bug-self.tar.bz2 which 
is a silly code i wrote to reproduce the issue. a 3x3 matrix is 
constructed, a corresponding 1-vector and the system is solved.
then the objects are teared down.

the tarball has a cmake buildsystem. the buildsystem relies on the 
standard PETSC_DIR / PETSC_ARCH env variables. if you prefer to build by 
hand, there is no requirement on using the build system. just watch out 
for the define it adds.

the application takes one required command line parameter which should 
be 'self' or 'world'. this controls which comm is duplicated. if I run 
with self, i get

[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Corrupt argument: 
http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
[0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer 
comm
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
[0]PETSC ERROR: ./bug_comm_self on a linux-gnu-cxx-dbg named akvarium by 
akva Mon Oct  5 14:05:52 2015
[0]PETSC ERROR: Configure options --with-precision=double 
--with-scalar-type=real --with-debugging=1 
--with-blas-lib=/usr/lib/libblas.a 
--with-lapack-lib=/usr/lib/liblapack.a --with-64-bit-indices=0 
--with-clanguage=c++ --with-mpi=0 --LIBS=-ldl --with-ml=0 
--with-shared-libraries=0
[0]PETSC ERROR: #1 Petsc_DelComm_Outer() line 360 in 
/home/akva/kode/petsc/petsc-3.6.2/src/sys/objects/pinit.c

if i run with world, no error is generated.

this is with the recetly released 3.6.2, but i can reproduce atleast 
back to 3.5.2 (haven't tried older).

i also noticed that if i add a VecView call, errors will also be 
generated with world. this line is commented out in the source code. 
might be some valuable info for those who know the system in that fact.

since i made it reproducable through the test app, i skipped logs for 
now. if they or some other info is required from me side, i'll gladly 
provide it.

thanks in advance

arnem

From gforsyth at gwu.edu  Mon Oct  5 09:06:03 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Mon, 5 Oct 2015 10:06:03 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
	<CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
Message-ID: <CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>

Hi Mark,

I've lost it, too.  I was bisecting to find the change that started
returning the indefinite PC error to our code that has previously worked --
but this was using KSPSetNullSpace.
Increasing the number of steps was in an effort to see if it impacted the
error or not, partially based on this thread from PETSC-users (
http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023653.html)
with one of the previous authors of our code.

Updating to PETSc master briefly eliminated the error in the poisson
solver, although this was only the case in serial, it still failed with an
indefinite PC error in parallel.

I'll confess that I'm not sure what to bisect between, as we don't have a
"good" version after the switch from KSPSetNullSpace -> MatSetNullSpace.
That's what prompted the initial bisection search in and around the 3.5.4
commit range.  I'm going to take another crack at that today in a more
automated fashion, since I expect I inserted some human error somewhere
along the way.

Compiled against petsc v3.6.2, it shows again that the coarse grid solver
is using GMRES even when using -mg_coarse_ksp_type preonly.  Logs are
attached.

$PETSC_ARCH/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
-poisson_mg_coarse_ksp_type preonly -poisson_pc_type gamg
-poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view
-poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info

On Sun, Oct 4, 2015 at 8:57 AM, Mark Adams <mfadams at lbl.gov> wrote:

> I've lost this thread a bit, but you seem to be bisecting to find where a
> problem started and you are noticing the gmres coarse grid solver.  We
> fixed a bug where PETSc was resetting the coarse grid solver to GMRES when
> it should not. So older versions have this, but the current version, and I
> think this has been in place for all of v3.6, but it might have missed
> v3.6.1, have the fix of not resetting the coarse grid solver type.  GAMG
> sets the coarse grid solver type to preonly, but you can override it.  Let
> me know if I'm missing something here.
>
> I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This is
> the number of smoothing steps of the prolongation operator and you should
> not use more than 1.  In fact for CFD, you should use no smoothing (0),
> probably.
>
> Mark
>
> On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> Hi Mark,
>>
>> I just noticed that in the previous commit 7743f89, it's also using GMRES
>> in the multigrid solve but doesn't complain until the 2nd timestep, so my
>> bisection criteria is off, since I was giving commits a PASS if they made
>> it through the one timestep without complaining about the indefinite PC. I
>> think I'm still close to the problem commit, but it's probably a little bit
>> before 25a145a.  Apologies for the goose chase.
>>
>> Thanks,
>> Gil Forsyth
>>
>> On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>
>>> I ran for one timestep against 3.5.4 with
>>> #+BEGIN_SRC
>>> petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>>> -poisson_pc_type gamg -poisson_pc_gamg_type agg
>>> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
>>> kspview_3.5.4.log
>>> #+END_SRC
>>>
>>> and then against 25a145a with the same inputs.  I notice that the
>>> poisson multigrid solve in 25a145a is using GMRES again while 3.5.4 is
>>> using preonly.
>>>
>>> Logs from both runs are attached.
>>>
>>>
>>> On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>>
>>>> Can you please send a good log also, with the ksp_view.
>>>> Mark
>>>>
>>>> On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>
>>>>> Using PETSc master branch solved the problem in serial, but I'm still
>>>>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>>>>> runs to completion when I don't use GAMG.  Log is attached for the
>>>>> following run.
>>>>>
>>>>> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>>>>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>>>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>>>>>
>>>>>
>>>>> Thanks again,
>>>>> Gil Forsyth
>>>>>
>>>>>
>>>>> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>>>>
>>>>>> Ah, got it.  I'll checkout the master branch and see if the behavior
>>>>>> persists.
>>>>>>
>>>>>> Many thanks,
>>>>>> Gil
>>>>>>
>>>>>> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show
>>>>>>>> that the behavior seems to have changed between versions.  The only
>>>>>>>> difference in our code between 3.5.4 and 3.6.1 is the change from
>>>>>>>> KSPSetNullSpace to MatSetNullSpace.
>>>>>>>>
>>>>>>>
>>>>>>> Mark made some GAMG changes which were later reversed because they
>>>>>>> had unintended consequences like this.
>>>>>>> I think what Barry means is, "you should get the behavior you expect
>>>>>>> using the master branch from PETSc development"
>>>>>>>
>>>>>>>   Thanks,
>>>>>>>
>>>>>>>      Matt
>>>>>>>
>>>>>>>
>>>>>>>> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>>
>>>>>>>>>   Update your PETSc
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > Hi Barry,
>>>>>>>>> >
>>>>>>>>> > We aren't explicitly setting GMRES anywhere in the code and I'm
>>>>>>>>> not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>>>>>>>> works with:
>>>>>>>>> >
>>>>>>>>> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>>>> -poisson_ksp_converged_reason > kspview3.5.4
>>>>>>>>> >
>>>>>>>>> > and shows that the coarse grid solver is of type:preonly
>>>>>>>>> >
>>>>>>>>> > running the newer version that uses MatSetNullSpace in its stead
>>>>>>>>> and adding in -poisson_mg_coarse_ksp_type preonly
>>>>>>>>> >
>>>>>>>>> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>>>>>>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>>>>>>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>>>>>>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>>>>>>>> kspview3.6.1
>>>>>>>>> >
>>>>>>>>> > still shows
>>>>>>>>> >
>>>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>>>> >   type: cg
>>>>>>>>> >   maximum iterations=10000
>>>>>>>>> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>>>>>>>> >   left preconditioning
>>>>>>>>> >   using nonzero initial guess
>>>>>>>>> >   using PRECONDITIONED norm type for convergence test
>>>>>>>>> > PC Object:(poisson_) 1 MPI processes
>>>>>>>>> >   type: gamg
>>>>>>>>> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>>>>>>>> >       Cycles per PCApply=1
>>>>>>>>> >       Using Galerkin computed coarse grid matrices
>>>>>>>>> >       GAMG specific options
>>>>>>>>> >         Threshold for dropping small values from graph 0
>>>>>>>>> >         AGG specific options
>>>>>>>>> >           Symmetric graph false
>>>>>>>>> >   Coarse grid solver -- level -------------------------------
>>>>>>>>> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>>>> >       type: gmres
>>>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>>>> >       tolerances:  relative=1e-05, absolute=1e-50,
>>>>>>>>> divergence=10000
>>>>>>>>> >       left preconditioning
>>>>>>>>> >       using NONE norm type for convergence test
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > both logs are attached.
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <
>>>>>>>>> bsmith at mcs.anl.gov> wrote:
>>>>>>>>> >
>>>>>>>>> >    This can't work. You can't use a GMRES inside a CG.   Try
>>>>>>>>> changing to -poisson_mg_coarse_ksp_type preonly
>>>>>>>>> >
>>>>>>>>> > KSP Object:(poisson_) 1 MPI processes
>>>>>>>>> >   type: cg
>>>>>>>>> >
>>>>>>>>> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>>>>>>>> >       type: gmres
>>>>>>>>> >         GMRES: restart=30, using Classical (unmodified)
>>>>>>>>> Gram-Schmidt Orthogonalization with no iterative refinement
>>>>>>>>> >         GMRES: happy breakdown tolerance 1e-30
>>>>>>>>> >       maximum iterations=1, initial guess is zero
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>>>>>>>> wrote:
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>>>>>>>> knepley at gmail.com> wrote:
>>>>>>>>> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <
>>>>>>>>> gforsyth at gwu.edu> wrote:
>>>>>>>>> > > Hi all,
>>>>>>>>> > >
>>>>>>>>> > > I've been having some trouble with what should be a relatively
>>>>>>>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>>>>>>>> > >
>>>>>>>>> > > I'm getting indefinite PC errors for a simple lid-driven
>>>>>>>>> cavity test problem, 32x32 at Re 100
>>>>>>>>> > >
>>>>>>>>> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following
>>>>>>>>> to set the null space.  This is for a 2D Poisson system with no immersed
>>>>>>>>> boundary and so the null space is the constant vector.
>>>>>>>>> > >
>>>>>>>>> > >   MatNullSpace nsp;
>>>>>>>>> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0,
>>>>>>>>> NULL, &nsp); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>>>>>>>> > >
>>>>>>>>> > > Clearly this has to happen in the reverse order, since ksp2
>>>>>>>>> would not be created yet.
>>>>>>>>> > >
>>>>>>>>> > > For questions about solvers, we HAVE to see the complete
>>>>>>>>> output of -ksp_view so we
>>>>>>>>> > > know what we are dealing with. Its also nice to have
>>>>>>>>> -ksp_monitor_true_residual -ksp_converged_reason
>>>>>>>>> > >
>>>>>>>>> > >   Matt
>>>>>>>>> > >
>>>>>>>>> > > Yes -- sorry, those are both in inline files and are called in
>>>>>>>>> the reverse order that I wrote them out.
>>>>>>>>> > >
>>>>>>>>> > > I've attached the output of
>>>>>>>>> > >
>>>>>>>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory .
>>>>>>>>> -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths
>>>>>>>>> 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>>>>>>>> -poisson_ksp_converged_reason > kspview.log
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > And then setup the KSP with
>>>>>>>>> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>>>>>>>> CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>>>>>>>> CHKERRQ(ierr);
>>>>>>>>> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>>>>>>>> > >
>>>>>>>>> > > The matrix QTBNQ does not change, only the rhs of the system
>>>>>>>>> is updated.
>>>>>>>>> > >
>>>>>>>>> > > We run this with `-pc_type gamg -pc_gamg_type agg
>>>>>>>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>>>>>>>> > >
>>>>>>>>> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>>>>>>>> > >
>>>>>>>>> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>>>>>>>> > >
>>>>>>>>> > > and the same code diverges after 1 timestep and returns a -8
>>>>>>>>> KSP_DIVERGED_INDEFINITE_PC
>>>>>>>>> > >
>>>>>>>>> > > This is weird, especially because if we change nsmooths to 2,
>>>>>>>>> it runs for 264 timesteps and the returns the same error.  But we have
>>>>>>>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>>>>>>>> right?
>>>>>>>>> > >
>>>>>>>>> > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>>>>>>>> > >
>>>>>>>>> > > Change nsmooths to 4 and it runs to completion.
>>>>>>>>> > >
>>>>>>>>> > > It seems like either gamg's behavior has changed, or that
>>>>>>>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>>>>>>>> explicitly in addition to MatSetNullSpace?
>>>>>>>>> > >
>>>>>>>>> > > Thanks,
>>>>>>>>> > > Gil Forsyth
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > >
>>>>>>>>> > > --
>>>>>>>>> > > What most experimenters take for granted before they begin
>>>>>>>>> their experiments is infinitely more interesting than any results to which
>>>>>>>>> their experiments lead.
>>>>>>>>> > > -- Norbert Wiener
>>>>>>>>> > >
>>>>>>>>> > > <kspview.log>
>>>>>>>>> >
>>>>>>>>> >
>>>>>>>>> > <kspview3.5.4><kspview3.6.1>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> What most experimenters take for granted before they begin their
>>>>>>> experiments is infinitely more interesting than any results to which their
>>>>>>> experiments lead.
>>>>>>> -- Norbert Wiener
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/51463956/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: kspview_arch-3264318.log
Type: text/x-log
Size: 10817 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/51463956/attachment-0001.bin>

From danyang.su at gmail.com  Mon Oct  5 13:14:30 2015
From: danyang.su at gmail.com (Danyang Su)
Date: Mon, 5 Oct 2015 11:14:30 -0700
Subject: [petsc-users] KSP iterations increase rapidly after 8 processors
Message-ID: <5612BE06.1050602@gmail.com>

Hi All,

I have a complex flow, heat transport and reactive transport problem 
that is not scalable after 8 processors. When using less than 8 
processors, it has almost linear speedup. But after 8 processor, the 
number of KSP iteration steps increases to 2 times more than using 1 
processors or 4 processor (47 ksp steps vs 15 ksp steps, attached 
below). Do you have any suggestion to improve the performance?

convergence summary using 1 or 4 processors
  ------------------------------------------------------------------------
  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  
1 max.sia: 0.000E+00 tol.sia: 1.000E-01
   0 KSP preconditioned resid norm 1.137005501778e+06 true resid norm 
9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 1.506085526342e+05 true resid norm 
9.597161643502e+05 ||r(i)||/||b|| 9.958492960384e-02
   2 KSP preconditioned resid norm 5.456879627980e+04 true resid norm 
1.406907438671e+05 ||r(i)||/||b|| 1.459877237079e-02
   3 KSP preconditioned resid norm 2.710407097622e+04 true resid norm 
1.291826672621e+05 ||r(i)||/||b|| 1.340463701999e-02
   4 KSP preconditioned resid norm 9.897001667954e+03 true resid norm 
1.994507102853e+04 ||r(i)||/||b|| 2.069599917247e-03
   5 KSP preconditioned resid norm 4.429538402026e+03 true resid norm 
1.048435199832e+04 ||r(i)||/||b|| 1.087908586391e-03
   6 KSP preconditioned resid norm 2.183344020397e+03 true resid norm 
5.418816689871e+03 ||r(i)||/||b|| 5.622834111190e-04
   7 KSP preconditioned resid norm 1.322492709422e+03 true resid norm 
3.186218374195e+03 ||r(i)||/||b|| 3.306178892084e-04
   8 KSP preconditioned resid norm 6.738694120116e+02 true resid norm 
1.461978701106e+03 ||r(i)||/||b|| 1.517021922107e-04
   9 KSP preconditioned resid norm 3.720835435101e+02 true resid norm 
4.069516281820e+02 ||r(i)||/||b|| 4.222732798516e-05
  10 KSP preconditioned resid norm 1.874689621103e+02 true resid norm 
5.563177554677e+02 ||r(i)||/||b|| 5.772630135195e-05
  11 KSP preconditioned resid norm 8.612569469037e+01 true resid norm 
1.054136865792e+02 ||r(i)||/||b|| 1.093824918994e-05
  12 KSP preconditioned resid norm 4.181701635083e+01 true resid norm 
1.125544728389e+02 ||r(i)||/||b|| 1.167921274083e-05
  13 KSP preconditioned resid norm 2.181470261073e+01 true resid norm 
2.314218289439e+01 ||r(i)||/||b|| 2.401348169412e-06
  14 KSP preconditioned resid norm 1.180501182018e+01 true resid norm 
2.818614875745e+01 ||r(i)||/||b|| 2.924735191592e-06
  15 KSP preconditioned resid norm 6.607882346065e+00 true resid norm 
7.039364165934e+00 ||r(i)||/||b|| 7.304394892577e-07
Linear solve converged due to CONVERGED_RTOL iterations 15


convergence summary using 8 or 16 processors
  ------------------------------------------------------------------------
  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  
1 max.sia: 0.000E+00 tol.sia: 1.000E-01
   0 KSP preconditioned resid norm 1.172246902854e+06 true resid norm 
9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
   1 KSP preconditioned resid norm 2.111438753837e+05 true resid norm 
3.422907872797e+06 ||r(i)||/||b|| 3.551779705448e-01
   2 KSP preconditioned resid norm 9.189117316053e+04 true resid norm 
1.197217247701e+06 ||r(i)||/||b|| 1.242292250163e-01
   3 KSP preconditioned resid norm 3.567188384324e+04 true resid norm 
3.070624730376e+05 ||r(i)||/||b|| 3.186233169485e-02
   4 KSP preconditioned resid norm 1.629589856587e+04 true resid norm 
1.857713260583e+05 ||r(i)||/||b|| 1.927655812743e-02
   5 KSP preconditioned resid norm 8.006759563515e+03 true resid norm 
1.180288951151e+05 ||r(i)||/||b|| 1.224726606456e-02
   6 KSP preconditioned resid norm 5.358897276807e+03 true resid norm 
1.244255645533e+05 ||r(i)||/||b|| 1.291101634758e-02
   7 KSP preconditioned resid norm 4.389072625628e+03 true resid norm 
1.090628475655e+05 ||r(i)||/||b|| 1.131690431051e-02
   8 KSP preconditioned resid norm 3.744362402037e+03 true resid norm 
1.068327247800e+05 ||r(i)||/||b|| 1.108549566194e-02
   9 KSP preconditioned resid norm 3.329272298367e+03 true resid norm 
1.000736812587e+05 ||r(i)||/||b|| 1.038414363906e-02
  10 KSP preconditioned resid norm 2.973880694381e+03 true resid norm 
9.519213877340e+04 ||r(i)||/||b|| 9.877610475599e-03
  11 KSP preconditioned resid norm 2.672681833195e+03 true resid norm 
8.484262500566e+04 ||r(i)||/||b|| 8.803693375649e-03
  12 KSP preconditioned resid norm 2.174736539891e+03 true resid norm 
7.637960549691e+04 ||r(i)||/||b|| 7.925528316728e-03
  13 KSP preconditioned resid norm 1.920508771340e+03 true resid norm 
6.875465496734e+04 ||r(i)||/||b|| 7.134325469547e-03
  14 KSP preconditioned resid norm 1.748769817174e+03 true resid norm 
6.402474320438e+04 ||r(i)||/||b|| 6.643526265111e-03
  15 KSP preconditioned resid norm 1.668781376573e+03 true resid norm 
5.938074141272e+04 ||r(i)||/||b|| 6.161641507219e-03
  16 KSP preconditioned resid norm 1.507934446836e+03 true resid norm 
5.376897976816e+04 ||r(i)||/||b|| 5.579337166534e-03
  17 KSP preconditioned resid norm 1.341780677843e+03 true resid norm 
4.418909196385e+04 ||r(i)||/||b|| 4.585280290092e-03
  18 KSP preconditioned resid norm 1.096322610019e+03 true resid norm 
3.281273489858e+04 ||r(i)||/||b|| 3.404812814836e-03
  19 KSP preconditioned resid norm 8.572434892128e+02 true resid norm 
2.162598538334e+04 ||r(i)||/||b|| 2.244019963415e-03
  20 KSP preconditioned resid norm 6.507523424872e+02 true resid norm 
1.670057090484e+04 ||r(i)||/||b|| 1.732934423406e-03
  21 KSP preconditioned resid norm 5.390162236367e+02 true resid norm 
1.360673589185e+04 ||r(i)||/||b|| 1.411902691922e-03
  22 KSP preconditioned resid norm 4.424143347792e+02 true resid norm 
1.137879849375e+04 ||r(i)||/||b|| 1.180720810037e-03
  23 KSP preconditioned resid norm 3.720257830024e+02 true resid norm 
9.449031285412e+03 ||r(i)||/||b|| 9.804785522388e-04
  24 KSP preconditioned resid norm 3.030947404532e+02 true resid norm 
8.347069391513e+03 ||r(i)||/||b|| 8.661334972044e-04
  25 KSP preconditioned resid norm 2.667874861724e+02 true resid norm 
7.530125164047e+03 ||r(i)||/||b|| 7.813632949253e-04
  26 KSP preconditioned resid norm 2.367701662092e+02 true resid norm 
6.792074618833e+03 ||r(i)||/||b|| 7.047794940898e-04
  27 KSP preconditioned resid norm 2.083554010658e+02 true resid norm 
5.833539191175e+03 ||r(i)||/||b|| 6.053170836065e-04
  28 KSP preconditioned resid norm 1.735399579409e+02 true resid norm 
4.883873002196e+03 ||r(i)||/||b|| 5.067749895065e-04
  29 KSP preconditioned resid norm 1.467413383315e+02 true resid norm 
4.058954246436e+03 ||r(i)||/||b|| 4.211773104502e-04
  30 KSP preconditioned resid norm 1.244235694540e+02 true resid norm 
3.337590813452e+03 ||r(i)||/||b|| 3.463250474989e-04
  31 KSP preconditioned resid norm 1.198432875407e+02 true resid norm 
3.199014077607e+03 ||r(i)||/||b|| 3.319456351305e-04
  32 KSP preconditioned resid norm 1.027669478022e+02 true resid norm 
3.150305508056e+03 ||r(i)||/||b|| 3.268913913342e-04
  33 KSP preconditioned resid norm 9.663893782079e+01 true resid norm 
2.857716117586e+03 ||r(i)||/||b|| 2.965308587777e-04
  34 KSP preconditioned resid norm 8.531640537674e+01 true resid norm 
2.584725514230e+03 ||r(i)||/||b|| 2.682039939945e-04
  35 KSP preconditioned resid norm 7.610279175833e+01 true resid norm 
2.212227973785e+03 ||r(i)||/||b|| 2.295517937704e-04
  36 KSP preconditioned resid norm 6.331885100455e+01 true resid norm 
1.858910367331e+03 ||r(i)||/||b|| 1.928897990333e-04
  37 KSP preconditioned resid norm 5.635209736607e+01 true resid norm 
1.571087791205e+03 ||r(i)||/||b|| 1.630238948769e-04
  38 KSP preconditioned resid norm 4.815528377271e+01 true resid norm 
1.313627294747e+03 ||r(i)||/||b|| 1.363085113416e-04
  39 KSP preconditioned resid norm 4.314593066310e+01 true resid norm 
1.141179411417e+03 ||r(i)||/||b|| 1.184144599963e-04
  40 KSP preconditioned resid norm 3.700552222031e+01 true resid norm 
9.643354964443e+02 ||r(i)||/||b|| 1.000642545110e-04
  41 KSP preconditioned resid norm 3.059145971331e+01 true resid norm 
7.755572523134e+02 ||r(i)||/||b|| 8.047568358679e-05
  42 KSP preconditioned resid norm 2.553811607631e+01 true resid norm 
6.248805390867e+02 ||r(i)||/||b|| 6.484071729467e-05
  43 KSP preconditioned resid norm 2.016013790111e+01 true resid norm 
5.415908243972e+02 ||r(i)||/||b|| 5.619816162855e-05
  44 KSP preconditioned resid norm 1.757243660826e+01 true resid norm 
4.812748098035e+02 ||r(i)||/||b|| 4.993947151743e-05
  45 KSP preconditioned resid norm 1.498503590085e+01 true resid norm 
4.478420094581e+02 ||r(i)||/||b|| 4.647031762326e-05
  46 KSP preconditioned resid norm 1.348574164098e+01 true resid norm 
3.944236184677e+02 ||r(i)||/||b|| 4.092735929461e-05
  47 KSP preconditioned resid norm 1.156613274720e+01 true resid norm 
3.682508444462e+02 ||r(i)||/||b|| 3.821154189432e-05
Linear solve converged due to CONVERGED_RTOL iterations 47

Thanks and regards,

Danyang


From knepley at gmail.com  Mon Oct  5 13:47:22 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 5 Oct 2015 13:47:22 -0500
Subject: [petsc-users] KSP iterations increase rapidly after 8 processors
In-Reply-To: <5612BE06.1050602@gmail.com>
References: <5612BE06.1050602@gmail.com>
Message-ID: <CAMYG4GngS_KpxVQX8Eh+xwCywt4UNOjd+cHWC-HACrZ94+eRnQ@mail.gmail.com>

We cannot say anything without also having -ksp_view and knowing something
about the system

    Matt

On Mon, Oct 5, 2015 at 1:14 PM, Danyang Su <danyang.su at gmail.com> wrote:

> Hi All,
>
> I have a complex flow, heat transport and reactive transport problem that
> is not scalable after 8 processors. When using less than 8 processors, it
> has almost linear speedup. But after 8 processor, the number of KSP
> iteration steps increases to 2 times more than using 1 processors or 4
> processor (47 ksp steps vs 15 ksp steps, attached below). Do you have any
> suggestion to improve the performance?
>
> convergence summary using 1 or 4 processors
>  ------------------------------------------------------------------------
>  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  1
> max.sia: 0.000E+00 tol.sia: 1.000E-01
>   0 KSP preconditioned resid norm 1.137005501778e+06 true resid norm
> 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.506085526342e+05 true resid norm
> 9.597161643502e+05 ||r(i)||/||b|| 9.958492960384e-02
>   2 KSP preconditioned resid norm 5.456879627980e+04 true resid norm
> 1.406907438671e+05 ||r(i)||/||b|| 1.459877237079e-02
>   3 KSP preconditioned resid norm 2.710407097622e+04 true resid norm
> 1.291826672621e+05 ||r(i)||/||b|| 1.340463701999e-02
>   4 KSP preconditioned resid norm 9.897001667954e+03 true resid norm
> 1.994507102853e+04 ||r(i)||/||b|| 2.069599917247e-03
>   5 KSP preconditioned resid norm 4.429538402026e+03 true resid norm
> 1.048435199832e+04 ||r(i)||/||b|| 1.087908586391e-03
>   6 KSP preconditioned resid norm 2.183344020397e+03 true resid norm
> 5.418816689871e+03 ||r(i)||/||b|| 5.622834111190e-04
>   7 KSP preconditioned resid norm 1.322492709422e+03 true resid norm
> 3.186218374195e+03 ||r(i)||/||b|| 3.306178892084e-04
>   8 KSP preconditioned resid norm 6.738694120116e+02 true resid norm
> 1.461978701106e+03 ||r(i)||/||b|| 1.517021922107e-04
>   9 KSP preconditioned resid norm 3.720835435101e+02 true resid norm
> 4.069516281820e+02 ||r(i)||/||b|| 4.222732798516e-05
>  10 KSP preconditioned resid norm 1.874689621103e+02 true resid norm
> 5.563177554677e+02 ||r(i)||/||b|| 5.772630135195e-05
>  11 KSP preconditioned resid norm 8.612569469037e+01 true resid norm
> 1.054136865792e+02 ||r(i)||/||b|| 1.093824918994e-05
>  12 KSP preconditioned resid norm 4.181701635083e+01 true resid norm
> 1.125544728389e+02 ||r(i)||/||b|| 1.167921274083e-05
>  13 KSP preconditioned resid norm 2.181470261073e+01 true resid norm
> 2.314218289439e+01 ||r(i)||/||b|| 2.401348169412e-06
>  14 KSP preconditioned resid norm 1.180501182018e+01 true resid norm
> 2.818614875745e+01 ||r(i)||/||b|| 2.924735191592e-06
>  15 KSP preconditioned resid norm 6.607882346065e+00 true resid norm
> 7.039364165934e+00 ||r(i)||/||b|| 7.304394892577e-07
> Linear solve converged due to CONVERGED_RTOL iterations 15
>
>
> convergence summary using 8 or 16 processors
>  ------------------------------------------------------------------------
>  timestep:       1 time: 1.000E-02 years   delt: 1.000E-02 years iter:  1
> max.sia: 0.000E+00 tol.sia: 1.000E-01
>   0 KSP preconditioned resid norm 1.172246902854e+06 true resid norm
> 9.637162652703e+06 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 2.111438753837e+05 true resid norm
> 3.422907872797e+06 ||r(i)||/||b|| 3.551779705448e-01
>   2 KSP preconditioned resid norm 9.189117316053e+04 true resid norm
> 1.197217247701e+06 ||r(i)||/||b|| 1.242292250163e-01
>   3 KSP preconditioned resid norm 3.567188384324e+04 true resid norm
> 3.070624730376e+05 ||r(i)||/||b|| 3.186233169485e-02
>   4 KSP preconditioned resid norm 1.629589856587e+04 true resid norm
> 1.857713260583e+05 ||r(i)||/||b|| 1.927655812743e-02
>   5 KSP preconditioned resid norm 8.006759563515e+03 true resid norm
> 1.180288951151e+05 ||r(i)||/||b|| 1.224726606456e-02
>   6 KSP preconditioned resid norm 5.358897276807e+03 true resid norm
> 1.244255645533e+05 ||r(i)||/||b|| 1.291101634758e-02
>   7 KSP preconditioned resid norm 4.389072625628e+03 true resid norm
> 1.090628475655e+05 ||r(i)||/||b|| 1.131690431051e-02
>   8 KSP preconditioned resid norm 3.744362402037e+03 true resid norm
> 1.068327247800e+05 ||r(i)||/||b|| 1.108549566194e-02
>   9 KSP preconditioned resid norm 3.329272298367e+03 true resid norm
> 1.000736812587e+05 ||r(i)||/||b|| 1.038414363906e-02
>  10 KSP preconditioned resid norm 2.973880694381e+03 true resid norm
> 9.519213877340e+04 ||r(i)||/||b|| 9.877610475599e-03
>  11 KSP preconditioned resid norm 2.672681833195e+03 true resid norm
> 8.484262500566e+04 ||r(i)||/||b|| 8.803693375649e-03
>  12 KSP preconditioned resid norm 2.174736539891e+03 true resid norm
> 7.637960549691e+04 ||r(i)||/||b|| 7.925528316728e-03
>  13 KSP preconditioned resid norm 1.920508771340e+03 true resid norm
> 6.875465496734e+04 ||r(i)||/||b|| 7.134325469547e-03
>  14 KSP preconditioned resid norm 1.748769817174e+03 true resid norm
> 6.402474320438e+04 ||r(i)||/||b|| 6.643526265111e-03
>  15 KSP preconditioned resid norm 1.668781376573e+03 true resid norm
> 5.938074141272e+04 ||r(i)||/||b|| 6.161641507219e-03
>  16 KSP preconditioned resid norm 1.507934446836e+03 true resid norm
> 5.376897976816e+04 ||r(i)||/||b|| 5.579337166534e-03
>  17 KSP preconditioned resid norm 1.341780677843e+03 true resid norm
> 4.418909196385e+04 ||r(i)||/||b|| 4.585280290092e-03
>  18 KSP preconditioned resid norm 1.096322610019e+03 true resid norm
> 3.281273489858e+04 ||r(i)||/||b|| 3.404812814836e-03
>  19 KSP preconditioned resid norm 8.572434892128e+02 true resid norm
> 2.162598538334e+04 ||r(i)||/||b|| 2.244019963415e-03
>  20 KSP preconditioned resid norm 6.507523424872e+02 true resid norm
> 1.670057090484e+04 ||r(i)||/||b|| 1.732934423406e-03
>  21 KSP preconditioned resid norm 5.390162236367e+02 true resid norm
> 1.360673589185e+04 ||r(i)||/||b|| 1.411902691922e-03
>  22 KSP preconditioned resid norm 4.424143347792e+02 true resid norm
> 1.137879849375e+04 ||r(i)||/||b|| 1.180720810037e-03
>  23 KSP preconditioned resid norm 3.720257830024e+02 true resid norm
> 9.449031285412e+03 ||r(i)||/||b|| 9.804785522388e-04
>  24 KSP preconditioned resid norm 3.030947404532e+02 true resid norm
> 8.347069391513e+03 ||r(i)||/||b|| 8.661334972044e-04
>  25 KSP preconditioned resid norm 2.667874861724e+02 true resid norm
> 7.530125164047e+03 ||r(i)||/||b|| 7.813632949253e-04
>  26 KSP preconditioned resid norm 2.367701662092e+02 true resid norm
> 6.792074618833e+03 ||r(i)||/||b|| 7.047794940898e-04
>  27 KSP preconditioned resid norm 2.083554010658e+02 true resid norm
> 5.833539191175e+03 ||r(i)||/||b|| 6.053170836065e-04
>  28 KSP preconditioned resid norm 1.735399579409e+02 true resid norm
> 4.883873002196e+03 ||r(i)||/||b|| 5.067749895065e-04
>  29 KSP preconditioned resid norm 1.467413383315e+02 true resid norm
> 4.058954246436e+03 ||r(i)||/||b|| 4.211773104502e-04
>  30 KSP preconditioned resid norm 1.244235694540e+02 true resid norm
> 3.337590813452e+03 ||r(i)||/||b|| 3.463250474989e-04
>  31 KSP preconditioned resid norm 1.198432875407e+02 true resid norm
> 3.199014077607e+03 ||r(i)||/||b|| 3.319456351305e-04
>  32 KSP preconditioned resid norm 1.027669478022e+02 true resid norm
> 3.150305508056e+03 ||r(i)||/||b|| 3.268913913342e-04
>  33 KSP preconditioned resid norm 9.663893782079e+01 true resid norm
> 2.857716117586e+03 ||r(i)||/||b|| 2.965308587777e-04
>  34 KSP preconditioned resid norm 8.531640537674e+01 true resid norm
> 2.584725514230e+03 ||r(i)||/||b|| 2.682039939945e-04
>  35 KSP preconditioned resid norm 7.610279175833e+01 true resid norm
> 2.212227973785e+03 ||r(i)||/||b|| 2.295517937704e-04
>  36 KSP preconditioned resid norm 6.331885100455e+01 true resid norm
> 1.858910367331e+03 ||r(i)||/||b|| 1.928897990333e-04
>  37 KSP preconditioned resid norm 5.635209736607e+01 true resid norm
> 1.571087791205e+03 ||r(i)||/||b|| 1.630238948769e-04
>  38 KSP preconditioned resid norm 4.815528377271e+01 true resid norm
> 1.313627294747e+03 ||r(i)||/||b|| 1.363085113416e-04
>  39 KSP preconditioned resid norm 4.314593066310e+01 true resid norm
> 1.141179411417e+03 ||r(i)||/||b|| 1.184144599963e-04
>  40 KSP preconditioned resid norm 3.700552222031e+01 true resid norm
> 9.643354964443e+02 ||r(i)||/||b|| 1.000642545110e-04
>  41 KSP preconditioned resid norm 3.059145971331e+01 true resid norm
> 7.755572523134e+02 ||r(i)||/||b|| 8.047568358679e-05
>  42 KSP preconditioned resid norm 2.553811607631e+01 true resid norm
> 6.248805390867e+02 ||r(i)||/||b|| 6.484071729467e-05
>  43 KSP preconditioned resid norm 2.016013790111e+01 true resid norm
> 5.415908243972e+02 ||r(i)||/||b|| 5.619816162855e-05
>  44 KSP preconditioned resid norm 1.757243660826e+01 true resid norm
> 4.812748098035e+02 ||r(i)||/||b|| 4.993947151743e-05
>  45 KSP preconditioned resid norm 1.498503590085e+01 true resid norm
> 4.478420094581e+02 ||r(i)||/||b|| 4.647031762326e-05
>  46 KSP preconditioned resid norm 1.348574164098e+01 true resid norm
> 3.944236184677e+02 ||r(i)||/||b|| 4.092735929461e-05
>  47 KSP preconditioned resid norm 1.156613274720e+01 true resid norm
> 3.682508444462e+02 ||r(i)||/||b|| 3.821154189432e-05
> Linear solve converged due to CONVERGED_RTOL iterations 47
>
> Thanks and regards,
>
> Danyang
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/f835110e/attachment.html>

From bsmith at mcs.anl.gov  Mon Oct  5 13:51:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 5 Oct 2015 13:51:48 -0500
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
	<CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
	<CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>
Message-ID: <898D5190-4C0A-484F-BB0F-9482038AABBA@mcs.anl.gov>


   Looks like the bug of using gmres on the coarse mesh is still there in the latest patch release. 

   If you switch to PETSc master http://www.mcs.anl.gov/petsc/developers/index.html  it will not use gmres

  Barry

> On Oct 5, 2015, at 9:06 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> 
> Hi Mark, 
> 
> I've lost it, too.  I was bisecting to find the change that started returning the indefinite PC error to our code that has previously worked -- but this was using KSPSetNullSpace.  
> Increasing the number of steps was in an effort to see if it impacted the error or not, partially based on this thread from PETSC-users (http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023653.html) with one of the previous authors of our code.  
> 
> Updating to PETSc master briefly eliminated the error in the poisson solver, although this was only the case in serial, it still failed with an indefinite PC error in parallel.  
> 
> I'll confess that I'm not sure what to bisect between, as we don't have a "good" version after the switch from KSPSetNullSpace -> MatSetNullSpace.  That's what prompted the initial bisection search in and around the 3.5.4 commit range.  I'm going to take another crack at that today in a more automated fashion, since I expect I inserted some human error somewhere along the way.
> 
> Compiled against petsc v3.6.2, it shows again that the coarse grid solver is using GMRES even when using -mg_coarse_ksp_type preonly.  Logs are attached.
> 
> $PETSC_ARCH/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100 -poisson_mg_coarse_ksp_type preonly -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info
> 
> On Sun, Oct 4, 2015 at 8:57 AM, Mark Adams <mfadams at lbl.gov> wrote:
> I've lost this thread a bit, but you seem to be bisecting to find where a problem started and you are noticing the gmres coarse grid solver.  We fixed a bug where PETSc was resetting the coarse grid solver to GMRES when it should not. So older versions have this, but the current version, and I think this has been in place for all of v3.6, but it might have missed v3.6.1, have the fix of not resetting the coarse grid solver type.  GAMG sets the coarse grid solver type to preonly, but you can override it.  Let me know if I'm missing something here.
> 
> I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This is the number of smoothing steps of the prolongation operator and you should not use more than 1.  In fact for CFD, you should use no smoothing (0), probably.  
> 
> Mark
> 
> On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> Hi Mark, 
> 
> I just noticed that in the previous commit 7743f89, it's also using GMRES in the multigrid solve but doesn't complain until the 2nd timestep, so my bisection criteria is off, since I was giving commits a PASS if they made it through the one timestep without complaining about the indefinite PC. I think I'm still close to the problem commit, but it's probably a little bit before 25a145a.  Apologies for the goose chase.
> 
> Thanks,
> Gil Forsyth
> 
> On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> I ran for one timestep against 3.5.4 with 
> #+BEGIN_SRC
> petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100 -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info > kspview_3.5.4.log
> #+END_SRC
> 
> and then against 25a145a with the same inputs.  I notice that the poisson multigrid solve in 25a145a is using GMRES again while 3.5.4 is using preonly.  
> 
> Logs from both runs are attached.
> 
> 
> On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
> Can you please send a good log also, with the ksp_view.
> Mark
> 
> On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> Using PETSc master branch solved the problem in serial, but I'm still seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This runs to completion when I don't use GAMG.  Log is attached for the following run.
> 
> $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2 $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
> 
> 
> Thanks again,
> Gil Forsyth
> 
> 
> On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> Ah, got it.  I'll checkout the master branch and see if the behavior persists.  
> 
> Many thanks,
> Gil
> 
> On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that the behavior seems to have changed between versions.  The only difference in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to MatSetNullSpace.
> 
> Mark made some GAMG changes which were later reversed because they had unintended consequences like this.
> I think what Barry means is, "you should get the behavior you expect using the master branch from PETSc development"
> 
>   Thanks,
> 
>      Matt
>  
> On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Update your PETSc
> 
> 
> > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> > Hi Barry,
> >
> > We aren't explicitly setting GMRES anywhere in the code and I'm not sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace works with:
> >
> > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.5.4
> >
> > and shows that the coarse grid solver is of type:preonly
> >
> > running the newer version that uses MatSetNullSpace in its stead and adding in -poisson_mg_coarse_ksp_type preonly
> >
> > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview3.6.1
> >
> > still shows
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >   maximum iterations=10000
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >   left preconditioning
> >   using nonzero initial guess
> >   using PRECONDITIONED norm type for convergence test
> > PC Object:(poisson_) 1 MPI processes
> >   type: gamg
> >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
> >       Cycles per PCApply=1
> >       Using Galerkin computed coarse grid matrices
> >       GAMG specific options
> >         Threshold for dropping small values from graph 0
> >         AGG specific options
> >           Symmetric graph false
> >   Coarse grid solver -- level -------------------------------
> >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> >       left preconditioning
> >       using NONE norm type for convergence test
> >
> >
> > both logs are attached.
> >
> >
> > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >    This can't work. You can't use a GMRES inside a CG.   Try changing to -poisson_mg_coarse_ksp_type preonly
> >
> > KSP Object:(poisson_) 1 MPI processes
> >   type: cg
> >
> > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> >       type: gmres
> >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
> >         GMRES: happy breakdown tolerance 1e-30
> >       maximum iterations=1, initial guess is zero
> >
> >
> > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > >
> > >
> > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > > Hi all,
> > >
> > > I've been having some trouble with what should be a relatively simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> > >
> > > I'm getting indefinite PC errors for a simple lid-driven cavity test problem, 32x32 at Re 100
> > >
> > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set the null space.  This is for a 2D Poisson system with no immersed boundary and so the null space is the constant vector.
> > >
> > >   MatNullSpace nsp;
> > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL, &nsp); CHKERRQ(ierr);
> > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> > >
> > > Clearly this has to happen in the reverse order, since ksp2 would not be created yet.
> > >
> > > For questions about solvers, we HAVE to see the complete output of -ksp_view so we
> > > know what we are dealing with. Its also nice to have -ksp_monitor_true_residual -ksp_converged_reason
> > >
> > >   Matt
> > >
> > > Yes -- sorry, those are both in inline files and are called in the reverse order that I wrote them out.
> > >
> > > I've attached the output of
> > >
> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1 -poisson_ksp_view -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason > kspview.log
> > >
> > >
> > >
> > > And then setup the KSP with
> > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> > >
> > > The matrix QTBNQ does not change, only the rhs of the system is updated.
> > >
> > > We run this with `-pc_type gamg -pc_gamg_type agg -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> > >
> > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> > >
> > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> > >
> > > and the same code diverges after 1 timestep and returns a -8 KSP_DIVERGED_INDEFINITE_PC
> > >
> > > This is weird, especially because if we change nsmooths to 2, it runs for 264 timesteps and the returns the same error.  But we have explicitly set KSPSetReusePreconditioner so it should be using the same PC, right?
> > >
> > > Change nsmooths to 3 and it again diverges after 1 timestep.
> > >
> > > Change nsmooths to 4 and it runs to completion.
> > >
> > > It seems like either gamg's behavior has changed, or that KSPSetNullSpace was doing something implicitly that we now need to do explicitly in addition to MatSetNullSpace?
> > >
> > > Thanks,
> > > Gil Forsyth
> > >
> > >
> > >
> > > --
> > > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > > -- Norbert Wiener
> > >
> > > <kspview.log>
> >
> >
> > <kspview3.5.4><kspview3.6.1>
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> 
> 
> 
> <kspview_arch-3264318.log>


From gforsyth at gwu.edu  Mon Oct  5 14:53:02 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Mon, 5 Oct 2015 15:53:02 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <898D5190-4C0A-484F-BB0F-9482038AABBA@mcs.anl.gov>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
	<CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
	<CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>
	<898D5190-4C0A-484F-BB0F-9482038AABBA@mcs.anl.gov>
Message-ID: <CAHDu6k5tp-vbdPpf3EdGYcJtrLzjszY4EpzuymOHHO_UJ7uf6A@mail.gmail.com>

Hi Barry and Mark,

Everything is now working as expected on PETSc master, both in serial and
parallel.  Many thanks for all of your help.

Gil Forsyth

On Mon, Oct 5, 2015 at 2:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    Looks like the bug of using gmres on the coarse mesh is still there in
> the latest patch release.
>
>    If you switch to PETSc master
> http://www.mcs.anl.gov/petsc/developers/index.html  it will not use gmres
>
>   Barry
>
> > On Oct 5, 2015, at 9:06 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> >
> > Hi Mark,
> >
> > I've lost it, too.  I was bisecting to find the change that started
> returning the indefinite PC error to our code that has previously worked --
> but this was using KSPSetNullSpace.
> > Increasing the number of steps was in an effort to see if it impacted
> the error or not, partially based on this thread from PETSC-users (
> http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023653.html)
> with one of the previous authors of our code.
> >
> > Updating to PETSc master briefly eliminated the error in the poisson
> solver, although this was only the case in serial, it still failed with an
> indefinite PC error in parallel.
> >
> > I'll confess that I'm not sure what to bisect between, as we don't have
> a "good" version after the switch from KSPSetNullSpace -> MatSetNullSpace.
> That's what prompted the initial bisection search in and around the 3.5.4
> commit range.  I'm going to take another crack at that today in a more
> automated fashion, since I expect I inserted some human error somewhere
> along the way.
> >
> > Compiled against petsc v3.6.2, it shows again that the coarse grid
> solver is using GMRES even when using -mg_coarse_ksp_type preonly.  Logs
> are attached.
> >
> > $PETSC_ARCH/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
> -poisson_mg_coarse_ksp_type preonly -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info
> >
> > On Sun, Oct 4, 2015 at 8:57 AM, Mark Adams <mfadams at lbl.gov> wrote:
> > I've lost this thread a bit, but you seem to be bisecting to find where
> a problem started and you are noticing the gmres coarse grid solver.  We
> fixed a bug where PETSc was resetting the coarse grid solver to GMRES when
> it should not. So older versions have this, but the current version, and I
> think this has been in place for all of v3.6, but it might have missed
> v3.6.1, have the fix of not resetting the coarse grid solver type.  GAMG
> sets the coarse grid solver type to preonly, but you can override it.  Let
> me know if I'm missing something here.
> >
> > I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This is
> the number of smoothing steps of the prolongation operator and you should
> not use more than 1.  In fact for CFD, you should use no smoothing (0),
> probably.
> >
> > Mark
> >
> > On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Hi Mark,
> >
> > I just noticed that in the previous commit 7743f89, it's also using
> GMRES in the multigrid solve but doesn't complain until the 2nd timestep,
> so my bisection criteria is off, since I was giving commits a PASS if they
> made it through the one timestep without complaining about the indefinite
> PC. I think I'm still close to the problem commit, but it's probably a
> little bit before 25a145a.  Apologies for the goose chase.
> >
> > Thanks,
> > Gil Forsyth
> >
> > On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > I ran for one timestep against 3.5.4 with
> > #+BEGIN_SRC
> > petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
> -poisson_pc_type gamg -poisson_pc_gamg_type agg
> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
> kspview_3.5.4.log
> > #+END_SRC
> >
> > and then against 25a145a with the same inputs.  I notice that the
> poisson multigrid solve in 25a145a is using GMRES again while 3.5.4 is
> using preonly.
> >
> > Logs from both runs are attached.
> >
> >
> > On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
> > Can you please send a good log also, with the ksp_view.
> > Mark
> >
> > On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Using PETSc master branch solved the problem in serial, but I'm still
> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
> runs to completion when I don't use GAMG.  Log is attached for the
> following run.
> >
> > $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
> >
> >
> > Thanks again,
> > Gil Forsyth
> >
> >
> > On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > Ah, got it.  I'll checkout the master branch and see if the behavior
> persists.
> >
> > Many thanks,
> > Gil
> >
> > On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
> the behavior seems to have changed between versions.  The only difference
> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
> MatSetNullSpace.
> >
> > Mark made some GAMG changes which were later reversed because they had
> unintended consequences like this.
> > I think what Barry means is, "you should get the behavior you expect
> using the master branch from PETSc development"
> >
> >   Thanks,
> >
> >      Matt
> >
> > On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >   Update your PETSc
> >
> >
> > > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > >
> > > Hi Barry,
> > >
> > > We aren't explicitly setting GMRES anywhere in the code and I'm not
> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
> works with:
> > >
> > > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.5.4
> > >
> > > and shows that the coarse grid solver is of type:preonly
> > >
> > > running the newer version that uses MatSetNullSpace in its stead and
> adding in -poisson_mg_coarse_ksp_type preonly
> > >
> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
> kspview3.6.1
> > >
> > > still shows
> > >
> > > KSP Object:(poisson_) 1 MPI processes
> > >   type: cg
> > >   maximum iterations=10000
> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> > >   left preconditioning
> > >   using nonzero initial guess
> > >   using PRECONDITIONED norm type for convergence test
> > > PC Object:(poisson_) 1 MPI processes
> > >   type: gamg
> > >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
> > >       Cycles per PCApply=1
> > >       Using Galerkin computed coarse grid matrices
> > >       GAMG specific options
> > >         Threshold for dropping small values from graph 0
> > >         AGG specific options
> > >           Symmetric graph false
> > >   Coarse grid solver -- level -------------------------------
> > >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> > >       type: gmres
> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > >         GMRES: happy breakdown tolerance 1e-30
> > >       maximum iterations=1, initial guess is zero
> > >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
> > >       left preconditioning
> > >       using NONE norm type for convergence test
> > >
> > >
> > > both logs are attached.
> > >
> > >
> > > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >    This can't work. You can't use a GMRES inside a CG.   Try changing
> to -poisson_mg_coarse_ksp_type preonly
> > >
> > > KSP Object:(poisson_) 1 MPI processes
> > >   type: cg
> > >
> > > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
> > >       type: gmres
> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> > >         GMRES: happy breakdown tolerance 1e-30
> > >       maximum iterations=1, initial guess is zero
> > >
> > >
> > > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
> > > >
> > > >
> > > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
> wrote:
> > > > Hi all,
> > > >
> > > > I've been having some trouble with what should be a relatively
> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
> > > >
> > > > I'm getting indefinite PC errors for a simple lid-driven cavity test
> problem, 32x32 at Re 100
> > > >
> > > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to set
> the null space.  This is for a 2D Poisson system with no immersed boundary
> and so the null space is the constant vector.
> > > >
> > > >   MatNullSpace nsp;
> > > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
> &nsp); CHKERRQ(ierr);
> > > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
> > > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
> > > >
> > > > Clearly this has to happen in the reverse order, since ksp2 would
> not be created yet.
> > > >
> > > > For questions about solvers, we HAVE to see the complete output of
> -ksp_view so we
> > > > know what we are dealing with. Its also nice to have
> -ksp_monitor_true_residual -ksp_converged_reason
> > > >
> > > >   Matt
> > > >
> > > > Yes -- sorry, those are both in inline files and are called in the
> reverse order that I wrote them out.
> > > >
> > > > I've attached the output of
> > > >
> > > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
> -poisson_ksp_view -poisson_ksp_monitor_true_residual
> -poisson_ksp_converged_reason > kspview.log
> > > >
> > > >
> > > >
> > > > And then setup the KSP with
> > > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
> > > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
> > > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
> > > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
> > > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
> > > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
> > > >
> > > > The matrix QTBNQ does not change, only the rhs of the system is
> updated.
> > > >
> > > > We run this with `-pc_type gamg -pc_gamg_type agg
> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
> > > >
> > > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
> > > >
> > > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
> > > >
> > > > and the same code diverges after 1 timestep and returns a -8
> KSP_DIVERGED_INDEFINITE_PC
> > > >
> > > > This is weird, especially because if we change nsmooths to 2, it
> runs for 264 timesteps and the returns the same error.  But we have
> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
> right?
> > > >
> > > > Change nsmooths to 3 and it again diverges after 1 timestep.
> > > >
> > > > Change nsmooths to 4 and it runs to completion.
> > > >
> > > > It seems like either gamg's behavior has changed, or that
> KSPSetNullSpace was doing something implicitly that we now need to do
> explicitly in addition to MatSetNullSpace?
> > > >
> > > > Thanks,
> > > > Gil Forsyth
> > > >
> > > >
> > > >
> > > > --
> > > > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > > > -- Norbert Wiener
> > > >
> > > > <kspview.log>
> > >
> > >
> > > <kspview3.5.4><kspview3.6.1>
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> >
> >
> >
> > <kspview_arch-3264318.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/43940bc7/attachment.html>

From mfadams at lbl.gov  Mon Oct  5 15:16:41 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Mon, 5 Oct 2015 16:16:41 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CAHDu6k5tp-vbdPpf3EdGYcJtrLzjszY4EpzuymOHHO_UJ7uf6A@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
	<CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
	<CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>
	<898D5190-4C0A-484F-BB0F-9482038AABBA@mcs.anl.gov>
	<CAHDu6k5tp-vbdPpf3EdGYcJtrLzjszY4EpzuymOHHO_UJ7uf6A@mail.gmail.com>
Message-ID: <CADOhEh4PvJPg5da=J_FhF2Kk-DbYO_z9zEo_a6_O87J6JZcoxw@mail.gmail.com>

Good, sorry for the confusion, I thought this stuff was squared away in
v3.6 and certainly in v3.6.2. (its a little disconcerting that it failed.
 it would be interesting to see if master fails if you set gmres for the
coarse grid solver, ie, was gmres really the problem?)
Mark

On Mon, Oct 5, 2015 at 3:53 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:

> Hi Barry and Mark,
>
> Everything is now working as expected on PETSc master, both in serial and
> parallel.  Many thanks for all of your help.
>
> Gil Forsyth
>
> On Mon, Oct 5, 2015 at 2:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>    Looks like the bug of using gmres on the coarse mesh is still there in
>> the latest patch release.
>>
>>    If you switch to PETSc master
>> http://www.mcs.anl.gov/petsc/developers/index.html  it will not use gmres
>>
>>   Barry
>>
>> > On Oct 5, 2015, at 9:06 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> >
>> > Hi Mark,
>> >
>> > I've lost it, too.  I was bisecting to find the change that started
>> returning the indefinite PC error to our code that has previously worked --
>> but this was using KSPSetNullSpace.
>> > Increasing the number of steps was in an effort to see if it impacted
>> the error or not, partially based on this thread from PETSC-users (
>> http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023653.html)
>> with one of the previous authors of our code.
>> >
>> > Updating to PETSc master briefly eliminated the error in the poisson
>> solver, although this was only the case in serial, it still failed with an
>> indefinite PC error in parallel.
>> >
>> > I'll confess that I'm not sure what to bisect between, as we don't have
>> a "good" version after the switch from KSPSetNullSpace -> MatSetNullSpace.
>> That's what prompted the initial bisection search in and around the 3.5.4
>> commit range.  I'm going to take another crack at that today in a more
>> automated fashion, since I expect I inserted some human error somewhere
>> along the way.
>> >
>> > Compiled against petsc v3.6.2, it shows again that the coarse grid
>> solver is using GMRES even when using -mg_coarse_ksp_type preonly.  Logs
>> are attached.
>> >
>> > $PETSC_ARCH/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>> -poisson_mg_coarse_ksp_type preonly -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info
>> >
>> > On Sun, Oct 4, 2015 at 8:57 AM, Mark Adams <mfadams at lbl.gov> wrote:
>> > I've lost this thread a bit, but you seem to be bisecting to find where
>> a problem started and you are noticing the gmres coarse grid solver.  We
>> fixed a bug where PETSc was resetting the coarse grid solver to GMRES when
>> it should not. So older versions have this, but the current version, and I
>> think this has been in place for all of v3.6, but it might have missed
>> v3.6.1, have the fix of not resetting the coarse grid solver type.  GAMG
>> sets the coarse grid solver type to preonly, but you can override it.  Let
>> me know if I'm missing something here.
>> >
>> > I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This is
>> the number of smoothing steps of the prolongation operator and you should
>> not use more than 1.  In fact for CFD, you should use no smoothing (0),
>> probably.
>> >
>> > Mark
>> >
>> > On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > Hi Mark,
>> >
>> > I just noticed that in the previous commit 7743f89, it's also using
>> GMRES in the multigrid solve but doesn't complain until the 2nd timestep,
>> so my bisection criteria is off, since I was giving commits a PASS if they
>> made it through the one timestep without complaining about the indefinite
>> PC. I think I'm still close to the problem commit, but it's probably a
>> little bit before 25a145a.  Apologies for the goose chase.
>> >
>> > Thanks,
>> > Gil Forsyth
>> >
>> > On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > I ran for one timestep against 3.5.4 with
>> > #+BEGIN_SRC
>> > petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>> -poisson_pc_type gamg -poisson_pc_gamg_type agg
>> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
>> kspview_3.5.4.log
>> > #+END_SRC
>> >
>> > and then against 25a145a with the same inputs.  I notice that the
>> poisson multigrid solve in 25a145a is using GMRES again while 3.5.4 is
>> using preonly.
>> >
>> > Logs from both runs are attached.
>> >
>> >
>> > On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
>> > Can you please send a good log also, with the ksp_view.
>> > Mark
>> >
>> > On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > Using PETSc master branch solved the problem in serial, but I'm still
>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>> runs to completion when I don't use GAMG.  Log is attached for the
>> following run.
>> >
>> > $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>> >
>> >
>> > Thanks again,
>> > Gil Forsyth
>> >
>> >
>> > On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > Ah, got it.  I'll checkout the master branch and see if the behavior
>> persists.
>> >
>> > Many thanks,
>> > Gil
>> >
>> > On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>> > On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show that
>> the behavior seems to have changed between versions.  The only difference
>> in our code between 3.5.4 and 3.6.1 is the change from KSPSetNullSpace to
>> MatSetNullSpace.
>> >
>> > Mark made some GAMG changes which were later reversed because they had
>> unintended consequences like this.
>> > I think what Barry means is, "you should get the behavior you expect
>> using the master branch from PETSc development"
>> >
>> >   Thanks,
>> >
>> >      Matt
>> >
>> > On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> >
>> >   Update your PETSc
>> >
>> >
>> > > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > >
>> > > Hi Barry,
>> > >
>> > > We aren't explicitly setting GMRES anywhere in the code and I'm not
>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>> works with:
>> > >
>> > > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.5.4
>> > >
>> > > and shows that the coarse grid solver is of type:preonly
>> > >
>> > > running the newer version that uses MatSetNullSpace in its stead and
>> adding in -poisson_mg_coarse_ksp_type preonly
>> > >
>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type gamg
>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>> kspview3.6.1
>> > >
>> > > still shows
>> > >
>> > > KSP Object:(poisson_) 1 MPI processes
>> > >   type: cg
>> > >   maximum iterations=10000
>> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> > >   left preconditioning
>> > >   using nonzero initial guess
>> > >   using PRECONDITIONED norm type for convergence test
>> > > PC Object:(poisson_) 1 MPI processes
>> > >   type: gamg
>> > >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>> > >       Cycles per PCApply=1
>> > >       Using Galerkin computed coarse grid matrices
>> > >       GAMG specific options
>> > >         Threshold for dropping small values from graph 0
>> > >         AGG specific options
>> > >           Symmetric graph false
>> > >   Coarse grid solver -- level -------------------------------
>> > >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> > >       type: gmres
>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> > >         GMRES: happy breakdown tolerance 1e-30
>> > >       maximum iterations=1, initial guess is zero
>> > >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>> > >       left preconditioning
>> > >       using NONE norm type for convergence test
>> > >
>> > >
>> > > both logs are attached.
>> > >
>> > >
>> > > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>> wrote:
>> > >
>> > >    This can't work. You can't use a GMRES inside a CG.   Try changing
>> to -poisson_mg_coarse_ksp_type preonly
>> > >
>> > > KSP Object:(poisson_) 1 MPI processes
>> > >   type: cg
>> > >
>> > > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>> > >       type: gmres
>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>> Orthogonalization with no iterative refinement
>> > >         GMRES: happy breakdown tolerance 1e-30
>> > >       maximum iterations=1, initial guess is zero
>> > >
>> > >
>> > > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>> > > >
>> > > >
>> > > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>> knepley at gmail.com> wrote:
>> > > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>> wrote:
>> > > > Hi all,
>> > > >
>> > > > I've been having some trouble with what should be a relatively
>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>> > > >
>> > > > I'm getting indefinite PC errors for a simple lid-driven cavity
>> test problem, 32x32 at Re 100
>> > > >
>> > > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>> set the null space.  This is for a 2D Poisson system with no immersed
>> boundary and so the null space is the constant vector.
>> > > >
>> > > >   MatNullSpace nsp;
>> > > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>> &nsp); CHKERRQ(ierr);
>> > > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>> > > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>> > > >
>> > > > Clearly this has to happen in the reverse order, since ksp2 would
>> not be created yet.
>> > > >
>> > > > For questions about solvers, we HAVE to see the complete output of
>> -ksp_view so we
>> > > > know what we are dealing with. Its also nice to have
>> -ksp_monitor_true_residual -ksp_converged_reason
>> > > >
>> > > >   Matt
>> > > >
>> > > > Yes -- sorry, those are both in inline files and are called in the
>> reverse order that I wrote them out.
>> > > >
>> > > > I've attached the output of
>> > > >
>> > > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>> -poisson_ksp_converged_reason > kspview.log
>> > > >
>> > > >
>> > > >
>> > > > And then setup the KSP with
>> > > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>> > > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>> > > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>> > > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>> > > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE); CHKERRQ(ierr);
>> > > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>> > > >
>> > > > The matrix QTBNQ does not change, only the rhs of the system is
>> updated.
>> > > >
>> > > > We run this with `-pc_type gamg -pc_gamg_type agg
>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>> > > >
>> > > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>> > > >
>> > > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>> > > >
>> > > > and the same code diverges after 1 timestep and returns a -8
>> KSP_DIVERGED_INDEFINITE_PC
>> > > >
>> > > > This is weird, especially because if we change nsmooths to 2, it
>> runs for 264 timesteps and the returns the same error.  But we have
>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>> right?
>> > > >
>> > > > Change nsmooths to 3 and it again diverges after 1 timestep.
>> > > >
>> > > > Change nsmooths to 4 and it runs to completion.
>> > > >
>> > > > It seems like either gamg's behavior has changed, or that
>> KSPSetNullSpace was doing something implicitly that we now need to do
>> explicitly in addition to MatSetNullSpace?
>> > > >
>> > > > Thanks,
>> > > > Gil Forsyth
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > > > -- Norbert Wiener
>> > > >
>> > > > <kspview.log>
>> > >
>> > >
>> > > <kspview3.5.4><kspview3.6.1>
>> >
>> >
>> >
>> >
>> >
>> > --
>> > What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> > -- Norbert Wiener
>> >
>> >
>> >
>> >
>> >
>> >
>> >
>> > <kspview_arch-3264318.log>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/4a3a5bc9/attachment-0001.html>

From bsmith at mcs.anl.gov  Mon Oct  5 15:46:56 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Mon, 5 Oct 2015 15:46:56 -0500
Subject: [petsc-users] Combining MPIUNI with duplicated comms
In-Reply-To: <56126957.4030707@sintef.no>
References: <56126957.4030707@sintef.no>
Message-ID: <C5D49A6B-5A39-4C10-BD34-6E77D7C821C1@mcs.anl.gov>


  Arnem,

   Please find attached a patch that allows you to create additional communicators and use them and free them with MPI-Uni.  This is also in the development branch barry/fix-mpiuni; once it is tested completely we will put it into the maint branch and it will be in the next patch release.

   Please let us know if the patch does not resolve your difficulties.

  Barry

Note that is an arbitrary limit of 120 communicators allowed.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix-mpiuni.patch
Type: application/octet-stream
Size: 3638 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/86fcd13c/attachment.obj>
-------------- next part --------------

> On Oct 5, 2015, at 7:13 AM, Arne Morten Kvarving <arne.morten.kvarving at sintef.no> wrote:
> 
> hi there.
> 
> first of all, i'm not 100% this is supposed to work. if it is not, feel free to tell me just that and i'll find a way to workaround.
> 
> in our code we use MPI_Comm_dup for various reasons. this works fine. except when i run in serial, i.e., using the MPUNI wrapper. if the PETSC_COMM_SELF communicator is duplicated, nastyness results when tearing down the KSP object. while this isn't really fatal as such, it is rather annoying.
> 
> please consider http://www.math.ntnu.no/~arnemort/bug-self.tar.bz2 which is a silly code i wrote to reproduce the issue. a 3x3 matrix is constructed, a corresponding 1-vector and the system is solved.
> then the objects are teared down.
> 
> the tarball has a cmake buildsystem. the buildsystem relies on the standard PETSC_DIR / PETSC_ARCH env variables. if you prefer to build by hand, there is no requirement on using the build system. just watch out for the define it adds.
> 
> the application takes one required command line parameter which should be 'self' or 'world'. this controls which comm is duplicated. if I run with self, i get
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Corrupt argument: http://www.mcs.anl.gov/petsc/documentation/faq.html#valgrind
> [0]PETSC ERROR: Inner MPI_Comm does not have expected reference to outer comm
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
> [0]PETSC ERROR: ./bug_comm_self on a linux-gnu-cxx-dbg named akvarium by akva Mon Oct  5 14:05:52 2015
> [0]PETSC ERROR: Configure options --with-precision=double --with-scalar-type=real --with-debugging=1 --with-blas-lib=/usr/lib/libblas.a --with-lapack-lib=/usr/lib/liblapack.a --with-64-bit-indices=0 --with-clanguage=c++ --with-mpi=0 --LIBS=-ldl --with-ml=0 --with-shared-libraries=0
> [0]PETSC ERROR: #1 Petsc_DelComm_Outer() line 360 in /home/akva/kode/petsc/petsc-3.6.2/src/sys/objects/pinit.c
> 
> if i run with world, no error is generated.
> 
> this is with the recetly released 3.6.2, but i can reproduce atleast back to 3.5.2 (haven't tried older).
> 
> i also noticed that if i add a VecView call, errors will also be generated with world. this line is commented out in the source code. might be some valuable info for those who know the system in that fact.
> 
> since i made it reproducable through the test app, i skipped logs for now. if they or some other info is required from me side, i'll gladly provide it.
> 
> thanks in advance
> 
> arnem


From gforsyth at gwu.edu  Mon Oct  5 16:50:50 2015
From: gforsyth at gwu.edu (Gil Forsyth)
Date: Mon, 5 Oct 2015 17:50:50 -0400
Subject: [petsc-users] Moving from KSPSetNullSpace to MatSetNullSpace
In-Reply-To: <CADOhEh4PvJPg5da=J_FhF2Kk-DbYO_z9zEo_a6_O87J6JZcoxw@mail.gmail.com>
References: <CAHDu6k6w5Wa4D0_YDs2Hmaqk4FFA_zN4ZapXLo-0g9UuPVifoQ@mail.gmail.com>
	<CAMYG4Gk9A70r0R47zTgfHfJv=bsxRV1d8C-vw==gY3FMf+dHWg@mail.gmail.com>
	<CAHDu6k60JLvRExZY5ZCe44RMXtMjoJhsdAkaJro1VPs64wc-cQ@mail.gmail.com>
	<BCB8890C-EFF4-4670-82F4-5BBC9EED6374@mcs.anl.gov>
	<CAHDu6k6_d6zHWVNKThMknNEioLoWFZd07vBCN+SSyg=45pQCtA@mail.gmail.com>
	<F9DC127E-2025-46E0-B692-66B82DC8965F@mcs.anl.gov>
	<CAHDu6k7yckKMK_2Z=4v7ySik3z4LmiO5ocVJOEsAPb6Kj3=WqQ@mail.gmail.com>
	<CAMYG4Gkz6GXzdySH6pCVcRyGOTpZ0coE3ApJ19B_vx8p0-YEiQ@mail.gmail.com>
	<CAHDu6k5jHHZSPzhwdR-1FX3D-yBKsJmx8nkiBfEcS6hxYvPyaA@mail.gmail.com>
	<CAHDu6k7XaCdjLsN2AXOwRaMO1h5Lmr+dsdXMV69GqxLsSGWBrA@mail.gmail.com>
	<CADOhEh4H-NJjuoDtOiwkMdSQibdugM88cQNLruY_6O2+dod-NA@mail.gmail.com>
	<CAHDu6k6yYY=cTRu9mdQc_hLfxgay853Ai4FgLg4ytfKuis_i6w@mail.gmail.com>
	<CAHDu6k4zHWh7fLhZk=hmAkiUUuW-0PS8aCwn6WYtCLYaMyh=HQ@mail.gmail.com>
	<CADOhEh7Ep3oAFNEtge-whQSsgFS2UggkoHsxmnUx411gPaJh9g@mail.gmail.com>
	<CAHDu6k4O9=Rz7qC-=_hX6K6h3TOu8edNKJELWD8tvkjB45EzoA@mail.gmail.com>
	<898D5190-4C0A-484F-BB0F-9482038AABBA@mcs.anl.gov>
	<CAHDu6k5tp-vbdPpf3EdGYcJtrLzjszY4EpzuymOHHO_UJ7uf6A@mail.gmail.com>
	<CADOhEh4PvJPg5da=J_FhF2Kk-DbYO_z9zEo_a6_O87J6JZcoxw@mail.gmail.com>
Message-ID: <CAHDu6k7myQdbiSFTqGdotCWPqNRLZMmyO_C=7qyVR+jLm4m5ew@mail.gmail.com>

Hi Mark,

I've attached the log of

petibm-master/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
-poisson_pc_type gamg -poisson_pc_gamg_type agg
-poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view
-poisson_mg_coarse_ksp_type gmres > gmrescoarse.log

and it does fail straight away when setting gmres as the coarse grid
solver.  If I give it nsmooths=1, the smoothing saves it for about 15
timesteps but then it still errors out with the indefinite PC error.  If
there's anything else you'd like me to try out, just let me know.

Thanks,
Gil Forsyth

On Mon, Oct 5, 2015 at 4:16 PM, Mark Adams <mfadams at lbl.gov> wrote:

> Good, sorry for the confusion, I thought this stuff was squared away in
> v3.6 and certainly in v3.6.2. (its a little disconcerting that it failed.
>  it would be interesting to see if master fails if you set gmres for the
> coarse grid solver, ie, was gmres really the problem?)
> Mark
>
> On Mon, Oct 5, 2015 at 3:53 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>
>> Hi Barry and Mark,
>>
>> Everything is now working as expected on PETSc master, both in serial and
>> parallel.  Many thanks for all of your help.
>>
>> Gil Forsyth
>>
>> On Mon, Oct 5, 2015 at 2:51 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>    Looks like the bug of using gmres on the coarse mesh is still there
>>> in the latest patch release.
>>>
>>>    If you switch to PETSc master
>>> http://www.mcs.anl.gov/petsc/developers/index.html  it will not use
>>> gmres
>>>
>>>   Barry
>>>
>>> > On Oct 5, 2015, at 9:06 AM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> >
>>> > Hi Mark,
>>> >
>>> > I've lost it, too.  I was bisecting to find the change that started
>>> returning the indefinite PC error to our code that has previously worked --
>>> but this was using KSPSetNullSpace.
>>> > Increasing the number of steps was in an effort to see if it impacted
>>> the error or not, partially based on this thread from PETSC-users (
>>> http://lists.mcs.anl.gov/pipermail/petsc-users/2014-November/023653.html)
>>> with one of the previous authors of our code.
>>> >
>>> > Updating to PETSc master briefly eliminated the error in the poisson
>>> solver, although this was only the case in serial, it still failed with an
>>> indefinite PC error in parallel.
>>> >
>>> > I'll confess that I'm not sure what to bisect between, as we don't
>>> have a "good" version after the switch from KSPSetNullSpace ->
>>> MatSetNullSpace.  That's what prompted the initial bisection search in and
>>> around the 3.5.4 commit range.  I'm going to take another crack at that
>>> today in a more automated fashion, since I expect I inserted some human
>>> error somewhere along the way.
>>> >
>>> > Compiled against petsc v3.6.2, it shows again that the coarse grid
>>> solver is using GMRES even when using -mg_coarse_ksp_type preonly.  Logs
>>> are attached.
>>> >
>>> > $PETSC_ARCH/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>>> -poisson_mg_coarse_ksp_type preonly -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 0 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info
>>> >
>>> > On Sun, Oct 4, 2015 at 8:57 AM, Mark Adams <mfadams at lbl.gov> wrote:
>>> > I've lost this thread a bit, but you seem to be bisecting to find
>>> where a problem started and you are noticing the gmres coarse grid solver.
>>> We fixed a bug where PETSc was resetting the coarse grid solver to GMRES
>>> when it should not. So older versions have this, but the current version,
>>> and I think this has been in place for all of v3.6, but it might have
>>> missed v3.6.1, have the fix of not resetting the coarse grid solver type.
>>> GAMG sets the coarse grid solver type to preonly, but you can override it.
>>> Let me know if I'm missing something here.
>>> >
>>> > I also see that you are setting -pc_gamg_agg_nsmooths 1,2,3,4.  This
>>> is the number of smoothing steps of the prolongation operator and you
>>> should not use more than 1.  In fact for CFD, you should use no smoothing
>>> (0), probably.
>>> >
>>> > Mark
>>> >
>>> > On Thu, Oct 1, 2015 at 4:27 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > Hi Mark,
>>> >
>>> > I just noticed that in the previous commit 7743f89, it's also using
>>> GMRES in the multigrid solve but doesn't complain until the 2nd timestep,
>>> so my bisection criteria is off, since I was giving commits a PASS if they
>>> made it through the one timestep without complaining about the indefinite
>>> PC. I think I'm still close to the problem commit, but it's probably a
>>> little bit before 25a145a.  Apologies for the goose chase.
>>> >
>>> > Thanks,
>>> > Gil Forsyth
>>> >
>>> > On Thu, Oct 1, 2015 at 4:21 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > I ran for one timestep against 3.5.4 with
>>> > #+BEGIN_SRC
>>> > petibm-3.5.4/bin/petibm2d -directory examples/2d/lidDrivenCavity/Re100
>>> -poisson_pc_type gamg -poisson_pc_gamg_type agg
>>> -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason -info >
>>> kspview_3.5.4.log
>>> > #+END_SRC
>>> >
>>> > and then against 25a145a with the same inputs.  I notice that the
>>> poisson multigrid solve in 25a145a is using GMRES again while 3.5.4 is
>>> using preonly.
>>> >
>>> > Logs from both runs are attached.
>>> >
>>> >
>>> > On Thu, Oct 1, 2015 at 3:52 PM, Mark Adams <mfadams at lbl.gov> wrote:
>>> > Can you please send a good log also, with the ksp_view.
>>> > Mark
>>> >
>>> > On Wed, Sep 30, 2015 at 3:11 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > Using PETSc master branch solved the problem in serial, but I'm still
>>> seeing the same KSP_DIVERGED_INDEFINITE_PC error when I run with MPI.  This
>>> runs to completion when I don't use GAMG.  Log is attached for the
>>> following run.
>>> >
>>> > $PETSC_DIR/$PETSC_ARCH/bin/mpirun -n 2
>>> $PETIBM_DIR/petibm-git/bin/petibm2d -directory . -poisson_pc_type gamg
>>> -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1 -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason
>>> >
>>> >
>>> > Thanks again,
>>> > Gil Forsyth
>>> >
>>> >
>>> > On Tue, Sep 29, 2015 at 1:12 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > Ah, got it.  I'll checkout the master branch and see if the behavior
>>> persists.
>>> >
>>> > Many thanks,
>>> > Gil
>>> >
>>> > On Tue, Sep 29, 2015 at 1:10 PM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>> > On Tue, Sep 29, 2015 at 12:08 PM, Gil Forsyth <gforsyth at gwu.edu>
>>> wrote:
>>> > PETSc is version 3.6.1 -- I just included a log from 3.5.4 to show
>>> that the behavior seems to have changed between versions.  The only
>>> difference in our code between 3.5.4 and 3.6.1 is the change from
>>> KSPSetNullSpace to MatSetNullSpace.
>>> >
>>> > Mark made some GAMG changes which were later reversed because they had
>>> unintended consequences like this.
>>> > I think what Barry means is, "you should get the behavior you expect
>>> using the master branch from PETSc development"
>>> >
>>> >   Thanks,
>>> >
>>> >      Matt
>>> >
>>> > On Tue, Sep 29, 2015 at 1:04 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >   Update your PETSc
>>> >
>>> >
>>> > > On Sep 29, 2015, at 12:00 PM, Gil Forsyth <gforsyth at gwu.edu> wrote:
>>> > >
>>> > > Hi Barry,
>>> > >
>>> > > We aren't explicitly setting GMRES anywhere in the code and I'm not
>>> sure why it's being used.  Running our 3.5.4 code using KSPSetNullSpace
>>> works with:
>>> > >
>>> > > $PETIBM_DIR/petibm3.5/bin/petibm2d -directory . -poisson_pc_type
>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>> -poisson_ksp_converged_reason > kspview3.5.4
>>> > >
>>> > > and shows that the coarse grid solver is of type:preonly
>>> > >
>>> > > running the newer version that uses MatSetNullSpace in its stead and
>>> adding in -poisson_mg_coarse_ksp_type preonly
>>> > >
>>> > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>> gamg -poisson_pc_gamg_type agg -poisson_pc_gamg_agg_nsmooths 1
>>> -poisson_mg_coarse_ksp_type preonly -poisson_ksp_view
>>> -poisson_ksp_monitor_true_residual -poisson_ksp_converged_reason >
>>> kspview3.6.1
>>> > >
>>> > > still shows
>>> > >
>>> > > KSP Object:(poisson_) 1 MPI processes
>>> > >   type: cg
>>> > >   maximum iterations=10000
>>> > >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> > >   left preconditioning
>>> > >   using nonzero initial guess
>>> > >   using PRECONDITIONED norm type for convergence test
>>> > > PC Object:(poisson_) 1 MPI processes
>>> > >   type: gamg
>>> > >     MG: type is MULTIPLICATIVE, levels=3 cycles=v
>>> > >       Cycles per PCApply=1
>>> > >       Using Galerkin computed coarse grid matrices
>>> > >       GAMG specific options
>>> > >         Threshold for dropping small values from graph 0
>>> > >         AGG specific options
>>> > >           Symmetric graph false
>>> > >   Coarse grid solver -- level -------------------------------
>>> > >     KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> > >       type: gmres
>>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> > >         GMRES: happy breakdown tolerance 1e-30
>>> > >       maximum iterations=1, initial guess is zero
>>> > >       tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
>>> > >       left preconditioning
>>> > >       using NONE norm type for convergence test
>>> > >
>>> > >
>>> > > both logs are attached.
>>> > >
>>> > >
>>> > > On Tue, Sep 29, 2015 at 12:37 PM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> > >
>>> > >    This can't work. You can't use a GMRES inside a CG.   Try
>>> changing to -poisson_mg_coarse_ksp_type preonly
>>> > >
>>> > > KSP Object:(poisson_) 1 MPI processes
>>> > >   type: cg
>>> > >
>>> > > KSP Object:    (poisson_mg_coarse_)     1 MPI processes
>>> > >       type: gmres
>>> > >         GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
>>> Orthogonalization with no iterative refinement
>>> > >         GMRES: happy breakdown tolerance 1e-30
>>> > >       maximum iterations=1, initial guess is zero
>>> > >
>>> > >
>>> > > > On Sep 29, 2015, at 10:53 AM, Gil Forsyth <gforsyth at gwu.edu>
>>> wrote:
>>> > > >
>>> > > >
>>> > > > On Tue, Sep 29, 2015 at 11:42 AM, Matthew Knepley <
>>> knepley at gmail.com> wrote:
>>> > > > On Tue, Sep 29, 2015 at 10:28 AM, Gil Forsyth <gforsyth at gwu.edu>
>>> wrote:
>>> > > > Hi all,
>>> > > >
>>> > > > I've been having some trouble with what should be a relatively
>>> simple update to an immersed boundary CFD solver from PETSc 3.5.4 to 3.6.1
>>> > > >
>>> > > > I'm getting indefinite PC errors for a simple lid-driven cavity
>>> test problem, 32x32 at Re 100
>>> > > >
>>> > > > Under PETSc 3.5.4 using KSPSetNullSpace we used the following to
>>> set the null space.  This is for a 2D Poisson system with no immersed
>>> boundary and so the null space is the constant vector.
>>> > > >
>>> > > >   MatNullSpace nsp;
>>> > > >   ierr = MatNullSpaceCreate(PETSC_COMM_WORLD, PETSC_TRUE, 0, NULL,
>>> &nsp); CHKERRQ(ierr);
>>> > > >   ierr = KSPSetNullSpace(ksp2, nsp); CHKERRQ(ierr);
>>> > > >   ierr = MatNullSpaceDestroy(&nsp); CHKERRQ(ierr);
>>> > > >
>>> > > > Clearly this has to happen in the reverse order, since ksp2 would
>>> not be created yet.
>>> > > >
>>> > > > For questions about solvers, we HAVE to see the complete output of
>>> -ksp_view so we
>>> > > > know what we are dealing with. Its also nice to have
>>> -ksp_monitor_true_residual -ksp_converged_reason
>>> > > >
>>> > > >   Matt
>>> > > >
>>> > > > Yes -- sorry, those are both in inline files and are called in the
>>> reverse order that I wrote them out.
>>> > > >
>>> > > > I've attached the output of
>>> > > >
>>> > > > $PETIBM_DIR/petibm3.6/bin/petibm2d -directory . -poisson_pc_type
>>> gamg -poisson_pc_gamg_type agg -poisson_gamg_agg_nsmooths 1
>>> -poisson_ksp_view -poisson_ksp_monitor_true_residual
>>> -poisson_ksp_converged_reason > kspview.log
>>> > > >
>>> > > >
>>> > > >
>>> > > > And then setup the KSP with
>>> > > >   ierr = KSPCreate(PETSC_COMM_WORLD, &ksp2); CHKERRQ(ierr);
>>> > > >   ierr = KSPSetOptionsPrefix(ksp2, "poisson_"); CHKERRQ(ierr);
>>> > > >   ierr = KSPSetOperators(ksp2, QTBNQ, QTBNQ); CHKERRQ(ierr);
>>> > > >   ierr = KSPSetInitialGuessNonzero(ksp2, PETSC_TRUE);
>>> CHKERRQ(ierr);
>>> > > >   ierr = KSPSetType(ksp2, KSPCG); CHKERRQ(ierr);
>>> > > >   ierr = KSPSetReusePreconditioner(ksp2, PETSC_TRUE);
>>> CHKERRQ(ierr);
>>> > > >   ierr = KSPSetFromOptions(ksp2); CHKERRQ(ierr);
>>> > > >
>>> > > > The matrix QTBNQ does not change, only the rhs of the system is
>>> updated.
>>> > > >
>>> > > > We run this with `-pc_type gamg -pc_gamg_type agg
>>> -pc_gamg_agg_nsmooths 1` and everything seems to work as expected.
>>> > > >
>>> > > > Under PETSc 3.6.1, we change only the KSPSetNullSpace line, to
>>> > > >
>>> > > >   ierr = MatSetNullSpace(QTBNQ, nsp); CHKERRQ(ierr);
>>> > > >
>>> > > > and the same code diverges after 1 timestep and returns a -8
>>> KSP_DIVERGED_INDEFINITE_PC
>>> > > >
>>> > > > This is weird, especially because if we change nsmooths to 2, it
>>> runs for 264 timesteps and the returns the same error.  But we have
>>> explicitly set KSPSetReusePreconditioner so it should be using the same PC,
>>> right?
>>> > > >
>>> > > > Change nsmooths to 3 and it again diverges after 1 timestep.
>>> > > >
>>> > > > Change nsmooths to 4 and it runs to completion.
>>> > > >
>>> > > > It seems like either gamg's behavior has changed, or that
>>> KSPSetNullSpace was doing something implicitly that we now need to do
>>> explicitly in addition to MatSetNullSpace?
>>> > > >
>>> > > > Thanks,
>>> > > > Gil Forsyth
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > > > -- Norbert Wiener
>>> > > >
>>> > > > <kspview.log>
>>> > >
>>> > >
>>> > > <kspview3.5.4><kspview3.6.1>
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > --
>>> > What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> > -- Norbert Wiener
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > <kspview_arch-3264318.log>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/78a7efbd/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: gmrescoarse.log
Type: text/x-log
Size: 9621 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/78a7efbd/attachment-0001.bin>

From jychang48 at gmail.com  Tue Oct  6 00:23:14 2015
From: jychang48 at gmail.com (Justin Chang)
Date: Mon, 5 Oct 2015 23:23:14 -0600
Subject: [petsc-users] make streams error
Message-ID: <CAP2=TMgTuDzfLMuHzL1qWurLB_ga_zAbtqThztjgBB-SsANYag@mail.gmail.com>

Hi everyone,

I tried running the streams benchmark on the latest petsc version and got
this error:

$ make streams NPMAX=1
cd src/benchmarks/streams; /usr/bin/gmake  --no-print-directory
PETSC_DIR=/users/jychang48/petsc PETSC_ARCH=arch-linux2-c-opt streams
mpicc -o MPIVersion.o -c -fPIC  -Wall -Wwrite-strings -Wno-strict-aliasing
-Wno-unknown-pragmas -g -O   -I/turquoise/users/jychang48/petsc/include
-I/turquoise/users/jychang48/petsc/arch-linux2-c-opt/include
-I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include/openmpi/opal/mca/hwloc/hwloc132/hwloc/include
-I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include
-I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include/openmpi
`pwd`/MPIVersion.c
Reported: 1 (out of 1) daemons - 1 (out of 1) procs
Number of MPI processes 1 Processor names  wf612.localdomain
Triad:        13797.0526   Rate (MB/s)
------------------------------------------------
Traceback (most recent call last):
  File "process.py", line 110, in <module>
    process(len(sys.argv)-1)
  File "process.py", line 24, in process
    size = int(fields[0].split()[0])
ValueError: invalid literal for int() with base 10: 'Reported:'
gmake[1]: [stream] Error 1 (ignored)
Traceback (most recent call last):
  File "process.py", line 110, in <module>
    process(len(sys.argv)-1)
  File "process.py", line 24, in process
    size = int(fields[0].split()[0])
ValueError: invalid literal for int() with base 10: 'Reported:'
gmake[1]: [streams] Error 1 (ignored)


Did something break, or am I missing something?

Thanks,
Justin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151005/0253deab/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct  6 10:17:12 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 6 Oct 2015 10:17:12 -0500
Subject: [petsc-users] make streams error
In-Reply-To: <CAP2=TMgTuDzfLMuHzL1qWurLB_ga_zAbtqThztjgBB-SsANYag@mail.gmail.com>
References: <CAP2=TMgTuDzfLMuHzL1qWurLB_ga_zAbtqThztjgBB-SsANYag@mail.gmail.com>
Message-ID: <CF2A6FAC-6BFE-4E29-A376-D098DF50F061@mcs.anl.gov>


  Something must be going wrong with the processing of there results.  The benchmark ran ok producing Triad:        13797.0526   Rate (MB/s)  which is all you really need to know.

  Barry

 
> On Oct 6, 2015, at 12:23 AM, Justin Chang <jychang48 at gmail.com> wrote:
> 
> Hi everyone,
> 
> I tried running the streams benchmark on the latest petsc version and got this error:
> 
> $ make streams NPMAX=1
> cd src/benchmarks/streams; /usr/bin/gmake  --no-print-directory PETSC_DIR=/users/jychang48/petsc PETSC_ARCH=arch-linux2-c-opt streams
> mpicc -o MPIVersion.o -c -fPIC  -Wall -Wwrite-strings -Wno-strict-aliasing -Wno-unknown-pragmas -g -O   -I/turquoise/users/jychang48/petsc/include -I/turquoise/users/jychang48/petsc/arch-linux2-c-opt/include -I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include/openmpi/opal/mca/hwloc/hwloc132/hwloc/include -I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include -I/usr/projects/hpcsoft/toss2.2/wolf/openmpi/1.6.5-gcc-4.8/include/openmpi    `pwd`/MPIVersion.c
> Reported: 1 (out of 1) daemons - 1 (out of 1) procs
> Number of MPI processes 1 Processor names  wf612.localdomain 
> Triad:        13797.0526   Rate (MB/s) 
> ------------------------------------------------
> Traceback (most recent call last):
>   File "process.py", line 110, in <module>
>     process(len(sys.argv)-1)
>   File "process.py", line 24, in process
>     size = int(fields[0].split()[0])
> ValueError: invalid literal for int() with base 10: 'Reported:'
> gmake[1]: [stream] Error 1 (ignored)
> Traceback (most recent call last):
>   File "process.py", line 110, in <module>
>     process(len(sys.argv)-1)
>   File "process.py", line 24, in process
>     size = int(fields[0].split()[0])
> ValueError: invalid literal for int() with base 10: 'Reported:'
> gmake[1]: [streams] Error 1 (ignored)
> 
> 
> Did something break, or am I missing something?
> 
> Thanks,
> Justin


From ling.zou at inl.gov  Tue Oct  6 15:29:59 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Tue, 6 Oct 2015 14:29:59 -0600
Subject: [petsc-users] A question on finite difference Jacobian
Message-ID: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>

Hi All,

If the non-zero pattern of a finite difference Jacobian needs 20 colors to
color it (20 comes from MatFDColoringView, the non-zero pattern is
pre-determined from mesh connectivity), is it true that PETSc needs 40
functions evaluation to get the full Jacobian matrix filled? This is
because that a perturbation per color needs two function evaluation
according to PETSc manual (ver 3.6, page 123, equations shown in the middle
of the page).
But I only see 20 function evaluations. I probably have some
misunderstanding somewhere. Any suggestions?

Ling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151006/3609f208/attachment.html>

From jed at jedbrown.org  Tue Oct  6 15:33:31 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 06 Oct 2015 14:33:31 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
Message-ID: <87pp0rka90.fsf@jedbrown.org>

"Zou (Non-US), Ling" <ling.zou at inl.gov> writes:

> Hi All,
>
> If the non-zero pattern of a finite difference Jacobian needs 20 colors to
> color it (20 comes from MatFDColoringView, the non-zero pattern is
> pre-determined from mesh connectivity), is it true that PETSc needs 40
> functions evaluation to get the full Jacobian matrix filled? This is
> because that a perturbation per color needs two function evaluation
> according to PETSc manual (ver 3.6, page 123, equations shown in the middle
> of the page).

Note that F(u) has already been evaluated and is the same for all the
perturbation directions.  A centered difference would require two
evaluations per direction, as you say.

> But I only see 20 function evaluations. I probably have some
> misunderstanding somewhere. Any suggestions?
>
> Ling
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151006/48b38ee3/attachment.pgp>

From bsmith at mcs.anl.gov  Tue Oct  6 15:38:21 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 6 Oct 2015 15:38:21 -0500
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
Message-ID: <CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>


> On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> 
> Hi All,
> 
> If the non-zero pattern of a finite difference Jacobian needs 20 colors to color it (20 comes from MatFDColoringView, the non-zero pattern is pre-determined from mesh connectivity), is it true that PETSc needs 40 functions evaluation to get the full Jacobian matrix filled? This is because that a perturbation per color needs two function evaluation according to PETSc manual (ver 3.6, page 123, equations shown in the middle of the page).
> But I only see 20 function evaluations. I probably have some misunderstanding somewhere. Any suggestions?

   PETSc uses forward differencing to compute the derivatives, hence it needs a single function evaluation at the given point (which has almost always been previously computed in Newton's method) and then one function evaluation for each color. This is why it reports 20 function evaluations.

  Barry


> 
> Ling


From ling.zou at inl.gov  Tue Oct  6 15:43:58 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Tue, 6 Oct 2015 14:43:58 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <87pp0rka90.fsf@jedbrown.org>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<87pp0rka90.fsf@jedbrown.org>
Message-ID: <CAJWxZDRmVTd2D9F2PY0HvE6ZNquHUEUN8H7Fn-dGs8FXc2FfWg@mail.gmail.com>

Ah! That's correct. I forgot that the base vector always keeps the same.
Thanks.

Ling

On Tue, Oct 6, 2015 at 2:33 PM, Jed Brown <jed at jedbrown.org> wrote:

> "Zou (Non-US), Ling" <ling.zou at inl.gov> writes:
>
> > Hi All,
> >
> > If the non-zero pattern of a finite difference Jacobian needs 20 colors
> to
> > color it (20 comes from MatFDColoringView, the non-zero pattern is
> > pre-determined from mesh connectivity), is it true that PETSc needs 40
> > functions evaluation to get the full Jacobian matrix filled? This is
> > because that a perturbation per color needs two function evaluation
> > according to PETSc manual (ver 3.6, page 123, equations shown in the
> middle
> > of the page).
>
> Note that F(u) has already been evaluated and is the same for all the
> perturbation directions.  A centered difference would require two
> evaluations per direction, as you say.
>
> > But I only see 20 function evaluations. I probably have some
> > misunderstanding somewhere. Any suggestions?
> >
> > Ling
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151006/4440ac3c/attachment.html>

From ling.zou at inl.gov  Tue Oct  6 16:22:30 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Tue, 6 Oct 2015 15:22:30 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
Message-ID: <CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>

On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> >
> > Hi All,
> >
> > If the non-zero pattern of a finite difference Jacobian needs 20 colors
> to color it (20 comes from MatFDColoringView, the non-zero pattern is
> pre-determined from mesh connectivity), is it true that PETSc needs 40
> functions evaluation to get the full Jacobian matrix filled? This is
> because that a perturbation per color needs two function evaluation
> according to PETSc manual (ver 3.6, page 123, equations shown in the middle
> of the page).
> > But I only see 20 function evaluations. I probably have some
> misunderstanding somewhere. Any suggestions?
>
>    PETSc uses forward differencing to compute the derivatives, hence it
> needs a single function evaluation at the given point (which has almost
> always been previously computed in Newton's method)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Is it a potential problem if the user chooses to use a different (e.g.
simplified) residual function as the function for MatFDColoringSetFunction?



> and then one function evaluation for each color. This is why it reports 20
> function evaluations.
>
>   Barry
>
>
> >
> > Ling
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151006/31239bcd/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct  6 20:09:41 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 6 Oct 2015 20:09:41 -0500
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
	<CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
Message-ID: <24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>


> On Oct 6, 2015, at 4:22 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> 
> 
> 
> On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> >
> > Hi All,
> >
> > If the non-zero pattern of a finite difference Jacobian needs 20 colors to color it (20 comes from MatFDColoringView, the non-zero pattern is pre-determined from mesh connectivity), is it true that PETSc needs 40 functions evaluation to get the full Jacobian matrix filled? This is because that a perturbation per color needs two function evaluation according to PETSc manual (ver 3.6, page 123, equations shown in the middle of the page).
> > But I only see 20 function evaluations. I probably have some misunderstanding somewhere. Any suggestions?
> 
>    PETSc uses forward differencing to compute the derivatives, hence it needs a single function evaluation at the given point (which has almost always been previously computed in Newton's method)
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Is it a potential problem if the user chooses to use a different (e.g. simplified) residual function as the function for MatFDColoringSetFunction?

   Yes, you can do that. But this may result in a "Newton" direction that is not a descent direction hence Newton stalls. If you have 20 colors I doubt that it would be a good idea to use a cheaper function there. If you have several hundred colors then you can use a simpler function PLUS -snes_mf_operator to insure that the Newton direction is correct.


  Barry

> 
>  
> and then one function evaluation for each color. This is why it reports 20 function evaluations.
> 
>   Barry
> 
> 
> >
> > Ling
> 
> 


From ggiuntoli at tecna.com  Wed Oct  7 07:08:43 2015
From: ggiuntoli at tecna.com (Giuntoli Guido Alfredo)
Date: Wed, 7 Oct 2015 12:08:43 +0000
Subject: [petsc-users] working-with-submatrix
Message-ID: <1D3C873DD26F4D4896443D2C76263D5762D42E@ARGTC-VEX03-000.tecna.com>

Hi! I have a problem of the form Ax=kBx were x is a scalar variable that is 0 in the border of a domain (x_D=0).
Using FEM the problem that you have to solve is a reduced problem were you don't have to include the coefficients of the matrix that multiply the x_Ds. So, I have two alternatives here, one is to construct a reduce matrix A and B from the beginning... this could result in a slow code. The other alternative I want to try is to use a sub matrix of A and B and pass them to slepc. How can I do this is I have Mat A and Mat b ?

Thank you, Guido!



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/c979d108/attachment-0001.html>

From knepley at gmail.com  Wed Oct  7 07:17:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 7 Oct 2015 07:17:45 -0500
Subject: [petsc-users] working-with-submatrix
In-Reply-To: <1D3C873DD26F4D4896443D2C76263D5762D42E@ARGTC-VEX03-000.tecna.com>
References: <1D3C873DD26F4D4896443D2C76263D5762D42E@ARGTC-VEX03-000.tecna.com>
Message-ID: <CAMYG4GkdUct3tcRYvOMVT8HKyEWw_MhDz-RhdM6bc44R4pr4NQ@mail.gmail.com>

On Wed, Oct 7, 2015 at 7:08 AM, Giuntoli Guido Alfredo <ggiuntoli at tecna.com>
wrote:

> Hi! I have a problem of the form Ax=kBx were x is a scalar variable that
> is 0 in the border of a domain (x_D=0).
>
> Using FEM the problem that you have to solve is a reduced problem were you
> don?t have to include the coefficients of the matrix that multiply the
> x_Ds. So, I have two alternatives here, one is to construct a reduce matrix
> A and B from the beginning? this could result in a slow code. The other
> alternative I want to try is to use a sub matrix of A and B and pass them
> to slepc. How can I do this is I have Mat A and Mat b ?
>

1) Can't you just use MatZeroRows() or MatZeroRowsColumns() to apply your
BC?

2) I am not sure why you think eliminating variables will be slower. My
experiments and theoretical understanding say it is at least as fast as
zeroing the rows.

3) If you want to extract a submatrix, you can use MatGetSubmatrix().

  Thanks,

     Matt


>
>
> Thank you, Guido!
>
> ------------------------------
> [image: Argentina Oil & Gas Expo] <http://www.aogexpo.com.ar> *?Venga a
> visitarnos!* *en Argentina Oil & Gas Expo*
> 5 ? 8 Octubre, 2015 La Rural Predio Ferial
> Stand 2F-30 - Hall 2
>
> Trabajamos con seguridad y velamos por la protecci?n del Medio Ambiente.
> Antes de Imprimir este mail confirme que sea necesario. Gracias
>
> El presente e-mail y cualquier documento adjunto al mismo, pertenecen al
> emisor y puede contener informaci?n confidencial legalmente protegida. La
> informaci?n contenida en este e-mail es remitida ?nicamente para uso del
> destinatario indicado en el mismo. La divulgaci?n, copia, distribuci?n o
> cualquier otro uso de la informaci?n contenida en el presente e-mail por
> parte de personas distintas al destinatario se encuentra prohibido. Si Ud.
> ha recibido este e-mail por error, rogamos borrar el mismo e informar dicha
> circunstancia por mail a la direcci?n info at tecna.com Muchas gracias.
>
> This electronic mail transmission and any attached documents contain
> information belonging to the sender which may be confidential and legally
> privileged. The information is intended for the use of the named
> recipient(s) only. If you are not the intended recipient(s), any
> disclosure, copying, distribution or action taken regarding the information
> on this email is strictly prohibited. If you have received this email in
> error please notify info at tecna.com and delete the email from your
> computer. Thank you.
> ------------------------------
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/419bd1fd/attachment.html>

From juan at tf.uni-kiel.de  Wed Oct  7 07:27:09 2015
From: juan at tf.uni-kiel.de (Julian Andrej)
Date: Wed, 7 Oct 2015 14:27:09 +0200
Subject: [petsc-users] working-with-submatrix
In-Reply-To: <CAMYG4GkdUct3tcRYvOMVT8HKyEWw_MhDz-RhdM6bc44R4pr4NQ@mail.gmail.com>
References: <1D3C873DD26F4D4896443D2C76263D5762D42E@ARGTC-VEX03-000.tecna.com>
	<CAMYG4GkdUct3tcRYvOMVT8HKyEWw_MhDz-RhdM6bc44R4pr4NQ@mail.gmail.com>
Message-ID: <CABFzUT0OEppQw-NkohR0oKKT2_5BXarC8k7HUBVy4YVRZkSySw@mail.gmail.com>

Especially in eigenproblems, I would suggest using Matthews approach, or
applying Dirichlet boundary conditions via the penalty method (penalized
robin for example). This would avoid spurious eigenvalues which occur when
you use MatZeroRows().

On Wed, Oct 7, 2015 at 2:17 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 7, 2015 at 7:08 AM, Giuntoli Guido Alfredo <
> ggiuntoli at tecna.com> wrote:
>
>> Hi! I have a problem of the form Ax=kBx were x is a scalar variable that
>> is 0 in the border of a domain (x_D=0).
>>
>> Using FEM the problem that you have to solve is a reduced problem were
>> you don?t have to include the coefficients of the matrix that multiply the
>> x_Ds. So, I have two alternatives here, one is to construct a reduce matrix
>> A and B from the beginning? this could result in a slow code. The other
>> alternative I want to try is to use a sub matrix of A and B and pass them
>> to slepc. How can I do this is I have Mat A and Mat b ?
>>
>
> 1) Can't you just use MatZeroRows() or MatZeroRowsColumns() to apply your
> BC?
>
> 2) I am not sure why you think eliminating variables will be slower. My
> experiments and theoretical understanding say it is at least as fast as
> zeroing the rows.
>
> 3) If you want to extract a submatrix, you can use MatGetSubmatrix().
>
>   Thanks,
>
>      Matt
>
>
>>
>>
>> Thank you, Guido!
>>
>> ------------------------------
>> [image: Argentina Oil & Gas Expo] <http://www.aogexpo.com.ar> *?Venga a
>> visitarnos!* *en Argentina Oil & Gas Expo*
>> 5 ? 8 Octubre, 2015 La Rural Predio Ferial
>> Stand 2F-30 - Hall 2
>>
>> Trabajamos con seguridad y velamos por la protecci?n del Medio Ambiente.
>> Antes de Imprimir este mail confirme que sea necesario. Gracias
>>
>> El presente e-mail y cualquier documento adjunto al mismo, pertenecen al
>> emisor y puede contener informaci?n confidencial legalmente protegida. La
>> informaci?n contenida en este e-mail es remitida ?nicamente para uso del
>> destinatario indicado en el mismo. La divulgaci?n, copia, distribuci?n o
>> cualquier otro uso de la informaci?n contenida en el presente e-mail por
>> parte de personas distintas al destinatario se encuentra prohibido. Si Ud.
>> ha recibido este e-mail por error, rogamos borrar el mismo e informar dicha
>> circunstancia por mail a la direcci?n info at tecna.com Muchas gracias.
>>
>> This electronic mail transmission and any attached documents contain
>> information belonging to the sender which may be confidential and legally
>> privileged. The information is intended for the use of the named
>> recipient(s) only. If you are not the intended recipient(s), any
>> disclosure, copying, distribution or action taken regarding the information
>> on this email is strictly prohibited. If you have received this email in
>> error please notify info at tecna.com and delete the email from your
>> computer. Thank you.
>> ------------------------------
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/10af7d14/attachment.html>

From ling.zou at inl.gov  Wed Oct  7 09:54:10 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Wed, 7 Oct 2015 08:54:10 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
	<CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
	<24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>
Message-ID: <CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>

Thank you Barry.

The background I am asking this question is that I want to reduce (or you
can say optimize) the cost of my finite difference Jacobian evaluation,
which is used for preconditioning purpose. The concept is based on my
understanding of the problem I am solving, but I am not sure if it will
work, thus I want to do some test.

Here is the concept, assume that my residual reads,

F(\vec{U}) = F[\vec{U}, g(\vec{U})]


in which, g(\vec{U}) is a quite complicated and thus expensive function
evaluation. This function, however, is not very sensitive to \vec{U}, i.e.,
\partial{g(\vec{U})}/\partial{g(\vec{U})} is not that important.



Normally, a finite difference Jacobian is evaluated as (as discussed in
PETSc manual),


J(\vec{u}) \approx \frac{F(\vec{U}+\epsilon \vec{v}) - F(\vec{U})}
{\epsilon}


In my case, it reads,


J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U}+\epsilon
\vec{v})] - F[(\vec{U}), g(\vec{U})]} {\epsilon}


Because \partial{g(\vec{U})}/\partial{g(\vec{U})} is not important, the
simplification I want to make is, when finite difference Jacobian (as
preconditioner) is evaluated, it can be further simplified as,


J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U})] -
F[(\vec{U}), g(\vec{U})]} {\epsilon}


Thus, the re-evaluation on g(\vec{U}+\epsilon \vec{v}) is removed. It seems
to me that I need some kind of signal from PETSc so I can tell the code not
to update g(\vec{U}). However, I never tested it and I don't know if
anybody did similar things before.


Thanks,


Ling



On Tue, Oct 6, 2015 at 7:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 6, 2015, at 4:22 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> >
> >
> >
> > On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
> wrote:
> > >
> > > Hi All,
> > >
> > > If the non-zero pattern of a finite difference Jacobian needs 20
> colors to color it (20 comes from MatFDColoringView, the non-zero pattern
> is pre-determined from mesh connectivity), is it true that PETSc needs 40
> functions evaluation to get the full Jacobian matrix filled? This is
> because that a perturbation per color needs two function evaluation
> according to PETSc manual (ver 3.6, page 123, equations shown in the middle
> of the page).
> > > But I only see 20 function evaluations. I probably have some
> misunderstanding somewhere. Any suggestions?
> >
> >    PETSc uses forward differencing to compute the derivatives, hence it
> needs a single function evaluation at the given point (which has almost
> always been previously computed in Newton's method)
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Is it a potential problem if the user chooses to use a different (e.g.
> simplified) residual function as the function for MatFDColoringSetFunction?
>
>    Yes, you can do that. But this may result in a "Newton" direction that
> is not a descent direction hence Newton stalls. If you have 20 colors I
> doubt that it would be a good idea to use a cheaper function there. If you
> have several hundred colors then you can use a simpler function PLUS
> -snes_mf_operator to insure that the Newton direction is correct.
>
>
>   Barry
>
> >
> >
> > and then one function evaluation for each color. This is why it reports
> 20 function evaluations.
> >
> >   Barry
> >
> >
> > >
> > > Ling
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/26a2ba8b/attachment-0001.html>

From ling.zou at inl.gov  Wed Oct  7 09:57:53 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Wed, 7 Oct 2015 08:57:53 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
	<CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
	<24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>
	<CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>
Message-ID: <CAJWxZDTffYimcmCi-wyi31MsxPDO857utL+EwO49XU=y=WYLLw@mail.gmail.com>

well, it has the same risk that Newton direction is not good due to the
simplification. However, it worth my trying.

On Wed, Oct 7, 2015 at 8:54 AM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:

> Thank you Barry.
>
> The background I am asking this question is that I want to reduce (or you
> can say optimize) the cost of my finite difference Jacobian evaluation,
> which is used for preconditioning purpose. The concept is based on my
> understanding of the problem I am solving, but I am not sure if it will
> work, thus I want to do some test.
>
> Here is the concept, assume that my residual reads,
>
> F(\vec{U}) = F[\vec{U}, g(\vec{U})]
>
>
> in which, g(\vec{U}) is a quite complicated and thus expensive function
> evaluation. This function, however, is not very sensitive to \vec{U}, i.e.,
> \partial{g(\vec{U})}/\partial{g(\vec{U})} is not that important.
>
>
>
> Normally, a finite difference Jacobian is evaluated as (as discussed in
> PETSc manual),
>
>
> J(\vec{u}) \approx \frac{F(\vec{U}+\epsilon \vec{v}) - F(\vec{U})}
> {\epsilon}
>
>
> In my case, it reads,
>
>
> J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U}+\epsilon
> \vec{v})] - F[(\vec{U}), g(\vec{U})]} {\epsilon}
>
>
> Because \partial{g(\vec{U})}/\partial{g(\vec{U})} is not important, the
> simplification I want to make is, when finite difference Jacobian (as
> preconditioner) is evaluated, it can be further simplified as,
>
>
> J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U})] -
> F[(\vec{U}), g(\vec{U})]} {\epsilon}
>
>
> Thus, the re-evaluation on g(\vec{U}+\epsilon \vec{v}) is removed. It
> seems to me that I need some kind of signal from PETSc so I can tell the
> code not to update g(\vec{U}). However, I never tested it and I don't know
> if anybody did similar things before.
>
>
> Thanks,
>
>
> Ling
>
>
>
> On Tue, Oct 6, 2015 at 7:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> > On Oct 6, 2015, at 4:22 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
>> wrote:
>> >
>> >
>> >
>> > On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> > > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
>> wrote:
>> > >
>> > > Hi All,
>> > >
>> > > If the non-zero pattern of a finite difference Jacobian needs 20
>> colors to color it (20 comes from MatFDColoringView, the non-zero pattern
>> is pre-determined from mesh connectivity), is it true that PETSc needs 40
>> functions evaluation to get the full Jacobian matrix filled? This is
>> because that a perturbation per color needs two function evaluation
>> according to PETSc manual (ver 3.6, page 123, equations shown in the middle
>> of the page).
>> > > But I only see 20 function evaluations. I probably have some
>> misunderstanding somewhere. Any suggestions?
>> >
>> >    PETSc uses forward differencing to compute the derivatives, hence it
>> needs a single function evaluation at the given point (which has almost
>> always been previously computed in Newton's method)
>> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> > Is it a potential problem if the user chooses to use a different (e.g.
>> simplified) residual function as the function for MatFDColoringSetFunction?
>>
>>    Yes, you can do that. But this may result in a "Newton" direction that
>> is not a descent direction hence Newton stalls. If you have 20 colors I
>> doubt that it would be a good idea to use a cheaper function there. If you
>> have several hundred colors then you can use a simpler function PLUS
>> -snes_mf_operator to insure that the Newton direction is correct.
>>
>>
>>   Barry
>>
>> >
>> >
>> > and then one function evaluation for each color. This is why it reports
>> 20 function evaluations.
>> >
>> >   Barry
>> >
>> >
>> > >
>> > > Ling
>> >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/9b50be7b/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct  7 10:30:54 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 7 Oct 2015 10:30:54 -0500
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
	<CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
	<24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>
	<CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>
Message-ID: <77A2AFA2-A8A5-4B48-8094-7E172378BC31@mcs.anl.gov>


> On Oct 7, 2015, at 9:54 AM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> 
> Thank you Barry.
> 
> The background I am asking this question is that I want to reduce (or you can say optimize) the cost of my finite difference Jacobian evaluation, which is used for preconditioning purpose. The concept is based on my understanding of the problem I am solving, but I am not sure if it will work, thus I want to do some test.
> 
> Here is the concept, assume that my residual reads,
> 
> F(\vec{U}) = F[\vec{U}, g(\vec{U})]
> 
> in which, g(\vec{U}) is a quite complicated and thus expensive function evaluation. This function, however, is not very sensitive to \vec{U}, i.e., \partial{g(\vec{U})}/\partial{g(\vec{U})} is not that important.
> 
> 
> Normally, a finite difference Jacobian is evaluated as (as discussed in PETSc manual),
> 
> J(\vec{u}) \approx \frac{F(\vec{U}+\epsilon \vec{v}) - F(\vec{U})} {\epsilon}
> 
> In my case, it reads,
> 
> J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U}+\epsilon \vec{v})] - F[(\vec{U}), g(\vec{U})]} {\epsilon}
> 
> Because \partial{g(\vec{U})}/\partial{g(\vec{U})} is not important, the simplification I want to make is, when finite difference Jacobian (as preconditioner) is evaluated, it can be further simplified as,
> 
> J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U})] - F[(\vec{U}), g(\vec{U})]} {\epsilon}
> 
> Thus, the re-evaluation on g(\vec{U}+\epsilon \vec{v}) is removed. It seems to me that I need some kind of signal from PETSc so I can tell the code not to update g(\vec{U}). However, I never tested it and I don't know if anybody did similar things before.

   This is tricky because you are not simply providing a different function to difference; you are skipping part of the differencing procedure. MatFDColoringSetFunction() is only designed to allow you to change the function, not monkey with removing the differencing value sometimes.

   Of course it is possible for you to do what you want but I think it would require you copying our differencing code and modifying exactly for your needs. For AIJ matrices the function is MatFDColoringApply_AIJ() it is a bit obscure and you would have to figure out what rows correspond to the g(\vec{U}) part so you handle them differently. 

  Barry

> 
> Thanks,
> 
> Ling
> 
> 
> On Tue, Oct 6, 2015 at 7:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Oct 6, 2015, at 4:22 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> >
> >
> >
> > On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> > >
> > > Hi All,
> > >
> > > If the non-zero pattern of a finite difference Jacobian needs 20 colors to color it (20 comes from MatFDColoringView, the non-zero pattern is pre-determined from mesh connectivity), is it true that PETSc needs 40 functions evaluation to get the full Jacobian matrix filled? This is because that a perturbation per color needs two function evaluation according to PETSc manual (ver 3.6, page 123, equations shown in the middle of the page).
> > > But I only see 20 function evaluations. I probably have some misunderstanding somewhere. Any suggestions?
> >
> >    PETSc uses forward differencing to compute the derivatives, hence it needs a single function evaluation at the given point (which has almost always been previously computed in Newton's method)
> > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > Is it a potential problem if the user chooses to use a different (e.g. simplified) residual function as the function for MatFDColoringSetFunction?
> 
>    Yes, you can do that. But this may result in a "Newton" direction that is not a descent direction hence Newton stalls. If you have 20 colors I doubt that it would be a good idea to use a cheaper function there. If you have several hundred colors then you can use a simpler function PLUS -snes_mf_operator to insure that the Newton direction is correct.
> 
> 
>   Barry
> 
> >
> >
> > and then one function evaluation for each color. This is why it reports 20 function evaluations.
> >
> >   Barry
> >
> >
> > >
> > > Ling
> >
> >
> 
> 


From ling.zou at inl.gov  Wed Oct  7 10:38:38 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Wed, 7 Oct 2015 09:38:38 -0600
Subject: [petsc-users] A question on finite difference Jacobian
In-Reply-To: <77A2AFA2-A8A5-4B48-8094-7E172378BC31@mcs.anl.gov>
References: <CAJWxZDSw7r6TX-FvHzKBvy1URrTVMSjdTBfsB24uXz374Wrh2w@mail.gmail.com>
	<CB8287AD-1A95-4A1C-9302-2F8F657EDBA3@mcs.anl.gov>
	<CAJWxZDT1mm_16KRu2OvJyUDbq2nWxhRHs5XJ5AWSo0V52fXWTg@mail.gmail.com>
	<24DD7D79-1DA1-4456-964C-850E82AE62CA@mcs.anl.gov>
	<CAJWxZDSVarOim=9n=2fs6K2FJkFTHi+dgzc7qi_t_JVNFoWwOw@mail.gmail.com>
	<77A2AFA2-A8A5-4B48-8094-7E172378BC31@mcs.anl.gov>
Message-ID: <CAJWxZDT5wJWDp5UvTu9g-E0LY=Z-CsE7U_uT9k1=3XtsGk_xaw@mail.gmail.com>

Thank you Jed.
This seems to be hard :)

Ling

On Wed, Oct 7, 2015 at 9:30 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 7, 2015, at 9:54 AM, Zou (Non-US), Ling <ling.zou at inl.gov> wrote:
> >
> > Thank you Barry.
> >
> > The background I am asking this question is that I want to reduce (or
> you can say optimize) the cost of my finite difference Jacobian evaluation,
> which is used for preconditioning purpose. The concept is based on my
> understanding of the problem I am solving, but I am not sure if it will
> work, thus I want to do some test.
> >
> > Here is the concept, assume that my residual reads,
> >
> > F(\vec{U}) = F[\vec{U}, g(\vec{U})]
> >
> > in which, g(\vec{U}) is a quite complicated and thus expensive function
> evaluation. This function, however, is not very sensitive to \vec{U}, i.e.,
> \partial{g(\vec{U})}/\partial{g(\vec{U})} is not that important.
> >
> >
> > Normally, a finite difference Jacobian is evaluated as (as discussed in
> PETSc manual),
> >
> > J(\vec{u}) \approx \frac{F(\vec{U}+\epsilon \vec{v}) - F(\vec{U})}
> {\epsilon}
> >
> > In my case, it reads,
> >
> > J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}),
> g(\vec{U}+\epsilon \vec{v})] - F[(\vec{U}), g(\vec{U})]} {\epsilon}
> >
> > Because \partial{g(\vec{U})}/\partial{g(\vec{U})} is not important, the
> simplification I want to make is, when finite difference Jacobian (as
> preconditioner) is evaluated, it can be further simplified as,
> >
> > J(\vec{u}) \approx \frac{F[(\vec{U}+\epsilon \vec{v}), g(\vec{U})] -
> F[(\vec{U}), g(\vec{U})]} {\epsilon}
> >
> > Thus, the re-evaluation on g(\vec{U}+\epsilon \vec{v}) is removed. It
> seems to me that I need some kind of signal from PETSc so I can tell the
> code not to update g(\vec{U}). However, I never tested it and I don't know
> if anybody did similar things before.
>
>    This is tricky because you are not simply providing a different
> function to difference; you are skipping part of the differencing
> procedure. MatFDColoringSetFunction() is only designed to allow you to
> change the function, not monkey with removing the differencing value
> sometimes.
>
>    Of course it is possible for you to do what you want but I think it
> would require you copying our differencing code and modifying exactly for
> your needs. For AIJ matrices the function is MatFDColoringApply_AIJ() it is
> a bit obscure and you would have to figure out what rows correspond to the
> g(\vec{U}) part so you handle them differently.
>
>   Barry
>
> >
> > Thanks,
> >
> > Ling
> >
> >
> > On Tue, Oct 6, 2015 at 7:09 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Oct 6, 2015, at 4:22 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
> wrote:
> > >
> > >
> > >
> > > On Tue, Oct 6, 2015 at 2:38 PM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > > > On Oct 6, 2015, at 3:29 PM, Zou (Non-US), Ling <ling.zou at inl.gov>
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > If the non-zero pattern of a finite difference Jacobian needs 20
> colors to color it (20 comes from MatFDColoringView, the non-zero pattern
> is pre-determined from mesh connectivity), is it true that PETSc needs 40
> functions evaluation to get the full Jacobian matrix filled? This is
> because that a perturbation per color needs two function evaluation
> according to PETSc manual (ver 3.6, page 123, equations shown in the middle
> of the page).
> > > > But I only see 20 function evaluations. I probably have some
> misunderstanding somewhere. Any suggestions?
> > >
> > >    PETSc uses forward differencing to compute the derivatives, hence
> it needs a single function evaluation at the given point (which has almost
> always been previously computed in Newton's method)
> > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> > > Is it a potential problem if the user chooses to use a different (e.g.
> simplified) residual function as the function for MatFDColoringSetFunction?
> >
> >    Yes, you can do that. But this may result in a "Newton" direction
> that is not a descent direction hence Newton stalls. If you have 20 colors
> I doubt that it would be a good idea to use a cheaper function there. If
> you have several hundred colors then you can use a simpler function PLUS
> -snes_mf_operator to insure that the Newton direction is correct.
> >
> >
> >   Barry
> >
> > >
> > >
> > > and then one function evaluation for each color. This is why it
> reports 20 function evaluations.
> > >
> > >   Barry
> > >
> > >
> > > >
> > > > Ling
> > >
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/b56609d4/attachment.html>

From timothee.nicolas at gmail.com  Wed Oct  7 13:19:33 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 7 Oct 2015 20:19:33 +0200
Subject: [petsc-users] MatShellGetContext in Fortran
Message-ID: <CAGi1ndSG0Tu-E7eb=zZhq4dWywYoaiBW-BYzd6R2-xpYe9JKUg@mail.gmail.com>

Hi all,

In the manual, it says, about MatShellSetContext and MatShellGetContext,
that in FORTRAN we cannot set a context other than integer or PetscObject.

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateShell.html#MatCreateShell

In my understanding, it means that I would be allowed to do, say

PetscInt :: ione
ione=1
call MatCreateShell(PETSC_COMM_WORLD,m,m,N,N,PETSC_NULL_INTEGER,A,ierr)
call MatShellSetContext(A,ione,ierr)
call MatShellSetOperation(A,MATOP_MULT,MyMult,ierr)

in the routine where I define the matrix A, and then recover the integer in
the routine MyMult with

PetscInt :: intval
call MatShellGetContext(A,intval,ierr)

This would allow me to circumvent the interdiction to pass a real context,
by e.g. using a module at the beginning of MyMult, containing contexts and
discriminating which context to use with the value of the integer (I want
to use Multigrid, so I cannot have a different routine for each level
obviously). However, this approach did not work, the value in intval stays
undefined. How could I solve this ?

Best

Timothee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/acd63d9d/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct  7 13:50:45 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 7 Oct 2015 13:50:45 -0500
Subject: [petsc-users] MatShellGetContext in Fortran
In-Reply-To: <CAGi1ndSG0Tu-E7eb=zZhq4dWywYoaiBW-BYzd6R2-xpYe9JKUg@mail.gmail.com>
References: <CAGi1ndSG0Tu-E7eb=zZhq4dWywYoaiBW-BYzd6R2-xpYe9JKUg@mail.gmail.com>
Message-ID: <B14390C3-14CE-4671-9454-9781626CD771@mcs.anl.gov>


  Timothee,

   That information in the manual is not really accurate. 

   Here is how you should do it. Take a look at  src/snes/examples/tutorials/ex5f90.F and see the definition of the userctx data type and then further down the interface
definitions for SNESSetApplicationContext() and SNESGetApplicationContext(). You need to mimic that same structure in your code but with MatShellSet/GetContext() instead.
In this way you can do things properly and not need to deal with integer keys into data etc. Note that each level can have its own userctx data.

  Barry



> On Oct 7, 2015, at 1:19 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Hi all,
> 
> In the manual, it says, about MatShellSetContext and MatShellGetContext, that in FORTRAN we cannot set a context other than integer or PetscObject. 
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateShell.html#MatCreateShell
> 
> In my understanding, it means that I would be allowed to do, say
> 
> PetscInt :: ione
> ione=1
> call MatCreateShell(PETSC_COMM_WORLD,m,m,N,N,PETSC_NULL_INTEGER,A,ierr)
> call MatShellSetContext(A,ione,ierr)
> call MatShellSetOperation(A,MATOP_MULT,MyMult,ierr)
> 
> in the routine where I define the matrix A, and then recover the integer in the routine MyMult with
> 
> PetscInt :: intval
> call MatShellGetContext(A,intval,ierr)
> 
> This would allow me to circumvent the interdiction to pass a real context, by e.g. using a module at the beginning of MyMult, containing contexts and discriminating which context to use with the value of the integer (I want to use Multigrid, so I cannot have a different routine for each level obviously). However, this approach did not work, the value in intval stays undefined. How could I solve this ?
> 
> Best
> 
> Timothee


From brbfernandes at utexas.edu  Wed Oct  7 14:13:29 2015
From: brbfernandes at utexas.edu (Bruno Ramon Batista Fernande)
Date: Wed, 7 Oct 2015 14:13:29 -0500
Subject: [petsc-users] Speeding-up petsc
Message-ID: <CAEpUo3it58WQ1c3k5abRmFxyvTjireoDv5doj7B+9TtqWP+a6Q@mail.gmail.com>

Hello guys,

I am using petsc is a solver for a finite difference PDE solver.

Basically the coefficient matrix (say A in Ax = b) is well conditioned,
symmetric, and up to 7 diagonals. Also, no pivoting is necessary for such
problem and it is always diagonally dominant.

I am using all default options of petsc but I still feel like I could get
better results by choosing different answers. By default I mean: GMRES
solver preconditioned on left with ILU(0) and a classical Gram-Schmidt to
orthogonalize against krylov vector.

I am using MAT_STRUCTURALLY_SYMMETRIC option and
atol = 1.0e-30
dtol = 1.0e5

You guys have any idea of how to further improve performance of the solver?

Regards,
Bruno
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/d646d56a/attachment.html>

From jed at jedbrown.org  Wed Oct  7 14:25:52 2015
From: jed at jedbrown.org (Jed Brown)
Date: Wed, 07 Oct 2015 13:25:52 -0600
Subject: [petsc-users] Speeding-up petsc
In-Reply-To: <CAEpUo3it58WQ1c3k5abRmFxyvTjireoDv5doj7B+9TtqWP+a6Q@mail.gmail.com>
References: <CAEpUo3it58WQ1c3k5abRmFxyvTjireoDv5doj7B+9TtqWP+a6Q@mail.gmail.com>
Message-ID: <87si5miipr.fsf@jedbrown.org>

Bruno Ramon Batista Fernande <brbfernandes at utexas.edu> writes:

> Hello guys,
>
> I am using petsc is a solver for a finite difference PDE solver.
>
> Basically the coefficient matrix (say A in Ax = b) is well conditioned,
> symmetric, and up to 7 diagonals. Also, no pivoting is necessary for such
> problem and it is always diagonally dominant.
>
> I am using all default options of petsc but I still feel like I could get
> better results by choosing different answers. By default I mean: GMRES
> solver preconditioned on left with ILU(0) and a classical Gram-Schmidt to
> orthogonalize against krylov vector.

You have told us nothing about the performance of this solver.  Always
include -log_summary -ksp_view.

> I am using MAT_STRUCTURALLY_SYMMETRIC option and
> atol = 1.0e-30
> dtol = 1.0e5

Why these options?

How accurate of a solution do you need?
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/1dae2f67/attachment.pgp>

From timothee.nicolas at gmail.com  Wed Oct  7 14:28:13 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 7 Oct 2015 21:28:13 +0200
Subject: [petsc-users] MatShellGetContext in Fortran
In-Reply-To: <B14390C3-14CE-4671-9454-9781626CD771@mcs.anl.gov>
References: <CAGi1ndSG0Tu-E7eb=zZhq4dWywYoaiBW-BYzd6R2-xpYe9JKUg@mail.gmail.com>
	<B14390C3-14CE-4671-9454-9781626CD771@mcs.anl.gov>
Message-ID: <CAGi1ndT2-Kh07Sd8Eqh6-ZP0Pi_Msc35qC1_9yQ1QYide0EiSQ@mail.gmail.com>

Thank you very much Barry. It is working smoothly. Apparently it is very
important that for the Get context routine, the context be declared as a
pointer, and consistently, also in the routine where it is called (as in
ex5f90.F indeed).

Best

Timothee

2015-10-07 20:50 GMT+02:00 Barry Smith <bsmith at mcs.anl.gov>:

>
>   Timothee,
>
>    That information in the manual is not really accurate.
>
>    Here is how you should do it. Take a look at
> src/snes/examples/tutorials/ex5f90.F and see the definition of the userctx
> data type and then further down the interface
> definitions for SNESSetApplicationContext() and
> SNESGetApplicationContext(). You need to mimic that same structure in your
> code but with MatShellSet/GetContext() instead.
> In this way you can do things properly and not need to deal with integer
> keys into data etc. Note that each level can have its own userctx data.
>
>   Barry
>
>
>
> > On Oct 7, 2015, at 1:19 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > In the manual, it says, about MatShellSetContext and MatShellGetContext,
> that in FORTRAN we cannot set a context other than integer or PetscObject.
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreateShell.html#MatCreateShell
> >
> > In my understanding, it means that I would be allowed to do, say
> >
> > PetscInt :: ione
> > ione=1
> > call MatCreateShell(PETSC_COMM_WORLD,m,m,N,N,PETSC_NULL_INTEGER,A,ierr)
> > call MatShellSetContext(A,ione,ierr)
> > call MatShellSetOperation(A,MATOP_MULT,MyMult,ierr)
> >
> > in the routine where I define the matrix A, and then recover the integer
> in the routine MyMult with
> >
> > PetscInt :: intval
> > call MatShellGetContext(A,intval,ierr)
> >
> > This would allow me to circumvent the interdiction to pass a real
> context, by e.g. using a module at the beginning of MyMult, containing
> contexts and discriminating which context to use with the value of the
> integer (I want to use Multigrid, so I cannot have a different routine for
> each level obviously). However, this approach did not work, the value in
> intval stays undefined. How could I solve this ?
> >
> > Best
> >
> > Timothee
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/3248b453/attachment.html>

From vencels at kth.se  Wed Oct  7 15:30:31 2015
From: vencels at kth.se (Juris Vencels)
Date: Wed, 7 Oct 2015 14:30:31 -0600
Subject: [petsc-users] Coloring modifies user defined Jacobian at the first
	time step
Message-ID: <561580E7.1020004@kth.se>

Hello,

I am using such options:
    -ksp_type gmres -snes_monitor_short -snes_mf_operator -info

I construct Jacobian and pass it to SNES
SNESSetJacobian(snes,Jac,Jac,PETSC_NULL_FUNCTION,PETSC_NULL_OBJECT,ierr)

Jacobian is not changing during simulation.

At the first time step I get the following message
    [0] ISColoringCreate(): Number of colors 3
    [0] PetscCommDuplicate(): Duplicating a communicator 1 3 max tags = 
100000000
    [0] PetscCommDuplicate(): Using internal PETSc communicator 1 3
    [0] VecScatterCreate(): Special case: sequential vector general to 
stride
    [0] MatFDColoringSetUp_SeqXAIJ(): ncolors 3, brows 48 and bcols 1 
are used.

When I print Jacobian matrix after the first time step, it is different 
from one that I set at the beginning of simulation. Any ideas why 
coloring (which I am not using) changes user defined Jacobian?

Petsc Release Version 3.6.1 from Git
Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran 
--download-fblaslapack --with-fortran-kernels=generic 
--with-scalar-type=complex --with-debugging=0 COPTFLAGS="-O3 
-march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native 
-mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-mpi=0 
--with-openmp --with-threadcomm --with-pthreadclasses --with-openmpclasses

Thanks!

From bsmith at mcs.anl.gov  Wed Oct  7 15:44:00 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 7 Oct 2015 15:44:00 -0500
Subject: [petsc-users] Coloring modifies user defined Jacobian at the
	first time step
In-Reply-To: <561580E7.1020004@kth.se>
References: <561580E7.1020004@kth.se>
Message-ID: <78C68493-1D23-4ED3-AEAD-8B5696B91418@mcs.anl.gov>


   The defaults have become rather convoluted and it looks like since you provided a matrix but not a function it is computing the jacobin for you using coloring.
Create a dummy computejacobian function that does nothing but returns and pass that in to snessetjacobian.

   Barry

> On Oct 7, 2015, at 3:30 PM, Juris Vencels <vencels at kth.se> wrote:
> 
> Hello,
> 
> I am using such options:
>   -ksp_type gmres -snes_monitor_short -snes_mf_operator -info
> 
> I construct Jacobian and pass it to SNES
> SNESSetJacobian(snes,Jac,Jac,PETSC_NULL_FUNCTION,PETSC_NULL_OBJECT,ierr)
> 
> Jacobian is not changing during simulation.
> 
> At the first time step I get the following message
>   [0] ISColoringCreate(): Number of colors 3
>   [0] PetscCommDuplicate(): Duplicating a communicator 1 3 max tags = 100000000
>   [0] PetscCommDuplicate(): Using internal PETSc communicator 1 3
>   [0] VecScatterCreate(): Special case: sequential vector general to stride
>   [0] MatFDColoringSetUp_SeqXAIJ(): ncolors 3, brows 48 and bcols 1 are used.
> 
> When I print Jacobian matrix after the first time step, it is different from one that I set at the beginning of simulation. Any ideas why coloring (which I am not using) changes user defined Jacobian?
> 
> Petsc Release Version 3.6.1 from Git
> Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --with-fortran-kernels=generic --with-scalar-type=complex --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-mpi=0 --with-openmp --with-threadcomm --with-pthreadclasses --with-openmpclasses
> 
> Thanks!


From vencels at kth.se  Wed Oct  7 16:13:38 2015
From: vencels at kth.se (Juris Vencels)
Date: Wed, 7 Oct 2015 15:13:38 -0600
Subject: [petsc-users] Coloring modifies user defined Jacobian at the
 first time step
In-Reply-To: <78C68493-1D23-4ED3-AEAD-8B5696B91418@mcs.anl.gov>
References: <561580E7.1020004@kth.se>
	<78C68493-1D23-4ED3-AEAD-8B5696B91418@mcs.anl.gov>
Message-ID: <56158B02.1080200@kth.se>

Works. Thank you!

On 10/07/2015 02:44 PM, Barry Smith wrote:
>     The defaults have become rather convoluted and it looks like since you provided a matrix but not a function it is computing the jacobin for you using coloring.
> Create a dummy computejacobian function that does nothing but returns and pass that in to snessetjacobian.
>
>     Barry
>
>> On Oct 7, 2015, at 3:30 PM, Juris Vencels <vencels at kth.se> wrote:
>>
>> Hello,
>>
>> I am using such options:
>>    -ksp_type gmres -snes_monitor_short -snes_mf_operator -info
>>
>> I construct Jacobian and pass it to SNES
>> SNESSetJacobian(snes,Jac,Jac,PETSC_NULL_FUNCTION,PETSC_NULL_OBJECT,ierr)
>>
>> Jacobian is not changing during simulation.
>>
>> At the first time step I get the following message
>>    [0] ISColoringCreate(): Number of colors 3
>>    [0] PetscCommDuplicate(): Duplicating a communicator 1 3 max tags = 100000000
>>    [0] PetscCommDuplicate(): Using internal PETSc communicator 1 3
>>    [0] VecScatterCreate(): Special case: sequential vector general to stride
>>    [0] MatFDColoringSetUp_SeqXAIJ(): ncolors 3, brows 48 and bcols 1 are used.
>>
>> When I print Jacobian matrix after the first time step, it is different from one that I set at the beginning of simulation. Any ideas why coloring (which I am not using) changes user defined Jacobian?
>>
>> Petsc Release Version 3.6.1 from Git
>> Configure options --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-fblaslapack --with-fortran-kernels=generic --with-scalar-type=complex --with-debugging=0 COPTFLAGS="-O3 -march=native -mtune=native" CXXOPTFLAGS="-O3 -march=native -mtune=native" FOPTFLAGS="-O3 -march=native -mtune=native" --with-mpi=0 --with-openmp --with-threadcomm --with-pthreadclasses --with-openmpclasses
>>
>> Thanks!


From William.Coirier at kratosdefense.com  Wed Oct  7 17:12:13 2015
From: William.Coirier at kratosdefense.com (William Coirier)
Date: Wed, 7 Oct 2015 22:12:13 +0000
Subject: [petsc-users] SNESSetConvergenceTest usage?
Message-ID: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A20@SD-MB02.wfinet.com>

Without going into too many details regarding the application, here is a problem I've encountered and wonder if anyone has any suggestions.

I have supplied a convergence test function for SNES using the SNESSetConvergenceTest API. My convergence test function sets the SNESConvergedReason variable referenced in the argument list to a handful of SNESConvergedReasons (and does other stuff, but that is not relevant...)

In my application code, after I call SNESSolve, I call SNESGetConvergedReason, and for some reason I will occasionally see SNESConvergedReasons that are NOT in the subset of those I supply with my convergence test function.

It's as if my test function setting of the converged reason is either being overwritten, or ignored.

As an example, I'm using the newtonls (default) scheme, and (an inexact, but workable) Jacobian. My test function only will set SNESConvergedReason to SNES_CONVERGED_ITERATING, SNES_CONVERGED_SNORM_RELATIVE or SNES_DIVERGED_MAX_IT. But, I see DIVERGED_LINE_SEARCH from the call to SNESGetConvergedReason.

Is the line search diverging (most likely?!) and blowing by my convergence test function?? Maybe (most likely?) I'm doing something stupid...

Otherwise things are working great. PETSc is awesome!

Any help/suggestions/input is appreciated.


-----------------------------------------------------------------------
William J. Coirier, Ph.D.
Director, Aerosciences and Engineering Analysis
Advanced Technology Division
Kratos/Digital Fusion, Inc.
4904 Research Drive
Huntsville, AL 35805
256-327-8170
256-327-8120 (fax)


 --------------------------------------------------------------------------------------------------------------------

***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/eba1a226/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct  7 17:21:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 7 Oct 2015 17:21:46 -0500
Subject: [petsc-users] SNESSetConvergenceTest usage?
In-Reply-To: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A20@SD-MB02.wfinet.com>
References: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A20@SD-MB02.wfinet.com>
Message-ID: <AA3C49B0-30CB-4011-BF7A-5EF96C1377D7@mcs.anl.gov>


> On Oct 7, 2015, at 5:12 PM, William Coirier <William.Coirier at kratosdefense.com> wrote:
> 
> Without going into too many details regarding the application, here is a problem I?ve encountered and wonder if anyone has any suggestions.
>  
> I have supplied a convergence test function for SNES using the SNESSetConvergenceTest API. My convergence test function sets the SNESConvergedReason variable referenced in the argument list to a handful of SNESConvergedReasons (and does other stuff, but that is not relevant?)
>  
> In my application code, after I call SNESSolve, I call SNESGetConvergedReason, and for some reason I will occasionally see SNESConvergedReasons that are NOT in the subset of those I supply with my convergence test function.
>  
> It?s as if my test function setting of the converged reason is either being overwritten, or ignored.
>  
> As an example, I?m using the newtonls (default) scheme, and (an inexact, but workable) Jacobian. My test function only will set SNESConvergedReason to SNES_CONVERGED_ITERATING, SNES_CONVERGED_SNORM_RELATIVE or SNES_DIVERGED_MAX_IT. But, I see DIVERGED_LINE_SEARCH from the call to SNESGetConvergedReason.

   The line search routines themselves can set the converged reason, when the line search fails, this happens before it has the next solution so before the next time your line search tester is called. So your value is not been overwritten or ignored. 

   I am not sure what you want to do about this? It is just the way it is.

  Barry

>  
> Is the line search diverging (most likely?!) and blowing by my convergence test function?? Maybe (most likely?) I?m doing something stupid?
>  
> Otherwise things are working great. PETSc is awesome!
>  
> Any help/suggestions/input is appreciated.
>  
>  
> -----------------------------------------------------------------------
> William J. Coirier, Ph.D.
> Director, Aerosciences and Engineering Analysis
> Advanced Technology Division
> Kratos/Digital Fusion, Inc.
> 4904 Research Drive
> Huntsville, AL 35805
> 256-327-8170
> 256-327-8120 (fax)
>  
> 
> --------------------------------------------------------------------------------------------------------------------
> 
> ***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.


From William.Coirier at kratosdefense.com  Wed Oct  7 17:28:10 2015
From: William.Coirier at kratosdefense.com (William Coirier)
Date: Wed, 7 Oct 2015 22:28:10 +0000
Subject: [petsc-users] SNESSetConvergenceTest usage?
In-Reply-To: <AA3C49B0-30CB-4011-BF7A-5EF96C1377D7@mcs.anl.gov>
References: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A20@SD-MB02.wfinet.com>
	<AA3C49B0-30CB-4011-BF7A-5EF96C1377D7@mcs.anl.gov>
Message-ID: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A46@SD-MB02.wfinet.com>

Thank you Barry. This explains the behavior.

Maybe I should try a different search strategy? Perhaps newtontr?

Thanks for your help.

-----------------------------------------------------------------------
William J. Coirier, Ph.D.
Director, Aerosciences and Engineering Analysis
Advanced Technology Division
Kratos/Digital Fusion, Inc.
4904 Research Drive
Huntsville, AL 35805
256-327-8170
256-327-8120 (fax)


-----Original Message-----
From: Barry Smith [mailto:bsmith at mcs.anl.gov] 
Sent: Wednesday, October 07, 2015 5:22 PM
To: William Coirier
Cc: petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] SNESSetConvergenceTest usage?


> On Oct 7, 2015, at 5:12 PM, William Coirier <William.Coirier at kratosdefense.com> wrote:
> 
> Without going into too many details regarding the application, here is a problem I?ve encountered and wonder if anyone has any suggestions.
>  
> I have supplied a convergence test function for SNES using the 
> SNESSetConvergenceTest API. My convergence test function sets the 
> SNESConvergedReason variable referenced in the argument list to a 
> handful of SNESConvergedReasons (and does other stuff, but that is not 
> relevant?)
>  
> In my application code, after I call SNESSolve, I call SNESGetConvergedReason, and for some reason I will occasionally see SNESConvergedReasons that are NOT in the subset of those I supply with my convergence test function.
>  
> It?s as if my test function setting of the converged reason is either being overwritten, or ignored.
>  
> As an example, I?m using the newtonls (default) scheme, and (an inexact, but workable) Jacobian. My test function only will set SNESConvergedReason to SNES_CONVERGED_ITERATING, SNES_CONVERGED_SNORM_RELATIVE or SNES_DIVERGED_MAX_IT. But, I see DIVERGED_LINE_SEARCH from the call to SNESGetConvergedReason.

   The line search routines themselves can set the converged reason, when the line search fails, this happens before it has the next solution so before the next time your line search tester is called. So your value is not been overwritten or ignored. 

   I am not sure what you want to do about this? It is just the way it is.

  Barry

>  
> Is the line search diverging (most likely?!) and blowing by my 
> convergence test function?? Maybe (most likely?) I?m doing something 
> stupid?
>  
> Otherwise things are working great. PETSc is awesome!
>  
> Any help/suggestions/input is appreciated.
>  
>  
> ----------------------------------------------------------------------
> -
> William J. Coirier, Ph.D.
> Director, Aerosciences and Engineering Analysis Advanced Technology 
> Division Kratos/Digital Fusion, Inc.
> 4904 Research Drive
> Huntsville, AL 35805
> 256-327-8170
> 256-327-8120 (fax)
>  
> 
> ----------------------------------------------------------------------
> ----------------------------------------------
> 
> ***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.


 --------------------------------------------------------------------------------------------------------------------

***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.

From bsmith at mcs.anl.gov  Wed Oct  7 17:35:29 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 7 Oct 2015 17:35:29 -0500
Subject: [petsc-users] SNESSetConvergenceTest usage?
In-Reply-To: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A46@SD-MB02.wfinet.com>
References: <E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A20@SD-MB02.wfinet.com>
	<AA3C49B0-30CB-4011-BF7A-5EF96C1377D7@mcs.anl.gov>
	<E0E0BBE7E0725D4A94F11EF5C7BCB3EF3F7F2A46@SD-MB02.wfinet.com>
Message-ID: <E814701C-B9A0-4A0C-A540-68BA69A98C42@mcs.anl.gov>


> On Oct 7, 2015, at 5:28 PM, William Coirier <William.Coirier at kratosdefense.com> wrote:
> 
> Thank you Barry. This explains the behavior.
> 
> Maybe I should try a different search strategy? Perhaps newtontr?

   You can. But a failed line search is usually an indication of a bad Jacobian or not an accurate enough linear solve. http://www.mcs.anl.gov/petsc/documentation/faq.html#newton has many suggestions on how to debug failed convergence of Newton

  Barry

> 
> Thanks for your help.
> 
> -----------------------------------------------------------------------
> William J. Coirier, Ph.D.
> Director, Aerosciences and Engineering Analysis
> Advanced Technology Division
> Kratos/Digital Fusion, Inc.
> 4904 Research Drive
> Huntsville, AL 35805
> 256-327-8170
> 256-327-8120 (fax)
> 
> 
> -----Original Message-----
> From: Barry Smith [mailto:bsmith at mcs.anl.gov] 
> Sent: Wednesday, October 07, 2015 5:22 PM
> To: William Coirier
> Cc: petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] SNESSetConvergenceTest usage?
> 
> 
>> On Oct 7, 2015, at 5:12 PM, William Coirier <William.Coirier at kratosdefense.com> wrote:
>> 
>> Without going into too many details regarding the application, here is a problem I?ve encountered and wonder if anyone has any suggestions.
>> 
>> I have supplied a convergence test function for SNES using the 
>> SNESSetConvergenceTest API. My convergence test function sets the 
>> SNESConvergedReason variable referenced in the argument list to a 
>> handful of SNESConvergedReasons (and does other stuff, but that is not 
>> relevant?)
>> 
>> In my application code, after I call SNESSolve, I call SNESGetConvergedReason, and for some reason I will occasionally see SNESConvergedReasons that are NOT in the subset of those I supply with my convergence test function.
>> 
>> It?s as if my test function setting of the converged reason is either being overwritten, or ignored.
>> 
>> As an example, I?m using the newtonls (default) scheme, and (an inexact, but workable) Jacobian. My test function only will set SNESConvergedReason to SNES_CONVERGED_ITERATING, SNES_CONVERGED_SNORM_RELATIVE or SNES_DIVERGED_MAX_IT. But, I see DIVERGED_LINE_SEARCH from the call to SNESGetConvergedReason.
> 
>   The line search routines themselves can set the converged reason, when the line search fails, this happens before it has the next solution so before the next time your line search tester is called. So your value is not been overwritten or ignored. 
> 
>   I am not sure what you want to do about this? It is just the way it is.
> 
>  Barry
> 
>> 
>> Is the line search diverging (most likely?!) and blowing by my 
>> convergence test function?? Maybe (most likely?) I?m doing something 
>> stupid?
>> 
>> Otherwise things are working great. PETSc is awesome!
>> 
>> Any help/suggestions/input is appreciated.
>> 
>> 
>> ----------------------------------------------------------------------
>> -
>> William J. Coirier, Ph.D.
>> Director, Aerosciences and Engineering Analysis Advanced Technology 
>> Division Kratos/Digital Fusion, Inc.
>> 4904 Research Drive
>> Huntsville, AL 35805
>> 256-327-8170
>> 256-327-8120 (fax)
>> 
>> 
>> ----------------------------------------------------------------------
>> ----------------------------------------------
>> 
>> ***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.
> 
> 
> --------------------------------------------------------------------------------------------------------------------
> 
> ***NOTICE*** This e-mail and/or the attached documents may contain technical data within the definition of the International Traffic in Arms Regulations and/or Export Administration Regulations, and are subject to the export control laws of the U.S. Government. Transfer of this data by any means to a foreign person, whether in the United States or abroad, without an export license or other approval from the U.S. Department of State or Commerce, as applicable, is prohibited. No portion of this e-mail or its attachment(s) may be reproduced without written consent of Kratos Defense & Security Solutions, Inc. Any views expressed in this message are those of the individual sender, except where the message states otherwise and the sender is authorized to state them to be the views of any such entity. The information contained in this message and or attachments is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. If you are not the intended recipient or believe that you may have received this document in error, please notify the sender and delete this e-mail and any attachments immediately.


From borisbou at buffalo.edu  Wed Oct  7 18:04:14 2015
From: borisbou at buffalo.edu (Boris)
Date: Wed, 7 Oct 2015 19:04:14 -0400
Subject: [petsc-users] petsc error RE pc_mg_levels usage
Message-ID: <5615A4EE.8070801@buffalo.edu>

Hello,

I am working on a toy Laplace problem (p1.C attached) investigating some 
multigrid projection matrices when using finite differences. Things seem 
to work as expected when running with :

./p1 -da_grid_x 21 -da_grid_y 21 -mat_view -pc_type mg -pc_mg_levels 1

But if I change pc_mg_levels to 2 or 3, I generate the error below.

Any insights as to what I could be doing wrong would be much appreciated.

Thanks,
Boris


[0]PETSC ERROR: --------------------- Error Message 
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Local index 169 too large 168 (max) at 4
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html 
for trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
[0]PETSC ERROR: ./p1 on a gcc-4.8.4-mpich-3.1.4-openblas-0.2.14-opt 
named bender.eng.buffalo.edu by borisbou Wed Oct  7 18:06:35 2015
[0]PETSC ERROR: Configure options 
--prefix=/bender1/data/shared/software/libs/petsc/3.5.4/gcc/4.8.4/mpich/3.1.4/openblas/0.2.14/opt 
--with-debugging=false --COPTFLAGS="-O3 -mavx" --CXXOPTFLAGS="-O3 -mavx" 
--FOPTFLAGS=-O3 --with-shared-libraries=1 
--with-mpi-dir=/bender1/data/shared/software/libs/mpich/3.1.4/gcc/4.8.4 
--with-mumps=true --download-mumps=1 --with-metis=true 
--download-metis=1 --with-parmetis=true --download-parmetis=1 
--with-superlu=true --download-superlu=1 --with-superludir=true 
--download-superlu_dist=1 --with-blacs=true --download-blacs=1 
--with-scalapack=true --download-scalapack=1 --with-hypre=true 
--download-hypre=1 
--with-blas-lib="[/bender1/data/shared/software/libs/openblas/0.2.14/gcc/4.8.4/lib/libopenblas.so]" 
--with-lapack-lib="[/bender1/data/shared/software/libs/openblas/0.2.14/gcc/4.8.4/lib/libopenblas.so]"
[0]PETSC ERROR: #1 ISLocalToGlobalMappingApply() line 401 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/vec/is/utils/isltog.c
[0]PETSC ERROR: #2 MatSetValuesLocal() line 1982 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 MatSetValuesStencil() line 1383 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 FormJacobian() line 366 in p1.C
[0]PETSC ERROR: #5 SNESComputeJacobian() line 2193 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
[0]PETSC ERROR: #6 KSPComputeOperators_SNES() line 527 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
[0]PETSC ERROR: #7 KSPSetUp() line 256 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #8 PCSetUp_MG() line 803 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/pc/impls/mg/mg.c
[0]PETSC ERROR: #9 PCSetUp() line 902 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/pc/interface/precon.c
[0]PETSC ERROR: #10 KSPSetUp() line 306 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #11 KSPSolve() line 418 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #12 SNESSolve_NEWTONLS() line 232 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #13 SNESSolve() line 3743 in 
/bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
[0]PETSC ERROR: #14 main() line 147 in p1.C
[0]PETSC ERROR: ----------------End of Error Message -------send entire 
error message to petsc-maint at mcs.anl.gov----------
application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
-------------- next part --------------
A non-text attachment was scrubbed...
Name: p1.C
Type: text/x-c++src
Size: 14429 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/3ab85d3a/attachment.c>

From knepley at gmail.com  Wed Oct  7 19:20:47 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 7 Oct 2015 19:20:47 -0500
Subject: [petsc-users] petsc error RE pc_mg_levels usage
In-Reply-To: <5615A4EE.8070801@buffalo.edu>
References: <5615A4EE.8070801@buffalo.edu>
Message-ID: <CAMYG4G=bxXkFbZ5Hw2fopK4awhNnHxkxjKn1AhD499XRn6NMbg@mail.gmail.com>

On Wed, Oct 7, 2015 at 6:04 PM, Boris <borisbou at buffalo.edu> wrote:

> Hello,
>
> I am working on a toy Laplace problem (p1.C attached) investigating some
> multigrid projection matrices when using finite differences. Things seem to
> work as expected when running with :
>
> ./p1 -da_grid_x 21 -da_grid_y 21 -mat_view -pc_type mg -pc_mg_levels 1
>
> But if I change pc_mg_levels to 2 or 3, I generate the error below.
>
> Any insights as to what I could be doing wrong would be much appreciated.
>

It looks like your FormJacobian() is getting called with a DM that you do
not expect. I think its getting
called for a coarse grid, but you are getting the same fine DM from your
user context.

Take a look at SNES ex5 where we solve exactly this problem.

  Thanks,

    Matt


> Thanks,
> Boris
>
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Local index 169 too large 168 (max) at 4
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.4, May, 23, 2015
> [0]PETSC ERROR: ./p1 on a gcc-4.8.4-mpich-3.1.4-openblas-0.2.14-opt named
> bender.eng.buffalo.edu by borisbou Wed Oct  7 18:06:35 2015
> [0]PETSC ERROR: Configure options
> --prefix=/bender1/data/shared/software/libs/petsc/3.5.4/gcc/4.8.4/mpich/3.1.4/openblas/0.2.14/opt
> --with-debugging=false --COPTFLAGS="-O3 -mavx" --CXXOPTFLAGS="-O3 -mavx"
> --FOPTFLAGS=-O3 --with-shared-libraries=1
> --with-mpi-dir=/bender1/data/shared/software/libs/mpich/3.1.4/gcc/4.8.4
> --with-mumps=true --download-mumps=1 --with-metis=true --download-metis=1
> --with-parmetis=true --download-parmetis=1 --with-superlu=true
> --download-superlu=1 --with-superludir=true --download-superlu_dist=1
> --with-blacs=true --download-blacs=1 --with-scalapack=true
> --download-scalapack=1 --with-hypre=true --download-hypre=1
> --with-blas-lib="[/bender1/data/shared/software/libs/openblas/0.2.14/gcc/4.8.4/lib/libopenblas.so]"
> --with-lapack-lib="[/bender1/data/shared/software/libs/openblas/0.2.14/gcc/4.8.4/lib/libopenblas.so]"
> [0]PETSC ERROR: #1 ISLocalToGlobalMappingApply() line 401 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/vec/is/utils/isltog.c
> [0]PETSC ERROR: #2 MatSetValuesLocal() line 1982 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 MatSetValuesStencil() line 1383 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 FormJacobian() line 366 in p1.C
> [0]PETSC ERROR: #5 SNESComputeJacobian() line 2193 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #6 KSPComputeOperators_SNES() line 527 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #7 KSPSetUp() line 256 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #8 PCSetUp_MG() line 803 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/pc/impls/mg/mg.c
> [0]PETSC ERROR: #9 PCSetUp() line 902 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/pc/interface/precon.c
> [0]PETSC ERROR: #10 KSPSetUp() line 306 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #11 KSPSolve() line 418 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #12 SNESSolve_NEWTONLS() line 232 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #13 SNESSolve() line 3743 in
> /bender1/data/shared/software/builddir/petsc-1HCwjW/petsc-3.5.4/src/snes/interface/snes.c
> [0]PETSC ERROR: #14 main() line 147 in p1.C
> [0]PETSC ERROR: ----------------End of Error Message -------send entire
> error message to petsc-maint at mcs.anl.gov----------
> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151007/d7d2cbb7/attachment.html>

From Alexander.Korovin at lgep.supelec.fr  Thu Oct  8 03:54:51 2015
From: Alexander.Korovin at lgep.supelec.fr (Alexander Korovin)
Date: Thu, 8 Oct 2015 10:54:51 +0200
Subject: [petsc-users] MATLAB with PETSC
Message-ID: <1094435022.20151008105451@lgep.supelec.fr>

Dear, Petsc-users.

I'm sorry for your trouble.
I'm trying to test the ability of PETSc (PETSc-3.6.2), in MATLAB
(R2015a, version for Windows) using CYGWIN (latest version).
For the configure I used:
export PETSC_ARCH=win32-gnu
./configure PETSC_ARCH=win32-gnu --with-shared-libraries --with-matlab-engine --with-matlab --with-mpi=0 --download-fblaslapack=1 --with-x=0

There was an error in Matlab.py, because MATLAB does not work
in NoDisplay mode in Windows. To fix this, I put "r = 8.5"
after line 62 in Matlab.py.

Now  I  have  a lot of mistakes when I do make. They are all connected
with the compilation of sopen.c.
No error when I compile other files for MATLAB.

It would be nice if you could help me with this problem.
Thank you in advance.

***** copy of error messages *******
$ /cygdrive/d/R2015a/bin/win32/mex -I/cygdrive/d/R2015a/petsc-3.6.2/include -I/cygdrive/d/R2015a/petsc-3.6.2/win32-gnu/include -I/cygdrive/d/R2015a/extern/include -I/cygdrive/d/R2015a/petsc-3.6.2/include/petsc/mpiuni swrite.c bread.c
Warning: Legacy MEX infrastructure is provided for compatibility; it will be removed in a future version of MATLAB. For more information, consult the MEX release notes http://www.mathworks.com/help/matlab/release-notes.html.

$ /cygdrive/d/R2015a/bin/win32/mex -I/cygdrive/d/R2015a/petsc-3.6.2/include -I/cygdrive/d/R2015a/petsc-3.6.2/win32-gnu/include -I/cygdrive/d/R2015a/extern/include -I/cygdrive/d/R2015a/petsc-3.6.2/include/petsc/mpiuni sopen.c
In file included from /usr/include/w32api/winsock2.h:56:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_fd_types.h:100:2: warning: #warning "fd_set and associated macros have been defined in sys/types.      This can cause runtime problems with W32 sockets" [-Wcpp]
 #warning "fd_set and associated macros have been defined in sys/types.  \
  ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:25:8: error: redefinition of ?struct hostent?
 struct hostent {
        ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:79:8: note: originally defined here
 struct hostent {
        ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:33:8: error: redefinition of ?struct netent?
 struct netent {
        ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:93:8: note: originally defined here
 struct netent {
        ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:40:8: error: redefinition of ?struct servent?
 struct servent {
        ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:49:10: note: originally defined here
   struct servent *getservbyname (const char *__name, const char *__proto);
          ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:52:8: error: redefinition of ?struct protoent?
 struct protoent {
        ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:107:8: note: originally defined here
 struct protoent
        ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:63:8: error: redefinition of ?struct linger?
 struct linger {
        ^
In file included from /usr/include/sys/socket.h:16:0,
                 from sopen.c:34:
/usr/include/cygwin/socket.h:62:8: note: originally defined here
 struct linger {
        ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:70:8: error: redefinition of ?struct sockaddr?
 struct sockaddr {
        ^
In file included from /usr/include/sys/socket.h:16:0,
                 from sopen.c:34:
/usr/include/cygwin/socket.h:31:8: note: originally defined here
 struct sockaddr {
        ^
In file included from /usr/include/w32api/winsock2.h:57:0,
                 from /usr/include/w32api/Ws2tcpip.h:17,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_types.h:75:8: error: redefinition of ?struct sockaddr_in?
 struct sockaddr_in {
        ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:189:8: note: originally defined here
 struct sockaddr_in
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:260:10: error: redefinition of ?struct sockaddr_storage?
   struct sockaddr_storage {
          ^
In file included from /usr/include/sys/socket.h:16:0,
                 from sopen.c:34:
/usr/include/cygwin/socket.h:43:8: note: originally defined here
 struct sockaddr_storage {
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:973:37: error: conflicting types for ?accept?
   WINSOCK_API_LINKAGE SOCKET WSAAPI accept(SOCKET s,struct sockaddr *addr,int *addrlen);
                                     ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:25:7: note: previous declaration of ?accept? was here
   int accept (int, struct sockaddr *__peer, socklen_t *);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:974:34: error: conflicting types for ?bind?
   WINSOCK_API_LINKAGE int WSAAPI bind(SOCKET s,const struct sockaddr *name,int namelen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:27:7: note: previous declaration of ?bind? was here
   int bind (int, const struct sockaddr *__my_addr, socklen_t __addrlen);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:976:34: error: conflicting types for ?connect?
   WINSOCK_API_LINKAGE int WSAAPI connect(SOCKET s,const struct sockaddr *name,int namelen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:28:7: note: previous declaration of ?connect? was here
   int connect (int, const struct sockaddr *, socklen_t);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:978:34: error: conflicting types for ?getpeername?
   WINSOCK_API_LINKAGE int WSAAPI getpeername(SOCKET s,struct sockaddr *name,int *namelen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:29:7: note: previous declaration of ?getpeername? was here
   int getpeername (int, struct sockaddr *__peer, socklen_t *);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:979:34: error: conflicting types for ?getsockname?
   WINSOCK_API_LINKAGE int WSAAPI getsockname(SOCKET s,struct sockaddr *name,int *namelen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:30:7: note: previous declaration of ?getsockname? was here
   int getsockname (int, struct sockaddr *__addr, socklen_t *);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:980:34: error: conflicting types for ?getsockopt?
   WINSOCK_API_LINKAGE int WSAAPI getsockopt(SOCKET s,int level,int optname,char *optval,int *optlen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:42:7: note: previous declaration of ?getsockopt? was here
   int getsockopt (int __s, int __level, int __optname, void *__optval,
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:982:37: error: conflicting types for ?htonl?
   WINSOCK_API_LINKAGE u_long WSAAPI htonl(u_long hostlong);
                                     ^
In file included from /usr/include/cygwin/in.h:256:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/asm/byteorder.h:28:17: note: previous declaration of ?htonl? was here
 extern uint32_t htonl(uint32_t);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:983:38: error: conflicting types for ?htons?
   WINSOCK_API_LINKAGE u_short WSAAPI htons(u_short hostshort);
                                      ^
In file included from /usr/include/cygwin/in.h:256:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/asm/byteorder.h:29:17: note: previous declaration of ?htons? was here
 extern uint16_t htons(uint16_t);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:987:34: error: conflicting types for ?listen?
   WINSOCK_API_LINKAGE int WSAAPI listen(SOCKET s,int backlog);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:31:7: note: previous declaration of ?listen? was here
   int listen (int, int __n);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:989:37: error: conflicting types for ?ntohl?
   WINSOCK_API_LINKAGE u_long WSAAPI ntohl(u_long netlong);
                                     ^
In file included from /usr/include/cygwin/in.h:256:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/asm/byteorder.h:26:17: note: previous declaration of ?ntohl? was here
 extern uint32_t ntohl(uint32_t);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:990:38: error: conflicting types for ?ntohs?
   WINSOCK_API_LINKAGE u_short WSAAPI ntohs(u_short netshort);
                                      ^
In file included from /usr/include/cygwin/in.h:256:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/asm/byteorder.h:27:17: note: previous declaration of ?ntohs? was here
 extern uint16_t ntohs(uint16_t);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:992:34: error: conflicting types for ?recv?
   WINSOCK_API_LINKAGE int WSAAPI recv(SOCKET s,char *buf,int len,int flags);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:32:11: note: previous declaration of ?recv? was here
   ssize_t recv (int, void *__buff, size_t __len, int __flags);
           ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:993:34: error: conflicting types for ?recvfrom?
   WINSOCK_API_LINKAGE int WSAAPI recvfrom(SOCKET s,char *buf,int len,int flags,struct sockaddr *from,int *fromlen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:33:11: note: previous declaration of ?recvfrom? was here
   ssize_t recvfrom (int, void *__buff, size_t __len, int __flags,
           ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:995:34: error: conflicting types for ?select?
   WINSOCK_API_LINKAGE int WSAAPI select(int nfds,fd_set *readfds,fd_set *writefds,fd_set *exceptfds,const PTIMEVAL timeout);
                                  ^
In file included from /usr/include/sys/time.h:49:0,
                 from /usr/include/sys/socket.h:17,
                 from sopen.c:34:
/usr/include/sys/select.h:31:5: note: previous declaration of ?select? was here
 int select __P ((int __n, fd_set *__readfds, fd_set *__writefds,
     ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:997:34: error: conflicting types for ?send?
   WINSOCK_API_LINKAGE int WSAAPI send(SOCKET s,const char *buf,int len,int flags);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:36:11: note: previous declaration of ?send? was here
   ssize_t send (int, const void *__buff, size_t __len, int __flags);
           ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:998:34: error: conflicting types for ?sendto?
   WINSOCK_API_LINKAGE int WSAAPI sendto(SOCKET s,const char *buf,int len,int flags,const struct sockaddr *to,int tolen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:38:11: note: previous declaration of ?sendto? was here
   ssize_t sendto (int, const void *, size_t __len, int __flags,
           ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:999:34: error: conflicting types for ?setsockopt?
   WINSOCK_API_LINKAGE int WSAAPI setsockopt(SOCKET s,int level,int optname,const char *optval,int optlen);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:40:7: note: previous declaration of ?setsockopt? was here
   int setsockopt (int __s, int __level, int __optname, const void *optval,
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1000:34: error: conflicting types for ?shutdown?
   WINSOCK_API_LINKAGE int WSAAPI shutdown(SOCKET s,int how);
                                  ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:44:7: note: previous declaration of ?shutdown? was here
   int shutdown (int, int);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1001:37: error: conflicting types for ?socket?
   WINSOCK_API_LINKAGE SOCKET WSAAPI socket(int af,int type,int protocol);
                                     ^
In file included from sopen.c:34:0:
/usr/include/sys/socket.h:45:7: note: previous declaration of ?socket? was here
   int socket (int __family, int __type, int __protocol);
       ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1002:46: error: conflicting types for ?gethostbyaddr?
   WINSOCK_API_LINKAGE struct hostent *WSAAPI gethostbyaddr(const char *addr,int len,int type);
                                              ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:217:17: note: previous declaration of ?gethostbyaddr? was here
 struct hostent *gethostbyaddr (const char *, int, int);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1003:46: error: conflicting types for ?gethostbyname?
   WINSOCK_API_LINKAGE struct hostent *WSAAPI gethostbyname(const char *name);
                                              ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:218:17: note: previous declaration of ?gethostbyname? was here
 struct hostent *gethostbyname (const char *);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1004:34: error: conflicting types for ?gethostname?
   WINSOCK_API_LINKAGE int WSAAPI gethostname(char *name,int namelen);
                                  ^
In file included from /usr/include/stdio.h:29:0,
                 from /cygdrive/d/R2015a/petsc-3.6.2/include/petscsys.h:138,
                 from sopen.c:14:
/usr/include/sys/unistd.h:249:6: note: previous declaration of ?gethostname? was here
  int _EXFUN(gethostname, (char *__name, size_t __len));
      ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1005:46: error: conflicting types for ?getservbyport?
   WINSOCK_API_LINKAGE struct servent *WSAAPI getservbyport(int port,const char *proto);
                                              ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:228:17: note: previous declaration of ?getservbyport? was here
 struct servent *getservbyport (int, const char *);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1006:46: error: conflicting types for ?getservbyname?
   WINSOCK_API_LINKAGE struct servent *WSAAPI getservbyname(const char *name,const char *proto);
                                              ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:227:17: note: previous declaration of ?getservbyname? was here
 struct servent *getservbyname (const char *, const char *);
                 ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1007:47: error: conflicting types for ?getprotobynumber?
   WINSOCK_API_LINKAGE struct protoent *WSAAPI getprotobynumber(int number);
                                               ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:225:18: note: previous declaration of ?getprotobynumber? was here
 struct protoent *getprotobynumber (int);
                  ^
In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
                 from sopen.c:58:
/usr/include/w32api/winsock2.h:1008:47: error: conflicting types for ?getprotobyname?
   WINSOCK_API_LINKAGE struct protoent *WSAAPI getprotobyname(const char *name);
                                               ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:224:18: note: previous declaration of ?getprotobyname? was here
 struct protoent *getprotobyname (const char *);
                  ^
In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
                 from sopen.c:58:
/usr/include/w32api/ws2ipdef.h:23:16: error: redefinition of ?struct ipv6_mreq?
 typedef struct ipv6_mreq {
                ^
In file included from /usr/include/cygwin/in.h:267:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/cygwin/in6.h:88:8: note: originally defined here
 struct ipv6_mreq
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
                 from sopen.c:58:
/usr/include/w32api/ws2ipdef.h:41:8: error: redefinition of ?struct sockaddr_in6?
 struct sockaddr_in6 {
        ^
In file included from /usr/include/cygwin/in.h:267:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/cygwin/in6.h:112:8: note: originally defined here
 struct sockaddr_in6
        ^
In file included from /usr/include/sys/socket.h:16:0,
                 from sopen.c:34:
/usr/include/w32api/ws2ipdef.h:64:3: error: expected identifier before numeric constant
   MCAST_INCLUDE   = 0,
   ^
In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
                 from sopen.c:58:
/usr/include/w32api/ws2ipdef.h:79:16: error: redefinition of ?struct group_filter?
 typedef struct group_filter {
                ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:174:8: note: originally defined here
 struct group_filter
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
                 from sopen.c:58:
/usr/include/w32api/ws2ipdef.h:87:16: error: redefinition of ?struct group_req?
 typedef struct group_req {
                ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:161:8: note: originally defined here
 struct group_req
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
                 from sopen.c:58:
/usr/include/w32api/ws2ipdef.h:92:16: error: redefinition of ?struct group_source_req?
 typedef struct group_source_req {
                ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:167:8: note: originally defined here
 struct group_source_req
        ^
In file included from /usr/include/w32api/Ws2tcpip.h:19:0,
                 from sopen.c:58:
/usr/include/w32api/psdk_inc/_ip_mreq1.h:12:8: error: redefinition of ?struct ip_mreq?
 struct ip_mreq {
        ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:127:8: note: originally defined here
 struct ip_mreq
        ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:22:8: error: redefinition of ?struct ip_mreq_source?
 struct ip_mreq_source {
        ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:133:8: note: originally defined here
 struct ip_mreq_source
        ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:28:8: error: redefinition of ?struct ip_msfilter?
 struct ip_msfilter {
        ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:140:8: note: originally defined here
 struct ip_msfilter
        ^
In file included from /usr/include/cygwin/in.h:267:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/w32api/Ws2tcpip.h:86:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:86:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:86:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:87:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:87:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:87:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:88:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:88:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:88:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:89:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?==? token
 int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:90:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?==? token
 int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:91:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:91:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:91:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:92:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:92:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:92:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:93:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?&&? token
 int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:94:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?&&? token
 int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:95:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?&&? token
 int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:96:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?&&? token
 int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:97:5: error: expected identifier or ?(? before ?const?
 int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?(? token
 int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?[? token
 int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?&&? token
 int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
     ^
/usr/include/w32api/Ws2tcpip.h:105:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:105:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:105:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:106:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:106:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:106:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:107:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
                     ^
/usr/include/w32api/Ws2tcpip.h:107:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
                     ^
/usr/include/w32api/Ws2tcpip.h:107:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
                     ^
/usr/include/w32api/Ws2tcpip.h:108:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?==? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:109:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?==? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:110:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:110:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:110:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
                     ^
/usr/include/w32api/Ws2tcpip.h:111:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
                     ^
/usr/include/w32api/Ws2tcpip.h:111:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
                     ^
/usr/include/w32api/Ws2tcpip.h:111:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
                     ^
/usr/include/w32api/Ws2tcpip.h:112:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
                     ^
/usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
                     ^
/usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
                     ^
/usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?&&? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
                     ^
/usr/include/w32api/Ws2tcpip.h:113:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
                     ^
/usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
                     ^
/usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
                     ^
/usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?&&? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
                     ^
/usr/include/w32api/Ws2tcpip.h:114:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
                     ^
/usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
                     ^
/usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
                     ^
/usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?&&? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
                     ^
/usr/include/w32api/Ws2tcpip.h:115:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
                     ^
/usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
                     ^
/usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
                     ^
/usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?&&? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
                     ^
/usr/include/w32api/Ws2tcpip.h:116:21: error: expected identifier or ?(? before ?const?
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
                     ^
/usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?(? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
                     ^
/usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?[? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
                     ^
/usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?&&? token
 WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
                     ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h: In function ?IN6_SET_ADDR_UNSPECIFIED?:
/usr/include/w32api/Ws2tcpip.h:119:77: error: ?struct in6_addr? has no member named ?s6_bytes?
 WS2TCPIP_INLINE void IN6_SET_ADDR_UNSPECIFIED(struct in6_addr *a) { memset(a->s6_bytes,0,sizeof(struct in6_addr)); }
                                                                             ^
/usr/include/w32api/Ws2tcpip.h: In function ?IN6_SET_ADDR_LOOPBACK?:
/usr/include/w32api/Ws2tcpip.h:121:11: error: ?struct in6_addr? has no member named ?s6_bytes?
   memset(a->s6_bytes,0,sizeof(struct in6_addr));
           ^
/usr/include/w32api/Ws2tcpip.h:122:4: error: ?struct in6_addr? has no member named ?s6_bytes?
   a->s6_bytes[15] = 1;
    ^
/usr/include/w32api/Ws2tcpip.h: At top level:
/usr/include/w32api/Ws2tcpip.h:156:16: error: redefinition of ?struct in_pktinfo?
 typedef struct in_pktinfo {
                ^
In file included from /usr/include/netinet/in.h:14:0,
                 from sopen.c:40:
/usr/include/cygwin/in.h:153:8: note: originally defined here
 struct in_pktinfo
        ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:157:3: error: unknown type name ?IN_ADDR?
   IN_ADDR ipi_addr;
   ^
/usr/include/w32api/Ws2tcpip.h:163:16: error: redefinition of ?struct in6_pktinfo?
 typedef struct in6_pktinfo {
                ^
In file included from /usr/include/cygwin/in.h:267:0,
                 from /usr/include/netinet/in.h:14,
                 from sopen.c:40:
/usr/include/cygwin/in6.h:102:8: note: originally defined here
 struct in6_pktinfo
        ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:164:3: error: unknown type name ?IN6_ADDR?
   IN6_ADDR ipi6_addr;
   ^
/usr/include/w32api/Ws2tcpip.h:168:1: error: size of unnamed array is negative
 C_ASSERT(sizeof(IN6_PKTINFO)==20);
 ^
/usr/include/w32api/Ws2tcpip.h:182:16: error: redefinition of ?struct addrinfo?
 typedef struct addrinfo {
                ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:121:8: note: originally defined here
 struct addrinfo {
        ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:231:34: error: conflicting types for ?getaddrinfo?
   WINSOCK_API_LINKAGE int WSAAPI getaddrinfo(const char *nodename,const char *servname,const struct addrinfo *hints,struct addrinfo **res);
                                  ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:242:6: note: previous declaration of ?getaddrinfo? was here
 int  getaddrinfo (const char *, const char *,
      ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:247:35: error: conflicting types for ?freeaddrinfo?
   WINSOCK_API_LINKAGE void WSAAPI freeaddrinfo(LPADDRINFO pAddrInfo);
                                   ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:240:7: note: previous declaration of ?freeaddrinfo? was here
 void  freeaddrinfo (struct addrinfo *);
       ^
In file included from sopen.c:58:0:
/usr/include/w32api/Ws2tcpip.h:265:34: error: conflicting types for ?getnameinfo?
   WINSOCK_API_LINKAGE int WSAAPI getnameinfo(const struct sockaddr *sa,socklen_t salen,char *host,DWORD hostlen,char *serv,DWORD servlen,int flags);
                                  ^
In file included from sopen.c:43:0:
/usr/include/netdb.h:244:6: note: previous declaration of ?getnameinfo? was here
 int  getnameinfo (const struct sockaddr *, socklen_t, char *,
      ^
In file included from /usr/include/w32api/Ws2tcpip.h:298:0,
                 from sopen.c:58:
/usr/include/w32api/mstcpip.h:110:43: error: unknown type name ?IN6_ADDR?
 LPSTR NTAPI RtlIpv6AddressToStringA(const IN6_ADDR *Addr, LPSTR S);
                                           ^
/usr/include/w32api/mstcpip.h:111:44: error: unknown type name ?IN6_ADDR?
 LPWSTR NTAPI RtlIpv6AddressToStringW(const IN6_ADDR *Addr, LPWSTR S);
                                            ^
/usr/include/w32api/mstcpip.h:113:44: error: unknown type name ?IN6_ADDR?
 LONG NTAPI RtlIpv6AddressToStringExA(const IN6_ADDR *Address, ULONG ScopeId, USHORT Port, LPSTR AddressString, PULONG AddressStringLength);
                                            ^
In file included from /usr/include/w32api/Ws2tcpip.h:298:0,
                 from sopen.c:58:
/usr/include/w32api/mstcpip.h:114:44: error: unknown type name ?IN6_ADDR?
 LONG NTAPI RtlIpv6AddressToStringExW(const IN6_ADDR *Address, ULONG ScopeId, USHORT Port, LPWSTR AddressString, PULONG AddressStringLength);
                                            ^
/usr/include/w32api/mstcpip.h:117:43: error: unknown type name ?IN_ADDR?
 LPSTR NTAPI RtlIpv4AddressToStringA(const IN_ADDR *Addr, LPSTR S);
                                           ^
/usr/include/w32api/mstcpip.h:118:44: error: unknown type name ?IN_ADDR?
 LPWSTR NTAPI RtlIpv4AddressToStringW(const IN_ADDR *Addr, LPWSTR S);
                                            ^
/usr/include/w32api/mstcpip.h:121:44: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4AddressToStringExA(const IN_ADDR *Address, USHORT Port, LPSTR AddressString, PULONG AddressStringLength);
                                            ^
/usr/include/w32api/mstcpip.h:122:44: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4AddressToStringExW(const IN_ADDR *Address, USHORT Port, LPWSTR AddressString, PULONG AddressStringLength);
                                            ^
/usr/include/w32api/mstcpip.h:125:80: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4StringToAddressA(PCSTR S, BOOLEAN Strict, LPSTR *Terminator, IN_ADDR *Addr);
                                                                                ^
/usr/include/w32api/mstcpip.h:126:82: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4StringToAddressW(PCWSTR S, BOOLEAN Strict, LPWSTR *Terminator, IN_ADDR *Addr);
                                                                                  ^
/usr/include/w32api/mstcpip.h:129:75: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4StringToAddressExA(PCSTR AddressString, BOOLEAN Strict, IN_ADDR *Address, PUSHORT Port);
                                                                           ^
/usr/include/w32api/mstcpip.h:130:76: error: unknown type name ?IN_ADDR?
 LONG NTAPI RtlIpv4StringToAddressExW(PCWSTR AddressString, BOOLEAN Strict, IN_ADDR *Address, PUSHORT Port);
                                                                            ^
/usr/include/w32api/mstcpip.h:133:59: error: unknown type name ?IN6_ADDR?
 LONG NTAPI RtlIpv6StringToAddressExA(PCSTR AddressString, IN6_ADDR *Address, PULONG ScopeId, PUSHORT Port);
                                                           ^
/usr/include/w32api/mstcpip.h:134:60: error: unknown type name ?IN6_ADDR?
 LONG NTAPI RtlIpv6StringToAddressExW(PCWSTR AddressString, IN6_ADDR *Address, PULONG ScopeId, PUSHORT Port);
                                                            ^
sopen.c: In function ?establish?:
sopen.c:116:44: error: ?namelen? undeclared (first use in this function)
   GetComputerName((LPTSTR)myname,(LPDWORD)&namelen);
                                            ^
sopen.c:116:44: note: each undeclared identifier is reported only once for each function it appears in

  D:\R2015A\BIN\MEX.PL: Error: Compile of 'sopen.c' failed.

************

Sincerely yours,
 Alexander

-------------------------
 Dr. Alexander KOROVIN

 Lab. G?nie ?lectrique et ?lectronique - Paris (GeePs)
 11 rue Joliot-Curie
 Plateau de Moulon
 F-91192 Gif-sur-Yvette cedex
 France
 +33-1-69-85-16-83


From mfadams at lbl.gov  Thu Oct  8 08:56:28 2015
From: mfadams at lbl.gov (Mark Adams)
Date: Thu, 8 Oct 2015 09:56:28 -0400
Subject: [petsc-users] PETSc params
Message-ID: <CADOhEh7CRfXa1=0j_WP1kpeqVH1DRqeOyLKU8tPdSfd_VpgMYQ@mail.gmail.com>

Pat, if you can fold this into some large scale experiments, that would be
great.

-s2_pc_gamg_repartition true [false]
-s2_pc_gamg_mat_partitioning_type parmetis
-s2_pc_gamg_process_eq_limit 200 [10, 500]

duplicate this without s2_, for the other solver.

The first parameters is to repartition the coarse grids, or not.
The second parameter is not used if you are not repartitioning.
The third parameter governs the size of processor subdomains on coarse
grids.  So a larger value reduced the number of active processor faster.
Any kind of scan that you can do would be great. I am suggesting doing 10 &
500, as well as 200, which is a sort of default.

You don't have to do all of these permutations.  maybe just 4 tests: t/200,
F/200, t/10, t/500.

Also use -options_left, if you don't already have it, to test that these
parameters are spelled correctly.

I just need the log summary file.  The -ksp_view data might be useful
eventually, but you can start by just giving me the log files.

And remember to write up what you said about slow MPI collectives in
cray-petsc with multiple ranks per node on Titan ... and any data that you
might have. You can send petsc-users at mcs.anl.gov

Thanks,
Mark
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151008/be2aa302/attachment.html>

From mpovolot at purdue.edu  Thu Oct  8 12:26:16 2015
From: mpovolot at purdue.edu (Michael Povolotskyi)
Date: Thu, 08 Oct 2015 13:26:16 -0400
Subject: [petsc-users] user defined algorithm for newton solver
Message-ID: <5616A738.50704@purdue.edu>

Dear Petsc developers and users,
I'm solving a nonlinear system with PETSc.
Often simple Newton iterations do not work and I have to use either 
linear search or trust region.
I would like to use my own algorithm to find a next iteration 
approximation to solution if the Newton step does not improve the 
residual. As far as I can see I have to define my own SNELLineSearch 
object. Is there any example that shows how to do it?
Thank you,
Michael.

-- 
Michael Povolotskyi, PhD
Research Assistant Professor
Network for Computational Nanotechnology
Hall for Discover and Learning Research, Room 441
West Lafayette, IN 47907
Phone (765) 4949396


From bsmith at mcs.anl.gov  Thu Oct  8 12:58:01 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 8 Oct 2015 12:58:01 -0500
Subject: [petsc-users] user defined algorithm for newton solver
In-Reply-To: <5616A738.50704@purdue.edu>
References: <5616A738.50704@purdue.edu>
Message-ID: <E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>


> On Oct 8, 2015, at 12:26 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
> 
> Dear Petsc developers and users,
> I'm solving a nonlinear system with PETSc.
> Often simple Newton iterations do not work and I have to use either linear search or trust region.
> I would like to use my own algorithm to find a next iteration approximation to solution if the Newton step does not improve the residual. As far as I can see I have to define my own SNELLineSearch object. Is there any example that shows how to do it?

   The line search model is 1) select direction based on approximate solution of approximate Jacobian  2) search in THAT direction for a decrease in the function evaluation. You state "if the Newton step does not improve the residual." so will you be computing the DIRECTION in a different way than (1) or will you be using the same direction but trying somehow to find a decrease in the function evaluation using a different technique then the standard line searchs we provide?

   Frankly if the line searches we provide don't work that means the direction is not good and no "special" line search will recover.  I really recommend you go through our suggestions on the FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#newton on trying to figure out why Newton is not converging before you think about writing a special line search.

  Barry



> Thank you,
> Michael.
> 
> -- 
> Michael Povolotskyi, PhD
> Research Assistant Professor
> Network for Computational Nanotechnology
> Hall for Discover and Learning Research, Room 441
> West Lafayette, IN 47907
> Phone (765) 4949396
> 


From mpovolot at purdue.edu  Thu Oct  8 13:05:28 2015
From: mpovolot at purdue.edu (Michael Povolotskyi)
Date: Thu, 08 Oct 2015 14:05:28 -0400
Subject: [petsc-users] user defined algorithm for newton solver
In-Reply-To: <E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>
References: <5616A738.50704@purdue.edu>
	<E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>
Message-ID: <5616B068.7000008@purdue.edu>

Thank you.
The situation is as follows:
The system I need to solve does not have a unique solution, but only one 
makes sense from physical point of view.  I need to compute direction in 
a different way based on the physics.

In other words it should be like this:

1. Start with a solution guess
2. Do full Newton step
3. If converged, exit
     else if the solution improved go to step 2
     otherwise "update_solution_in_my_way" ang go to step 2

Is it possible to do this in PETSc?
Michael.

On 10/08/2015 01:58 PM, Barry Smith wrote:
>> On Oct 8, 2015, at 12:26 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
>>
>> Dear Petsc developers and users,
>> I'm solving a nonlinear system with PETSc.
>> Often simple Newton iterations do not work and I have to use either linear search or trust region.
>> I would like to use my own algorithm to find a next iteration approximation to solution if the Newton step does not improve the residual. As far as I can see I have to define my own SNELLineSearch object. Is there any example that shows how to do it?
>     The line search model is 1) select direction based on approximate solution of approximate Jacobian  2) search in THAT direction for a decrease in the function evaluation. You state "if the Newton step does not improve the residual." so will you be computing the DIRECTION in a different way than (1) or will you be using the same direction but trying somehow to find a decrease in the function evaluation using a different technique then the standard line searchs we provide?
>
>     Frankly if the line searches we provide don't work that means the direction is not good and no "special" line search will recover.  I really recommend you go through our suggestions on the FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#newton on trying to figure out why Newton is not converging before you think about writing a special line search.
>
>    Barry
>
>
>
>> Thank you,
>> Michael.
>>
>> -- 
>> Michael Povolotskyi, PhD
>> Research Assistant Professor
>> Network for Computational Nanotechnology
>> Hall for Discover and Learning Research, Room 441
>> West Lafayette, IN 47907
>> Phone (765) 4949396
>>

-- 
Michael Povolotskyi, PhD
Research Assistant Professor
Network for Computational Nanotechnology
Hall for Discover and Learning Research, Room 441
West Lafayette, IN 47907
Phone (765) 4949396


From bsmith at mcs.anl.gov  Thu Oct  8 13:37:44 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 8 Oct 2015 13:37:44 -0500
Subject: [petsc-users] MATLAB with PETSC
In-Reply-To: <1094435022.20151008105451@lgep.supelec.fr>
References: <1094435022.20151008105451@lgep.supelec.fr>
Message-ID: <75A74342-49F0-4291-BD5D-C24230BE309C@mcs.anl.gov>


  Alexander,

  Windows supports a very different API for sockets than Unix. If you want this socket stuff for Windows I think you have to more or less write a completely different piece of code that is Windows specific.

   Barry

> On Oct 8, 2015, at 3:54 AM, Alexander Korovin <Alexander.Korovin at lgep.supelec.fr> wrote:
> 
> Dear, Petsc-users.
> 
> I'm sorry for your trouble.
> I'm trying to test the ability of PETSc (PETSc-3.6.2), in MATLAB
> (R2015a, version for Windows) using CYGWIN (latest version).
> For the configure I used:
> export PETSC_ARCH=win32-gnu
> ./configure PETSC_ARCH=win32-gnu --with-shared-libraries --with-matlab-engine --with-matlab --with-mpi=0 --download-fblaslapack=1 --with-x=0
> 
> There was an error in Matlab.py, because MATLAB does not work
> in NoDisplay mode in Windows. To fix this, I put "r = 8.5"
> after line 62 in Matlab.py.
> 
> Now  I  have  a lot of mistakes when I do make. They are all connected
> with the compilation of sopen.c.
> No error when I compile other files for MATLAB.
> 
> It would be nice if you could help me with this problem.
> Thank you in advance.
> 
> ***** copy of error messages *******
> $ /cygdrive/d/R2015a/bin/win32/mex -I/cygdrive/d/R2015a/petsc-3.6.2/include -I/cygdrive/d/R2015a/petsc-3.6.2/win32-gnu/include -I/cygdrive/d/R2015a/extern/include -I/cygdrive/d/R2015a/petsc-3.6.2/include/petsc/mpiuni swrite.c bread.c
> Warning: Legacy MEX infrastructure is provided for compatibility; it will be removed in a future version of MATLAB. For more information, consult the MEX release notes http://www.mathworks.com/help/matlab/release-notes.html.
> 
> $ /cygdrive/d/R2015a/bin/win32/mex -I/cygdrive/d/R2015a/petsc-3.6.2/include -I/cygdrive/d/R2015a/petsc-3.6.2/win32-gnu/include -I/cygdrive/d/R2015a/extern/include -I/cygdrive/d/R2015a/petsc-3.6.2/include/petsc/mpiuni sopen.c
> In file included from /usr/include/w32api/winsock2.h:56:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_fd_types.h:100:2: warning: #warning "fd_set and associated macros have been defined in sys/types.      This can cause runtime problems with W32 sockets" [-Wcpp]
> #warning "fd_set and associated macros have been defined in sys/types.  \
>  ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:25:8: error: redefinition of ?struct hostent?
> struct hostent {
>        ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:79:8: note: originally defined here
> struct hostent {
>        ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:33:8: error: redefinition of ?struct netent?
> struct netent {
>        ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:93:8: note: originally defined here
> struct netent {
>        ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:40:8: error: redefinition of ?struct servent?
> struct servent {
>        ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:49:10: note: originally defined here
>   struct servent *getservbyname (const char *__name, const char *__proto);
>          ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:52:8: error: redefinition of ?struct protoent?
> struct protoent {
>        ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:107:8: note: originally defined here
> struct protoent
>        ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:63:8: error: redefinition of ?struct linger?
> struct linger {
>        ^
> In file included from /usr/include/sys/socket.h:16:0,
>                 from sopen.c:34:
> /usr/include/cygwin/socket.h:62:8: note: originally defined here
> struct linger {
>        ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:70:8: error: redefinition of ?struct sockaddr?
> struct sockaddr {
>        ^
> In file included from /usr/include/sys/socket.h:16:0,
>                 from sopen.c:34:
> /usr/include/cygwin/socket.h:31:8: note: originally defined here
> struct sockaddr {
>        ^
> In file included from /usr/include/w32api/winsock2.h:57:0,
>                 from /usr/include/w32api/Ws2tcpip.h:17,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_types.h:75:8: error: redefinition of ?struct sockaddr_in?
> struct sockaddr_in {
>        ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:189:8: note: originally defined here
> struct sockaddr_in
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:260:10: error: redefinition of ?struct sockaddr_storage?
>   struct sockaddr_storage {
>          ^
> In file included from /usr/include/sys/socket.h:16:0,
>                 from sopen.c:34:
> /usr/include/cygwin/socket.h:43:8: note: originally defined here
> struct sockaddr_storage {
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:973:37: error: conflicting types for ?accept?
>   WINSOCK_API_LINKAGE SOCKET WSAAPI accept(SOCKET s,struct sockaddr *addr,int *addrlen);
>                                     ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:25:7: note: previous declaration of ?accept? was here
>   int accept (int, struct sockaddr *__peer, socklen_t *);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:974:34: error: conflicting types for ?bind?
>   WINSOCK_API_LINKAGE int WSAAPI bind(SOCKET s,const struct sockaddr *name,int namelen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:27:7: note: previous declaration of ?bind? was here
>   int bind (int, const struct sockaddr *__my_addr, socklen_t __addrlen);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:976:34: error: conflicting types for ?connect?
>   WINSOCK_API_LINKAGE int WSAAPI connect(SOCKET s,const struct sockaddr *name,int namelen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:28:7: note: previous declaration of ?connect? was here
>   int connect (int, const struct sockaddr *, socklen_t);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:978:34: error: conflicting types for ?getpeername?
>   WINSOCK_API_LINKAGE int WSAAPI getpeername(SOCKET s,struct sockaddr *name,int *namelen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:29:7: note: previous declaration of ?getpeername? was here
>   int getpeername (int, struct sockaddr *__peer, socklen_t *);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:979:34: error: conflicting types for ?getsockname?
>   WINSOCK_API_LINKAGE int WSAAPI getsockname(SOCKET s,struct sockaddr *name,int *namelen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:30:7: note: previous declaration of ?getsockname? was here
>   int getsockname (int, struct sockaddr *__addr, socklen_t *);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:980:34: error: conflicting types for ?getsockopt?
>   WINSOCK_API_LINKAGE int WSAAPI getsockopt(SOCKET s,int level,int optname,char *optval,int *optlen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:42:7: note: previous declaration of ?getsockopt? was here
>   int getsockopt (int __s, int __level, int __optname, void *__optval,
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:982:37: error: conflicting types for ?htonl?
>   WINSOCK_API_LINKAGE u_long WSAAPI htonl(u_long hostlong);
>                                     ^
> In file included from /usr/include/cygwin/in.h:256:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/asm/byteorder.h:28:17: note: previous declaration of ?htonl? was here
> extern uint32_t htonl(uint32_t);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:983:38: error: conflicting types for ?htons?
>   WINSOCK_API_LINKAGE u_short WSAAPI htons(u_short hostshort);
>                                      ^
> In file included from /usr/include/cygwin/in.h:256:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/asm/byteorder.h:29:17: note: previous declaration of ?htons? was here
> extern uint16_t htons(uint16_t);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:987:34: error: conflicting types for ?listen?
>   WINSOCK_API_LINKAGE int WSAAPI listen(SOCKET s,int backlog);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:31:7: note: previous declaration of ?listen? was here
>   int listen (int, int __n);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:989:37: error: conflicting types for ?ntohl?
>   WINSOCK_API_LINKAGE u_long WSAAPI ntohl(u_long netlong);
>                                     ^
> In file included from /usr/include/cygwin/in.h:256:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/asm/byteorder.h:26:17: note: previous declaration of ?ntohl? was here
> extern uint32_t ntohl(uint32_t);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:990:38: error: conflicting types for ?ntohs?
>   WINSOCK_API_LINKAGE u_short WSAAPI ntohs(u_short netshort);
>                                      ^
> In file included from /usr/include/cygwin/in.h:256:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/asm/byteorder.h:27:17: note: previous declaration of ?ntohs? was here
> extern uint16_t ntohs(uint16_t);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:992:34: error: conflicting types for ?recv?
>   WINSOCK_API_LINKAGE int WSAAPI recv(SOCKET s,char *buf,int len,int flags);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:32:11: note: previous declaration of ?recv? was here
>   ssize_t recv (int, void *__buff, size_t __len, int __flags);
>           ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:993:34: error: conflicting types for ?recvfrom?
>   WINSOCK_API_LINKAGE int WSAAPI recvfrom(SOCKET s,char *buf,int len,int flags,struct sockaddr *from,int *fromlen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:33:11: note: previous declaration of ?recvfrom? was here
>   ssize_t recvfrom (int, void *__buff, size_t __len, int __flags,
>           ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:995:34: error: conflicting types for ?select?
>   WINSOCK_API_LINKAGE int WSAAPI select(int nfds,fd_set *readfds,fd_set *writefds,fd_set *exceptfds,const PTIMEVAL timeout);
>                                  ^
> In file included from /usr/include/sys/time.h:49:0,
>                 from /usr/include/sys/socket.h:17,
>                 from sopen.c:34:
> /usr/include/sys/select.h:31:5: note: previous declaration of ?select? was here
> int select __P ((int __n, fd_set *__readfds, fd_set *__writefds,
>     ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:997:34: error: conflicting types for ?send?
>   WINSOCK_API_LINKAGE int WSAAPI send(SOCKET s,const char *buf,int len,int flags);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:36:11: note: previous declaration of ?send? was here
>   ssize_t send (int, const void *__buff, size_t __len, int __flags);
>           ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:998:34: error: conflicting types for ?sendto?
>   WINSOCK_API_LINKAGE int WSAAPI sendto(SOCKET s,const char *buf,int len,int flags,const struct sockaddr *to,int tolen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:38:11: note: previous declaration of ?sendto? was here
>   ssize_t sendto (int, const void *, size_t __len, int __flags,
>           ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:999:34: error: conflicting types for ?setsockopt?
>   WINSOCK_API_LINKAGE int WSAAPI setsockopt(SOCKET s,int level,int optname,const char *optval,int optlen);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:40:7: note: previous declaration of ?setsockopt? was here
>   int setsockopt (int __s, int __level, int __optname, const void *optval,
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1000:34: error: conflicting types for ?shutdown?
>   WINSOCK_API_LINKAGE int WSAAPI shutdown(SOCKET s,int how);
>                                  ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:44:7: note: previous declaration of ?shutdown? was here
>   int shutdown (int, int);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1001:37: error: conflicting types for ?socket?
>   WINSOCK_API_LINKAGE SOCKET WSAAPI socket(int af,int type,int protocol);
>                                     ^
> In file included from sopen.c:34:0:
> /usr/include/sys/socket.h:45:7: note: previous declaration of ?socket? was here
>   int socket (int __family, int __type, int __protocol);
>       ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1002:46: error: conflicting types for ?gethostbyaddr?
>   WINSOCK_API_LINKAGE struct hostent *WSAAPI gethostbyaddr(const char *addr,int len,int type);
>                                              ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:217:17: note: previous declaration of ?gethostbyaddr? was here
> struct hostent *gethostbyaddr (const char *, int, int);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1003:46: error: conflicting types for ?gethostbyname?
>   WINSOCK_API_LINKAGE struct hostent *WSAAPI gethostbyname(const char *name);
>                                              ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:218:17: note: previous declaration of ?gethostbyname? was here
> struct hostent *gethostbyname (const char *);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1004:34: error: conflicting types for ?gethostname?
>   WINSOCK_API_LINKAGE int WSAAPI gethostname(char *name,int namelen);
>                                  ^
> In file included from /usr/include/stdio.h:29:0,
>                 from /cygdrive/d/R2015a/petsc-3.6.2/include/petscsys.h:138,
>                 from sopen.c:14:
> /usr/include/sys/unistd.h:249:6: note: previous declaration of ?gethostname? was here
>  int _EXFUN(gethostname, (char *__name, size_t __len));
>      ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1005:46: error: conflicting types for ?getservbyport?
>   WINSOCK_API_LINKAGE struct servent *WSAAPI getservbyport(int port,const char *proto);
>                                              ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:228:17: note: previous declaration of ?getservbyport? was here
> struct servent *getservbyport (int, const char *);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1006:46: error: conflicting types for ?getservbyname?
>   WINSOCK_API_LINKAGE struct servent *WSAAPI getservbyname(const char *name,const char *proto);
>                                              ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:227:17: note: previous declaration of ?getservbyname? was here
> struct servent *getservbyname (const char *, const char *);
>                 ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1007:47: error: conflicting types for ?getprotobynumber?
>   WINSOCK_API_LINKAGE struct protoent *WSAAPI getprotobynumber(int number);
>                                               ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:225:18: note: previous declaration of ?getprotobynumber? was here
> struct protoent *getprotobynumber (int);
>                  ^
> In file included from /usr/include/w32api/Ws2tcpip.h:17:0,
>                 from sopen.c:58:
> /usr/include/w32api/winsock2.h:1008:47: error: conflicting types for ?getprotobyname?
>   WINSOCK_API_LINKAGE struct protoent *WSAAPI getprotobyname(const char *name);
>                                               ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:224:18: note: previous declaration of ?getprotobyname? was here
> struct protoent *getprotobyname (const char *);
>                  ^
> In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
>                 from sopen.c:58:
> /usr/include/w32api/ws2ipdef.h:23:16: error: redefinition of ?struct ipv6_mreq?
> typedef struct ipv6_mreq {
>                ^
> In file included from /usr/include/cygwin/in.h:267:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/cygwin/in6.h:88:8: note: originally defined here
> struct ipv6_mreq
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
>                 from sopen.c:58:
> /usr/include/w32api/ws2ipdef.h:41:8: error: redefinition of ?struct sockaddr_in6?
> struct sockaddr_in6 {
>        ^
> In file included from /usr/include/cygwin/in.h:267:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/cygwin/in6.h:112:8: note: originally defined here
> struct sockaddr_in6
>        ^
> In file included from /usr/include/sys/socket.h:16:0,
>                 from sopen.c:34:
> /usr/include/w32api/ws2ipdef.h:64:3: error: expected identifier before numeric constant
>   MCAST_INCLUDE   = 0,
>   ^
> In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
>                 from sopen.c:58:
> /usr/include/w32api/ws2ipdef.h:79:16: error: redefinition of ?struct group_filter?
> typedef struct group_filter {
>                ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:174:8: note: originally defined here
> struct group_filter
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
>                 from sopen.c:58:
> /usr/include/w32api/ws2ipdef.h:87:16: error: redefinition of ?struct group_req?
> typedef struct group_req {
>                ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:161:8: note: originally defined here
> struct group_req
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:18:0,
>                 from sopen.c:58:
> /usr/include/w32api/ws2ipdef.h:92:16: error: redefinition of ?struct group_source_req?
> typedef struct group_source_req {
>                ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:167:8: note: originally defined here
> struct group_source_req
>        ^
> In file included from /usr/include/w32api/Ws2tcpip.h:19:0,
>                 from sopen.c:58:
> /usr/include/w32api/psdk_inc/_ip_mreq1.h:12:8: error: redefinition of ?struct ip_mreq?
> struct ip_mreq {
>        ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:127:8: note: originally defined here
> struct ip_mreq
>        ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:22:8: error: redefinition of ?struct ip_mreq_source?
> struct ip_mreq_source {
>        ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:133:8: note: originally defined here
> struct ip_mreq_source
>        ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:28:8: error: redefinition of ?struct ip_msfilter?
> struct ip_msfilter {
>        ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:140:8: note: originally defined here
> struct ip_msfilter
>        ^
> In file included from /usr/include/cygwin/in.h:267:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/w32api/Ws2tcpip.h:86:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:86:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:86:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:87:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:87:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:87:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:88:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:88:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:88:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MULTICAST(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:89:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:89:5: error: expected ?)? before ?==? token
> int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:90:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:90:5: error: expected ?)? before ?==? token
> int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:91:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:91:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:91:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:92:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:92:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:92:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:93:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:93:5: error: expected ?)? before ?&&? token
> int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:94:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:94:5: error: expected ?)? before ?&&? token
> int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:95:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:95:5: error: expected ?)? before ?&&? token
> int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:96:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:96:5: error: expected ?)? before ?&&? token
> int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:97:5: error: expected identifier or ?(? before ?const?
> int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?(? token
> int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?[? token
> int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:97:5: error: expected ?)? before ?&&? token
> int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *);
>     ^
> /usr/include/w32api/Ws2tcpip.h:105:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:105:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:105:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_UNSPECIFIED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:106:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:106:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:106:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_LOOPBACK(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && (a->s6_words[6]==0) && (a->s6_words[7]==0x0100)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:107:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:107:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:107:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MULTICAST(const struct in6_addr *a) { return (a->s6_bytes[0]==0xff); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:108:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:108:21: error: expected ?)? before ?==? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_LINKLOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0x80)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:109:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:109:21: error: expected ?)? before ?==? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_SITELOCAL(const struct in6_addr *a) { return ((a->s6_bytes[0]==0xfe) && ((a->s6_bytes[1] & 0xc0)==0xc0)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:110:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:110:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:110:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4MAPPED(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0xffff)); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:111:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:111:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:111:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_V4COMPAT(const struct in6_addr *a) { return ((a->s6_words[0]==0) && (a->s6_words[1]==0) && (a->s6_words[2]==0) && (a->s6_words[3]==0) && (a->s6_words[4]==0) && (a->s6_words[5]==0) && !((a->s6_words[6]==0) && (a->s6_addr[14]==0) && ((a->s6_addr[15]==0) || (a->s6_addr[15]==1)))); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:112:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:112:21: error: expected ?)? before ?&&? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_NODELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==1); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:113:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:113:21: error: expected ?)? before ?&&? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_LINKLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==2); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:114:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:114:21: error: expected ?)? before ?&&? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_SITELOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==5); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:115:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:115:21: error: expected ?)? before ?&&? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_ORGLOCAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==8); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:116:21: error: expected identifier or ?(? before ?const?
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?(? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?[? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
>                     ^
> /usr/include/w32api/Ws2tcpip.h:116:21: error: expected ?)? before ?&&? token
> WS2TCPIP_INLINE int IN6_IS_ADDR_MC_GLOBAL(const struct in6_addr *a) { return IN6_IS_ADDR_MULTICAST(a) && ((a->s6_bytes[1] & 0xf)==0xe); }
>                     ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h: In function ?IN6_SET_ADDR_UNSPECIFIED?:
> /usr/include/w32api/Ws2tcpip.h:119:77: error: ?struct in6_addr? has no member named ?s6_bytes?
> WS2TCPIP_INLINE void IN6_SET_ADDR_UNSPECIFIED(struct in6_addr *a) { memset(a->s6_bytes,0,sizeof(struct in6_addr)); }
>                                                                             ^
> /usr/include/w32api/Ws2tcpip.h: In function ?IN6_SET_ADDR_LOOPBACK?:
> /usr/include/w32api/Ws2tcpip.h:121:11: error: ?struct in6_addr? has no member named ?s6_bytes?
>   memset(a->s6_bytes,0,sizeof(struct in6_addr));
>           ^
> /usr/include/w32api/Ws2tcpip.h:122:4: error: ?struct in6_addr? has no member named ?s6_bytes?
>   a->s6_bytes[15] = 1;
>    ^
> /usr/include/w32api/Ws2tcpip.h: At top level:
> /usr/include/w32api/Ws2tcpip.h:156:16: error: redefinition of ?struct in_pktinfo?
> typedef struct in_pktinfo {
>                ^
> In file included from /usr/include/netinet/in.h:14:0,
>                 from sopen.c:40:
> /usr/include/cygwin/in.h:153:8: note: originally defined here
> struct in_pktinfo
>        ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:157:3: error: unknown type name ?IN_ADDR?
>   IN_ADDR ipi_addr;
>   ^
> /usr/include/w32api/Ws2tcpip.h:163:16: error: redefinition of ?struct in6_pktinfo?
> typedef struct in6_pktinfo {
>                ^
> In file included from /usr/include/cygwin/in.h:267:0,
>                 from /usr/include/netinet/in.h:14,
>                 from sopen.c:40:
> /usr/include/cygwin/in6.h:102:8: note: originally defined here
> struct in6_pktinfo
>        ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:164:3: error: unknown type name ?IN6_ADDR?
>   IN6_ADDR ipi6_addr;
>   ^
> /usr/include/w32api/Ws2tcpip.h:168:1: error: size of unnamed array is negative
> C_ASSERT(sizeof(IN6_PKTINFO)==20);
> ^
> /usr/include/w32api/Ws2tcpip.h:182:16: error: redefinition of ?struct addrinfo?
> typedef struct addrinfo {
>                ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:121:8: note: originally defined here
> struct addrinfo {
>        ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:231:34: error: conflicting types for ?getaddrinfo?
>   WINSOCK_API_LINKAGE int WSAAPI getaddrinfo(const char *nodename,const char *servname,const struct addrinfo *hints,struct addrinfo **res);
>                                  ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:242:6: note: previous declaration of ?getaddrinfo? was here
> int  getaddrinfo (const char *, const char *,
>      ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:247:35: error: conflicting types for ?freeaddrinfo?
>   WINSOCK_API_LINKAGE void WSAAPI freeaddrinfo(LPADDRINFO pAddrInfo);
>                                   ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:240:7: note: previous declaration of ?freeaddrinfo? was here
> void  freeaddrinfo (struct addrinfo *);
>       ^
> In file included from sopen.c:58:0:
> /usr/include/w32api/Ws2tcpip.h:265:34: error: conflicting types for ?getnameinfo?
>   WINSOCK_API_LINKAGE int WSAAPI getnameinfo(const struct sockaddr *sa,socklen_t salen,char *host,DWORD hostlen,char *serv,DWORD servlen,int flags);
>                                  ^
> In file included from sopen.c:43:0:
> /usr/include/netdb.h:244:6: note: previous declaration of ?getnameinfo? was here
> int  getnameinfo (const struct sockaddr *, socklen_t, char *,
>      ^
> In file included from /usr/include/w32api/Ws2tcpip.h:298:0,
>                 from sopen.c:58:
> /usr/include/w32api/mstcpip.h:110:43: error: unknown type name ?IN6_ADDR?
> LPSTR NTAPI RtlIpv6AddressToStringA(const IN6_ADDR *Addr, LPSTR S);
>                                           ^
> /usr/include/w32api/mstcpip.h:111:44: error: unknown type name ?IN6_ADDR?
> LPWSTR NTAPI RtlIpv6AddressToStringW(const IN6_ADDR *Addr, LPWSTR S);
>                                            ^
> /usr/include/w32api/mstcpip.h:113:44: error: unknown type name ?IN6_ADDR?
> LONG NTAPI RtlIpv6AddressToStringExA(const IN6_ADDR *Address, ULONG ScopeId, USHORT Port, LPSTR AddressString, PULONG AddressStringLength);
>                                            ^
> In file included from /usr/include/w32api/Ws2tcpip.h:298:0,
>                 from sopen.c:58:
> /usr/include/w32api/mstcpip.h:114:44: error: unknown type name ?IN6_ADDR?
> LONG NTAPI RtlIpv6AddressToStringExW(const IN6_ADDR *Address, ULONG ScopeId, USHORT Port, LPWSTR AddressString, PULONG AddressStringLength);
>                                            ^
> /usr/include/w32api/mstcpip.h:117:43: error: unknown type name ?IN_ADDR?
> LPSTR NTAPI RtlIpv4AddressToStringA(const IN_ADDR *Addr, LPSTR S);
>                                           ^
> /usr/include/w32api/mstcpip.h:118:44: error: unknown type name ?IN_ADDR?
> LPWSTR NTAPI RtlIpv4AddressToStringW(const IN_ADDR *Addr, LPWSTR S);
>                                            ^
> /usr/include/w32api/mstcpip.h:121:44: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4AddressToStringExA(const IN_ADDR *Address, USHORT Port, LPSTR AddressString, PULONG AddressStringLength);
>                                            ^
> /usr/include/w32api/mstcpip.h:122:44: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4AddressToStringExW(const IN_ADDR *Address, USHORT Port, LPWSTR AddressString, PULONG AddressStringLength);
>                                            ^
> /usr/include/w32api/mstcpip.h:125:80: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4StringToAddressA(PCSTR S, BOOLEAN Strict, LPSTR *Terminator, IN_ADDR *Addr);
>                                                                                ^
> /usr/include/w32api/mstcpip.h:126:82: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4StringToAddressW(PCWSTR S, BOOLEAN Strict, LPWSTR *Terminator, IN_ADDR *Addr);
>                                                                                  ^
> /usr/include/w32api/mstcpip.h:129:75: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4StringToAddressExA(PCSTR AddressString, BOOLEAN Strict, IN_ADDR *Address, PUSHORT Port);
>                                                                           ^
> /usr/include/w32api/mstcpip.h:130:76: error: unknown type name ?IN_ADDR?
> LONG NTAPI RtlIpv4StringToAddressExW(PCWSTR AddressString, BOOLEAN Strict, IN_ADDR *Address, PUSHORT Port);
>                                                                            ^
> /usr/include/w32api/mstcpip.h:133:59: error: unknown type name ?IN6_ADDR?
> LONG NTAPI RtlIpv6StringToAddressExA(PCSTR AddressString, IN6_ADDR *Address, PULONG ScopeId, PUSHORT Port);
>                                                           ^
> /usr/include/w32api/mstcpip.h:134:60: error: unknown type name ?IN6_ADDR?
> LONG NTAPI RtlIpv6StringToAddressExW(PCWSTR AddressString, IN6_ADDR *Address, PULONG ScopeId, PUSHORT Port);
>                                                            ^
> sopen.c: In function ?establish?:
> sopen.c:116:44: error: ?namelen? undeclared (first use in this function)
>   GetComputerName((LPTSTR)myname,(LPDWORD)&namelen);
>                                            ^
> sopen.c:116:44: note: each undeclared identifier is reported only once for each function it appears in
> 
>  D:\R2015A\BIN\MEX.PL: Error: Compile of 'sopen.c' failed.
> 
> ************
> 
> Sincerely yours,
> Alexander
> 
> -------------------------
> Dr. Alexander KOROVIN
> 
> Lab. G?nie ?lectrique et ?lectronique - Paris (GeePs)
> 11 rue Joliot-Curie
> Plateau de Moulon
> F-91192 Gif-sur-Yvette cedex
> France
> +33-1-69-85-16-83
> 


From bsmith at mcs.anl.gov  Thu Oct  8 14:41:13 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 8 Oct 2015 14:41:13 -0500
Subject: [petsc-users] user defined algorithm for newton solver
In-Reply-To: <5616B068.7000008@purdue.edu>
References: <5616A738.50704@purdue.edu>
	<E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>
	<5616B068.7000008@purdue.edu>
Message-ID: <6B661431-3D1A-4108-944B-B847BC863C30@mcs.anl.gov>


   The easiest way for you to implement this is to simply call your update to the solution and then call SNESSolve() again. For example

   while (1) {
     SNESSolve(snes,x,NULL);
     SNESGetConvergedReason(snes,&reason);
     if (reason > 0) break;  /* since snes has converged */
     change x per your way
  }

The problem with the code above is that each new call to SNESSolve() resets the rtol convergence factor based on the current latest norm of F so the code above will "oversolve" the problem. You can call SNESSetTolerance() above the while loop with an appropriate atol to get it to exit at the point you want to declare it converged.

  It is not worthwhile trying to "weave" your special update code inside the Newton method.

   Barry

> On Oct 8, 2015, at 1:05 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
> 
> Thank you.
> The situation is as follows:
> The system I need to solve does not have a unique solution, but only one makes sense from physical point of view.  I need to compute direction in a different way based on the physics.
> 
> In other words it should be like this:
> 
> 1. Start with a solution guess
> 2. Do full Newton step
> 3. If converged, exit
>    else if the solution improved go to step 2
>    otherwise "update_solution_in_my_way" ang go to step 2
> 
> Is it possible to do this in PETSc?
> Michael.
> 
> On 10/08/2015 01:58 PM, Barry Smith wrote:
>>> On Oct 8, 2015, at 12:26 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
>>> 
>>> Dear Petsc developers and users,
>>> I'm solving a nonlinear system with PETSc.
>>> Often simple Newton iterations do not work and I have to use either linear search or trust region.
>>> I would like to use my own algorithm to find a next iteration approximation to solution if the Newton step does not improve the residual. As far as I can see I have to define my own SNELLineSearch object. Is there any example that shows how to do it?
>>    The line search model is 1) select direction based on approximate solution of approximate Jacobian  2) search in THAT direction for a decrease in the function evaluation. You state "if the Newton step does not improve the residual." so will you be computing the DIRECTION in a different way than (1) or will you be using the same direction but trying somehow to find a decrease in the function evaluation using a different technique then the standard line searchs we provide?
>> 
>>    Frankly if the line searches we provide don't work that means the direction is not good and no "special" line search will recover.  I really recommend you go through our suggestions on the FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#newton on trying to figure out why Newton is not converging before you think about writing a special line search.
>> 
>>   Barry
>> 
>> 
>> 
>>> Thank you,
>>> Michael.
>>> 
>>> -- 
>>> Michael Povolotskyi, PhD
>>> Research Assistant Professor
>>> Network for Computational Nanotechnology
>>> Hall for Discover and Learning Research, Room 441
>>> West Lafayette, IN 47907
>>> Phone (765) 4949396
>>> 
> 
> -- 
> Michael Povolotskyi, PhD
> Research Assistant Professor
> Network for Computational Nanotechnology
> Hall for Discover and Learning Research, Room 441
> West Lafayette, IN 47907
> Phone (765) 4949396
> 


From knepley at gmail.com  Thu Oct  8 14:44:17 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 8 Oct 2015 14:44:17 -0500
Subject: [petsc-users] user defined algorithm for newton solver
In-Reply-To: <6B661431-3D1A-4108-944B-B847BC863C30@mcs.anl.gov>
References: <5616A738.50704@purdue.edu>
	<E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>
	<5616B068.7000008@purdue.edu>
	<6B661431-3D1A-4108-944B-B847BC863C30@mcs.anl.gov>
Message-ID: <CAMYG4GkFp5cKM16_F3UufaC0=Q5MgJeHDouRzUXH8tKoKiOCkg@mail.gmail.com>

On Thu, Oct 8, 2015 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    The easiest way for you to implement this is to simply call your update
> to the solution and then call SNESSolve() again. For example
>
>    while (1) {
>      SNESSolve(snes,x,NULL);
>      SNESGetConvergedReason(snes,&reason);
>      if (reason > 0) break;  /* since snes has converged */
>      change x per your way
>   }
>
> The problem with the code above is that each new call to SNESSolve()
> resets the rtol convergence factor based on the current latest norm of F so
> the code above will "oversolve" the problem. You can call
> SNESSetTolerance() above the while loop with an appropriate atol to get it
> to exit at the point you want to declare it converged.
>
>   It is not worthwhile trying to "weave" your special update code inside
> the Newton method.


Do you know what the non-physical solutions are?

   Matt


>
>    Barry
>
> > On Oct 8, 2015, at 1:05 PM, Michael Povolotskyi <mpovolot at purdue.edu>
> wrote:
> >
> > Thank you.
> > The situation is as follows:
> > The system I need to solve does not have a unique solution, but only one
> makes sense from physical point of view.  I need to compute direction in a
> different way based on the physics.
> >
> > In other words it should be like this:
> >
> > 1. Start with a solution guess
> > 2. Do full Newton step
> > 3. If converged, exit
> >    else if the solution improved go to step 2
> >    otherwise "update_solution_in_my_way" ang go to step 2
> >
> > Is it possible to do this in PETSc?
> > Michael.
> >
> > On 10/08/2015 01:58 PM, Barry Smith wrote:
> >>> On Oct 8, 2015, at 12:26 PM, Michael Povolotskyi <mpovolot at purdue.edu>
> wrote:
> >>>
> >>> Dear Petsc developers and users,
> >>> I'm solving a nonlinear system with PETSc.
> >>> Often simple Newton iterations do not work and I have to use either
> linear search or trust region.
> >>> I would like to use my own algorithm to find a next iteration
> approximation to solution if the Newton step does not improve the residual.
> As far as I can see I have to define my own SNELLineSearch object. Is there
> any example that shows how to do it?
> >>    The line search model is 1) select direction based on approximate
> solution of approximate Jacobian  2) search in THAT direction for a
> decrease in the function evaluation. You state "if the Newton step does not
> improve the residual." so will you be computing the DIRECTION in a
> different way than (1) or will you be using the same direction but trying
> somehow to find a decrease in the function evaluation using a different
> technique then the standard line searchs we provide?
> >>
> >>    Frankly if the line searches we provide don't work that means the
> direction is not good and no "special" line search will recover.  I really
> recommend you go through our suggestions on the FAQ
> http://www.mcs.anl.gov/petsc/documentation/faq.html#newton on trying to
> figure out why Newton is not converging before you think about writing a
> special line search.
> >>
> >>   Barry
> >>
> >>
> >>
> >>> Thank you,
> >>> Michael.
> >>>
> >>> --
> >>> Michael Povolotskyi, PhD
> >>> Research Assistant Professor
> >>> Network for Computational Nanotechnology
> >>> Hall for Discover and Learning Research, Room 441
> >>> West Lafayette, IN 47907
> >>> Phone (765) 4949396
> >>>
> >
> > --
> > Michael Povolotskyi, PhD
> > Research Assistant Professor
> > Network for Computational Nanotechnology
> > Hall for Discover and Learning Research, Room 441
> > West Lafayette, IN 47907
> > Phone (765) 4949396
> >
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151008/594cd0f0/attachment.html>

From valencia at astro.utoronto.ca  Thu Oct  8 15:26:56 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Thu, 8 Oct 2015 16:26:56 -0400
Subject: [petsc-users] ld: cannot find -lpetsc error
Message-ID: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>

Dear PETSc users,

In hopes that one of you can help me solve a compiling problem.

I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
After compiling all the subroutines, I get an error building the code. Here is the stout:

I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.

Thank you in advance for all your help.
Diana


.
.
.
mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
ld: cannot find -lpetsc
make: *** [stagyy] Error 1
mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
ld: cannot find -lpetsc
make: *** [stagyympi] Error 1
make: Target `both' not remade because of errors.

_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151008/7200ea34/attachment.html>

From balay at mcs.anl.gov  Thu Oct  8 15:31:30 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Thu, 8 Oct 2015 15:31:30 -0500
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
Message-ID: <alpine.LFD.2.20.1510081529520.3050@asterix>

1. Are you able to compile [using petsc makefile] and run PETSc examples?

2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib

[you should verify the output from the example compile/link commands -
and the compile/link command for this application.]

BTW: This is an old version of petsc.

Satish

On Thu, 8 Oct 2015, Diana Valencia wrote:

> Dear PETSc users,
> 
> In hopes that one of you can help me solve a compiling problem.
> 
> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
> After compiling all the subroutines, I get an error building the code. Here is the stout:
> 
> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
> 
> Thank you in advance for all your help.
> Diana
> 
> 
> .
> .
> .
> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> ld: cannot find -lpetsc
> make: *** [stagyy] Error 1
> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> ld: cannot find -lpetsc
> make: *** [stagyympi] Error 1
> make: Target `both' not remade because of errors.
> 
> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences 
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986
> 
> 

From filippo.leonardi at sam.math.ethz.ch  Fri Oct  9 05:04:53 2015
From: filippo.leonardi at sam.math.ethz.ch (Filippo Leonardi)
Date: Fri, 9 Oct 2015 12:04:53 +0200
Subject: [petsc-users] Compute initial guess multiple times
Message-ID: <1786668.RPbRGasB0H@besikowitch-iii>

So, I want to solve a system Ax = b multiple times, reuse A with different bs. 
It is natural to set differents initial guesses each time for iterative 
solvers.

And 
KSPSetComputeInitialGuess(ksp,ComputeGuess,&user);
is what PETSc provide for setting the Guess.

I am unsure if I am making something wrong, but it seems to me that this 
function is called only once the matrix is first setted up. I want to be 
called each time.

See my modified ex32.c for comparison.

./ex32 -pc_type mg -pc_mg_type full -ksp_type fgmres -ksp_monitor_short -
pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero -
ksp_initial_guess_nonzero

Thanks.

Best,
Filippo
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex32.c
Type: text/x-csrc
Size: 8151 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151009/76ca8f16/attachment.c>

From mpovolot at purdue.edu  Fri Oct  9 12:45:30 2015
From: mpovolot at purdue.edu (Michael Povolotskyi)
Date: Fri, 09 Oct 2015 13:45:30 -0400
Subject: [petsc-users] user defined algorithm for newton solver
In-Reply-To: <CAMYG4GkFp5cKM16_F3UufaC0=Q5MgJeHDouRzUXH8tKoKiOCkg@mail.gmail.com>
References: <5616A738.50704@purdue.edu>	<E315F9F2-BB6E-4B1C-BAF0-FB7CF998541D@mcs.anl.gov>	<5616B068.7000008@purdue.edu>	<6B661431-3D1A-4108-944B-B847BC863C30@mcs.anl.gov>
	<CAMYG4GkFp5cKM16_F3UufaC0=Q5MgJeHDouRzUXH8tKoKiOCkg@mail.gmail.com>
Message-ID: <5617FD3A.4080707@purdue.edu>

thank you,
I'll try to do that.

Yes, i have some criteria that can tell that the solution is a 
non-physical one.
Basically what I'm trying to do is very similar to a predictor corrector 
scheme.
Predictor should move solution to the right direction.

On 10/08/2015 03:44 PM, Matthew Knepley wrote:
> On Thu, Oct 8, 2015 at 2:41 PM, Barry Smith <bsmith at mcs.anl.gov 
> <mailto:bsmith at mcs.anl.gov>> wrote:
>
>
>        The easiest way for you to implement this is to simply call
>     your update to the solution and then call SNESSolve() again. For
>     example
>
>        while (1) {
>          SNESSolve(snes,x,NULL);
>          SNESGetConvergedReason(snes,&reason);
>          if (reason > 0) break;  /* since snes has converged */
>          change x per your way
>       }
>
>     The problem with the code above is that each new call to
>     SNESSolve() resets the rtol convergence factor based on the
>     current latest norm of F so the code above will "oversolve" the
>     problem. You can call SNESSetTolerance() above the while loop with
>     an appropriate atol to get it to exit at the point you want to
>     declare it converged.
>
>       It is not worthwhile trying to "weave" your special update code
>     inside the Newton method.
>
>
> Do you know what the non-physical solutions are?
>
>    Matt
>
>
>        Barry
>
>     > On Oct 8, 2015, at 1:05 PM, Michael Povolotskyi
>     <mpovolot at purdue.edu <mailto:mpovolot at purdue.edu>> wrote:
>     >
>     > Thank you.
>     > The situation is as follows:
>     > The system I need to solve does not have a unique solution, but
>     only one makes sense from physical point of view.  I need to
>     compute direction in a different way based on the physics.
>     >
>     > In other words it should be like this:
>     >
>     > 1. Start with a solution guess
>     > 2. Do full Newton step
>     > 3. If converged, exit
>     >    else if the solution improved go to step 2
>     >    otherwise "update_solution_in_my_way" ang go to step 2
>     >
>     > Is it possible to do this in PETSc?
>     > Michael.
>     >
>     > On 10/08/2015 01:58 PM, Barry Smith wrote:
>     >>> On Oct 8, 2015, at 12:26 PM, Michael Povolotskyi
>     <mpovolot at purdue.edu <mailto:mpovolot at purdue.edu>> wrote:
>     >>>
>     >>> Dear Petsc developers and users,
>     >>> I'm solving a nonlinear system with PETSc.
>     >>> Often simple Newton iterations do not work and I have to use
>     either linear search or trust region.
>     >>> I would like to use my own algorithm to find a next iteration
>     approximation to solution if the Newton step does not improve the
>     residual. As far as I can see I have to define my own
>     SNELLineSearch object. Is there any example that shows how to do it?
>     >>    The line search model is 1) select direction based on
>     approximate solution of approximate Jacobian  2) search in THAT
>     direction for a decrease in the function evaluation. You state "if
>     the Newton step does not improve the residual." so will you be
>     computing the DIRECTION in a different way than (1) or will you be
>     using the same direction but trying somehow to find a decrease in
>     the function evaluation using a different technique then the
>     standard line searchs we provide?
>     >>
>     >>    Frankly if the line searches we provide don't work that
>     means the direction is not good and no "special" line search will
>     recover.  I really recommend you go through our suggestions on the
>     FAQ http://www.mcs.anl.gov/petsc/documentation/faq.html#newton on
>     trying to figure out why Newton is not converging before you think
>     about writing a special line search.
>     >>
>     >>   Barry
>     >>
>     >>
>     >>
>     >>> Thank you,
>     >>> Michael.
>     >>>
>     >>> --
>     >>> Michael Povolotskyi, PhD
>     >>> Research Assistant Professor
>     >>> Network for Computational Nanotechnology
>     >>> Hall for Discover and Learning Research, Room 441
>     >>> West Lafayette, IN 47907
>     >>> Phone (765) 4949396 <tel:%28765%29%204949396>
>     >>>
>     >
>     > --
>     > Michael Povolotskyi, PhD
>     > Research Assistant Professor
>     > Network for Computational Nanotechnology
>     > Hall for Discover and Learning Research, Room 441
>     > West Lafayette, IN 47907
>     > Phone (765) 4949396 <tel:%28765%29%204949396>
>     >
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-- 
Michael Povolotskyi, PhD
Research Assistant Professor
Network for Computational Nanotechnology
Hall for Discover and Learning Research, Room 441
West Lafayette, IN 47907
Phone (765) 4949396

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151009/2562d7c6/attachment.html>

From bsmith at mcs.anl.gov  Fri Oct  9 13:49:25 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 9 Oct 2015 13:49:25 -0500
Subject: [petsc-users] Compute initial guess multiple times
In-Reply-To: <1786668.RPbRGasB0H@besikowitch-iii>
References: <1786668.RPbRGasB0H@besikowitch-iii>
Message-ID: <F8233B8B-DDEA-47BE-8669-9C7E1D14DB0C@mcs.anl.gov>


  The current model is a bit wonky.


    At the first call to KSPSolve after a call to KSPSetComputeOperators the computematrix, computerhs, and computeinitialguess are called. 

    In subsequent calls to KSPSolve ONLY the computerhs is called.

    So for example

      KSPSetComputeOperators()
      KSPSolve   -- computematrix, rhs, initial guess called
      KSPSolve  --- computerhs called
      KSPSetComputerOperators()
      KSPSolve   -- computematrix, rhs, initial guess called


   Note that the KSPSetCompute...() functions are just one particular way of using KSP.  You can also use the "standard" interface

     Set values into matrix and rhs and initial guess (call KSPSetInitialGuessNonzero) 
     KSPSolve(ksp,b,x)
     change values in b, this would use the current values in x as the initial guess for the next solve
     KSPSolve(ksp,b,x)
     change values in b and x
     KSPSolve()
     Change values in matrix and b and x (for initial guess)
     KSPSolve() 

    This gives you exact detailed control over when and what you set in initial guesses.

    Is there a particular reason for your problem that it is better to use the KSPSetCompute...() interface?

   Barry

  Note I wrote the KSPSetCompute...() interface so that PCMG could "build its own" coarse grid matrix values without the user needing to set values for all the levels before calling KSPSolve.


   
> On Oct 9, 2015, at 5:04 AM, Filippo Leonardi <filippo.leonardi at sam.math.ethz.ch> wrote:
> 
> So, I want to solve a system Ax = b multiple times, reuse A with different bs. 
> It is natural to set differents initial guesses each time for iterative 
> solvers.
> 
> And 
> KSPSetComputeInitialGuess(ksp,ComputeGuess,&user);
> is what PETSc provide for setting the Guess.
> 
> I am unsure if I am making something wrong, but it seems to me that this 
> function is called only once the matrix is first setted up. I want to be 
> called each time.
> 
> See my modified ex32.c for comparison.
> 
> ./ex32 -pc_type mg -pc_mg_type full -ksp_type fgmres -ksp_monitor_short -
> pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero -
> ksp_initial_guess_nonzero
> 
> Thanks.
> 
> Best,
> Filippo<ex32.c>


From filippo.leonardi at sam.math.ethz.ch  Fri Oct  9 13:58:57 2015
From: filippo.leonardi at sam.math.ethz.ch (Leonardi  Filippo)
Date: Fri, 9 Oct 2015 18:58:57 +0000
Subject: [petsc-users] Compute initial guess multiple times
In-Reply-To: <F8233B8B-DDEA-47BE-8669-9C7E1D14DB0C@mcs.anl.gov>
References: <1786668.RPbRGasB0H@besikowitch-iii>,
	<F8233B8B-DDEA-47BE-8669-9C7E1D14DB0C@mcs.anl.gov>
Message-ID: <F7BAED866D326642BD7DE49EB610754F441830BF@MBX212.d.ethz.ch>

Actually I used to use KSPSolve directly, then I've started using KSPSetCompute... because (if I remember correctly) (as you mention) I wanted KSPComputeMatrix to compute all the coarse matrices for me.

Now I switched to completely use algebraic multigrid or galerkin computed matrices, so I can revert back to use the KSPSolve directly, but, for testing purposes, it may be good to keep being able to use PCMG.

My question now is: can I use SetComputeMatrix() and then call KSPSolve(ksp,b,x)? (I forgot if i need to use b,x = NULL or not when using the compute matrix routine.)
________________________________________
Da: Barry Smith [bsmith at mcs.anl.gov]
Inviato: venerd? 9 ottobre 2015 20.49
A: Leonardi  Filippo
Cc: petsc-users at mcs.anl.gov
Oggetto: Re: [petsc-users] Compute initial guess multiple times

  The current model is a bit wonky.


    At the first call to KSPSolve after a call to KSPSetComputeOperators the computematrix, computerhs, and computeinitialguess are called.

    In subsequent calls to KSPSolve ONLY the computerhs is called.

    So for example

      KSPSetComputeOperators()
      KSPSolve   -- computematrix, rhs, initial guess called
      KSPSolve  --- computerhs called
      KSPSetComputerOperators()
      KSPSolve   -- computematrix, rhs, initial guess called


   Note that the KSPSetCompute...() functions are just one particular way of using KSP.  You can also use the "standard" interface

     Set values into matrix and rhs and initial guess (call KSPSetInitialGuessNonzero)
     KSPSolve(ksp,b,x)
     change values in b, this would use the current values in x as the initial guess for the next solve
     KSPSolve(ksp,b,x)
     change values in b and x
     KSPSolve()
     Change values in matrix and b and x (for initial guess)
     KSPSolve()

    This gives you exact detailed control over when and what you set in initial guesses.

    Is there a particular reason for your problem that it is better to use the KSPSetCompute...() interface?

   Barry

  Note I wrote the KSPSetCompute...() interface so that PCMG could "build its own" coarse grid matrix values without the user needing to set values for all the levels before calling KSPSolve.



> On Oct 9, 2015, at 5:04 AM, Filippo Leonardi <filippo.leonardi at sam.math.ethz.ch> wrote:
>
> So, I want to solve a system Ax = b multiple times, reuse A with different bs.
> It is natural to set differents initial guesses each time for iterative
> solvers.
>
> And
> KSPSetComputeInitialGuess(ksp,ComputeGuess,&user);
> is what PETSc provide for setting the Guess.
>
> I am unsure if I am making something wrong, but it seems to me that this
> function is called only once the matrix is first setted up. I want to be
> called each time.
>
> See my modified ex32.c for comparison.
>
> ./ex32 -pc_type mg -pc_mg_type full -ksp_type fgmres -ksp_monitor_short -
> pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero -
> ksp_initial_guess_nonzero
>
> Thanks.
>
> Best,
> Filippo<ex32.c>


From bsmith at mcs.anl.gov  Fri Oct  9 14:06:52 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 9 Oct 2015 14:06:52 -0500
Subject: [petsc-users] Compute initial guess multiple times
In-Reply-To: <F7BAED866D326642BD7DE49EB610754F441830BF@MBX212.d.ethz.ch>
References: <1786668.RPbRGasB0H@besikowitch-iii>
	<F8233B8B-DDEA-47BE-8669-9C7E1D14DB0C@mcs.anl.gov>
	<F7BAED866D326642BD7DE49EB610754F441830BF@MBX212.d.ethz.ch>
Message-ID: <E5D0B3A7-DE0F-4E90-9D7E-6158631C89A8@mcs.anl.gov>


> On Oct 9, 2015, at 1:58 PM, Leonardi Filippo <filippo.leonardi at sam.math.ethz.ch> wrote:
> 
> Actually I used to use KSPSolve directly, then I've started using KSPSetCompute... because (if I remember correctly) (as you mention) I wanted KSPComputeMatrix to compute all the coarse matrices for me.
> 
> Now I switched to completely use algebraic multigrid or galerkin computed matrices, so I can revert back to use the KSPSolve directly, but, for testing purposes, it may be good to keep being able to use PCMG.
> 
> My question now is: can I use SetComputeMatrix() and then call KSPSolve(ksp,b,x)? (I forgot if i need to use b,x = NULL or not when using the compute matrix routine.)

   What about providing the ComputeMatrix() but NOT the computerrhs or computeinitialguess?  Make sure to call KSPSetInitialGuessNonzero() so KSP doesn't zero out the x you pass in).

  Barry

> ________________________________________
> Da: Barry Smith [bsmith at mcs.anl.gov]
> Inviato: venerd? 9 ottobre 2015 20.49
> A: Leonardi  Filippo
> Cc: petsc-users at mcs.anl.gov
> Oggetto: Re: [petsc-users] Compute initial guess multiple times
> 
>  The current model is a bit wonky.
> 
> 
>    At the first call to KSPSolve after a call to KSPSetComputeOperators the computematrix, computerhs, and computeinitialguess are called.
> 
>    In subsequent calls to KSPSolve ONLY the computerhs is called.
> 
>    So for example
> 
>      KSPSetComputeOperators()
>      KSPSolve   -- computematrix, rhs, initial guess called
>      KSPSolve  --- computerhs called
>      KSPSetComputerOperators()
>      KSPSolve   -- computematrix, rhs, initial guess called
> 
> 
>   Note that the KSPSetCompute...() functions are just one particular way of using KSP.  You can also use the "standard" interface
> 
>     Set values into matrix and rhs and initial guess (call KSPSetInitialGuessNonzero)
>     KSPSolve(ksp,b,x)
>     change values in b, this would use the current values in x as the initial guess for the next solve
>     KSPSolve(ksp,b,x)
>     change values in b and x
>     KSPSolve()
>     Change values in matrix and b and x (for initial guess)
>     KSPSolve()
> 
>    This gives you exact detailed control over when and what you set in initial guesses.
> 
>    Is there a particular reason for your problem that it is better to use the KSPSetCompute...() interface?
> 
>   Barry
> 
>  Note I wrote the KSPSetCompute...() interface so that PCMG could "build its own" coarse grid matrix values without the user needing to set values for all the levels before calling KSPSolve.
> 
> 
> 
>> On Oct 9, 2015, at 5:04 AM, Filippo Leonardi <filippo.leonardi at sam.math.ethz.ch> wrote:
>> 
>> So, I want to solve a system Ax = b multiple times, reuse A with different bs.
>> It is natural to set differents initial guesses each time for iterative
>> solvers.
>> 
>> And
>> KSPSetComputeInitialGuess(ksp,ComputeGuess,&user);
>> is what PETSc provide for setting the Guess.
>> 
>> I am unsure if I am making something wrong, but it seems to me that this
>> function is called only once the matrix is first setted up. I want to be
>> called each time.
>> 
>> See my modified ex32.c for comparison.
>> 
>> ./ex32 -pc_type mg -pc_mg_type full -ksp_type fgmres -ksp_monitor_short -
>> pc_mg_levels 3 -mg_coarse_pc_factor_shift_type nonzero -
>> ksp_initial_guess_nonzero
>> 
>> Thanks.
>> 
>> Best,
>> Filippo<ex32.c>
> 


From xdavidliu at gmail.com  Sat Oct 10 14:37:27 2015
From: xdavidliu at gmail.com (David Liu)
Date: Sat, 10 Oct 2015 15:37:27 -0400
Subject: [petsc-users] repeatedly direct solving a matrix that changes?
Message-ID: <CAMSHvJG4V5gmztkVqXkA=Vq4YarLHhvgH60GTpG7Xz==NRacOA@mail.gmail.com>

Hi,

I have a matrix A that I am trying to solve (currently using Mumps).
Between solves, I'm changing some of the elements of A, but not changing
the nonzero pattern.

In the past, with Petsc 3.4 and below, I was able to pass options like
SAME_NONZERO_PATTERN and SAME_PRECONDITIONER to KSPSetOperators. By
fiddling around with this (I don't recall exactly how), as well as
re-calling KSPSetOperators between solves (iirc), I was able to switch at
will between one of two things:

1) re-doing the LU factorization at each solve, so that the iterative part
of the KSPSolve takes only one iteration to converge (because the
"preconditioner" is just the inverse matrix) (I believe that's what it's
doing, sorry if I am butchering the nomenclature)

2) doing the LU factorization on the first solve only, so that for the
subsequent solves, the LU factorized matrix from the FIRST solve is used,
so now the iterative part takes more than 1 iteration to converge.

I don't recall exactly the exact steps I took to switch between 1) and 2);
it had something to do with the very last option passed to KSPSetOperators.
All I know is that option 2) actually worked fine for me because my A
matrix was only changing slightly between solves.

However, now I updated to 3.5, and it looks like that very last option to
KSPSetOperators has been removed, as recorded in the following commit on
bitbucket:
https://bitbucket.org/petsc/petsc/issues/50/eliminate-the-need-for-matstructure-flag

Now it looks like my code is exclusively doing 2) (if I am not mistaken),
even though I am currently repeatedly calling KSPSetOperators between
solves. How do I get my code to do 1)?

thanks
best,
David Liu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151010/2c5521da/attachment.html>

From bsmith at mcs.anl.gov  Sat Oct 10 14:50:57 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 10 Oct 2015 14:50:57 -0500
Subject: [petsc-users] repeatedly direct solving a matrix that changes?
In-Reply-To: <CAMSHvJG4V5gmztkVqXkA=Vq4YarLHhvgH60GTpG7Xz==NRacOA@mail.gmail.com>
References: <CAMSHvJG4V5gmztkVqXkA=Vq4YarLHhvgH60GTpG7Xz==NRacOA@mail.gmail.com>
Message-ID: <78CE00A1-98CB-4E62-B9D2-53CC62605A69@mcs.anl.gov>


  If you change any values in the matrix then KSP will automatically refactor the matrix for the next linear solve. If you want to use the same factorization even though you have changed the matrix you can use KSPSetReusePreconditioner()

  Barry

> On Oct 10, 2015, at 2:37 PM, David Liu <xdavidliu at gmail.com> wrote:
> 
> Hi,
> 
> I have a matrix A that I am trying to solve (currently using Mumps). Between solves, I'm changing some of the elements of A, but not changing the nonzero pattern.
> 
> In the past, with Petsc 3.4 and below, I was able to pass options like SAME_NONZERO_PATTERN and SAME_PRECONDITIONER to KSPSetOperators. By fiddling around with this (I don't recall exactly how), as well as re-calling KSPSetOperators between solves (iirc), I was able to switch at will between one of two things:
> 
> 1) re-doing the LU factorization at each solve, so that the iterative part of the KSPSolve takes only one iteration to converge (because the "preconditioner" is just the inverse matrix) (I believe that's what it's doing, sorry if I am butchering the nomenclature)
> 
> 2) doing the LU factorization on the first solve only, so that for the subsequent solves, the LU factorized matrix from the FIRST solve is used, so now the iterative part takes more than 1 iteration to converge.
> 
> I don't recall exactly the exact steps I took to switch between 1) and 2); it had something to do with the very last option passed to KSPSetOperators. All I know is that option 2) actually worked fine for me because my A matrix was only changing slightly between solves. 
> 
> However, now I updated to 3.5, and it looks like that very last option to KSPSetOperators has been removed, as recorded in the following commit on bitbucket: https://bitbucket.org/petsc/petsc/issues/50/eliminate-the-need-for-matstructure-flag
> 
> Now it looks like my code is exclusively doing 2) (if I am not mistaken), even though I am currently repeatedly calling KSPSetOperators between solves. How do I get my code to do 1)?
> 
> thanks
> best,
> David Liu
> 


From knram06 at gmail.com  Sat Oct 10 19:51:59 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Sat, 10 Oct 2015 20:51:59 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
Message-ID: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>

Hello all,

I am a graduate student pursuing my Master's and I am trying to benchmark a
previous work by using PETSc for solving Poisson's Equation.

I am starting off with a serial code and I am trying to keep my code
modular, i.e. I generate the sparse matrix format and send it to PETSc or
any other solver. So I haven't built my code from the ground up using
PETSc's native data structures.

I am having trouble understanding the behavior of the solver and would like
your thoughts or inputs on what I can do better. I have both Dirichlet and
Neumann boundary conditions and my matrix size (number of rows) is around a
million but very sparse (~7 nonzeros per row), as can be expected from a
finite difference discretization of Poisson's equation.

I tried the methods outlined here
<http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging?rq=1>
and here
<http://scicomp.stackexchange.com/questions/34/how-can-i-estimate-the-condition-number-of-a-large-sparse-matrix-using-petsc>.
Reverting to a 41^3 grid, I got the approximate condition number
(using  -ksp_monitor_singular_value
-ksp_type gmres -ksp_gmres_restart 1000 -pc_type none) as ~9072, which
seems pretty large. Higher matrix sizes give a larger condition number.

1) My best performing solver + preconditioner is bcgs+ilu(0) (on 1e6 grid)
which solves in around 32 seconds, 196 iterations. How do I get a fix for
what the lower bound on the running time could be?

2) Initially -pc_type hypre just Diverged and I was never able to use it.
Looking at this thread
<http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/019127.html>,
I had tried the options and it no longer diverges, but the residuals reduce
and hover around a constant value. How do I work with hypre to get a useful
preconditioner?

Initially I solve Laplace's equation, so the mesh grid size has no effect
and even when I solve Poisson's equation, the spacing is carried over to
the RHS, so I am pretty sure the spacing is not affecting the condition
number calculation.

Hope this helps. Please let me know if you might need more information.

Thanking You,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151010/34360b02/attachment.html>

From knram06 at gmail.com  Sat Oct 10 19:56:11 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Sat, 10 Oct 2015 20:56:11 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
Message-ID: <CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>

Sorry, some more questions.

3) Also, for Dirichlet bc, I specify the value through Identity rows, i.e.
A_ii = 1 and the rhs value would correspond to the Dirichlet condition. I
am specifying it this way for my convenience. I am aware that
MatZerosRowColumns might help here, but would keeping it this way be
detrimental?

4) Can I expect symmetric matrices to perform better, i.e. if I eliminate
Dirichlet rows? But I would still be left with Neumann boundary conditions,
where I use the second order formulation. If I used the first order
formulation and made it symmetric, would that be an advantage? I tried the
latter, but I didn't see the condition number change much.



On Sat, Oct 10, 2015 at 8:51 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello all,
>
> I am a graduate student pursuing my Master's and I am trying to benchmark
> a previous work by using PETSc for solving Poisson's Equation.
>
> I am starting off with a serial code and I am trying to keep my code
> modular, i.e. I generate the sparse matrix format and send it to PETSc or
> any other solver. So I haven't built my code from the ground up using
> PETSc's native data structures.
>
> I am having trouble understanding the behavior of the solver and would
> like your thoughts or inputs on what I can do better. I have both Dirichlet
> and Neumann boundary conditions and my matrix size (number of rows) is
> around a million but very sparse (~7 nonzeros per row), as can be expected
> from a finite difference discretization of Poisson's equation.
>
> I tried the methods outlined here
> <http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging?rq=1>
> and here
> <http://scicomp.stackexchange.com/questions/34/how-can-i-estimate-the-condition-number-of-a-large-sparse-matrix-using-petsc>.
> Reverting to a 41^3 grid, I got the approximate condition number (using  -ksp_monitor_singular_value
> -ksp_type gmres -ksp_gmres_restart 1000 -pc_type none) as ~9072, which
> seems pretty large. Higher matrix sizes give a larger condition number.
>
> 1) My best performing solver + preconditioner is bcgs+ilu(0) (on 1e6
> grid) which solves in around 32 seconds, 196 iterations. How do I get a
> fix for what the lower bound on the running time could be?
>
> 2) Initially -pc_type hypre just Diverged and I was never able to use it.
> Looking at this thread
> <http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/019127.html>,
> I had tried the options and it no longer diverges, but the residuals reduce
> and hover around a constant value. How do I work with hypre to get a
> useful preconditioner?
>
> Initially I solve Laplace's equation, so the mesh grid size has no effect
> and even when I solve Poisson's equation, the spacing is carried over to
> the RHS, so I am pretty sure the spacing is not affecting the condition
> number calculation.
>
> Hope this helps. Please let me know if you might need more information.
>
> Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>


Thanking You,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151010/e222e011/attachment.html>

From coyigg at hotmail.com  Mon Oct 12 01:49:44 2015
From: coyigg at hotmail.com (keguoyi)
Date: Mon, 12 Oct 2015 14:49:44 +0800
Subject: [petsc-users] =?gb2312?Q?How_to_imp?= =?gb2312?Q?lement_pre?=
 =?gb2312?Q?ssure_conv?= =?gb2312?B?ZWN0aW9uqENkaWY=?=
 =?gb2312?Q?fusion_pre?= =?gb2312?Q?conditione?= =?gb2312?Q?r_in_petsz?= 
Message-ID: <BLU185-W60C1111F5BF88323F7984AD3310@phx.gbl>

Dear Petsc developers and users,  

This is Guoyi ke, a graduate student in Texas Tech University.  I have a 2D Navier Stokes problem that has block matrices: J=[F    B^T; B    0]. I want to build a pressure convection?Cdiffusion preconditioner (PCD) P=[F    B^T; 0   Sp]. Here, we let Sp=-Ap(Fp)^(-1)Mp approximate schur complement S=-BF^(-1)B^T, where Ap is pressure Laplacian matrix, Mp is pressure mass matrix, and Fp is convection-diffusion operators on pressure space.

 We use right preconditioner J*P^(-1)=[F    B^T; B    0] * [F    B^T; 0    Sp]^(-1), and is it possible for Petsz to build and implement this precondioner P? Since (Sp)^(-1)=-(Mp)^(-1) Fp(Ap)^(-1), is it possible that we can solve (Mp)^(-1) and (Ap)^(-1) by CG method separately inside preconditioner P. 

Any suggestion will be highly appreciated. Thank you so much!

Best,
Guoyi 
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/bdfebb67/attachment.html>

From knepley at gmail.com  Mon Oct 12 08:37:58 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 12 Oct 2015 08:37:58 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
Message-ID: <CAMYG4GnFUDr4qYXM-D2DevthmqPr=jfrSLW+UA9vufZj5EnHvQ@mail.gmail.com>

On Sat, Oct 10, 2015 at 7:51 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello all,
>
> I am a graduate student pursuing my Master's and I am trying to benchmark
> a previous work by using PETSc for solving Poisson's Equation.
>
> I am starting off with a serial code and I am trying to keep my code
> modular, i.e. I generate the sparse matrix format and send it to PETSc or
> any other solver. So I haven't built my code from the ground up using
> PETSc's native data structures.
>
> I am having trouble understanding the behavior of the solver and would
> like your thoughts or inputs on what I can do better. I have both Dirichlet
> and Neumann boundary conditions and my matrix size (number of rows) is
> around a million but very sparse (~7 nonzeros per row), as can be expected
> from a finite difference discretization of Poisson's equation.
>
> I tried the methods outlined here
> <http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging?rq=1>
> and here
> <http://scicomp.stackexchange.com/questions/34/how-can-i-estimate-the-condition-number-of-a-large-sparse-matrix-using-petsc>.
> Reverting to a 41^3 grid, I got the approximate condition number (using  -ksp_monitor_singular_value
> -ksp_type gmres -ksp_gmres_restart 1000 -pc_type none) as ~9072, which
> seems pretty large. Higher matrix sizes give a larger condition number.
>
> 1) My best performing solver + preconditioner is bcgs+ilu(0) (on 1e6
> grid) which solves in around 32 seconds, 196 iterations. How do I get a
> fix for what the lower bound on the running time could be?
>

Unfortunately, that is really slow. You can look at this Tutorial (
http://www.mcs.anl.gov/petsc/documentation/tutorials/ParisTutorial.pdf)
slides 125-126 where
I solve elliptic systems using MG for systems with > 1M unknowns in that
amount of time.

You need to use Multigrid. Its the only solver that should be used for this
problem. Anything else is a waste of time.


> 2) Initially -pc_type hypre just Diverged and I was never able to use it.
> Looking at this thread
> <http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/019127.html>,
> I had tried the options and it no longer diverges, but the residuals reduce
> and hover around a constant value. How do I work with hypre to get a
> useful preconditioner?
>

Something is wrong with your setup. Hypre is a fantastic solver for these
problems. Send the convergence output (-ksp_monitor_true_residual
-ksp_converged_reason -ksp_view).


> Initially I solve Laplace's equation, so the mesh grid size has no effect
> and even when I solve Poisson's equation, the spacing is carried over to
> the RHS, so I am pretty sure the spacing is not affecting the condition
> number calculation.
>

No, the Laplacian has a condition number that grows like h^{-2}. See
earlier slides in that tutorial.

  Thanks,

     Matt


> Hope this helps. Please let me know if you might need more information.
>
> Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/225a3af6/attachment.html>

From knepley at gmail.com  Mon Oct 12 08:39:55 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 12 Oct 2015 08:39:55 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
Message-ID: <CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>

On Sat, Oct 10, 2015 at 7:56 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Sorry, some more questions.
>
> 3) Also, for Dirichlet bc, I specify the value through Identity rows, i.e.
> A_ii = 1 and the rhs value would correspond to the Dirichlet condition. I
> am specifying it this way for my convenience. I am aware that
> MatZerosRowColumns might help here, but would keeping it this way be
> detrimental?
>

That is fine. However you would want to scale these entries to be
approximately the same size as the other diagonal entries.


> 4) Can I expect symmetric matrices to perform better, i.e. if I eliminate
> Dirichlet rows? But I would still be left with Neumann boundary conditions,
> where I use the second order formulation. If I used the first order
> formulation and made it symmetric, would that be an advantage? I tried the
> latter, but I didn't see the condition number change much.
>

Will not matter for MG.

    Matt


>
>
> On Sat, Oct 10, 2015 at 8:51 PM, K. N. Ramachandran <knram06 at gmail.com>
> wrote:
>
>> Hello all,
>>
>> I am a graduate student pursuing my Master's and I am trying to benchmark
>> a previous work by using PETSc for solving Poisson's Equation.
>>
>> I am starting off with a serial code and I am trying to keep my code
>> modular, i.e. I generate the sparse matrix format and send it to PETSc or
>> any other solver. So I haven't built my code from the ground up using
>> PETSc's native data structures.
>>
>> I am having trouble understanding the behavior of the solver and would
>> like your thoughts or inputs on what I can do better. I have both Dirichlet
>> and Neumann boundary conditions and my matrix size (number of rows) is
>> around a million but very sparse (~7 nonzeros per row), as can be expected
>> from a finite difference discretization of Poisson's equation.
>>
>> I tried the methods outlined here
>> <http://scicomp.stackexchange.com/questions/513/why-is-my-iterative-linear-solver-not-converging?rq=1>
>> and here
>> <http://scicomp.stackexchange.com/questions/34/how-can-i-estimate-the-condition-number-of-a-large-sparse-matrix-using-petsc>.
>> Reverting to a 41^3 grid, I got the approximate condition number (using  -ksp_monitor_singular_value
>> -ksp_type gmres -ksp_gmres_restart 1000 -pc_type none) as ~9072, which
>> seems pretty large. Higher matrix sizes give a larger condition number.
>>
>> 1) My best performing solver + preconditioner is bcgs+ilu(0) (on 1e6
>> grid) which solves in around 32 seconds, 196 iterations. How do I get a
>> fix for what the lower bound on the running time could be?
>>
>> 2) Initially -pc_type hypre just Diverged and I was never able to use
>> it. Looking at this thread
>> <http://lists.mcs.anl.gov/pipermail/petsc-users/2013-October/019127.html>,
>> I had tried the options and it no longer diverges, but the residuals reduce
>> and hover around a constant value. How do I work with hypre to get a
>> useful preconditioner?
>>
>> Initially I solve Laplace's equation, so the mesh grid size has no effect
>> and even when I solve Poisson's equation, the spacing is carried over to
>> the RHS, so I am pretty sure the spacing is not affecting the condition
>> number calculation.
>>
>> Hope this helps. Please let me know if you might need more information.
>>
>> Thanking You,
>> K.N.Ramachandran
>> Ph: 814-441-4279
>>
>
>
> Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/4afb9a9f/attachment.html>

From knepley at gmail.com  Mon Oct 12 08:46:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 12 Oct 2015 08:46:35 -0500
Subject: [petsc-users] =?utf-8?q?How_to_implement_pressure_convection?=
	=?utf-8?q?=E2=80=93diffusion_preconditioner_in_petsz?=
In-Reply-To: <BLU185-W60C1111F5BF88323F7984AD3310@phx.gbl>
References: <BLU185-W60C1111F5BF88323F7984AD3310@phx.gbl>
Message-ID: <CAMYG4Gmy0PpQPmX8k4Mp_PpCdaaPx3DW54O+1ZP5i1kGB7u+iQ@mail.gmail.com>

On Mon, Oct 12, 2015 at 1:49 AM, keguoyi <coyigg at hotmail.com> wrote:

> Dear Petsc developers and users,
>
> This is Guoyi ke, a graduate student in Texas Tech University.  I have a
> 2D Navier Stokes problem that has block matrices: J=[F    B^T; B    0]. I
> want to build a pressure convection?diffusion preconditioner (PCD) P=[F
> B^T; 0   Sp]. Here, we let Sp=-Ap(Fp)^(-1)Mp approximate schur complement
> S=-BF^(-1)B^T, where Ap is pressure Laplacian matrix, Mp is pressure mass
> matrix, and Fp is convection-diffusion operators on pressure space.
>
>  We use right preconditioner J*P^(-1)=[F    B^T; B    0] * [F    B^T; 0
> Sp]^(-1), and is it possible for Petsz to build and implement this
> precondioner P? Since (Sp)^(-1)=-(Mp)^(-1) Fp(Ap)^(-1), is it possible that
> we can solve (Mp)^(-1) and (Ap)^(-1) by CG method separately inside
> preconditioner P.
>

Take a look at the PCFIELDSPLIT preconditioner. I think you want the LSC
option for that (
http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCLSC.html)
if I am reading your mail correctly.

  Thanks,

    Matt


> Any suggestion will be highly appreciated. Thank you so much!
>
> Best,
> Guoyi
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/978d856a/attachment-0001.html>

From knram06 at gmail.com  Mon Oct 12 14:52:18 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Mon, 12 Oct 2015 15:52:18 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
Message-ID: <CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>

Hello Matt,

Thanks for your reply. I am actually retrying against the latest PETSc and
will revert once I get the information.

On Mon, Oct 12, 2015 at 9:39 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Sat, Oct 10, 2015 at 7:56 PM, K. N. Ramachandran <knram06 at gmail.com>
> wrote:
>
>> Sorry, some more questions.
>>
>> 3) Also, for Dirichlet bc, I specify the value through Identity rows,
>> i.e. A_ii = 1 and the rhs value would correspond to the Dirichlet
>> condition. I am specifying it this way for my convenience. I am aware that
>> MatZerosRowColumns might help here, but would keeping it this way be
>> detrimental?
>>
>
> That is fine. However you would want to scale these entries to be
> approximately the same size as the other diagonal entries.
>
>
>> 4) Can I expect symmetric matrices to perform better, i.e. if I eliminate
>> Dirichlet rows? But I would still be left with Neumann boundary conditions,
>> where I use the second order formulation. If I used the first order
>> formulation and made it symmetric, would that be an advantage? I tried the
>> latter, but I didn't see the condition number change much.
>>
>
> Will not matter for MG.
>
>     Matt
>
>
>>
>>

Regards,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/9ea66a0d/attachment.html>

From knram06 at gmail.com  Mon Oct 12 21:11:59 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Mon, 12 Oct 2015 22:11:59 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
Message-ID: <CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>

Hello Matt,

Actually I felt the boundary conditions were having a role to play and set
all the boundary conditions to Dirichlet. In this case, convergence was
almost immediate with the Hypre preconditioner, taking 17 seconds with 3
iterations. The MG method took about the same time though.

So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to
diverge. Please find attached the output for the Hypre run using Dirichlet
and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row.
Details of the options used before running are in the file. The solver used
in all cases is bcgs.

Also attached is the MG output for 101^3 grid
1) Dirichlet and Neumann
2) Dirichlet only

where it seems to take about the same time.

I was thinking if the Null space has something to do with this? From the
PETSc docs, I understand that the Null space should be removed/attached
when solving singular systems. My domain will have at least two Dirichlet
conditions, so it is not singular. But since the Neumann conditions seem to
affect the convergence with Hypre, perhaps they have a role to play in
terms of the null space?



On Mon, Oct 12, 2015 at 3:52 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Matt,
>
> Thanks for your reply. I am actually retrying against the latest PETSc and
> will revert once I get the information.
>
> On Mon, Oct 12, 2015 at 9:39 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Sat, Oct 10, 2015 at 7:56 PM, K. N. Ramachandran <knram06 at gmail.com>
>> wrote:
>>
>>> Sorry, some more questions.
>>>
>>> 3) Also, for Dirichlet bc, I specify the value through Identity rows,
>>> i.e. A_ii = 1 and the rhs value would correspond to the Dirichlet
>>> condition. I am specifying it this way for my convenience. I am aware that
>>> MatZerosRowColumns might help here, but would keeping it this way be
>>> detrimental?
>>>
>>
>> That is fine. However you would want to scale these entries to be
>> approximately the same size as the other diagonal entries.
>>
>>
>>> 4) Can I expect symmetric matrices to perform better, i.e. if I
>>> eliminate Dirichlet rows? But I would still be left with Neumann boundary
>>> conditions, where I use the second order formulation. If I used the first
>>> order formulation and made it symmetric, would that be an advantage? I
>>> tried the latter, but I didn't see the condition number change much.
>>>
>>
>> Will not matter for MG.
>>
>>     Matt
>>
>>
>>>
>>>
>
> Regards,
> K.N.Ramachandran
> Ph: 814-441-4279
>



Thanking You,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/f31c01ee/attachment-0001.html>
-------------- next part --------------
Number of lines read was 629.
  0 KSP preconditioned resid norm 2.746422809456e+13 true resid norm 2.565000000000e+04 ||r(i)||/||b|| 9.047619047619e-01
  0 KSP Residual norm 2.74642e+13 
  1 KSP preconditioned resid norm 3.068441506053e+12 true resid norm 2.565000365319e+04 ||r(i)||/||b|| 9.047620336223e-01
  1 KSP Residual norm 3.06844e+12 
  2 KSP preconditioned resid norm 5.231248714904e+11 true resid norm 2.564996677545e+04 ||r(i)||/||b|| 9.047607328202e-01
  2 KSP Residual norm 5.23125e+11 
  3 KSP preconditioned resid norm 8.886333686081e+09 true resid norm 2.564999949878e+04 ||r(i)||/||b|| 9.047618870822e-01
  3 KSP Residual norm 8.88633e+09 
  4 KSP preconditioned resid norm 8.839069909584e+09 true resid norm 2.564999939254e+04 ||r(i)||/||b|| 9.047618833347e-01
  4 KSP Residual norm 8.83907e+09 
  5 KSP preconditioned resid norm 9.614961589585e+09 true resid norm 2.565178081416e+04 ||r(i)||/||b|| 9.048247200763e-01
  5 KSP Residual norm 9.61496e+09 
  6 KSP preconditioned resid norm 8.741604166481e+09 true resid norm 2.565178072277e+04 ||r(i)||/||b|| 9.048247168526e-01
  6 KSP Residual norm 8.7416e+09 
  7 KSP preconditioned resid norm 8.235731151411e+09 true resid norm 2.565178075232e+04 ||r(i)||/||b|| 9.048247178947e-01
  7 KSP Residual norm 8.23573e+09 
  8 KSP preconditioned resid norm 1.621290699141e+10 true resid norm 2.575853377098e+04 ||r(i)||/||b|| 9.085902564719e-01
  8 KSP Residual norm 1.62129e+10 
  9 KSP preconditioned resid norm 1.620223144519e+10 true resid norm 2.575845320875e+04 ||r(i)||/||b|| 9.085874147706e-01
  9 KSP Residual norm 1.62022e+10 
 10 KSP preconditioned resid norm 1.547486240008e+10 true resid norm 2.575771305043e+04 ||r(i)||/||b|| 9.085613068934e-01
 10 KSP Residual norm 1.54749e+10 
 11 KSP preconditioned resid norm 7.715548867365e+09 true resid norm 2.565552991809e+04 ||r(i)||/||b|| 9.049569636010e-01
 11 KSP Residual norm 7.71555e+09 
 12 KSP preconditioned resid norm 3.889844790546e+09 true resid norm 2.565945115206e+04 ||r(i)||/||b|| 9.050952787324e-01
 12 KSP Residual norm 3.88984e+09 
 13 KSP preconditioned resid norm 2.073570348298e+09 true resid norm 2.565946414127e+04 ||r(i)||/||b|| 9.050957369054e-01
 13 KSP Residual norm 2.07357e+09 
 14 KSP preconditioned resid norm 2.071889747823e+09 true resid norm 2.565946415085e+04 ||r(i)||/||b|| 9.050957372435e-01
 14 KSP Residual norm 2.07189e+09 
 15 KSP preconditioned resid norm 2.073916663913e+09 true resid norm 2.565948414693e+04 ||r(i)||/||b|| 9.050964425726e-01
 15 KSP Residual norm 2.07392e+09 
 16 KSP preconditioned resid norm 3.589156239198e+08 true resid norm 2.565390234822e+04 ||r(i)||/||b|| 9.048995537292e-01
 16 KSP Residual norm 3.58916e+08 
 17 KSP preconditioned resid norm 1.282067102091e+08 true resid norm 2.565925054713e+04 ||r(i)||/||b|| 9.050882027206e-01
 17 KSP Residual norm 1.28207e+08 
 18 KSP preconditioned resid norm 1.273021891519e+08 true resid norm 2.565925054752e+04 ||r(i)||/||b|| 9.050882027344e-01
 18 KSP Residual norm 1.27302e+08 
 19 KSP preconditioned resid norm 1.268287715098e+08 true resid norm 2.565926001252e+04 ||r(i)||/||b|| 9.050885365969e-01
 19 KSP Residual norm 1.26829e+08 
 20 KSP preconditioned resid norm 2.345828930128e+08 true resid norm 2.566026335819e+04 ||r(i)||/||b|| 9.051239279783e-01
 20 KSP Residual norm 2.34583e+08 
 21 KSP preconditioned resid norm 1.630341247775e+08 true resid norm 2.565856686030e+04 ||r(i)||/||b|| 9.050640867831e-01
 21 KSP Residual norm 1.63034e+08 
 22 KSP preconditioned resid norm 1.523032674511e+08 true resid norm 2.565866052028e+04 ||r(i)||/||b|| 9.050673904862e-01
 22 KSP Residual norm 1.52303e+08 
 23 KSP preconditioned resid norm 7.511400932854e+08 true resid norm 2.566425258525e+04 ||r(i)||/||b|| 9.052646414550e-01
 23 KSP Residual norm 7.5114e+08 
 24 KSP preconditioned resid norm 7.487411421877e+08 true resid norm 2.566425257593e+04 ||r(i)||/||b|| 9.052646411264e-01
 24 KSP Residual norm 7.48741e+08 
 25 KSP preconditioned resid norm 7.517590916570e+08 true resid norm 2.566428660733e+04 ||r(i)||/||b|| 9.052658415283e-01
 25 KSP Residual norm 7.51759e+08 
 26 KSP preconditioned resid norm 6.712414388866e+08 true resid norm 2.566385823745e+04 ||r(i)||/||b|| 9.052507314796e-01
 26 KSP Residual norm 6.71241e+08 
 27 KSP preconditioned resid norm 6.711948000736e+08 true resid norm 2.566385821489e+04 ||r(i)||/||b|| 9.052507306840e-01
 27 KSP Residual norm 6.71195e+08 
 28 KSP preconditioned resid norm 6.777281637926e+08 true resid norm 2.566406931696e+04 ||r(i)||/||b|| 9.052581769649e-01
 28 KSP Residual norm 6.77728e+08 
 29 KSP preconditioned resid norm 6.816151312784e+08 true resid norm 2.566419142142e+04 ||r(i)||/||b|| 9.052624840008e-01
 29 KSP Residual norm 6.81615e+08 
 30 KSP preconditioned resid norm 3.453970716323e+08 true resid norm 2.565596719002e+04 ||r(i)||/||b|| 9.049723876550e-01
 30 KSP Residual norm 3.45397e+08 
 31 KSP preconditioned resid norm 3.440067141747e+08 true resid norm 2.565595358060e+04 ||r(i)||/||b|| 9.049719076050e-01
 31 KSP Residual norm 3.44007e+08 
 32 KSP preconditioned resid norm 3.440043333335e+08 true resid norm 2.565595350230e+04 ||r(i)||/||b|| 9.049719048430e-01
 32 KSP Residual norm 3.44004e+08 
 33 KSP preconditioned resid norm 3.440062912059e+08 true resid norm 2.565595341062e+04 ||r(i)||/||b|| 9.049719016093e-01
 33 KSP Residual norm 3.44006e+08 
 34 KSP preconditioned resid norm 1.056990033279e+08 true resid norm 2.565262526962e+04 ||r(i)||/||b|| 9.048545068650e-01
 34 KSP Residual norm 1.05699e+08 
 35 KSP preconditioned resid norm 8.874726811494e+07 true resid norm 2.565262526272e+04 ||r(i)||/||b|| 9.048545066215e-01
 35 KSP Residual norm 8.87473e+07 
 36 KSP preconditioned resid norm 8.853824300744e+07 true resid norm 2.565263428333e+04 ||r(i)||/||b|| 9.048548248087e-01
 36 KSP Residual norm 8.85382e+07 
 37 KSP preconditioned resid norm 8.847029828263e+07 true resid norm 2.565263461124e+04 ||r(i)||/||b|| 9.048548363752e-01
 37 KSP Residual norm 8.84703e+07 
 38 KSP preconditioned resid norm 8.827921818416e+07 true resid norm 2.565263507031e+04 ||r(i)||/||b|| 9.048548525683e-01
 38 KSP Residual norm 8.82792e+07 
 39 KSP preconditioned resid norm 8.810895958886e+07 true resid norm 2.565263192471e+04 ||r(i)||/||b|| 9.048547416125e-01
 39 KSP Residual norm 8.8109e+07 
 40 KSP preconditioned resid norm 8.810893250183e+07 true resid norm 2.565263192472e+04 ||r(i)||/||b|| 9.048547416125e-01
 40 KSP Residual norm 8.81089e+07 
 41 KSP preconditioned resid norm 2.832676882984e+08 true resid norm 2.587947231892e+04 ||r(i)||/||b|| 9.128561664522e-01
 41 KSP Residual norm 2.83268e+08 
 42 KSP preconditioned resid norm 5.094464612028e+08 true resid norm 2.632571610237e+04 ||r(i)||/||b|| 9.285966879144e-01
 42 KSP Residual norm 5.09446e+08 
 43 KSP preconditioned resid norm 5.094272471154e+08 true resid norm 2.632571759868e+04 ||r(i)||/||b|| 9.285967406940e-01
 43 KSP Residual norm 5.09427e+08 
 44 KSP preconditioned resid norm 5.094932045774e+08 true resid norm 2.632581903783e+04 ||r(i)||/||b|| 9.286003187947e-01
 44 KSP Residual norm 5.09493e+08 
 45 KSP preconditioned resid norm 5.094147162461e+08 true resid norm 2.632570228114e+04 ||r(i)||/||b|| 9.285962003929e-01
 45 KSP Residual norm 5.09415e+08 
 46 KSP preconditioned resid norm 5.392886146895e+08 true resid norm 2.637071253446e+04 ||r(i)||/||b|| 9.301838636494e-01
 46 KSP Residual norm 5.39289e+08 
 47 KSP preconditioned resid norm 5.394119316675e+08 true resid norm 2.637090361414e+04 ||r(i)||/||b|| 9.301906036734e-01
 47 KSP Residual norm 5.39412e+08 
 48 KSP preconditioned resid norm 6.619138323982e+08 true resid norm 2.656818352376e+04 ||r(i)||/||b|| 9.371493306440e-01
 48 KSP Residual norm 6.61914e+08 
 49 KSP preconditioned resid norm 5.750298780298e+08 true resid norm 2.642560433787e+04 ||r(i)||/||b|| 9.321200824644e-01
 49 KSP Residual norm 5.7503e+08 
 50 KSP preconditioned resid norm 5.904316293874e+08 true resid norm 2.645018122477e+04 ||r(i)||/||b|| 9.329869920555e-01
 50 KSP Residual norm 5.90432e+08 
 51 KSP preconditioned resid norm 5.908766007712e+08 true resid norm 2.645090453708e+04 ||r(i)||/||b|| 9.330125057172e-01
 51 KSP Residual norm 5.90877e+08 
 52 KSP preconditioned resid norm 5.473325996306e+09 true resid norm 4.605238258213e+04 ||r(i)||/||b|| 1.624422666036e+00
 52 KSP Residual norm 5.47333e+09 
 53 KSP preconditioned resid norm 5.545262905105e+09 true resid norm 4.644344293813e+04 ||r(i)||/||b|| 1.638216682121e+00
 53 KSP Residual norm 5.54526e+09 
 54 KSP preconditioned resid norm 6.985912203753e+09 true resid norm 5.465866950046e+04 ||r(i)||/||b|| 1.927995396842e+00
 54 KSP Residual norm 6.98591e+09 
 55 KSP preconditioned resid norm 7.028372253151e+09 true resid norm 5.491163709071e+04 ||r(i)||/||b|| 1.936918415898e+00
 55 KSP Residual norm 7.02837e+09 
 56 KSP preconditioned resid norm 6.956696069159e+09 true resid norm 5.449230060223e+04 ||r(i)||/||b|| 1.922127005370e+00
 56 KSP Residual norm 6.9567e+09 
 57 KSP preconditioned resid norm 6.956620270168e+09 true resid norm 5.449227495259e+04 ||r(i)||/||b|| 1.922126100620e+00
 57 KSP Residual norm 6.95662e+09 
 58 KSP preconditioned resid norm 7.557947501531e+09 true resid norm 5.790682041701e+04 ||r(i)||/||b|| 2.042568621411e+00
 58 KSP Residual norm 7.55795e+09 
 59 KSP preconditioned resid norm 7.557887191771e+09 true resid norm 5.790684802653e+04 ||r(i)||/||b|| 2.042569595292e+00
 59 KSP Residual norm 7.55789e+09 
 60 KSP preconditioned resid norm 5.922018057101e+09 true resid norm 4.877211157711e+04 ||r(i)||/||b|| 1.720356669386e+00
 60 KSP Residual norm 5.92202e+09 
 61 KSP preconditioned resid norm 5.921901618668e+09 true resid norm 4.877143775666e+04 ||r(i)||/||b|| 1.720332901470e+00
 61 KSP Residual norm 5.9219e+09 
 62 KSP preconditioned resid norm 6.206899718077e+09 true resid norm 5.048143067017e+04 ||r(i)||/||b|| 1.780650111823e+00
 62 KSP Residual norm 6.2069e+09 
 63 KSP preconditioned resid norm 5.915710507609e+09 true resid norm 4.875654343996e+04 ||r(i)||/||b|| 1.719807528746e+00
 63 KSP Residual norm 5.91571e+09 
 64 KSP preconditioned resid norm 5.874812416368e+09 true resid norm 4.851577496082e+04 ||r(i)||/||b|| 1.711314813433e+00
 64 KSP Residual norm 5.87481e+09 
 65 KSP preconditioned resid norm 4.282677980829e+09 true resid norm 1.263517579531e+05 ||r(i)||/||b|| 4.456852132386e+00
 65 KSP Residual norm 4.28268e+09 
 66 KSP preconditioned resid norm 3.399082369795e+09 true resid norm 1.263514198377e+05 ||r(i)||/||b|| 4.456840205915e+00
 66 KSP Residual norm 3.39908e+09 
 67 KSP preconditioned resid norm 3.398843450548e+09 true resid norm 1.263512159099e+05 ||r(i)||/||b|| 4.456833012693e+00
 67 KSP Residual norm 3.39884e+09 
 68 KSP preconditioned resid norm 3.340863917040e+09 true resid norm 1.251554785170e+05 ||r(i)||/||b|| 4.414655326879e+00
 68 KSP Residual norm 3.34086e+09 
 69 KSP preconditioned resid norm 3.341010231124e+09 true resid norm 1.251596104734e+05 ||r(i)||/||b|| 4.414801074901e+00
 69 KSP Residual norm 3.34101e+09 
 70 KSP preconditioned resid norm 3.413554710700e+09 true resid norm 1.267351623618e+05 ||r(i)||/||b|| 4.470376097419e+00
 70 KSP Residual norm 3.41355e+09 
 71 KSP preconditioned resid norm 3.406550397907e+09 true resid norm 1.267481752469e+05 ||r(i)||/||b|| 4.470835105711e+00
 71 KSP Residual norm 3.40655e+09 
 72 KSP preconditioned resid norm 3.405180126984e+09 true resid norm 1.267482110372e+05 ||r(i)||/||b|| 4.470836368155e+00
 72 KSP Residual norm 3.40518e+09 
 73 KSP preconditioned resid norm 3.202517222100e+09 true resid norm 1.227605456906e+05 ||r(i)||/||b|| 4.330177978503e+00
 73 KSP Residual norm 3.20252e+09 
 74 KSP preconditioned resid norm 3.200035599801e+09 true resid norm 1.227599047136e+05 ||r(i)||/||b|| 4.330155369088e+00
 74 KSP Residual norm 3.20004e+09 
 75 KSP preconditioned resid norm 5.800869198419e+09 true resid norm 1.750241638496e+05 ||r(i)||/||b|| 6.173691846546e+00
 75 KSP Residual norm 5.80087e+09 
 76 KSP preconditioned resid norm 5.786449097800e+09 true resid norm 1.750126667441e+05 ||r(i)||/||b|| 6.173286304907e+00
 76 KSP Residual norm 5.78645e+09 
 77 KSP preconditioned resid norm 4.835224177932e+09 true resid norm 1.562224322500e+05 ||r(i)||/||b|| 5.510491437389e+00
 77 KSP Residual norm 4.83522e+09 
 78 KSP preconditioned resid norm 4.835168801321e+09 true resid norm 1.562233476102e+05 ||r(i)||/||b|| 5.510523725226e+00
 78 KSP Residual norm 4.83517e+09 
 79 KSP preconditioned resid norm 6.347543402233e+09 true resid norm 7.481908590977e+04 ||r(i)||/||b|| 2.639121196112e+00
 79 KSP Residual norm 6.34754e+09 
 80 KSP preconditioned resid norm 6.326865493321e+09 true resid norm 7.443283156609e+04 ||r(i)||/||b|| 2.625496704271e+00
 80 KSP Residual norm 6.32687e+09 
 81 KSP preconditioned resid norm 1.955070438438e+09 true resid norm 3.296317972666e+04 ||r(i)||/||b|| 1.162722388948e+00
 81 KSP Residual norm 1.95507e+09 
 82 KSP preconditioned resid norm 1.879007613781e+09 true resid norm 3.391312036621e+04 ||r(i)||/||b|| 1.196229995281e+00
 82 KSP Residual norm 1.87901e+09 
 83 KSP preconditioned resid norm 1.878584017023e+09 true resid norm 3.390547232026e+04 ||r(i)||/||b|| 1.195960222937e+00
 83 KSP Residual norm 1.87858e+09 
 84 KSP preconditioned resid norm 1.878625570135e+09 true resid norm 3.390494811375e+04 ||r(i)||/||b|| 1.195941732407e+00
 84 KSP Residual norm 1.87863e+09 
 85 KSP preconditioned resid norm 1.878150970552e+09 true resid norm 3.392253157839e+04 ||r(i)||/||b|| 1.196561960437e+00
 85 KSP Residual norm 1.87815e+09 
 86 KSP preconditioned resid norm 1.872131248738e+09 true resid norm 3.398501170486e+04 ||r(i)||/||b|| 1.198765844968e+00
 86 KSP Residual norm 1.87213e+09 
 87 KSP preconditioned resid norm 1.862562589043e+09 true resid norm 3.410432855752e+04 ||r(i)||/||b|| 1.202974552293e+00
 87 KSP Residual norm 1.86256e+09 
 88 KSP preconditioned resid norm 1.861801920798e+09 true resid norm 3.411436806464e+04 ||r(i)||/||b|| 1.203328679529e+00
 88 KSP Residual norm 1.8618e+09 
 89 KSP preconditioned resid norm 1.861732815998e+09 true resid norm 3.411450834870e+04 ||r(i)||/||b|| 1.203333627820e+00
 89 KSP Residual norm 1.86173e+09 
 90 KSP preconditioned resid norm 1.861829113157e+09 true resid norm 3.417586491910e+04 ||r(i)||/||b|| 1.205497880744e+00
 90 KSP Residual norm 1.86183e+09 
 91 KSP preconditioned resid norm 1.856883125610e+09 true resid norm 3.417498104263e+04 ||r(i)||/||b|| 1.205466703444e+00
 91 KSP Residual norm 1.85688e+09 
 92 KSP preconditioned resid norm 1.847065929125e+09 true resid norm 3.428971796667e+04 ||r(i)||/||b|| 1.209513861258e+00
 92 KSP Residual norm 1.84707e+09 
 93 KSP preconditioned resid norm 1.848791512225e+09 true resid norm 3.426284189227e+04 ||r(i)||/||b|| 1.208565851579e+00
 93 KSP Residual norm 1.84879e+09 
 94 KSP preconditioned resid norm 1.827844234443e+09 true resid norm 3.453841929543e+04 ||r(i)||/||b|| 1.218286394901e+00
 94 KSP Residual norm 1.82784e+09 
 95 KSP preconditioned resid norm 1.827774709429e+09 true resid norm 3.453841962987e+04 ||r(i)||/||b|| 1.218286406697e+00
 95 KSP Residual norm 1.82777e+09 
 96 KSP preconditioned resid norm 6.414487075650e+09 true resid norm 7.391240064152e+04 ||r(i)||/||b|| 2.607139352435e+00
 96 KSP Residual norm 6.41449e+09 
 97 KSP preconditioned resid norm 6.415045121560e+09 true resid norm 7.392268206099e+04 ||r(i)||/||b|| 2.607502012733e+00
 97 KSP Residual norm 6.41505e+09 
 98 KSP preconditioned resid norm 6.401511248492e+09 true resid norm 7.367799137929e+04 ||r(i)||/||b|| 2.598870948123e+00
 98 KSP Residual norm 6.40151e+09 
 99 KSP preconditioned resid norm 5.823452922482e+09 true resid norm 6.324752016902e+04 ||r(i)||/||b|| 2.230953092382e+00
 99 KSP Residual norm 5.82345e+09 
100 KSP preconditioned resid norm 1.200549311182e+10 true resid norm 1.817187164268e+05 ||r(i)||/||b|| 6.409831267258e+00
100 KSP Residual norm 1.20055e+10 
101 KSP preconditioned resid norm 1.199021567525e+10 true resid norm 1.816659732185e+05 ||r(i)||/||b|| 6.407970836632e+00
101 KSP Residual norm 1.19902e+10 
102 KSP preconditioned resid norm 1.599940019890e+10 true resid norm 2.601826735101e+05 ||r(i)||/||b|| 9.177519347800e+00
102 KSP Residual norm 1.59994e+10 
103 KSP preconditioned resid norm 1.596880097390e+10 true resid norm 2.595841214331e+05 ||r(i)||/||b|| 9.156406399755e+00
103 KSP Residual norm 1.59688e+10 
104 KSP preconditioned resid norm 5.367460315778e+09 true resid norm 5.575620393927e+04 ||r(i)||/||b|| 1.966709133660e+00
104 KSP Residual norm 5.36746e+09 
105 KSP preconditioned resid norm 5.367159894930e+09 true resid norm 5.575117378768e+04 ||r(i)||/||b|| 1.966531703269e+00
105 KSP Residual norm 5.36716e+09 
106 KSP preconditioned resid norm 5.320615022748e+09 true resid norm 6.100403691365e+04 ||r(i)||/||b|| 2.151817880552e+00
106 KSP Residual norm 5.32062e+09 
107 KSP preconditioned resid norm 5.295090674561e+09 true resid norm 6.100343132478e+04 ||r(i)||/||b|| 2.151796519393e+00
107 KSP Residual norm 5.29509e+09 
108 KSP preconditioned resid norm 5.300575025668e+09 true resid norm 6.112708327182e+04 ||r(i)||/||b|| 2.156158140099e+00
108 KSP Residual norm 5.30058e+09 
109 KSP preconditioned resid norm 5.300311309732e+09 true resid norm 6.112184454233e+04 ||r(i)||/||b|| 2.155973352463e+00
109 KSP Residual norm 5.30031e+09 
110 KSP preconditioned resid norm 4.707463773424e+09 true resid norm 4.961969919811e+04 ||r(i)||/||b|| 1.750253939968e+00
110 KSP Residual norm 4.70746e+09 
111 KSP preconditioned resid norm 4.707435553722e+09 true resid norm 4.961917481621e+04 ||r(i)||/||b|| 1.750235443252e+00
111 KSP Residual norm 4.70744e+09 
112 KSP preconditioned resid norm 4.581779174905e+09 true resid norm 8.243536773251e+04 ||r(i)||/||b|| 2.907773112258e+00
112 KSP Residual norm 4.58178e+09 
113 KSP preconditioned resid norm 4.460704312961e+09 true resid norm 8.243667172369e+04 ||r(i)||/||b|| 2.907819108420e+00
113 KSP Residual norm 4.4607e+09 
114 KSP preconditioned resid norm 4.444429099697e+09 true resid norm 8.185303111185e+04 ||r(i)||/||b|| 2.887232137984e+00
114 KSP Residual norm 4.44443e+09 
115 KSP preconditioned resid norm 4.444180637628e+09 true resid norm 8.184349413475e+04 ||r(i)||/||b|| 2.886895736675e+00
115 KSP Residual norm 4.44418e+09 
116 KSP preconditioned resid norm 4.442501512234e+09 true resid norm 8.176511122736e+04 ||r(i)||/||b|| 2.884130907491e+00
116 KSP Residual norm 4.4425e+09 
117 KSP preconditioned resid norm 4.442500992893e+09 true resid norm 8.176511099161e+04 ||r(i)||/||b|| 2.884130899175e+00
117 KSP Residual norm 4.4425e+09 
118 KSP preconditioned resid norm 4.438588508271e+09 true resid norm 8.171332777261e+04 ||r(i)||/||b|| 2.882304330604e+00
118 KSP Residual norm 4.43859e+09 
119 KSP preconditioned resid norm 4.438401655459e+09 true resid norm 8.171332773633e+04 ||r(i)||/||b|| 2.882304329324e+00
119 KSP Residual norm 4.4384e+09 
120 KSP preconditioned resid norm 1.317525363949e+08 true resid norm 2.692106697168e+04 ||r(i)||/||b|| 9.495967185778e-01
120 KSP Residual norm 1.31753e+08 
121 KSP preconditioned resid norm 1.308800784727e+08 true resid norm 2.692692818159e+04 ||r(i)||/||b|| 9.498034631954e-01
121 KSP Residual norm 1.3088e+08 
122 KSP preconditioned resid norm 7.603842291444e+07 true resid norm 2.779457205567e+04 ||r(i)||/||b|| 9.804081853851e-01
122 KSP Residual norm 7.60384e+07 
123 KSP preconditioned resid norm 7.786999714125e+07 true resid norm 2.783487508098e+04 ||r(i)||/||b|| 9.818298088528e-01
123 KSP Residual norm 7.787e+07 
124 KSP preconditioned resid norm 7.588747018640e+07 true resid norm 2.779571139502e+04 ||r(i)||/||b|| 9.804483737220e-01
124 KSP Residual norm 7.58875e+07 
125 KSP preconditioned resid norm 7.883172110161e+07 true resid norm 2.785164157962e+04 ||r(i)||/||b|| 9.824212197397e-01
125 KSP Residual norm 7.88317e+07 
126 KSP preconditioned resid norm 7.895619286249e+07 true resid norm 2.785394896305e+04 ||r(i)||/||b|| 9.825026089261e-01
126 KSP Residual norm 7.89562e+07 
127 KSP preconditioned resid norm 1.419982565818e+08 true resid norm 2.685170592763e+04 ||r(i)||/||b|| 9.471501209040e-01
127 KSP Residual norm 1.41998e+08 
128 KSP preconditioned resid norm 1.571365751637e+08 true resid norm 2.675030023884e+04 ||r(i)||/||b|| 9.435732006644e-01
128 KSP Residual norm 1.57137e+08 
129 KSP preconditioned resid norm 1.547396298924e+08 true resid norm 2.674946866281e+04 ||r(i)||/||b|| 9.435438681766e-01
129 KSP Residual norm 1.5474e+08 
130 KSP preconditioned resid norm 1.550040600360e+08 true resid norm 2.674789513753e+04 ||r(i)||/||b|| 9.434883646394e-01
130 KSP Residual norm 1.55004e+08 
131 KSP preconditioned resid norm 1.473917053093e+08 true resid norm 2.679211609491e+04 ||r(i)||/||b|| 9.450481867694e-01
131 KSP Residual norm 1.47392e+08 
132 KSP preconditioned resid norm 1.447376699776e+08 true resid norm 2.680792549358e+04 ||r(i)||/||b|| 9.456058375161e-01
132 KSP Residual norm 1.44738e+08 
133 KSP preconditioned resid norm 9.392057756041e+07 true resid norm 2.717200095632e+04 ||r(i)||/||b|| 9.584480055138e-01
133 KSP Residual norm 9.39206e+07 
134 KSP preconditioned resid norm 9.365125751222e+07 true resid norm 2.717452342546e+04 ||r(i)||/||b|| 9.585369814978e-01
134 KSP Residual norm 9.36513e+07 
135 KSP preconditioned resid norm 9.357656851783e+07 true resid norm 2.717504278405e+04 ||r(i)||/||b|| 9.585553010248e-01
135 KSP Residual norm 9.35766e+07 
136 KSP preconditioned resid norm 9.432911703112e+07 true resid norm 2.716745072225e+04 ||r(i)||/||b|| 9.582875034304e-01
136 KSP Residual norm 9.43291e+07 
137 KSP preconditioned resid norm 9.208617884080e+07 true resid norm 2.718929189750e+04 ||r(i)||/||b|| 9.590579152557e-01
137 KSP Residual norm 9.20862e+07 
138 KSP preconditioned resid norm 9.203707831663e+07 true resid norm 2.718905167187e+04 ||r(i)||/||b|| 9.590494416886e-01
138 KSP Residual norm 9.20371e+07 
139 KSP preconditioned resid norm 9.337458782546e+07 true resid norm 2.717493537915e+04 ||r(i)||/||b|| 9.585515124920e-01
139 KSP Residual norm 9.33746e+07 
140 KSP preconditioned resid norm 6.901974077152e+08 true resid norm 2.623579156570e+04 ||r(i)||/||b|| 9.254247465856e-01
140 KSP Residual norm 6.90197e+08 
141 KSP preconditioned resid norm 1.975679753339e+08 true resid norm 2.646433994022e+04 ||r(i)||/||b|| 9.334864176446e-01
141 KSP Residual norm 1.97568e+08 
142 KSP preconditioned resid norm 1.954417782803e+08 true resid norm 2.647439384680e+04 ||r(i)||/||b|| 9.338410527972e-01
142 KSP Residual norm 1.95442e+08 
143 KSP preconditioned resid norm 1.960399893844e+08 true resid norm 2.647133991553e+04 ||r(i)||/||b|| 9.337333303539e-01
143 KSP Residual norm 1.9604e+08 
144 KSP preconditioned resid norm 1.955670584333e+08 true resid norm 2.647325367781e+04 ||r(i)||/||b|| 9.338008351960e-01
144 KSP Residual norm 1.95567e+08 
145 KSP preconditioned resid norm 1.950109040317e+08 true resid norm 2.647531183614e+04 ||r(i)||/||b|| 9.338734333736e-01
145 KSP Residual norm 1.95011e+08 
146 KSP preconditioned resid norm 2.044593716817e+08 true resid norm 2.642909995124e+04 ||r(i)||/||b|| 9.322433845234e-01
146 KSP Residual norm 2.04459e+08 
147 KSP preconditioned resid norm 2.029589861814e+08 true resid norm 2.643596010230e+04 ||r(i)||/||b|| 9.324853651604e-01
147 KSP Residual norm 2.02959e+08 
148 KSP preconditioned resid norm 2.021068171567e+08 true resid norm 2.644005464457e+04 ||r(i)||/||b|| 9.326297934594e-01
148 KSP Residual norm 2.02107e+08 
149 KSP preconditioned resid norm 3.913475546831e+08 true resid norm 2.585852316907e+04 ||r(i)||/||b|| 9.121172193676e-01
149 KSP Residual norm 3.91348e+08 
150 KSP preconditioned resid norm 2.936596720006e+08 true resid norm 2.610384627244e+04 ||r(i)||/||b|| 9.207705916204e-01
150 KSP Residual norm 2.9366e+08 
151 KSP preconditioned resid norm 2.388405097790e+08 true resid norm 2.610383006923e+04 ||r(i)||/||b|| 9.207700200785e-01
151 KSP Residual norm 2.38841e+08 
152 KSP preconditioned resid norm 2.388371833554e+08 true resid norm 2.610383007076e+04 ||r(i)||/||b|| 9.207700201326e-01
152 KSP Residual norm 2.38837e+08 
153 KSP preconditioned resid norm 2.338099680317e+08 true resid norm 2.611532371119e+04 ||r(i)||/||b|| 9.211754395483e-01
153 KSP Residual norm 2.3381e+08 
154 KSP preconditioned resid norm 2.309318034524e+08 true resid norm 2.611532410104e+04 ||r(i)||/||b|| 9.211754532994e-01
154 KSP Residual norm 2.30932e+08 
155 KSP preconditioned resid norm 2.321145313197e+08 true resid norm 2.611063178072e+04 ||r(i)||/||b|| 9.210099393553e-01
155 KSP Residual norm 2.32115e+08 
156 KSP preconditioned resid norm 2.321145291804e+08 true resid norm 2.611063178074e+04 ||r(i)||/||b|| 9.210099393557e-01
156 KSP Residual norm 2.32115e+08 
157 KSP preconditioned resid norm 2.320433411319e+08 true resid norm 2.611077036572e+04 ||r(i)||/||b|| 9.210148277151e-01
157 KSP Residual norm 2.32043e+08 
158 KSP preconditioned resid norm 2.320433270295e+08 true resid norm 2.611077036371e+04 ||r(i)||/||b|| 9.210148276441e-01
158 KSP Residual norm 2.32043e+08 
159 KSP preconditioned resid norm 1.978911170688e+08 true resid norm 2.618266522378e+04 ||r(i)||/||b|| 9.235508015441e-01
159 KSP Residual norm 1.97891e+08 
160 KSP preconditioned resid norm 1.978533948713e+08 true resid norm 2.618266545810e+04 ||r(i)||/||b|| 9.235508098096e-01
160 KSP Residual norm 1.97853e+08 
161 KSP preconditioned resid norm 2.038236960653e+08 true resid norm 2.616971533406e+04 ||r(i)||/||b|| 9.230940153108e-01
161 KSP Residual norm 2.03824e+08 
162 KSP preconditioned resid norm 2.030406249754e+08 true resid norm 2.616971498086e+04 ||r(i)||/||b|| 9.230940028521e-01
162 KSP Residual norm 2.03041e+08 
163 KSP preconditioned resid norm 2.022418418224e+08 true resid norm 2.616964828616e+04 ||r(i)||/||b|| 9.230916503055e-01
163 KSP Residual norm 2.02242e+08 
164 KSP preconditioned resid norm 2.022395853195e+08 true resid norm 2.616965000510e+04 ||r(i)||/||b|| 9.230917109381e-01
164 KSP Residual norm 2.0224e+08 
165 KSP preconditioned resid norm 2.038571322569e+08 true resid norm 2.616457641497e+04 ||r(i)||/||b|| 9.229127483236e-01
165 KSP Residual norm 2.03857e+08 
166 KSP preconditioned resid norm 2.037514413417e+08 true resid norm 2.616457640363e+04 ||r(i)||/||b|| 9.229127479233e-01
166 KSP Residual norm 2.03751e+08 
167 KSP preconditioned resid norm 2.053085730433e+08 true resid norm 2.615946980360e+04 ||r(i)||/||b|| 9.227326209381e-01
167 KSP Residual norm 2.05309e+08 
168 KSP preconditioned resid norm 2.053032672180e+08 true resid norm 2.615946978529e+04 ||r(i)||/||b|| 9.227326202924e-01
168 KSP Residual norm 2.05303e+08 
169 KSP preconditioned resid norm 1.828123515158e+08 true resid norm 2.623703546424e+04 ||r(i)||/||b|| 9.254686230771e-01
169 KSP Residual norm 1.82812e+08 
170 KSP preconditioned resid norm 1.828098122925e+08 true resid norm 2.623704378628e+04 ||r(i)||/||b|| 9.254689166236e-01
170 KSP Residual norm 1.8281e+08 
171 KSP preconditioned resid norm 1.834673936846e+08 true resid norm 2.623437488930e+04 ||r(i)||/||b|| 9.253747756368e-01
171 KSP Residual norm 1.83467e+08 
172 KSP preconditioned resid norm 1.916771577850e+08 true resid norm 2.620533558988e+04 ||r(i)||/||b|| 9.243504617242e-01
172 KSP Residual norm 1.91677e+08 
173 KSP preconditioned resid norm 1.686641180349e+08 true resid norm 2.628665878348e+04 ||r(i)||/||b|| 9.272190047084e-01
173 KSP Residual norm 1.68664e+08 
174 KSP preconditioned resid norm 1.709601402020e+08 true resid norm 2.627821316745e+04 ||r(i)||/||b|| 9.269210993811e-01
174 KSP Residual norm 1.7096e+08 
175 KSP preconditioned resid norm 7.846116202614e+07 true resid norm 2.642395683327e+04 ||r(i)||/||b|| 9.320619694275e-01
175 KSP Residual norm 7.84612e+07 
176 KSP preconditioned resid norm 5.522839992102e+07 true resid norm 2.642395687169e+04 ||r(i)||/||b|| 9.320619707828e-01
176 KSP Residual norm 5.52284e+07 
177 KSP preconditioned resid norm 5.194245628852e+07 true resid norm 2.642320086058e+04 ||r(i)||/||b|| 9.320353037241e-01
177 KSP Residual norm 5.19425e+07 
178 KSP preconditioned resid norm 5.194058661224e+07 true resid norm 2.642319769115e+04 ||r(i)||/||b|| 9.320351919277e-01
178 KSP Residual norm 5.19406e+07 
179 KSP preconditioned resid norm 5.204188143404e+07 true resid norm 2.642431651695e+04 ||r(i)||/||b|| 9.320746566825e-01
179 KSP Residual norm 5.20419e+07 
180 KSP preconditioned resid norm 5.204080009408e+07 true resid norm 2.642431651631e+04 ||r(i)||/||b|| 9.320746566601e-01
180 KSP Residual norm 5.20408e+07 
181 KSP preconditioned resid norm 5.204864539664e+07 true resid norm 2.642449262498e+04 ||r(i)||/||b|| 9.320808686060e-01
181 KSP Residual norm 5.20486e+07 
182 KSP preconditioned resid norm 5.204844807607e+07 true resid norm 2.642449262510e+04 ||r(i)||/||b|| 9.320808686102e-01
182 KSP Residual norm 5.20484e+07 
183 KSP preconditioned resid norm 3.493548636434e+07 true resid norm 2.571292620106e+04 ||r(i)||/||b|| 9.069815238469e-01
183 KSP Residual norm 3.49355e+07 
184 KSP preconditioned resid norm 4.110412050847e+07 true resid norm 2.573035498554e+04 ||r(i)||/||b|| 9.075962957862e-01
184 KSP Residual norm 4.11041e+07 
185 KSP preconditioned resid norm 4.110493904669e+07 true resid norm 2.573035852944e+04 ||r(i)||/||b|| 9.075964207915e-01
185 KSP Residual norm 4.11049e+07 
186 KSP preconditioned resid norm 4.665669250590e+07 true resid norm 2.575925122542e+04 ||r(i)||/||b|| 9.086155635068e-01
186 KSP Residual norm 4.66567e+07 
187 KSP preconditioned resid norm 4.665810410629e+07 true resid norm 2.575925994183e+04 ||r(i)||/||b|| 9.086158709641e-01
187 KSP Residual norm 4.66581e+07 
188 KSP preconditioned resid norm 4.666263340378e+07 true resid norm 2.575928786768e+04 ||r(i)||/||b|| 9.086168560029e-01
188 KSP Residual norm 4.66626e+07 
189 KSP preconditioned resid norm 4.626669770877e+07 true resid norm 2.575678257532e+04 ||r(i)||/||b|| 9.085284859019e-01
189 KSP Residual norm 4.62667e+07 
190 KSP preconditioned resid norm 4.620748644461e+07 true resid norm 2.575646291068e+04 ||r(i)||/||b|| 9.085172102534e-01
190 KSP Residual norm 4.62075e+07 
191 KSP preconditioned resid norm 4.632759772995e+07 true resid norm 2.575717683196e+04 ||r(i)||/||b|| 9.085423926619e-01
191 KSP Residual norm 4.63276e+07 
192 KSP preconditioned resid norm 4.631864478318e+07 true resid norm 2.575712422051e+04 ||r(i)||/||b|| 9.085405368787e-01
192 KSP Residual norm 4.63186e+07 
193 KSP preconditioned resid norm 4.632152855513e+07 true resid norm 2.575713994644e+04 ||r(i)||/||b|| 9.085410915852e-01
193 KSP Residual norm 4.63215e+07 
194 KSP preconditioned resid norm 4.632228862988e+07 true resid norm 2.575714466753e+04 ||r(i)||/||b|| 9.085412581140e-01
194 KSP Residual norm 4.63223e+07 
195 KSP preconditioned resid norm 4.632227766263e+07 true resid norm 2.575714466849e+04 ||r(i)||/||b|| 9.085412581477e-01
195 KSP Residual norm 4.63223e+07 
196 KSP preconditioned resid norm 4.631487133291e+07 true resid norm 2.575703630753e+04 ||r(i)||/||b|| 9.085374358916e-01
196 KSP Residual norm 4.63149e+07 
197 KSP preconditioned resid norm 4.631488029126e+07 true resid norm 2.575704583406e+04 ||r(i)||/||b|| 9.085377719244e-01
197 KSP Residual norm 4.63149e+07 
198 KSP preconditioned resid norm 4.632566198860e+07 true resid norm 2.575699519567e+04 ||r(i)||/||b|| 9.085359857378e-01
198 KSP Residual norm 4.63257e+07 
199 KSP preconditioned resid norm 4.632226688620e+07 true resid norm 2.575699473520e+04 ||r(i)||/||b|| 9.085359694956e-01
199 KSP Residual norm 4.63223e+07 
200 KSP preconditioned resid norm 4.632225655099e+07 true resid norm 2.575699469705e+04 ||r(i)||/||b|| 9.085359681500e-01
200 KSP Residual norm 4.63223e+07 
201 KSP preconditioned resid norm 4.633086727249e+07 true resid norm 2.575702312484e+04 ||r(i)||/||b|| 9.085369708940e-01
201 KSP Residual norm 4.63309e+07 
202 KSP preconditioned resid norm 4.630387372316e+07 true resid norm 2.575692877878e+04 ||r(i)||/||b|| 9.085336429905e-01
202 KSP Residual norm 4.63039e+07 
203 KSP preconditioned resid norm 4.613960410187e+07 true resid norm 2.575635581328e+04 ||r(i)||/||b|| 9.085134325674e-01
203 KSP Residual norm 4.61396e+07 
204 KSP preconditioned resid norm 4.611641031108e+07 true resid norm 2.575627327847e+04 ||r(i)||/||b|| 9.085105212864e-01
204 KSP Residual norm 4.61164e+07 
205 KSP preconditioned resid norm 4.611624745847e+07 true resid norm 2.575627303518e+04 ||r(i)||/||b|| 9.085105127049e-01
205 KSP Residual norm 4.61162e+07 
206 KSP preconditioned resid norm 4.522026679241e+07 true resid norm 2.574084144820e+04 ||r(i)||/||b|| 9.079661886489e-01
206 KSP Residual norm 4.52203e+07 
207 KSP preconditioned resid norm 4.440654191250e+07 true resid norm 2.574083736616e+04 ||r(i)||/||b|| 9.079660446617e-01
207 KSP Residual norm 4.44065e+07 
208 KSP preconditioned resid norm 4.435532670230e+07 true resid norm 2.574084204541e+04 ||r(i)||/||b|| 9.079662097145e-01
208 KSP Residual norm 4.43553e+07 
209 KSP preconditioned resid norm 4.435529161412e+07 true resid norm 2.574084205091e+04 ||r(i)||/||b|| 9.079662099086e-01
209 KSP Residual norm 4.43553e+07 
210 KSP preconditioned resid norm 5.008864960243e+07 true resid norm 2.570344416601e+04 ||r(i)||/||b|| 9.066470605296e-01
210 KSP Residual norm 5.00886e+07 
211 KSP preconditioned resid norm 5.008665719311e+07 true resid norm 2.570345353484e+04 ||r(i)||/||b|| 9.066473909995e-01
211 KSP Residual norm 5.00867e+07 
212 KSP preconditioned resid norm 4.873328531855e+07 true resid norm 2.571033463076e+04 ||r(i)||/||b|| 9.068901104325e-01
212 KSP Residual norm 4.87333e+07 
213 KSP preconditioned resid norm 4.872391062208e+07 true resid norm 2.571033841344e+04 ||r(i)||/||b|| 9.068902438603e-01
213 KSP Residual norm 4.87239e+07 
214 KSP preconditioned resid norm 1.285333990139e+07 true resid norm 2.633261896477e+04 ||r(i)||/||b|| 9.288401751240e-01
214 KSP Residual norm 1.28533e+07 
215 KSP preconditioned resid norm 1.375301658571e+07 true resid norm 2.630366478461e+04 ||r(i)||/||b|| 9.278188636548e-01
215 KSP Residual norm 1.3753e+07 
216 KSP preconditioned resid norm 6.463279915453e+07 true resid norm 2.570172838821e+04 ||r(i)||/||b|| 9.065865392666e-01
216 KSP Residual norm 6.46328e+07 
217 KSP preconditioned resid norm 6.462779374862e+07 true resid norm 2.570170607518e+04 ||r(i)||/||b|| 9.065857522108e-01
217 KSP Residual norm 6.46278e+07 
218 KSP preconditioned resid norm 6.291711309111e+07 true resid norm 2.569509155679e+04 ||r(i)||/||b|| 9.063524358656e-01
218 KSP Residual norm 6.29171e+07 
219 KSP preconditioned resid norm 6.285783026106e+07 true resid norm 2.569506812611e+04 ||r(i)||/||b|| 9.063516093866e-01
219 KSP Residual norm 6.28578e+07 
220 KSP preconditioned resid norm 6.303127974998e+07 true resid norm 2.569568787704e+04 ||r(i)||/||b|| 9.063734700895e-01
220 KSP Residual norm 6.30313e+07 
221 KSP preconditioned resid norm 6.297318826605e+07 true resid norm 2.569549448922e+04 ||r(i)||/||b|| 9.063666486496e-01
221 KSP Residual norm 6.29732e+07 
222 KSP preconditioned resid norm 6.260949091108e+07 true resid norm 2.569428853522e+04 ||r(i)||/||b|| 9.063241105899e-01
222 KSP Residual norm 6.26095e+07 
223 KSP preconditioned resid norm 6.261557064067e+07 true resid norm 2.569432546223e+04 ||r(i)||/||b|| 9.063254131296e-01
223 KSP Residual norm 6.26156e+07 
224 KSP preconditioned resid norm 5.951473226337e+06 true resid norm 2.667511060136e+04 ||r(i)||/||b|| 9.409210088664e-01
224 KSP Residual norm 5.95147e+06 
225 KSP preconditioned resid norm 5.951452921487e+06 true resid norm 2.667498289716e+04 ||r(i)||/||b|| 9.409165043091e-01
225 KSP Residual norm 5.95145e+06 
226 KSP preconditioned resid norm 5.951436438422e+06 true resid norm 2.667473247750e+04 ||r(i)||/||b|| 9.409076711640e-01
226 KSP Residual norm 5.95144e+06 
227 KSP preconditioned resid norm 5.951443951506e+06 true resid norm 2.667474724075e+04 ||r(i)||/||b|| 9.409081919136e-01
227 KSP Residual norm 5.95144e+06 
228 KSP preconditioned resid norm 5.951380888414e+06 true resid norm 2.667461051769e+04 ||r(i)||/||b|| 9.409033692306e-01
228 KSP Residual norm 5.95138e+06 
229 KSP preconditioned resid norm 5.952408132430e+06 true resid norm 2.667643831927e+04 ||r(i)||/||b|| 9.409678419496e-01
229 KSP Residual norm 5.95241e+06 
230 KSP preconditioned resid norm 5.953624680884e+06 true resid norm 2.667802076716e+04 ||r(i)||/||b|| 9.410236602173e-01
230 KSP Residual norm 5.95362e+06 
231 KSP preconditioned resid norm 5.954824237410e+06 true resid norm 2.667928505223e+04 ||r(i)||/||b|| 9.410682558107e-01
231 KSP Residual norm 5.95482e+06 
232 KSP preconditioned resid norm 5.952281807267e+06 true resid norm 2.667627989768e+04 ||r(i)||/||b|| 9.409622538866e-01
232 KSP Residual norm 5.95228e+06 
233 KSP preconditioned resid norm 5.960861208790e+06 true resid norm 2.666047881324e+04 ||r(i)||/||b|| 9.404048964107e-01
233 KSP Residual norm 5.96086e+06 
234 KSP preconditioned resid norm 5.956181940673e+06 true resid norm 2.666366484470e+04 ||r(i)||/||b|| 9.405172784728e-01
234 KSP Residual norm 5.95618e+06 
235 KSP preconditioned resid norm 5.954538882390e+06 true resid norm 2.666564387831e+04 ||r(i)||/||b|| 9.405870856547e-01
235 KSP Residual norm 5.95454e+06 
236 KSP preconditioned resid norm 5.954547537849e+06 true resid norm 2.666562361166e+04 ||r(i)||/||b|| 9.405863707816e-01
236 KSP Residual norm 5.95455e+06 
237 KSP preconditioned resid norm 5.953797604275e+06 true resid norm 2.666685112942e+04 ||r(i)||/||b|| 9.406296694681e-01
237 KSP Residual norm 5.9538e+06 
238 KSP preconditioned resid norm 5.954425350666e+06 true resid norm 2.666571870149e+04 ||r(i)||/||b|| 9.405897249204e-01
238 KSP Residual norm 5.95443e+06 
239 KSP preconditioned resid norm 5.955718319708e+06 true resid norm 2.666397343198e+04 ||r(i)||/||b|| 9.405281633856e-01
239 KSP Residual norm 5.95572e+06 
240 KSP preconditioned resid norm 5.954436274133e+06 true resid norm 2.666543763666e+04 ||r(i)||/||b|| 9.405798108168e-01
240 KSP Residual norm 5.95444e+06 
241 KSP preconditioned resid norm 5.954724290986e+06 true resid norm 2.666512479036e+04 ||r(i)||/||b|| 9.405687756742e-01
241 KSP Residual norm 5.95472e+06 
242 KSP preconditioned resid norm 5.954892822775e+06 true resid norm 2.666497154237e+04 ||r(i)||/||b|| 9.405633701011e-01
242 KSP Residual norm 5.95489e+06 
243 KSP preconditioned resid norm 5.955127697992e+06 true resid norm 2.666482901545e+04 ||r(i)||/||b|| 9.405583426967e-01
243 KSP Residual norm 5.95513e+06 
244 KSP preconditioned resid norm 5.955115610210e+06 true resid norm 2.666483414335e+04 ||r(i)||/||b|| 9.405585235748e-01
244 KSP Residual norm 5.95512e+06 
245 KSP preconditioned resid norm 5.954981617954e+06 true resid norm 2.666486482168e+04 ||r(i)||/||b|| 9.405596057030e-01
245 KSP Residual norm 5.95498e+06 
246 KSP preconditioned resid norm 5.955009279295e+06 true resid norm 2.666486597277e+04 ||r(i)||/||b|| 9.405596463057e-01
246 KSP Residual norm 5.95501e+06 
247 KSP preconditioned resid norm 5.955037520489e+06 true resid norm 2.666486789677e+04 ||r(i)||/||b|| 9.405597141717e-01
247 KSP Residual norm 5.95504e+06 
248 KSP preconditioned resid norm 5.955036670997e+06 true resid norm 2.666486859340e+04 ||r(i)||/||b|| 9.405597387444e-01
248 KSP Residual norm 5.95504e+06 
249 KSP preconditioned resid norm 5.954620763678e+06 true resid norm 2.666525495155e+04 ||r(i)||/||b|| 9.405733668976e-01
249 KSP Residual norm 5.95462e+06 
250 KSP preconditioned resid norm 5.955265027524e+06 true resid norm 2.666465617073e+04 ||r(i)||/||b|| 9.405522458810e-01
250 KSP Residual norm 5.95527e+06 
251 KSP preconditioned resid norm 5.953687790205e+06 true resid norm 2.666656383150e+04 ||r(i)||/||b|| 9.406195355026e-01
251 KSP Residual norm 5.95369e+06 
252 KSP preconditioned resid norm 5.953643908144e+06 true resid norm 2.666661477855e+04 ||r(i)||/||b|| 9.406213325767e-01
252 KSP Residual norm 5.95364e+06 
253 KSP preconditioned resid norm 5.957968961882e+06 true resid norm 2.666267474144e+04 ||r(i)||/||b|| 9.404823541953e-01
253 KSP Residual norm 5.95797e+06 
254 KSP preconditioned resid norm 5.967273485925e+06 true resid norm 2.665739200142e+04 ||r(i)||/||b|| 9.402960141594e-01
254 KSP Residual norm 5.96727e+06 
255 KSP preconditioned resid norm 5.968317576988e+06 true resid norm 2.665691903179e+04 ||r(i)||/||b|| 9.402793309275e-01
255 KSP Residual norm 5.96832e+06 
256 KSP preconditioned resid norm 5.966055859533e+06 true resid norm 2.665796386003e+04 ||r(i)||/||b|| 9.403161855390e-01
256 KSP Residual norm 5.96606e+06 
257 KSP preconditioned resid norm 5.952086969114e+06 true resid norm 2.666902557238e+04 ||r(i)||/||b|| 9.407063693962e-01
257 KSP Residual norm 5.95209e+06 
258 KSP preconditioned resid norm 5.955066442083e+06 true resid norm 2.667906281435e+04 ||r(i)||/||b|| 9.410604167318e-01
258 KSP Residual norm 5.95507e+06 
259 KSP preconditioned resid norm 5.952126835134e+06 true resid norm 2.666833747433e+04 ||r(i)||/||b|| 9.406820978600e-01
259 KSP Residual norm 5.95213e+06 
260 KSP preconditioned resid norm 5.952141519067e+06 true resid norm 2.666830175508e+04 ||r(i)||/||b|| 9.406808379219e-01
260 KSP Residual norm 5.95214e+06 
261 KSP preconditioned resid norm 5.952673555778e+06 true resid norm 2.666732663998e+04 ||r(i)||/||b|| 9.406464423273e-01
261 KSP Residual norm 5.95267e+06 
262 KSP preconditioned resid norm 5.952461287632e+06 true resid norm 2.666758592323e+04 ||r(i)||/||b|| 9.406555881210e-01
262 KSP Residual norm 5.95246e+06 
263 KSP preconditioned resid norm 5.952476013743e+06 true resid norm 2.666756669234e+04 ||r(i)||/||b|| 9.406549097827e-01
263 KSP Residual norm 5.95248e+06 
264 KSP preconditioned resid norm 6.012511087576e+06 true resid norm 2.670228582800e+04 ||r(i)||/||b|| 9.418795706526e-01
264 KSP Residual norm 6.01251e+06 
265 KSP preconditioned resid norm 5.952693706888e+06 true resid norm 2.666759108019e+04 ||r(i)||/||b|| 9.406557700245e-01
265 KSP Residual norm 5.95269e+06 
266 KSP preconditioned resid norm 5.952793800274e+06 true resid norm 2.666747297033e+04 ||r(i)||/||b|| 9.406516038917e-01
266 KSP Residual norm 5.95279e+06 
267 KSP preconditioned resid norm 5.953160266475e+06 true resid norm 2.666704190510e+04 ||r(i)||/||b|| 9.406363987689e-01
267 KSP Residual norm 5.95316e+06 
268 KSP preconditioned resid norm 5.952817182579e+06 true resid norm 2.666757659066e+04 ||r(i)||/||b|| 9.406552589297e-01
268 KSP Residual norm 5.95282e+06 
269 KSP preconditioned resid norm 5.952829056250e+06 true resid norm 2.666756727152e+04 ||r(i)||/||b|| 9.406549302124e-01
269 KSP Residual norm 5.95283e+06 
270 KSP preconditioned resid norm 5.952993865769e+06 true resid norm 2.666742456291e+04 ||r(i)||/||b|| 9.406498963990e-01
270 KSP Residual norm 5.95299e+06 
271 KSP preconditioned resid norm 5.952743614295e+06 true resid norm 2.666765842124e+04 ||r(i)||/||b|| 9.406581453701e-01
271 KSP Residual norm 5.95274e+06 
272 KSP preconditioned resid norm 5.954012172749e+06 true resid norm 2.666654603506e+04 ||r(i)||/||b|| 9.406189077622e-01
272 KSP Residual norm 5.95401e+06 
273 KSP preconditioned resid norm 5.992303612445e+06 true resid norm 2.669997738981e+04 ||r(i)||/||b|| 9.417981442614e-01
273 KSP Residual norm 5.9923e+06 
274 KSP preconditioned resid norm 6.001038839709e+06 true resid norm 2.670238837979e+04 ||r(i)||/||b|| 9.418831879996e-01
274 KSP Residual norm 6.00104e+06 
275 KSP preconditioned resid norm 8.864701512063e+06 true resid norm 2.691280962987e+04 ||r(i)||/||b|| 9.493054543163e-01
275 KSP Residual norm 8.8647e+06 
276 KSP preconditioned resid norm 6.837184933510e+06 true resid norm 2.656250168656e+04 ||r(i)||/||b|| 9.369489131062e-01
276 KSP Residual norm 6.83718e+06 
277 KSP preconditioned resid norm 2.012493473966e+07 true resid norm 2.743622005883e+04 ||r(i)||/||b|| 9.677679033098e-01
277 KSP Residual norm 2.01249e+07 
278 KSP preconditioned resid norm 2.220353164993e+07 true resid norm 2.753421762421e+04 ||r(i)||/||b|| 9.712246075558e-01
278 KSP Residual norm 2.22035e+07 
279 KSP preconditioned resid norm 5.965928653692e+06 true resid norm 2.669063560619e+04 ||r(i)||/||b|| 9.414686280842e-01
279 KSP Residual norm 5.96593e+06 
280 KSP preconditioned resid norm 5.958031121970e+06 true resid norm 2.666372707266e+04 ||r(i)||/||b|| 9.405194734625e-01
280 KSP Residual norm 5.95803e+06 
281 KSP preconditioned resid norm 9.347845981984e+06 true resid norm 2.693713593549e+04 ||r(i)||/||b|| 9.501635250615e-01
281 KSP Residual norm 9.34785e+06 
282 KSP preconditioned resid norm 9.434290706765e+06 true resid norm 2.694142914095e+04 ||r(i)||/||b|| 9.503149608802e-01
282 KSP Residual norm 9.43429e+06 
283 KSP preconditioned resid norm 9.444985258628e+06 true resid norm 2.694195811744e+04 ||r(i)||/||b|| 9.503336196630e-01
283 KSP Residual norm 9.44499e+06 
284 KSP preconditioned resid norm 8.902004262959e+06 true resid norm 2.691469745841e+04 ||r(i)||/||b|| 9.493720443883e-01
284 KSP Residual norm 8.902e+06 
285 KSP preconditioned resid norm 9.190592344455e+06 true resid norm 2.692929705960e+04 ||r(i)||/||b|| 9.498870215026e-01
285 KSP Residual norm 9.19059e+06 
286 KSP preconditioned resid norm 6.516380355017e+06 true resid norm 2.676843818641e+04 ||r(i)||/||b|| 9.442129871751e-01
286 KSP Residual norm 6.51638e+06 
287 KSP preconditioned resid norm 6.626119704895e+06 true resid norm 2.677771617775e+04 ||r(i)||/||b|| 9.445402531833e-01
287 KSP Residual norm 6.62612e+06 
288 KSP preconditioned resid norm 6.452387585878e+06 true resid norm 2.676267557617e+04 ||r(i)||/||b|| 9.440097204998e-01
288 KSP Residual norm 6.45239e+06 
289 KSP preconditioned resid norm 6.210111022107e+06 true resid norm 2.673720628869e+04 ||r(i)||/||b|| 9.431113329345e-01
289 KSP Residual norm 6.21011e+06 
290 KSP preconditioned resid norm 1.625917943102e+07 true resid norm 2.725770495958e+04 ||r(i)||/||b|| 9.614710744120e-01
290 KSP Residual norm 1.62592e+07 
291 KSP preconditioned resid norm 1.130001123670e+07 true resid norm 2.637115751978e+04 ||r(i)||/||b|| 9.301995597805e-01
291 KSP Residual norm 1.13e+07 
292 KSP preconditioned resid norm 6.815561788034e+06 true resid norm 2.679242945846e+04 ||r(i)||/||b|| 9.450592401575e-01
292 KSP Residual norm 6.81556e+06 
293 KSP preconditioned resid norm 6.532669400752e+06 true resid norm 2.676985263634e+04 ||r(i)||/||b|| 9.442628795888e-01
293 KSP Residual norm 6.53267e+06 
294 KSP preconditioned resid norm 6.048470192747e+06 true resid norm 2.671301519507e+04 ||r(i)||/||b|| 9.422580315721e-01
294 KSP Residual norm 6.04847e+06 
295 KSP preconditioned resid norm 6.072339843866e+06 true resid norm 2.671738980538e+04 ||r(i)||/||b|| 9.424123388142e-01
295 KSP Residual norm 6.07234e+06 
296 KSP preconditioned resid norm 6.116777518701e+06 true resid norm 2.672458449878e+04 ||r(i)||/||b|| 9.426661198865e-01
296 KSP Residual norm 6.11678e+06 
297 KSP preconditioned resid norm 3.364141322146e+07 true resid norm 2.810600636459e+04 ||r(i)||/||b|| 9.913935225606e-01
297 KSP Residual norm 3.36414e+07 
298 KSP preconditioned resid norm 5.948890530157e+06 true resid norm 2.667441749736e+04 ||r(i)||/||b|| 9.408965607533e-01
298 KSP Residual norm 5.94889e+06 
299 KSP preconditioned resid norm 5.951518748847e+06 true resid norm 2.668119731354e+04 ||r(i)||/||b|| 9.411357077088e-01
299 KSP Residual norm 5.95152e+06 
300 KSP preconditioned resid norm 1.044927296976e+07 true resid norm 2.640074675427e+04 ||r(i)||/||b|| 9.312432717557e-01
300 KSP Residual norm 1.04493e+07 
301 KSP preconditioned resid norm 1.056430681083e+07 true resid norm 2.639663904991e+04 ||r(i)||/||b|| 9.310983791857e-01
301 KSP Residual norm 1.05643e+07 
302 KSP preconditioned resid norm 1.004111410662e+07 true resid norm 2.641559875415e+04 ||r(i)||/||b|| 9.317671518219e-01
302 KSP Residual norm 1.00411e+07 
303 KSP preconditioned resid norm 5.956968881367e+06 true resid norm 2.666434231230e+04 ||r(i)||/||b|| 9.405411750370e-01
303 KSP Residual norm 5.95697e+06 
304 KSP preconditioned resid norm 6.138256681452e+06 true resid norm 2.672771306926e+04 ||r(i)||/||b|| 9.427764751063e-01
304 KSP Residual norm 6.13826e+06 
305 KSP preconditioned resid norm 5.954241714167e+06 true resid norm 2.668390888394e+04 ||r(i)||/||b|| 9.412313539307e-01
305 KSP Residual norm 5.95424e+06 
306 KSP preconditioned resid norm 6.010159270590e+06 true resid norm 2.664581405688e+04 ||r(i)||/||b|| 9.398876210538e-01
306 KSP Residual norm 6.01016e+06 
307 KSP preconditioned resid norm 5.956518225731e+06 true resid norm 2.668555037080e+04 ||r(i)||/||b|| 9.412892547018e-01
307 KSP Residual norm 5.95652e+06 
308 KSP preconditioned resid norm 6.159309785014e+06 true resid norm 2.673064838351e+04 ||r(i)||/||b|| 9.428800135276e-01
308 KSP Residual norm 6.15931e+06 
309 KSP preconditioned resid norm 1.194967720748e+07 true resid norm 2.706085956022e+04 ||r(i)||/||b|| 9.545276740819e-01
309 KSP Residual norm 1.19497e+07 
310 KSP preconditioned resid norm 1.336145584855e+07 true resid norm 2.712563259064e+04 ||r(i)||/||b|| 9.568124370597e-01
310 KSP Residual norm 1.33615e+07 
311 KSP preconditioned resid norm 8.964691191590e+06 true resid norm 2.691806481035e+04 ||r(i)||/||b|| 9.494908222347e-01
311 KSP Residual norm 8.96469e+06 
312 KSP preconditioned resid norm 8.736376917576e+06 true resid norm 2.690635239853e+04 ||r(i)||/||b|| 9.490776860152e-01
312 KSP Residual norm 8.73638e+06 
313 KSP preconditioned resid norm 1.069242992348e+07 true resid norm 2.700227746742e+04 ||r(i)||/||b|| 9.524612863286e-01
313 KSP Residual norm 1.06924e+07 
314 KSP preconditioned resid norm 9.023469334642e+06 true resid norm 2.692105099290e+04 ||r(i)||/||b|| 9.495961549523e-01
314 KSP Residual norm 9.02347e+06 
315 KSP preconditioned resid norm 7.648645489992e+06 true resid norm 2.684675795438e+04 ||r(i)||/||b|| 9.469755892199e-01
315 KSP Residual norm 7.64865e+06 
316 KSP preconditioned resid norm 7.317627423305e+06 true resid norm 2.682661073347e+04 ||r(i)||/||b|| 9.462649288703e-01
316 KSP Residual norm 7.31763e+06 
317 KSP preconditioned resid norm 6.192705956930e+06 true resid norm 2.661676761094e+04 ||r(i)||/||b|| 9.388630550596e-01
317 KSP Residual norm 6.19271e+06 
318 KSP preconditioned resid norm 9.176999099012e+06 true resid norm 2.644874439384e+04 ||r(i)||/||b|| 9.329363101884e-01
318 KSP Residual norm 9.177e+06 
319 KSP preconditioned resid norm 5.949267576414e+06 true resid norm 2.667205809462e+04 ||r(i)||/||b|| 9.408133366709e-01
319 KSP Residual norm 5.94927e+06 
320 KSP preconditioned resid norm 5.960257875352e+06 true resid norm 2.668791677393e+04 ||r(i)||/||b|| 9.413727257120e-01
320 KSP Residual norm 5.96026e+06 
321 KSP preconditioned resid norm 6.017228573506e+06 true resid norm 2.664418369792e+04 ||r(i)||/||b|| 9.398301128014e-01
321 KSP Residual norm 6.01723e+06 
322 KSP preconditioned resid norm 5.975714381365e+06 true resid norm 2.665564605683e+04 ||r(i)||/||b|| 9.402344288124e-01
322 KSP Residual norm 5.97571e+06 
323 KSP preconditioned resid norm 6.068465133376e+06 true resid norm 2.671671260134e+04 ||r(i)||/||b|| 9.423884515465e-01
323 KSP Residual norm 6.06847e+06 
324 KSP preconditioned resid norm 6.076606616885e+06 true resid norm 2.671812883072e+04 ||r(i)||/||b|| 9.424384067273e-01
324 KSP Residual norm 6.07661e+06 
325 KSP preconditioned resid norm 6.045965191463e+06 true resid norm 2.671251770386e+04 ||r(i)||/||b|| 9.422404833814e-01
325 KSP Residual norm 6.04597e+06 
326 KSP preconditioned resid norm 6.103168657457e+06 true resid norm 2.672249404203e+04 ||r(i)||/||b|| 9.425923824349e-01
326 KSP Residual norm 6.10317e+06 
327 KSP preconditioned resid norm 6.407529905136e+06 true resid norm 2.675856096747e+04 ||r(i)||/||b|| 9.438645843905e-01
327 KSP Residual norm 6.40753e+06 
328 KSP preconditioned resid norm 6.415532719976e+06 true resid norm 2.675932855985e+04 ||r(i)||/||b|| 9.438916599593e-01
328 KSP Residual norm 6.41553e+06 
329 KSP preconditioned resid norm 7.248185051371e+06 true resid norm 2.682233611942e+04 ||r(i)||/||b|| 9.461141488332e-01
329 KSP Residual norm 7.24819e+06 
330 KSP preconditioned resid norm 6.059876109036e+06 true resid norm 2.671517058161e+04 ||r(i)||/||b|| 9.423340593161e-01
330 KSP Residual norm 6.05988e+06 
331 KSP preconditioned resid norm 6.136391329972e+06 true resid norm 2.672751195572e+04 ||r(i)||/||b|| 9.427693811541e-01
331 KSP Residual norm 6.13639e+06 
332 KSP preconditioned resid norm 6.118047286073e+06 true resid norm 2.672481461828e+04 ||r(i)||/||b|| 9.426742369764e-01
332 KSP Residual norm 6.11805e+06 
333 KSP preconditioned resid norm 6.101072366720e+06 true resid norm 2.672219610437e+04 ||r(i)||/||b|| 9.425818731700e-01
333 KSP Residual norm 6.10107e+06 
334 KSP preconditioned resid norm 5.973355326508e+06 true resid norm 2.669388593460e+04 ||r(i)||/||b|| 9.415832781164e-01
334 KSP Residual norm 5.97336e+06 
335 KSP preconditioned resid norm 5.977018243292e+06 true resid norm 2.669523195786e+04 ||r(i)||/||b|| 9.416307568909e-01
335 KSP Residual norm 5.97702e+06 
336 KSP preconditioned resid norm 6.023483038848e+06 true resid norm 2.664271168322e+04 ||r(i)||/||b|| 9.397781898843e-01
336 KSP Residual norm 6.02348e+06 
337 KSP preconditioned resid norm 5.948640374266e+06 true resid norm 2.667679485995e+04 ||r(i)||/||b|| 9.409804183402e-01
337 KSP Residual norm 5.94864e+06 
338 KSP preconditioned resid norm 5.963777397633e+06 true resid norm 2.668981879637e+04 ||r(i)||/||b|| 9.414398164506e-01
338 KSP Residual norm 5.96378e+06 
339 KSP preconditioned resid norm 6.029601820345e+06 true resid norm 2.670923687534e+04 ||r(i)||/||b|| 9.421247575077e-01
339 KSP Residual norm 6.0296e+06 
340 KSP preconditioned resid norm 6.004680660719e+06 true resid norm 2.670343680259e+04 ||r(i)||/||b|| 9.419201694036e-01
340 KSP Residual norm 6.00468e+06 
341 KSP preconditioned resid norm 6.017010571083e+06 true resid norm 2.670643792295e+04 ||r(i)||/||b|| 9.420260290281e-01
341 KSP Residual norm 6.01701e+06 
342 KSP preconditioned resid norm 6.150883985772e+06 true resid norm 2.672959784238e+04 ||r(i)||/||b|| 9.428429574030e-01
342 KSP Residual norm 6.15088e+06 
343 KSP preconditioned resid norm 6.021905215561e+06 true resid norm 2.670754920299e+04 ||r(i)||/||b|| 9.420652276186e-01
343 KSP Residual norm 6.02191e+06 
344 KSP preconditioned resid norm 5.952861183777e+06 true resid norm 2.668278211691e+04 ||r(i)||/||b|| 9.411916090620e-01
344 KSP Residual norm 5.95286e+06 
345 KSP preconditioned resid norm 6.106338548056e+06 true resid norm 2.672296766614e+04 ||r(i)||/||b|| 9.426090887526e-01
345 KSP Residual norm 6.10634e+06 
346 KSP preconditioned resid norm 6.105492496792e+06 true resid norm 2.672283652613e+04 ||r(i)||/||b|| 9.426044630029e-01
346 KSP Residual norm 6.10549e+06 
347 KSP preconditioned resid norm 6.085358538112e+06 true resid norm 2.671951882044e+04 ||r(i)||/||b|| 9.424874363472e-01
347 KSP Residual norm 6.08536e+06 
348 KSP preconditioned resid norm 6.085943222038e+06 true resid norm 2.671961863242e+04 ||r(i)||/||b|| 9.424909570517e-01
348 KSP Residual norm 6.08594e+06 
349 KSP preconditioned resid norm 6.092130919547e+06 true resid norm 2.672064456099e+04 ||r(i)||/||b|| 9.425271450086e-01
349 KSP Residual norm 6.09213e+06 
350 KSP preconditioned resid norm 6.104606007013e+06 true resid norm 2.672264975742e+04 ||r(i)||/||b|| 9.425978750413e-01
350 KSP Residual norm 6.10461e+06 
351 KSP preconditioned resid norm 6.108639544613e+06 true resid norm 2.672327741800e+04 ||r(i)||/||b|| 9.426200147444e-01
351 KSP Residual norm 6.10864e+06 
352 KSP preconditioned resid norm 6.085018979329e+06 true resid norm 2.671950171148e+04 ||r(i)||/||b|| 9.424868328565e-01
352 KSP Residual norm 6.08502e+06 
353 KSP preconditioned resid norm 6.101028132550e+06 true resid norm 2.672210491830e+04 ||r(i)||/||b|| 9.425786567303e-01
353 KSP Residual norm 6.10103e+06 
354 KSP preconditioned resid norm 6.099765000056e+06 true resid norm 2.672190651832e+04 ||r(i)||/||b|| 9.425716584946e-01
354 KSP Residual norm 6.09977e+06 
355 KSP preconditioned resid norm 6.170677159819e+06 true resid norm 2.673208202732e+04 ||r(i)||/||b|| 9.429305829742e-01
355 KSP Residual norm 6.17068e+06 
356 KSP preconditioned resid norm 6.232971193231e+06 true resid norm 2.673983718918e+04 ||r(i)||/||b|| 9.432041336570e-01
356 KSP Residual norm 6.23297e+06 
357 KSP preconditioned resid norm 6.153050902055e+06 true resid norm 2.672957050094e+04 ||r(i)||/||b|| 9.428419929784e-01
357 KSP Residual norm 6.15305e+06 
358 KSP preconditioned resid norm 6.152441267711e+06 true resid norm 2.672949013517e+04 ||r(i)||/||b|| 9.428391582070e-01
358 KSP Residual norm 6.15244e+06 
359 KSP preconditioned resid norm 6.158132650376e+06 true resid norm 2.673022700816e+04 ||r(i)||/||b|| 9.428651501995e-01
359 KSP Residual norm 6.15813e+06 
360 KSP preconditioned resid norm 6.157183384449e+06 true resid norm 2.673010013641e+04 ||r(i)||/||b|| 9.428606750056e-01
360 KSP Residual norm 6.15718e+06 
361 KSP preconditioned resid norm 6.157416773931e+06 true resid norm 2.673013303514e+04 ||r(i)||/||b|| 9.428618354546e-01
361 KSP Residual norm 6.15742e+06 
362 KSP preconditioned resid norm 6.162295931170e+06 true resid norm 2.673098446477e+04 ||r(i)||/||b|| 9.428918682460e-01
362 KSP Residual norm 6.1623e+06 
363 KSP preconditioned resid norm 6.162169979866e+06 true resid norm 2.673097792379e+04 ||r(i)||/||b|| 9.428916375234e-01
363 KSP Residual norm 6.16217e+06 
364 KSP preconditioned resid norm 6.162182881845e+06 true resid norm 2.673097940205e+04 ||r(i)||/||b|| 9.428916896665e-01
364 KSP Residual norm 6.16218e+06 
365 KSP preconditioned resid norm 6.162529906891e+06 true resid norm 2.673102643865e+04 ||r(i)||/||b|| 9.428933488061e-01
365 KSP Residual norm 6.16253e+06 
366 KSP preconditioned resid norm 6.176179388176e+06 true resid norm 2.673285332274e+04 ||r(i)||/||b|| 9.429577891617e-01
366 KSP Residual norm 6.17618e+06 
367 KSP preconditioned resid norm 6.179708231435e+06 true resid norm 2.673326995504e+04 ||r(i)||/||b|| 9.429724851867e-01
367 KSP Residual norm 6.17971e+06 
368 KSP preconditioned resid norm 6.180460605632e+06 true resid norm 2.673336143910e+04 ||r(i)||/||b|| 9.429757121377e-01
368 KSP Residual norm 6.18046e+06 
369 KSP preconditioned resid norm 3.990429967198e+07 true resid norm 2.836032319013e+04 ||r(i)||/||b|| 1.000364133691e+00
369 KSP Residual norm 3.99043e+07 
370 KSP preconditioned resid norm 6.188023267733e+06 true resid norm 2.673424210235e+04 ||r(i)||/||b|| 9.430067760972e-01
370 KSP Residual norm 6.18802e+06 
371 KSP preconditioned resid norm 6.174162483987e+06 true resid norm 2.673256620393e+04 ||r(i)||/||b|| 9.429476615142e-01
371 KSP Residual norm 6.17416e+06 
372 KSP preconditioned resid norm 6.745485893120e+06 true resid norm 2.678395908023e+04 ||r(i)||/||b|| 9.447604613839e-01
372 KSP Residual norm 6.74549e+06 
373 KSP preconditioned resid norm 6.723191749363e+06 true resid norm 2.678232940896e+04 ||r(i)||/||b|| 9.447029773885e-01
373 KSP Residual norm 6.72319e+06 
374 KSP preconditioned resid norm 6.459654470197e+06 true resid norm 2.676137199795e+04 ||r(i)||/||b|| 9.439637389048e-01
374 KSP Residual norm 6.45965e+06 
375 KSP preconditioned resid norm 6.438944635358e+06 true resid norm 2.675955927385e+04 ||r(i)||/||b|| 9.438997980194e-01
375 KSP Residual norm 6.43894e+06 
376 KSP preconditioned resid norm 1.644524650187e+07 true resid norm 2.724141842109e+04 ||r(i)||/||b|| 9.608965933365e-01
376 KSP Residual norm 1.64452e+07 
377 KSP preconditioned resid norm 6.233771932173e+06 true resid norm 2.673945213827e+04 ||r(i)||/||b|| 9.431905516146e-01
377 KSP Residual norm 6.23377e+06 
378 KSP preconditioned resid norm 6.139347294498e+06 true resid norm 2.672809538720e+04 ||r(i)||/||b|| 9.427899607477e-01
378 KSP Residual norm 6.13935e+06 
379 KSP preconditioned resid norm 6.126283910384e+06 true resid norm 2.672635255979e+04 ||r(i)||/||b|| 9.427284853542e-01
379 KSP Residual norm 6.12628e+06 
380 KSP preconditioned resid norm 6.144087308685e+06 true resid norm 2.672871127174e+04 ||r(i)||/||b|| 9.428116850701e-01
380 KSP Residual norm 6.14409e+06 
381 KSP preconditioned resid norm 6.162121380642e+06 true resid norm 2.673102806989e+04 ||r(i)||/||b|| 9.428934063452e-01
381 KSP Residual norm 6.16212e+06 
382 KSP preconditioned resid norm 6.158572929408e+06 true resid norm 2.673058448026e+04 ||r(i)||/||b|| 9.428777594447e-01
382 KSP Residual norm 6.15857e+06 
383 KSP preconditioned resid norm 6.156595492465e+06 true resid norm 2.673033133593e+04 ||r(i)||/||b|| 9.428688301915e-01
383 KSP Residual norm 6.1566e+06 
384 KSP preconditioned resid norm 6.301780916700e+06 true resid norm 2.674681049680e+04 ||r(i)||/||b|| 9.434501057071e-01
384 KSP Residual norm 6.30178e+06 
385 KSP preconditioned resid norm 6.043288762072e+06 true resid norm 2.671361085048e+04 ||r(i)||/||b|| 9.422790423451e-01
385 KSP Residual norm 6.04329e+06 
386 KSP preconditioned resid norm 6.042773373963e+06 true resid norm 2.671352169258e+04 ||r(i)||/||b|| 9.422758974456e-01
386 KSP Residual norm 6.04277e+06 
387 KSP preconditioned resid norm 6.043271156395e+06 true resid norm 2.671361118941e+04 ||r(i)||/||b|| 9.422790543003e-01
387 KSP Residual norm 6.04327e+06 
388 KSP preconditioned resid norm 6.195046113929e+06 true resid norm 2.673529946324e+04 ||r(i)||/||b|| 9.430440727773e-01
388 KSP Residual norm 6.19505e+06 
389 KSP preconditioned resid norm 6.245467132962e+06 true resid norm 2.674106717309e+04 ||r(i)||/||b|| 9.432475193329e-01
389 KSP Residual norm 6.24547e+06 
390 KSP preconditioned resid norm 6.157744861629e+06 true resid norm 2.673100181401e+04 ||r(i)||/||b|| 9.428924802118e-01
390 KSP Residual norm 6.15774e+06 
391 KSP preconditioned resid norm 6.157825546529e+06 true resid norm 2.673102670870e+04 ||r(i)||/||b|| 9.428933583315e-01
391 KSP Residual norm 6.15783e+06 
392 KSP preconditioned resid norm 6.162710568295e+06 true resid norm 2.673162180574e+04 ||r(i)||/||b|| 9.429143494087e-01
392 KSP Residual norm 6.16271e+06 
393 KSP preconditioned resid norm 6.163289339885e+06 true resid norm 2.673169402058e+04 ||r(i)||/||b|| 9.429168966695e-01
393 KSP Residual norm 6.16329e+06 
394 KSP preconditioned resid norm 6.275036548124e+06 true resid norm 2.674441045036e+04 ||r(i)||/||b|| 9.433654479845e-01
394 KSP Residual norm 6.27504e+06 
395 KSP preconditioned resid norm 6.060232206207e+06 true resid norm 2.673618896793e+04 ||r(i)||/||b|| 9.430754486043e-01
395 KSP Residual norm 6.06023e+06 
396 KSP preconditioned resid norm 5.886959913877e+06 true resid norm 2.673618896584e+04 ||r(i)||/||b|| 9.430754485306e-01
396 KSP Residual norm 5.88696e+06 
397 KSP preconditioned resid norm 5.882454384223e+06 true resid norm 2.673605351925e+04 ||r(i)||/||b|| 9.430706708728e-01
397 KSP Residual norm 5.88245e+06 
398 KSP preconditioned resid norm 5.866351182740e+06 true resid norm 2.673545538251e+04 ||r(i)||/||b|| 9.430495725755e-01
398 KSP Residual norm 5.86635e+06 
399 KSP preconditioned resid norm 5.866994121511e+06 true resid norm 2.673576697924e+04 ||r(i)||/||b|| 9.430605636416e-01
399 KSP Residual norm 5.86699e+06 
400 KSP preconditioned resid norm 5.865996585067e+06 true resid norm 2.673484474655e+04 ||r(i)||/||b|| 9.430280333879e-01
400 KSP Residual norm 5.866e+06 
401 KSP preconditioned resid norm 5.862902513516e+06 true resid norm 2.673408093666e+04 ||r(i)||/||b|| 9.430010912403e-01
401 KSP Residual norm 5.8629e+06 
402 KSP preconditioned resid norm 5.862852756009e+06 true resid norm 2.673409825349e+04 ||r(i)||/||b|| 9.430017020632e-01
402 KSP Residual norm 5.86285e+06 
403 KSP preconditioned resid norm 5.865913216249e+06 true resid norm 2.673494667501e+04 ||r(i)||/||b|| 9.430316287481e-01
403 KSP Residual norm 5.86591e+06 
404 KSP preconditioned resid norm 5.868232992976e+06 true resid norm 2.673479640020e+04 ||r(i)||/||b|| 9.430263280493e-01
404 KSP Residual norm 5.86823e+06 
405 KSP preconditioned resid norm 5.868161684410e+06 true resid norm 2.673477300762e+04 ||r(i)||/||b|| 9.430255029142e-01
405 KSP Residual norm 5.86816e+06 
406 KSP preconditioned resid norm 5.872447589118e+06 true resid norm 2.673596385423e+04 ||r(i)||/||b|| 9.430675080856e-01
406 KSP Residual norm 5.87245e+06 
407 KSP preconditioned resid norm 5.869411778615e+06 true resid norm 2.673543733742e+04 ||r(i)||/||b|| 9.430489360641e-01
407 KSP Residual norm 5.86941e+06 
408 KSP preconditioned resid norm 5.869375278981e+06 true resid norm 2.673543478003e+04 ||r(i)||/||b|| 9.430488458564e-01
408 KSP Residual norm 5.86938e+06 
409 KSP preconditioned resid norm 5.870084978448e+06 true resid norm 2.673557480366e+04 ||r(i)||/||b|| 9.430537849615e-01
409 KSP Residual norm 5.87008e+06 
410 KSP preconditioned resid norm 5.870852177283e+06 true resid norm 2.673579080032e+04 ||r(i)||/||b|| 9.430614038912e-01
410 KSP Residual norm 5.87085e+06 
411 KSP preconditioned resid norm 5.871152607221e+06 true resid norm 2.673584922973e+04 ||r(i)||/||b|| 9.430634648934e-01
411 KSP Residual norm 5.87115e+06 
412 KSP preconditioned resid norm 5.872092409618e+06 true resid norm 2.673603313134e+04 ||r(i)||/||b|| 9.430699517226e-01
412 KSP Residual norm 5.87209e+06 
413 KSP preconditioned resid norm 5.870304292003e+06 true resid norm 2.673570437114e+04 ||r(i)||/||b|| 9.430583552429e-01
413 KSP Residual norm 5.8703e+06 
414 KSP preconditioned resid norm 5.870313225844e+06 true resid norm 2.673572816302e+04 ||r(i)||/||b|| 9.430591944627e-01
414 KSP Residual norm 5.87031e+06 
415 KSP preconditioned resid norm 5.870319043942e+06 true resid norm 2.673573415500e+04 ||r(i)||/||b|| 9.430594058201e-01
415 KSP Residual norm 5.87032e+06 
416 KSP preconditioned resid norm 5.870722861882e+06 true resid norm 2.673616290295e+04 ||r(i)||/||b|| 9.430745292048e-01
416 KSP Residual norm 5.87072e+06 
417 KSP preconditioned resid norm 5.870712110655e+06 true resid norm 2.673628777644e+04 ||r(i)||/||b|| 9.430789339133e-01
417 KSP Residual norm 5.87071e+06 
418 KSP preconditioned resid norm 5.870710521847e+06 true resid norm 2.673629029179e+04 ||r(i)||/||b|| 9.430790226380e-01
418 KSP Residual norm 5.87071e+06 
419 KSP preconditioned resid norm 5.870613304447e+06 true resid norm 2.673631527168e+04 ||r(i)||/||b|| 9.430799037628e-01
419 KSP Residual norm 5.87061e+06 
420 KSP preconditioned resid norm 5.872614386167e+06 true resid norm 2.673622946174e+04 ||r(i)||/||b|| 9.430768769573e-01
420 KSP Residual norm 5.87261e+06 
421 KSP preconditioned resid norm 5.872539494282e+06 true resid norm 2.673623001125e+04 ||r(i)||/||b|| 9.430768963405e-01
421 KSP Residual norm 5.87254e+06 
422 KSP preconditioned resid norm 5.872550209733e+06 true resid norm 2.673622851161e+04 ||r(i)||/||b|| 9.430768434432e-01
422 KSP Residual norm 5.87255e+06 
423 KSP preconditioned resid norm 5.874750656566e+06 true resid norm 2.673591517716e+04 ||r(i)||/||b|| 9.430657910816e-01
423 KSP Residual norm 5.87475e+06 
424 KSP preconditioned resid norm 5.874967373367e+06 true resid norm 2.673588401701e+04 ||r(i)||/||b|| 9.430646919582e-01
424 KSP Residual norm 5.87497e+06 
425 KSP preconditioned resid norm 5.911629538031e+06 true resid norm 2.673063363534e+04 ||r(i)||/||b|| 9.428794933102e-01
425 KSP Residual norm 5.91163e+06 
426 KSP preconditioned resid norm 5.948281044839e+06 true resid norm 2.672543806441e+04 ||r(i)||/||b|| 9.426962280214e-01
426 KSP Residual norm 5.94828e+06 
427 KSP preconditioned resid norm 5.910068655757e+06 true resid norm 2.673086553849e+04 ||r(i)||/||b|| 9.428876733152e-01
427 KSP Residual norm 5.91007e+06 
428 KSP preconditioned resid norm 5.914064632057e+06 true resid norm 2.673029505437e+04 ||r(i)||/||b|| 9.428675504187e-01
428 KSP Residual norm 5.91406e+06 
429 KSP preconditioned resid norm 5.927350852902e+06 true resid norm 2.672837375799e+04 ||r(i)||/||b|| 9.427997798232e-01
429 KSP Residual norm 5.92735e+06 
430 KSP preconditioned resid norm 5.928399314681e+06 true resid norm 2.672822118816e+04 ||r(i)||/||b|| 9.427943981713e-01
430 KSP Residual norm 5.9284e+06 
431 KSP preconditioned resid norm 5.919866502126e+06 true resid norm 2.672945841744e+04 ||r(i)||/||b|| 9.428380394160e-01
431 KSP Residual norm 5.91987e+06 
432 KSP preconditioned resid norm 5.849919401647e+06 true resid norm 2.673972528732e+04 ||r(i)||/||b|| 9.432001865014e-01
432 KSP Residual norm 5.84992e+06 
433 KSP preconditioned resid norm 5.886835954180e+06 true resid norm 2.673427575418e+04 ||r(i)||/||b|| 9.430079631104e-01
433 KSP Residual norm 5.88684e+06 
434 KSP preconditioned resid norm 5.420593468173e+06 true resid norm 2.680844351757e+04 ||r(i)||/||b|| 9.456241099673e-01
434 KSP Residual norm 5.42059e+06 
435 KSP preconditioned resid norm 6.040695319549e+06 true resid norm 2.671221565991e+04 ||r(i)||/||b|| 9.422298292739e-01
435 KSP Residual norm 6.0407e+06 
436 KSP preconditioned resid norm 5.962578763028e+06 true resid norm 2.672329339848e+04 ||r(i)||/||b|| 9.426205784295e-01
436 KSP Residual norm 5.96258e+06 
437 KSP preconditioned resid norm 5.928683402351e+06 true resid norm 2.672818818263e+04 ||r(i)||/||b|| 9.427932339553e-01
437 KSP Residual norm 5.92868e+06 
438 KSP preconditioned resid norm 5.947807075642e+06 true resid norm 2.672542049841e+04 ||r(i)||/||b|| 9.426956084095e-01
438 KSP Residual norm 5.94781e+06 
439 KSP preconditioned resid norm 5.973371195166e+06 true resid norm 2.672174327283e+04 ||r(i)||/||b|| 9.425659002762e-01
439 KSP Residual norm 5.97337e+06 
440 KSP preconditioned resid norm 5.921543788702e+06 true resid norm 2.672922848178e+04 ||r(i)||/||b|| 9.428299288106e-01
440 KSP Residual norm 5.92154e+06 
441 KSP preconditioned resid norm 5.887372979855e+06 true resid norm 2.673416865240e+04 ||r(i)||/||b|| 9.430041852697e-01
441 KSP Residual norm 5.88737e+06 
442 KSP preconditioned resid norm 5.885220882800e+06 true resid norm 2.673448852179e+04 ||r(i)||/||b|| 9.430154681408e-01
442 KSP Residual norm 5.88522e+06 
443 KSP preconditioned resid norm 5.887695484610e+06 true resid norm 2.673412478186e+04 ||r(i)||/||b|| 9.430026378082e-01
443 KSP Residual norm 5.8877e+06 
444 KSP preconditioned resid norm 5.914882063326e+06 true resid norm 2.673014150251e+04 ||r(i)||/||b|| 9.428621341275e-01
444 KSP Residual norm 5.91488e+06 
445 KSP preconditioned resid norm 5.917861145830e+06 true resid norm 2.672970900293e+04 ||r(i)||/||b|| 9.428468784104e-01
445 KSP Residual norm 5.91786e+06 
446 KSP preconditioned resid norm 5.876638144030e+06 true resid norm 2.673574656826e+04 ||r(i)||/||b|| 9.430598436777e-01
446 KSP Residual norm 5.87664e+06 
447 KSP preconditioned resid norm 5.897899345913e+06 true resid norm 2.673262596706e+04 ||r(i)||/||b|| 9.429497695611e-01
447 KSP Residual norm 5.8979e+06 
448 KSP preconditioned resid norm 5.864754809823e+06 true resid norm 2.673749545145e+04 ||r(i)||/||b|| 9.431215326792e-01
448 KSP Residual norm 5.86475e+06 
449 KSP preconditioned resid norm 6.046049235198e+06 true resid norm 2.671145984820e+04 ||r(i)||/||b|| 9.422031692486e-01
449 KSP Residual norm 6.04605e+06 
450 KSP preconditioned resid norm 5.831499222021e+06 true resid norm 2.674246554899e+04 ||r(i)||/||b|| 9.432968447616e-01
450 KSP Residual norm 5.8315e+06 
451 KSP preconditioned resid norm 5.821752090917e+06 true resid norm 2.674392170063e+04 ||r(i)||/||b|| 9.433482081352e-01
451 KSP Residual norm 5.82175e+06 
452 KSP preconditioned resid norm 5.619904638176e+06 true resid norm 2.677517566648e+04 ||r(i)||/||b|| 9.444506407931e-01
452 KSP Residual norm 5.6199e+06 
453 KSP preconditioned resid norm 5.659430027936e+06 true resid norm 2.676888436656e+04 ||r(i)||/||b|| 9.442287254519e-01
453 KSP Residual norm 5.65943e+06 
454 KSP preconditioned resid norm 5.090840094209e+06 true resid norm 2.687035380244e+04 ||r(i)||/||b|| 9.478078942661e-01
454 KSP Residual norm 5.09084e+06 
455 KSP preconditioned resid norm 5.059438412116e+06 true resid norm 2.687685701225e+04 ||r(i)||/||b|| 9.480372843828e-01
455 KSP Residual norm 5.05944e+06 
456 KSP preconditioned resid norm 4.354075731287e+06 true resid norm 2.710550680461e+04 ||r(i)||/||b|| 9.561025327903e-01
456 KSP Residual norm 4.35408e+06 
457 KSP preconditioned resid norm 4.332147364834e+06 true resid norm 2.712309210809e+04 ||r(i)||/||b|| 9.567228256822e-01
457 KSP Residual norm 4.33215e+06 
458 KSP preconditioned resid norm 1.252283757306e+07 true resid norm 2.923278795156e+04 ||r(i)||/||b|| 1.031138904817e+00
458 KSP Residual norm 1.25228e+07 
459 KSP preconditioned resid norm 5.330263954308e+07 true resid norm 4.172881583021e+04 ||r(i)||/||b|| 1.471915902300e+00
459 KSP Residual norm 5.33026e+07 
460 KSP preconditioned resid norm 3.426586316442e+07 true resid norm 2.741216468015e+04 ||r(i)||/||b|| 9.669193890706e-01
460 KSP Residual norm 3.42659e+07 
461 KSP preconditioned resid norm 3.081884008909e+07 true resid norm 2.696083881750e+04 ||r(i)||/||b|| 9.509996055555e-01
461 KSP Residual norm 3.08188e+07 
462 KSP preconditioned resid norm 4.726753587093e+07 true resid norm 3.954735358786e+04 ||r(i)||/||b|| 1.394968380524e+00
462 KSP Residual norm 4.72675e+07 
463 KSP preconditioned resid norm 4.727002625087e+07 true resid norm 3.954903049287e+04 ||r(i)||/||b|| 1.395027530613e+00
463 KSP Residual norm 4.727e+07 
464 KSP preconditioned resid norm 1.115241415121e+08 true resid norm 5.174746662674e+04 ||r(i)||/||b|| 1.825307464788e+00
464 KSP Residual norm 1.11524e+08 
465 KSP preconditioned resid norm 1.106657129072e+08 true resid norm 5.139986237805e+04 ||r(i)||/||b|| 1.813046291995e+00
465 KSP Residual norm 1.10666e+08 
466 KSP preconditioned resid norm 2.036616378433e+07 true resid norm 2.617109195465e+04 ||r(i)||/||b|| 9.231425733561e-01
466 KSP Residual norm 2.03662e+07 
467 KSP preconditioned resid norm 2.431743441682e+07 true resid norm 2.636313424720e+04 ||r(i)||/||b|| 9.299165519296e-01
467 KSP Residual norm 2.43174e+07 
468 KSP preconditioned resid norm 1.293223038219e+08 true resid norm 7.308863326511e+04 ||r(i)||/||b|| 2.578082302120e+00
468 KSP Residual norm 1.29322e+08 
469 KSP preconditioned resid norm 1.306505584193e+08 true resid norm 7.367141718136e+04 ||r(i)||/||b|| 2.598639054016e+00
469 KSP Residual norm 1.30651e+08 
470 KSP preconditioned resid norm 6.017859579428e+07 true resid norm 4.429976110464e+04 ||r(i)||/||b|| 1.562601802633e+00
470 KSP Residual norm 6.01786e+07 
471 KSP preconditioned resid norm 6.020643494613e+07 true resid norm 4.431035081579e+04 ||r(i)||/||b|| 1.562975337418e+00
471 KSP Residual norm 6.02064e+07 
472 KSP preconditioned resid norm 1.648190636840e+07 true resid norm 3.015795127640e+04 ||r(i)||/||b|| 1.063772531796e+00
472 KSP Residual norm 1.64819e+07 
473 KSP preconditioned resid norm 1.678654654542e+07 true resid norm 3.023222623119e+04 ||r(i)||/||b|| 1.066392459654e+00
473 KSP Residual norm 1.67865e+07 
474 KSP preconditioned resid norm 2.990585661773e+07 true resid norm 3.382931975141e+04 ||r(i)||/||b|| 1.193274065305e+00
474 KSP Residual norm 2.99059e+07 
475 KSP preconditioned resid norm 2.794642572076e+07 true resid norm 3.324605421179e+04 ||r(i)||/||b|| 1.172700324931e+00
475 KSP Residual norm 2.79464e+07 
476 KSP preconditioned resid norm 3.220771804545e+07 true resid norm 3.453294177537e+04 ||r(i)||/||b|| 1.218093184316e+00
476 KSP Residual norm 3.22077e+07 
477 KSP preconditioned resid norm 3.226637187575e+07 true resid norm 3.455111842912e+04 ||r(i)||/||b|| 1.218734336124e+00
477 KSP Residual norm 3.22664e+07 
478 KSP preconditioned resid norm 4.568794069470e+07 true resid norm 3.899614019644e+04 ||r(i)||/||b|| 1.375525227388e+00
478 KSP Residual norm 4.56879e+07 
479 KSP preconditioned resid norm 4.597439118136e+07 true resid norm 3.909648230444e+04 ||r(i)||/||b|| 1.379064631550e+00
479 KSP Residual norm 4.59744e+07 
480 KSP preconditioned resid norm 4.531242206955e+07 true resid norm 3.886495166911e+04 ||r(i)||/||b|| 1.370897766106e+00
480 KSP Residual norm 4.53124e+07 
481 KSP preconditioned resid norm 4.597886084014e+07 true resid norm 3.909815559480e+04 ||r(i)||/||b|| 1.379123654137e+00
481 KSP Residual norm 4.59789e+07 
482 KSP preconditioned resid norm 4.583299079565e+07 true resid norm 3.904702091007e+04 ||r(i)||/||b|| 1.377319961554e+00
482 KSP Residual norm 4.5833e+07 
483 KSP preconditioned resid norm 4.744922463061e+07 true resid norm 3.961651731898e+04 ||r(i)||/||b|| 1.397408018306e+00
483 KSP Residual norm 4.74492e+07 
484 KSP preconditioned resid norm 4.810709096378e+07 true resid norm 3.985011621074e+04 ||r(i)||/||b|| 1.405647838121e+00
484 KSP Residual norm 4.81071e+07 
485 KSP preconditioned resid norm 2.720424239504e+07 true resid norm 3.302915485504e+04 ||r(i)||/||b|| 1.165049553970e+00
485 KSP Residual norm 2.72042e+07 
486 KSP preconditioned resid norm 2.217433585684e+07 true resid norm 3.161834184189e+04 ||r(i)||/||b|| 1.115285426522e+00
486 KSP Residual norm 2.21743e+07 
487 KSP preconditioned resid norm 2.624897393387e+07 true resid norm 2.650454492327e+04 ||r(i)||/||b|| 9.349045828313e-01
487 KSP Residual norm 2.6249e+07 
488 KSP preconditioned resid norm 2.887097128217e+07 true resid norm 2.674576102580e+04 ||r(i)||/||b|| 9.434130873300e-01
488 KSP Residual norm 2.8871e+07 
489 KSP preconditioned resid norm 1.473389763804e+07 true resid norm 2.974048308785e+04 ||r(i)||/||b|| 1.049047022499e+00
489 KSP Residual norm 1.47339e+07 
490 KSP preconditioned resid norm 2.012896977572e+07 true resid norm 3.107627361113e+04 ||r(i)||/||b|| 1.096164854008e+00
490 KSP Residual norm 2.0129e+07 
491 KSP preconditioned resid norm 4.082640569840e+07 true resid norm 2.850706801897e+04 ||r(i)||/||b|| 1.005540318129e+00
491 KSP Residual norm 4.08264e+07 
492 KSP preconditioned resid norm 6.554163268985e+07 true resid norm 4.636032750297e+04 ||r(i)||/||b|| 1.635284920740e+00
492 KSP Residual norm 6.55416e+07 
493 KSP preconditioned resid norm 3.849244987216e+08 true resid norm 1.749495763051e+05 ||r(i)||/||b|| 6.171060892596e+00
493 KSP Residual norm 3.84924e+08 
494 KSP preconditioned resid norm 3.386629995338e+08 true resid norm 1.535181905433e+05 ||r(i)||/||b|| 5.415103722866e+00
494 KSP Residual norm 3.38663e+08 
495 KSP preconditioned resid norm 8.086332641230e+08 true resid norm 3.879274952755e+05 ||r(i)||/||b|| 1.368350953353e+01
495 KSP Residual norm 8.08633e+08 
496 KSP preconditioned resid norm 3.877702250967e+08 true resid norm 1.913505876741e+05 ||r(i)||/||b|| 6.749579812136e+00
496 KSP Residual norm 3.8777e+08 
497 KSP preconditioned resid norm 3.701470147674e+08 true resid norm 1.680941988281e+05 ||r(i)||/||b|| 5.929248635913e+00
497 KSP Residual norm 3.70147e+08 
498 KSP preconditioned resid norm 4.636258873424e+08 true resid norm 2.115366133631e+05 ||r(i)||/||b|| 7.461608936970e+00
498 KSP Residual norm 4.63626e+08 
499 KSP preconditioned resid norm 8.234093186304e+07 true resid norm 5.307149600231e+04 ||r(i)||/||b|| 1.872010440998e+00
499 KSP Residual norm 8.23409e+07 
500 KSP preconditioned resid norm 6.441236333647e+07 true resid norm 3.449592541526e+04 ||r(i)||/||b|| 1.216787492602e+00
500 KSP Residual norm 6.44124e+07 
501 KSP preconditioned resid norm 1.392807652779e+08 true resid norm 6.333982561045e+04 ||r(i)||/||b|| 2.234209016242e+00
501 KSP Residual norm 1.39281e+08 
502 KSP preconditioned resid norm 2.698937429089e+09 true resid norm 1.274306574171e+06 ||r(i)||/||b|| 4.494908550869e+01
502 KSP Residual norm 2.69894e+09 
503 KSP preconditioned resid norm 2.048382897103e+09 true resid norm 9.538585553353e+05 ||r(i)||/||b|| 3.364580442100e+01
503 KSP Residual norm 2.04838e+09 
504 KSP preconditioned resid norm 2.058664760080e+09 true resid norm 9.586818953937e+05 ||r(i)||/||b|| 3.381593987279e+01
504 KSP Residual norm 2.05866e+09 
505 KSP preconditioned resid norm 5.534694162960e+09 true resid norm 2.604761121022e+06 ||r(i)||/||b|| 9.187869915420e+01
505 KSP Residual norm 5.53469e+09 
506 KSP preconditioned resid norm 5.426384764347e+09 true resid norm 2.553942578487e+06 ||r(i)||/||b|| 9.008615797131e+01
506 KSP Residual norm 5.42638e+09 
507 KSP preconditioned resid norm 7.894951650351e+08 true resid norm 3.789619412597e+05 ||r(i)||/||b|| 1.336726424197e+01
507 KSP Residual norm 7.89495e+08 
508 KSP preconditioned resid norm 8.013576136793e+08 true resid norm 3.845152298676e+05 ||r(i)||/||b|| 1.356314743801e+01
508 KSP Residual norm 8.01358e+08 
509 KSP preconditioned resid norm 3.260202552312e+09 true resid norm 1.522375685622e+06 ||r(i)||/||b|| 5.369931871681e+01
509 KSP Residual norm 3.2602e+09 
510 KSP preconditioned resid norm 3.441702865892e+09 true resid norm 1.607529645431e+06 ||r(i)||/||b|| 5.670298572950e+01
510 KSP Residual norm 3.4417e+09 
511 KSP preconditioned resid norm 6.580055878775e+09 true resid norm 3.080021790848e+06 ||r(i)||/||b|| 1.086427439452e+02
511 KSP Residual norm 6.58006e+09 
512 KSP preconditioned resid norm 7.237093000147e+09 true resid norm 3.388309904924e+06 ||r(i)||/||b|| 1.195171042301e+02
512 KSP Residual norm 7.23709e+09 
513 KSP preconditioned resid norm 1.391360509505e+09 true resid norm 6.457709485268e+05 ||r(i)||/||b|| 2.277851670289e+01
513 KSP Residual norm 1.39136e+09 
514 KSP preconditioned resid norm 1.090652424906e+09 true resid norm 5.048172245720e+05 ||r(i)||/||b|| 1.780660404134e+01
514 KSP Residual norm 1.09065e+09 
515 KSP preconditioned resid norm 1.559753442542e+09 true resid norm 7.399559857659e+05 ||r(i)||/||b|| 2.610074023866e+01
515 KSP Residual norm 1.55975e+09 
516 KSP preconditioned resid norm 3.613425354107e+08 true resid norm 1.640154414100e+05 ||r(i)||/||b|| 5.785377122046e+00
516 KSP Residual norm 3.61343e+08 
517 KSP preconditioned resid norm 6.435082514763e+08 true resid norm 2.954852966539e+05 ||r(i)||/||b|| 1.042276178673e+01
517 KSP Residual norm 6.43508e+08 
518 KSP preconditioned resid norm 9.059161989933e+09 true resid norm 4.258444164145e+06 ||r(i)||/||b|| 1.502096706930e+02
518 KSP Residual norm 9.05916e+09 
519 KSP preconditioned resid norm 7.736894009760e+09 true resid norm 3.638020139393e+06 ||r(i)||/||b|| 1.283252253754e+02
519 KSP Residual norm 7.73689e+09 
520 KSP preconditioned resid norm 2.972208716440e+09 true resid norm 1.387272556206e+06 ||r(i)||/||b|| 4.893377623303e+01
520 KSP Residual norm 2.97221e+09 
521 KSP preconditioned resid norm 3.157436604231e+09 true resid norm 1.474171980196e+06 ||r(i)||/||b|| 5.199901164713e+01
521 KSP Residual norm 3.15744e+09 
522 KSP preconditioned resid norm 4.837209443760e+09 true resid norm 2.262284647511e+06 ||r(i)||/||b|| 7.979840026494e+01
522 KSP Residual norm 4.83721e+09 
523 KSP preconditioned resid norm 5.479313026769e+09 true resid norm 2.563558678782e+06 ||r(i)||/||b|| 9.042535022158e+01
523 KSP Residual norm 5.47931e+09 
524 KSP preconditioned resid norm 5.187907042841e+09 true resid norm 2.426829243870e+06 ||r(i)||/||b|| 8.560244246455e+01
524 KSP Residual norm 5.18791e+09 
525 KSP preconditioned resid norm 5.232858994678e+09 true resid norm 2.447919943576e+06 ||r(i)||/||b|| 8.634638248946e+01
525 KSP Residual norm 5.23286e+09 
526 KSP preconditioned resid norm 5.321560227271e+09 true resid norm 2.489540541444e+06 ||r(i)||/||b|| 8.781448117969e+01
526 KSP Residual norm 5.32156e+09 
527 KSP preconditioned resid norm 5.483163075138e+09 true resid norm 2.565372093266e+06 ||r(i)||/||b|| 9.048931545911e+01
527 KSP Residual norm 5.48316e+09 
528 KSP preconditioned resid norm 5.260197590801e+09 true resid norm 2.460749594656e+06 ||r(i)||/||b|| 8.679892750110e+01
528 KSP Residual norm 5.2602e+09 
529 KSP preconditioned resid norm 5.269970551915e+09 true resid norm 2.465330380892e+06 ||r(i)||/||b|| 8.696050726249e+01
529 KSP Residual norm 5.26997e+09 
530 KSP preconditioned resid norm 5.271283260202e+09 true resid norm 2.465946175103e+06 ||r(i)||/||b|| 8.698222839869e+01
530 KSP Residual norm 5.27128e+09 
531 KSP preconditioned resid norm 5.222434288611e+09 true resid norm 2.443026724154e+06 ||r(i)||/||b|| 8.617378215709e+01
531 KSP Residual norm 5.22243e+09 
532 KSP preconditioned resid norm 5.371199205748e+09 true resid norm 2.512827502335e+06 ||r(i)||/||b|| 8.863589073491e+01
532 KSP Residual norm 5.3712e+09 
533 KSP preconditioned resid norm 5.779732758359e+09 true resid norm 2.704505819818e+06 ||r(i)||/||b|| 9.539703068142e+01
533 KSP Residual norm 5.77973e+09 
534 KSP preconditioned resid norm 5.778811964051e+09 true resid norm 2.704074419189e+06 ||r(i)||/||b|| 9.538181372801e+01
534 KSP Residual norm 5.77881e+09 
535 KSP preconditioned resid norm 5.985007831013e+09 true resid norm 2.800823362961e+06 ||r(i)||/||b|| 9.879447488400e+01
535 KSP Residual norm 5.98501e+09 
536 KSP preconditioned resid norm 5.635276464513e+09 true resid norm 2.636726114866e+06 ||r(i)||/||b|| 9.300621216460e+01
536 KSP Residual norm 5.63528e+09 
537 KSP preconditioned resid norm 3.471027330483e+09 true resid norm 1.621273123200e+06 ||r(i)||/||b|| 5.718776448679e+01
537 KSP Residual norm 3.47103e+09 
538 KSP preconditioned resid norm 3.345964648464e+09 true resid norm 1.562598326233e+06 ||r(i)||/||b|| 5.511810674544e+01
538 KSP Residual norm 3.34596e+09 
539 KSP preconditioned resid norm 3.677254003360e+09 true resid norm 1.718027010859e+06 ||r(i)||/||b|| 6.060060003029e+01
539 KSP Residual norm 3.67725e+09 
540 KSP preconditioned resid norm 3.612498275204e+09 true resid norm 1.687644896510e+06 ||r(i)||/||b|| 5.952892051182e+01
540 KSP Residual norm 3.6125e+09 
541 KSP preconditioned resid norm 3.504125433539e+09 true resid norm 1.636799973378e+06 ||r(i)||/||b|| 5.773544879639e+01
541 KSP Residual norm 3.50413e+09 
542 KSP preconditioned resid norm 3.743210500103e+09 true resid norm 1.748971809565e+06 ||r(i)||/||b|| 6.169212732152e+01
542 KSP Residual norm 3.74321e+09 
543 KSP preconditioned resid norm 3.410957948831e+09 true resid norm 1.593081149856e+06 ||r(i)||/||b|| 5.619333861926e+01
543 KSP Residual norm 3.41096e+09 
544 KSP preconditioned resid norm 3.411652140993e+09 true resid norm 1.593407053399e+06 ||r(i)||/||b|| 5.620483433507e+01
544 KSP Residual norm 3.41165e+09 
545 KSP preconditioned resid norm 8.402437936396e+08 true resid norm 3.875095416312e+05 ||r(i)||/||b|| 1.366876690057e+01
545 KSP Residual norm 8.40244e+08 
546 KSP preconditioned resid norm 1.078009044619e+09 true resid norm 4.988791680064e+05 ||r(i)||/||b|| 1.759714878330e+01
546 KSP Residual norm 1.07801e+09 
547 KSP preconditioned resid norm 1.329195937405e+10 true resid norm 6.229264087396e+06 ||r(i)||/||b|| 2.197271283032e+02
547 KSP Residual norm 1.3292e+10 
548 KSP preconditioned resid norm 1.271109607840e+10 true resid norm 5.971877262550e+06 ||r(i)||/||b|| 2.106482279559e+02
548 KSP Residual norm 1.27111e+10 
549 KSP preconditioned resid norm 4.352821212827e+09 true resid norm 2.034987196610e+06 ||r(i)||/||b|| 7.178085349596e+01
549 KSP Residual norm 4.35282e+09 
550 KSP preconditioned resid norm 4.523714137688e+09 true resid norm 2.115167794848e+06 ||r(i)||/||b|| 7.460909329271e+01
550 KSP Residual norm 4.52371e+09 
551 KSP preconditioned resid norm 6.258929719479e+08 true resid norm 2.872449451975e+05 ||r(i)||/||b|| 1.013209683236e+01
551 KSP Residual norm 6.25893e+08 
552 KSP preconditioned resid norm 9.050704917967e+08 true resid norm 4.178724793739e+05 ||r(i)||/||b|| 1.473976999555e+01
552 KSP Residual norm 9.0507e+08 
553 KSP preconditioned resid norm 1.257937276433e+09 true resid norm 5.984477952388e+05 ||r(i)||/||b|| 2.110926967332e+01
553 KSP Residual norm 1.25794e+09 
554 KSP preconditioned resid norm 5.759657659036e+09 true resid norm 2.695073526986e+06 ||r(i)||/||b|| 9.506432193953e+01
554 KSP Residual norm 5.75966e+09 
555 KSP preconditioned resid norm 2.941817101359e+09 true resid norm 1.373004741053e+06 ||r(i)||/||b|| 4.843050232992e+01
555 KSP Residual norm 2.94182e+09 
556 KSP preconditioned resid norm 4.416137615440e+09 true resid norm 2.064710048852e+06 ||r(i)||/||b|| 7.282927861910e+01
556 KSP Residual norm 4.41614e+09 
557 KSP preconditioned resid norm 4.875454526944e+09 true resid norm 2.280215972031e+06 ||r(i)||/||b|| 8.043089848435e+01
557 KSP Residual norm 4.87545e+09 
558 KSP preconditioned resid norm 6.621026478002e+09 true resid norm 3.099241977930e+06 ||r(i)||/||b|| 1.093207046889e+02
558 KSP Residual norm 6.62103e+09 
559 KSP preconditioned resid norm 1.010174436666e+10 true resid norm 4.732447823323e+06 ||r(i)||/||b|| 1.669293764840e+02
559 KSP Residual norm 1.01017e+10 
560 KSP preconditioned resid norm 4.119820460107e+09 true resid norm 1.925644112913e+06 ||r(i)||/||b|| 6.792395460012e+01
560 KSP Residual norm 4.11982e+09 
561 KSP preconditioned resid norm 4.132408041792e+09 true resid norm 1.931550063039e+06 ||r(i)||/||b|| 6.813227735586e+01
561 KSP Residual norm 4.13241e+09 
562 KSP preconditioned resid norm 1.680803660677e+09 true resid norm 7.814806720868e+05 ||r(i)||/||b|| 2.756545580553e+01
562 KSP Residual norm 1.6808e+09 
563 KSP preconditioned resid norm 1.618876050993e+09 true resid norm 7.524392792784e+05 ||r(i)||/||b|| 2.654106805215e+01
563 KSP Residual norm 1.61888e+09 
564 KSP preconditioned resid norm 3.534234544482e+09 true resid norm 1.650964997302e+06 ||r(i)||/||b|| 5.823509690659e+01
564 KSP Residual norm 3.53423e+09 
565 KSP preconditioned resid norm 3.541628605087e+09 true resid norm 1.654434390218e+06 ||r(i)||/||b|| 5.835747408177e+01
565 KSP Residual norm 3.54163e+09 
566 KSP preconditioned resid norm 2.872588851518e+09 true resid norm 1.340541692644e+06 ||r(i)||/||b|| 4.728542125728e+01
566 KSP Residual norm 2.87259e+09 
567 KSP preconditioned resid norm 2.901441355312e+09 true resid norm 1.354077783274e+06 ||r(i)||/||b|| 4.776288477156e+01
567 KSP Residual norm 2.90144e+09 
568 KSP preconditioned resid norm 1.390864852589e+09 true resid norm 6.455390794249e+05 ||r(i)||/||b|| 2.277033789859e+01
568 KSP Residual norm 1.39086e+09 
569 KSP preconditioned resid norm 1.348330473517e+09 true resid norm 6.255983068763e+05 ||r(i)||/||b|| 2.206695967818e+01
569 KSP Residual norm 1.34833e+09 
570 KSP preconditioned resid norm 1.301999659102e+09 true resid norm 6.038783359575e+05 ||r(i)||/||b|| 2.130082313783e+01
570 KSP Residual norm 1.302e+09 
571 KSP preconditioned resid norm 1.576916673224e+09 true resid norm 7.327782529348e+05 ||r(i)||/||b|| 2.584755742274e+01
571 KSP Residual norm 1.57692e+09 
572 KSP preconditioned resid norm 1.313020443781e+09 true resid norm 6.090445238700e+05 ||r(i)||/||b|| 2.148305198836e+01
572 KSP Residual norm 1.31302e+09 
573 KSP preconditioned resid norm 1.351594836754e+09 true resid norm 6.271301800096e+05 ||r(i)||/||b|| 2.212099400387e+01
573 KSP Residual norm 1.35159e+09 
574 KSP preconditioned resid norm 1.344752288108e+09 true resid norm 6.239222779806e+05 ||r(i)||/||b|| 2.200784049314e+01
574 KSP Residual norm 1.34475e+09 
575 KSP preconditioned resid norm 3.035602811150e+10 true resid norm 1.423612303742e+07 ||r(i)||/||b|| 5.021560154291e+02
575 KSP Residual norm 3.0356e+10 
576 KSP preconditioned resid norm 1.378571700159e+10 true resid norm 6.460992934568e+06 ||r(i)||/||b|| 2.279009853463e+02
576 KSP Residual norm 1.37857e+10 
577 KSP preconditioned resid norm 9.080063716650e+09 true resid norm 4.268279509638e+06 ||r(i)||/||b|| 1.505565964599e+02
577 KSP Residual norm 9.08006e+09 
578 KSP preconditioned resid norm 1.803605799739e+10 true resid norm 8.470716537288e+06 ||r(i)||/||b|| 2.987907067826e+02
578 KSP Residual norm 1.80361e+10 
579 KSP preconditioned resid norm 1.822987649743e+10 true resid norm 8.561662011548e+06 ||r(i)||/||b|| 3.019986600193e+02
579 KSP Residual norm 1.82299e+10 
580 KSP preconditioned resid norm 8.745530772803e+08 true resid norm 4.187788157201e+05 ||r(i)||/||b|| 1.477173953157e+01
580 KSP Residual norm 8.74553e+08 
581 KSP preconditioned resid norm 9.712023024459e+08 true resid norm 4.640497968378e+05 ||r(i)||/||b|| 1.636859953573e+01
581 KSP Residual norm 9.71202e+08 
582 KSP preconditioned resid norm 5.936533329743e+09 true resid norm 2.793299676843e+06 ||r(i)||/||b|| 9.852908913028e+01
582 KSP Residual norm 5.93653e+09 
583 KSP preconditioned resid norm 5.669506252006e+09 true resid norm 2.668017260388e+06 ||r(i)||/||b|| 9.410995627472e+01
583 KSP Residual norm 5.66951e+09 
584 KSP preconditioned resid norm 5.889175183760e+09 true resid norm 2.771086510478e+06 ||r(i)||/||b|| 9.774555592514e+01
584 KSP Residual norm 5.88918e+09 
585 KSP preconditioned resid norm 2.234226929620e+10 true resid norm 1.049120587455e+07 ||r(i)||/||b|| 3.700601719419e+02
585 KSP Residual norm 2.23423e+10 
586 KSP preconditioned resid norm 2.006596838170e+09 true resid norm 9.495182044295e+05 ||r(i)||/||b|| 3.349270562361e+01
586 KSP Residual norm 2.0066e+09 
587 KSP preconditioned resid norm 2.166607085629e+08 true resid norm 1.123338527548e+05 ||r(i)||/||b|| 3.962393395231e+00
587 KSP Residual norm 2.16661e+08 
588 KSP preconditioned resid norm 2.568245189078e+09 true resid norm 1.213025616509e+06 ||r(i)||/||b|| 4.278749970049e+01
588 KSP Residual norm 2.56825e+09 
589 KSP preconditioned resid norm 2.642305233460e+09 true resid norm 1.247768569849e+06 ||r(i)||/||b|| 4.401300070013e+01
589 KSP Residual norm 2.64231e+09 
590 KSP preconditioned resid norm 8.707832682505e+09 true resid norm 4.093623674581e+06 ||r(i)||/||b|| 1.443958968106e+02
590 KSP Residual norm 8.70783e+09 
591 KSP preconditioned resid norm 8.631006014000e+09 true resid norm 4.057554618108e+06 ||r(i)||/||b|| 1.431236196863e+02
591 KSP Residual norm 8.63101e+09 
592 KSP preconditioned resid norm 8.624486395790e+09 true resid norm 4.054495558330e+06 ||r(i)||/||b|| 1.430157163432e+02
592 KSP Residual norm 8.62449e+09 
593 KSP preconditioned resid norm 4.004531107957e+09 true resid norm 1.871528680658e+06 ||r(i)||/||b|| 6.601512101087e+01
593 KSP Residual norm 4.00453e+09 
594 KSP preconditioned resid norm 3.973908626931e+09 true resid norm 1.857162197672e+06 ||r(i)||/||b|| 6.550836676093e+01
594 KSP Residual norm 3.97391e+09 
595 KSP preconditioned resid norm 2.479182468996e+09 true resid norm 1.171186582854e+06 ||r(i)||/||b|| 4.131169604423e+01
595 KSP Residual norm 2.47918e+09 
596 KSP preconditioned resid norm 2.717901477814e+09 true resid norm 1.283173789509e+06 ||r(i)||/||b|| 4.526186206379e+01
596 KSP Residual norm 2.7179e+09 
597 KSP preconditioned resid norm 2.788802060454e+09 true resid norm 1.301208677550e+06 ||r(i)||/||b|| 4.589801331747e+01
597 KSP Residual norm 2.7888e+09 
598 KSP preconditioned resid norm 1.899757041458e+10 true resid norm 8.906523807098e+06 ||r(i)||/||b|| 3.141630972521e+02
598 KSP Residual norm 1.89976e+10 
599 KSP preconditioned resid norm 6.391932268100e+10 true resid norm 3.000014757986e+07 ||r(i)||/||b|| 1.058206263840e+03
599 KSP Residual norm 6.39193e+10 
600 KSP preconditioned resid norm 2.126602497342e+11 true resid norm 9.979332763513e+07 ||r(i)||/||b|| 3.520046830163e+03
600 KSP Residual norm 2.1266e+11 
601 KSP preconditioned resid norm 1.389826377513e+11 true resid norm 6.522189343371e+07 ||r(i)||/||b|| 2.300595888314e+03
601 KSP Residual norm 1.38983e+11 
602 KSP preconditioned resid norm 3.130803748803e+10 true resid norm 1.468275247873e+07 ||r(i)||/||b|| 5.179101403431e+02
602 KSP Residual norm 3.1308e+10 
603 KSP preconditioned resid norm 1.468849729238e+11 true resid norm 6.891407055349e+07 ||r(i)||/||b|| 2.430831412822e+03
603 KSP Residual norm 1.46885e+11 
604 KSP preconditioned resid norm 3.786805213360e+11 true resid norm 1.776777712037e+08 ||r(i)||/||b|| 6.267293516887e+03
604 KSP Residual norm 3.78681e+11 
605 KSP preconditioned resid norm 6.406846599292e+10 true resid norm 3.007343496095e+07 ||r(i)||/||b|| 1.060791356647e+03
605 KSP Residual norm 6.40685e+10 
606 KSP preconditioned resid norm 6.412061862863e+10 true resid norm 3.009803564720e+07 ||r(i)||/||b|| 1.061659105722e+03
606 KSP Residual norm 6.41206e+10 
607 KSP preconditioned resid norm 1.027315508480e+11 true resid norm 4.819613278430e+07 ||r(i)||/||b|| 1.700039957118e+03
607 KSP Residual norm 1.02732e+11 
608 KSP preconditioned resid norm 1.041168254611e+11 true resid norm 4.884614085303e+07 ||r(i)||/||b|| 1.722967931324e+03
608 KSP Residual norm 1.04117e+11 
609 KSP preconditioned resid norm 2.015350238956e+11 true resid norm 9.455585873981e+07 ||r(i)||/||b|| 3.335303659253e+03
609 KSP Residual norm 2.01535e+11 
610 KSP preconditioned resid norm 2.025936632560e+11 true resid norm 9.505262809988e+07 ||r(i)||/||b|| 3.352826388003e+03
610 KSP Residual norm 2.02594e+11 
611 KSP preconditioned resid norm 2.389238751714e+11 true resid norm 1.120993902007e+08 ||r(i)||/||b|| 3.954123111136e+03
611 KSP Residual norm 2.38924e+11 
612 KSP preconditioned resid norm 2.090143906283e+13 true resid norm 9.807318971822e+09 ||r(i)||/||b|| 3.459371771366e+05
612 KSP Residual norm 2.09014e+13 
613 KSP preconditioned resid norm 6.660277916237e+10 true resid norm 3.125401796874e+07 ||r(i)||/||b|| 1.102434496252e+03
613 KSP Residual norm 6.66028e+10 
614 KSP preconditioned resid norm 2.689509339169e+12 true resid norm 1.261953110023e+09 ||r(i)||/||b|| 4.451333721423e+04
614 KSP Residual norm 2.68951e+12 
615 KSP preconditioned resid norm 3.566249043550e+12 true resid norm 1.673344767935e+09 ||r(i)||/||b|| 5.902450680548e+04
615 KSP Residual norm 3.56625e+12 
616 KSP preconditioned resid norm 5.701969015285e+12 true resid norm 2.675463681414e+09 ||r(i)||/||b|| 9.437261662837e+04
616 KSP Residual norm 5.70197e+12 
617 KSP preconditioned resid norm 5.684152456409e+12 true resid norm 2.667104810678e+09 ||r(i)||/||b|| 9.407777109977e+04
617 KSP Residual norm 5.68415e+12 
618 KSP preconditioned resid norm 4.799690188440e+12 true resid norm 2.252093686126e+09 ||r(i)||/||b|| 7.943893072756e+04
618 KSP Residual norm 4.79969e+12 
619 KSP preconditioned resid norm 4.456285102277e+12 true resid norm 2.090967253175e+09 ||r(i)||/||b|| 7.375545866580e+04
619 KSP Residual norm 4.45629e+12 
620 KSP preconditioned resid norm 3.612710385324e+12 true resid norm 1.695147730074e+09 ||r(i)||/||b|| 5.979357072570e+04
620 KSP Residual norm 3.61271e+12 
621 KSP preconditioned resid norm 3.569097583385e+12 true resid norm 1.674683412493e+09 ||r(i)||/||b|| 5.907172530838e+04
621 KSP Residual norm 3.5691e+12 
622 KSP preconditioned resid norm 6.852397549533e+12 true resid norm 3.215270734877e+09 ||r(i)||/||b|| 1.134134298016e+05
622 KSP Residual norm 6.8524e+12 
623 KSP preconditioned resid norm 9.786811954989e+12 true resid norm 4.592158864976e+09 ||r(i)||/||b|| 1.619809123448e+05
623 KSP Residual norm 9.78681e+12 
624 KSP preconditioned resid norm 1.786511420356e+12 true resid norm 8.382773150656e+08 ||r(i)||/||b|| 2.956886472895e+04
624 KSP Residual norm 1.78651e+12 
625 KSP preconditioned resid norm 1.353469650693e+11 true resid norm 6.351398316650e+07 ||r(i)||/||b|| 2.240352139912e+03
625 KSP Residual norm 1.35347e+11 
626 KSP preconditioned resid norm 1.614568993214e+13 true resid norm 7.575927648219e+09 ||r(i)||/||b|| 2.672284884733e+05
626 KSP Residual norm 1.61457e+13 
627 KSP preconditioned resid norm 9.427769024718e+13 true resid norm 4.423735292175e+10 ||r(i)||/||b|| 1.560400455794e+06
627 KSP Residual norm 9.42777e+13 
628 KSP preconditioned resid norm 1.408841064859e+13 true resid norm 6.610675956973e+09 ||r(i)||/||b|| 2.331808097698e+05
628 KSP Residual norm 1.40884e+13 
629 KSP preconditioned resid norm 1.325568643299e+13 true resid norm 6.219942299464e+09 ||r(i)||/||b|| 2.193983174414e+05
629 KSP Residual norm 1.32557e+13 
630 KSP preconditioned resid norm 6.751324718244e+14 true resid norm 3.167846617313e+11 ||r(i)||/||b|| 1.117406214220e+07
630 KSP Residual norm 6.75132e+14 
631 KSP preconditioned resid norm 8.773027358359e+14 true resid norm 4.116469814677e+11 ||r(i)||/||b|| 1.452017571314e+07
631 KSP Residual norm 8.77303e+14 
632 KSP preconditioned resid norm 1.749776678543e+14 true resid norm 8.210295300280e+10 ||r(i)||/||b|| 2.896047724967e+06
632 KSP Residual norm 1.74978e+14 
633 KSP preconditioned resid norm 1.912746627652e+15 true resid norm 8.975060621240e+11 ||r(i)||/||b|| 3.165806215605e+07
633 KSP Residual norm 1.91275e+15 
634 KSP preconditioned resid norm 3.215318591555e+14 true resid norm 1.508688418674e+11 ||r(i)||/||b|| 5.321652270455e+06
634 KSP Residual norm 3.21532e+14 
635 KSP preconditioned resid norm 2.968948647718e+14 true resid norm 1.393086235816e+11 ||r(i)||/||b|| 4.913884429686e+06
635 KSP Residual norm 2.96895e+14 
636 KSP preconditioned resid norm 4.101923333653e+14 true resid norm 1.924699473117e+11 ||r(i)||/||b|| 6.789063397239e+06
636 KSP Residual norm 4.10192e+14 
637 KSP preconditioned resid norm 1.997583748488e+13 true resid norm 9.375436856608e+09 ||r(i)||/||b|| 3.307032400920e+05
637 KSP Residual norm 1.99758e+13 
638 KSP preconditioned resid norm 1.158879067561e+16 true resid norm 5.437637409580e+12 ||r(i)||/||b|| 1.918037886977e+08
638 KSP Residual norm 1.15888e+16 
639 KSP preconditioned resid norm 8.554993143100e+15 true resid norm 4.014135646375e+12 ||r(i)||/||b|| 1.415920862919e+08
639 KSP Residual norm 8.55499e+15 
640 KSP preconditioned resid norm 7.603266202419e+14 true resid norm 3.567658068106e+11 ||r(i)||/||b|| 1.258433180990e+07
640 KSP Residual norm 7.60327e+14 
641 KSP preconditioned resid norm 1.218457098690e+15 true resid norm 5.717349042883e+11 ||r(i)||/||b|| 2.016701602428e+07
641 KSP Residual norm 1.21846e+15 
642 KSP preconditioned resid norm 2.100593811892e+15 true resid norm 9.856541630757e+11 ||r(i)||/||b|| 3.476734261290e+07
642 KSP Residual norm 2.10059e+15 
643 KSP preconditioned resid norm 3.819735262072e+16 true resid norm 1.792333282474e+13 ||r(i)||/||b|| 6.322163253876e+08
643 KSP Residual norm 3.81974e+16 
644 KSP preconditioned resid norm 3.759692271758e+17 true resid norm 1.764154044728e+14 ||r(i)||/||b|| 6.222765589869e+09
644 KSP Residual norm 3.75969e+17 
Linear solve did not converge due to DIVERGED_DTOL iterations 644
KSP Object: 1 MPI processes
  type: bcgs
  maximum iterations=10000
  tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: hypre
    HYPRE BoomerAMG preconditioning
    HYPRE BoomerAMG: Cycle type V
    HYPRE BoomerAMG: Maximum number of levels 25
    HYPRE BoomerAMG: Maximum number of iterations PER hypre call 1
    HYPRE BoomerAMG: Convergence tolerance PER hypre call 0
    HYPRE BoomerAMG: Threshold for strong coupling 0.25
    HYPRE BoomerAMG: Interpolation truncation factor 0
    HYPRE BoomerAMG: Interpolation: max elements per row 0
    HYPRE BoomerAMG: Number of levels of aggressive coarsening 0
    HYPRE BoomerAMG: Number of paths for aggressive coarsening 1
    HYPRE BoomerAMG: Maximum row sums 0.9
    HYPRE BoomerAMG: Sweeps down         1
    HYPRE BoomerAMG: Sweeps up           1
    HYPRE BoomerAMG: Sweeps on coarse    1
    HYPRE BoomerAMG: Relax down          SOR/Jacobi
    HYPRE BoomerAMG: Relax up            SOR/Jacobi
    HYPRE BoomerAMG: Relax on coarse     Gaussian-elimination
    HYPRE BoomerAMG: Relax weight  (all)      1
    HYPRE BoomerAMG: Outer relax weight (all) 1
    HYPRE BoomerAMG: Using CF-relaxation
    HYPRE BoomerAMG: Measure type        local
    HYPRE BoomerAMG: Coarsen type        Falgout
    HYPRE BoomerAMG: Interpolation type  classical
  linear system matrix = precond matrix:
  Mat Object:   1 MPI processes
    type: seqaij
    rows=9261, cols=9261
    total: nonzeros=53607, allocated nonzeros=0
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      not using I-node routines

-- Ramachandran --
Ran with -pc_type hypre -pc_hypre_boomeramg_relax_type_all SOR/Jacobi -ksp_monitor_true_residual -ksp_converged_reason -ksp_view
Dirichlet and Neumann conditions. Full Dirichlet converged in 3 iterations.
--------------------------------------------------------------------------------------------------------------------------------

Calculating Electric Field.....done

Timing Info
     Solve 2.00000000e+00
 TimeSteps 0.00000000e+00
Poisson TimeSteps 0.00000000e+00
-------------- next part --------------
Number of lines read was 629.
  0 KSP preconditioned resid norm 1.540359898985e+05 true resid norm 1.336500000000e+05 ||r(i)||/||b|| 9.801980198020e-01
  0 KSP Residual norm 154036 
  1 KSP preconditioned resid norm 3.037580792249e+04 true resid norm 1.230353657616e+04 ||r(i)||/||b|| 9.023495838768e-02
  1 KSP Residual norm 30375.8 
  2 KSP preconditioned resid norm 1.431593445319e+04 true resid norm 5.747942627371e+03 ||r(i)||/||b|| 4.215579484687e-02
  2 KSP Residual norm 14315.9 
  3 KSP preconditioned resid norm 8.780397692453e+03 true resid norm 2.925899916695e+03 ||r(i)||/||b|| 2.145874526362e-02
  3 KSP Residual norm 8780.4 
  4 KSP preconditioned resid norm 6.032536676927e+03 true resid norm 2.087929190866e+03 ||r(i)||/||b|| 1.531301203422e-02
  4 KSP Residual norm 6032.54 
  5 KSP preconditioned resid norm 4.428872582246e+03 true resid norm 1.395551190031e+03 ||r(i)||/||b|| 1.023506556678e-02
  5 KSP Residual norm 4428.87 
  6 KSP preconditioned resid norm 3.389849551684e+03 true resid norm 1.110863580907e+03 ||r(i)||/||b|| 8.147147641413e-03
  6 KSP Residual norm 3389.85 
  7 KSP preconditioned resid norm 2.671367553815e+03 true resid norm 8.204302056768e+02 ||r(i)||/||b|| 6.017089883951e-03
  7 KSP Residual norm 2671.37 
  8 KSP preconditioned resid norm 2.148484718840e+03 true resid norm 6.837478708714e+02 ||r(i)||/||b|| 5.014652518309e-03
  8 KSP Residual norm 2148.48 
  9 KSP preconditioned resid norm 1.754406364449e+03 true resid norm 5.309456048213e+02 ||r(i)||/||b|| 3.893990501073e-03
  9 KSP Residual norm 1754.41 
 10 KSP preconditioned resid norm 1.448204946467e+03 true resid norm 4.528537657529e+02 ||r(i)||/||b|| 3.321259741495e-03
 10 KSP Residual norm 1448.2 
 11 KSP preconditioned resid norm 1.205130040738e+03 true resid norm 3.613967059714e+02 ||r(i)||/||b|| 2.650507561213e-03
 11 KSP Residual norm 1205.13 
 12 KSP preconditioned resid norm 1.008439047979e+03 true resid norm 3.118052248080e+02 ||r(i)||/||b|| 2.286800328625e-03
 12 KSP Residual norm 1008.44 
 13 KSP preconditioned resid norm 8.473262200569e+02 true resid norm 2.525352420624e+02 ||r(i)||/||b|| 1.852110319490e-03
 13 KSP Residual norm 847.326 
 14 KSP preconditioned resid norm 7.140693667767e+02 true resid norm 2.190350685949e+02 ||r(i)||/||b|| 1.606417811477e-03
 14 KSP Residual norm 714.069 
 15 KSP preconditioned resid norm 6.033399186408e+02 true resid norm 1.789989840810e+02 ||r(i)||/||b|| 1.312790495643e-03
 15 KSP Residual norm 603.34 
 16 KSP preconditioned resid norm 5.110035051480e+02 true resid norm 1.558072184780e+02 ||r(i)||/||b|| 1.142700538893e-03
 16 KSP Residual norm 511.004 
 17 KSP preconditioned resid norm 4.338624290250e+02 true resid norm 1.282619796561e+02 ||r(i)||/||b|| 9.406819190032e-04
 17 KSP Residual norm 433.862 
 18 KSP preconditioned resid norm 3.691700007838e+02 true resid norm 1.120008750407e+02 ||r(i)||/||b|| 8.214218924874e-04
 18 KSP Residual norm 369.17 
 19 KSP preconditioned resid norm 3.146566239746e+02 true resid norm 9.274071234852e+01 ||r(i)||/||b|| 6.801665738799e-04
 19 KSP Residual norm 314.657 
 20 KSP preconditioned resid norm 2.685107408726e+02 true resid norm 8.114285285810e+01 ||r(i)||/||b|| 5.951070983359e-04
 20 KSP Residual norm 268.511 
 21 KSP preconditioned resid norm 2.292316391640e+02 true resid norm 6.743621765417e+01 ||r(i)||/||b|| 4.945817209693e-04
 21 KSP Residual norm 229.232 
 22 KSP preconditioned resid norm 1.955080412755e+02 true resid norm 5.902702158239e+01 ||r(i)||/||b|| 4.329081157491e-04
 22 KSP Residual norm 195.508 
 23 KSP preconditioned resid norm 1.661358980302e+02 true resid norm 4.893016734462e+01 ||r(i)||/||b|| 3.588571129052e-04
 23 KSP Residual norm 166.136 
 24 KSP preconditioned resid norm 1.400334850437e+02 true resid norm 4.248822227459e+01 ||r(i)||/||b|| 3.116114578261e-04
 24 KSP Residual norm 140.033 
 25 KSP preconditioned resid norm 1.163318134976e+02 true resid norm 3.442152556522e+01 ||r(i)||/||b|| 2.524497657882e-04
 25 KSP Residual norm 116.332 
 26 KSP preconditioned resid norm 9.450622464265e+01 true resid norm 2.890197582903e+01 ||r(i)||/||b|| 2.119690196482e-04
 26 KSP Residual norm 94.5062 
 27 KSP preconditioned resid norm 7.434686035562e+01 true resid norm 2.209463071066e+01 ||r(i)||/||b|| 1.620434962278e-04
 27 KSP Residual norm 74.3469 
 28 KSP preconditioned resid norm 5.597944006410e+01 true resid norm 1.718058588584e+01 ||r(i)||/||b|| 1.260035635192e-04
 28 KSP Residual norm 55.9794 
 29 KSP preconditioned resid norm 3.962006611039e+01 true resid norm 1.177119536249e+01 ||r(i)||/||b|| 8.633073239820e-05
 29 KSP Residual norm 39.6201 
 30 KSP preconditioned resid norm 2.557015835542e+01 true resid norm 7.796897230909e+00 ||r(i)||/||b|| 5.718296465646e-05
 30 KSP Residual norm 25.5702 
 31 KSP preconditioned resid norm 1.412797909469e+01 true resid norm 4.167690769824e+00 ||r(i)||/||b|| 3.056612225760e-05
 31 KSP Residual norm 14.128 
 32 KSP preconditioned resid norm 6.123262077691e+00 true resid norm 1.859729204031e+00 ||r(i)||/||b|| 1.363937810070e-05
 32 KSP Residual norm 6.12326 
 33 KSP preconditioned resid norm 2.570847989582e+00 true resid norm 8.145410612592e-01 ||r(i)||/||b|| 5.973898505751e-06
 33 KSP Residual norm 2.57085 
 34 KSP preconditioned resid norm 1.695870150544e+00 true resid norm 6.562825967774e-01 ||r(i)||/||b|| 4.813220365071e-06
 34 KSP Residual norm 1.69587 
 35 KSP preconditioned resid norm 1.329958833488e+00 true resid norm 4.059950884053e-01 ||r(i)||/||b|| 2.977595074480e-06
 35 KSP Residual norm 1.32996 
 36 KSP preconditioned resid norm 9.747655138349e-01 true resid norm 3.465586076774e-01 ||r(i)||/||b|| 2.541683958030e-06
 36 KSP Residual norm 0.974766 
 37 KSP preconditioned resid norm 7.490324984936e-01 true resid norm 2.480074017552e-01 ||r(i)||/||b|| 1.818902836488e-06
 37 KSP Residual norm 0.749032 
 38 KSP preconditioned resid norm 4.943435684205e-01 true resid norm 1.470347806241e-01 ||r(i)||/||b|| 1.078362894199e-06
 38 KSP Residual norm 0.494344 
 39 KSP preconditioned resid norm 3.084788637285e-01 true resid norm 1.626408346511e-01 ||r(i)||/||b|| 1.192818735981e-06
 39 KSP Residual norm 0.308479 
 40 KSP preconditioned resid norm 2.606142931640e-02 true resid norm 7.663053412700e-03 ||r(i)||/||b|| 5.620134516098e-08
 40 KSP Residual norm 0.0260614 
 41 KSP preconditioned resid norm 4.654756581390e-01 true resid norm 2.690546086115e-01 ||r(i)||/||b|| 1.973264456263e-06
 41 KSP Residual norm 0.465476 
 42 KSP preconditioned resid norm 4.983426095489e-02 true resid norm 2.356674411751e-02 ||r(i)||/||b|| 1.728400742025e-07
 42 KSP Residual norm 0.0498343 
 43 KSP preconditioned resid norm 2.506106421374e-02 true resid norm 7.194416846674e-03 ||r(i)||/||b|| 5.276433330894e-08
 43 KSP Residual norm 0.0250611 
 44 KSP preconditioned resid norm 6.791580828568e-03 true resid norm 5.051841909812e-03 ||r(i)||/||b|| 3.705054572653e-08
 44 KSP Residual norm 0.00679158 
 45 KSP preconditioned resid norm 7.135736697854e-03 true resid norm 2.074481391256e-03 ||r(i)||/||b|| 1.521438497437e-08
 45 KSP Residual norm 0.00713574 
 46 KSP preconditioned resid norm 4.020631323929e-03 true resid norm 1.355065535193e-03 ||r(i)||/||b|| 9.938141072192e-09
 46 KSP Residual norm 0.00402063 
 47 KSP preconditioned resid norm 2.057039990439e-03 true resid norm 1.371727147000e-03 ||r(i)||/||b|| 1.006033844518e-08
 47 KSP Residual norm 0.00205704 
 48 KSP preconditioned resid norm 1.353446629420e-03 true resid norm 3.851655898770e-04 ||r(i)||/||b|| 2.824830142112e-09
 48 KSP Residual norm 0.00135345 
 49 KSP preconditioned resid norm 2.553647882477e-04 true resid norm 1.863188574285e-04 ||r(i)||/||b|| 1.366474935303e-09
 49 KSP Residual norm 0.000255365 
 50 KSP preconditioned resid norm 1.612173268317e-04 true resid norm 4.653561708770e-05 ||r(i)||/||b|| 3.412953215087e-10
 50 KSP Residual norm 0.000161217 
 51 KSP preconditioned resid norm 1.137051022483e-04 true resid norm 3.292772361909e-05 ||r(i)||/||b|| 2.414941226189e-10
 51 KSP Residual norm 0.000113705 
 52 KSP preconditioned resid norm 7.277578106383e-05 true resid norm 2.271587299792e-05 ||r(i)||/||b|| 1.665997286243e-10
 52 KSP Residual norm 7.27758e-05 
 53 KSP preconditioned resid norm 6.105615488606e-05 true resid norm 2.478243215255e-05 ||r(i)||/||b|| 1.817560113865e-10
 53 KSP Residual norm 6.10562e-05 
 54 KSP preconditioned resid norm 6.167791539833e-05 true resid norm 1.874624940809e-05 ||r(i)||/||b|| 1.374862442837e-10
 54 KSP Residual norm 6.16779e-05 
 55 KSP preconditioned resid norm 5.608616574890e-05 true resid norm 1.689919986660e-05 ||r(i)||/||b|| 1.239398596744e-10
 55 KSP Residual norm 5.60862e-05 
 56 KSP preconditioned resid norm 2.920504130814e-05 true resid norm 2.040313998501e-05 ||r(i)||/||b|| 1.496379903558e-10
 56 KSP Residual norm 2.9205e-05 
 57 KSP preconditioned resid norm 2.072538301751e-05 true resid norm 5.908651320822e-06 ||r(i)||/||b|| 4.333444313034e-11
 57 KSP Residual norm 2.07254e-05 
 58 KSP preconditioned resid norm 1.827939839330e-05 true resid norm 5.466508729321e-06 ||r(i)||/||b|| 4.009173985567e-11
 58 KSP Residual norm 1.82794e-05 
Linear solve converged due to CONVERGED_RTOL iterations 58
KSP Object: 1 MPI processes
  type: bcgs
  maximum iterations=10000
  tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: mg
    MG: type is MULTIPLICATIVE, levels=1 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_levels_0_)     1 MPI processes
      type: chebyshev
        Chebyshev: eigenvalue estimates:  min = 0.0959872, max = 1.05586
        Chebyshev: eigenvalues estimated using gmres with translations  [0 0.1; 0 1.1]
        KSP Object:        (mg_levels_0_esteig_)         1 MPI processes
          type: gmres
            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            GMRES: happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_levels_0_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=1030301, cols=1030301
        total: nonzeros=6.8521e+06, allocated nonzeros=0
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          not using I-node routines
  linear system matrix = precond matrix:
  Mat Object:   1 MPI processes
    type: seqaij
    rows=1030301, cols=1030301
    total: nonzeros=6.8521e+06, allocated nonzeros=0
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      not using I-node routines

-- Ramachandran --
Ran with -pc_type mg -pc_mg_galerkin -ksp_monitor_true_residual -ksp_converged_reason -ksp_view
-----------------------------------------------------------------------------------------------

Calculating Electric Field.....done

Timing Info
     Solve 3.30000000e+01
 TimeSteps 0.00000000e+00
Poisson TimeSteps 0.00000000e+00
-------------- next part --------------
Number of lines read was 629.
  0 KSP preconditioned resid norm 1.584346428381e+05 true resid norm 1.336500000000e+05 ||r(i)||/||b|| 9.801980198020e-01
  0 KSP Residual norm 158435 
  1 KSP preconditioned resid norm 3.329718554911e+04 true resid norm 1.314423605069e+04 ||r(i)||/||b|| 9.640070444218e-02
  1 KSP Residual norm 33297.2 
  2 KSP preconditioned resid norm 1.648141546540e+04 true resid norm 6.460322005614e+03 ||r(i)||/||b|| 4.738043275111e-02
  2 KSP Residual norm 16481.4 
  3 KSP preconditioned resid norm 1.061572613965e+04 true resid norm 3.490828142800e+03 ||r(i)||/||b|| 2.560196657719e-02
  3 KSP Residual norm 10615.7 
  4 KSP preconditioned resid norm 7.663980481149e+03 true resid norm 2.600584917522e+03 ||r(i)||/||b|| 1.907286334816e-02
  4 KSP Residual norm 7663.98 
  5 KSP preconditioned resid norm 5.922848799208e+03 true resid norm 1.847479847878e+03 ||r(i)||/||b|| 1.354954050515e-02
  5 KSP Residual norm 5922.85 
  6 KSP preconditioned resid norm 4.780746481930e+03 true resid norm 1.540624138254e+03 ||r(i)||/||b|| 1.129904025122e-02
  6 KSP Residual norm 4780.75 
  7 KSP preconditioned resid norm 3.982515401127e+03 true resid norm 1.212451741093e+03 ||r(i)||/||b|| 8.892201988217e-03
  7 KSP Residual norm 3982.52 
  8 KSP preconditioned resid norm 3.394848744858e+03 true resid norm 1.063430700112e+03 ||r(i)||/||b|| 7.799271727993e-03
  8 KSP Residual norm 3394.85 
  9 KSP preconditioned resid norm 2.946986523448e+03 true resid norm 8.843206088626e+02 ||r(i)||/||b|| 6.485666364962e-03
  9 KSP Residual norm 2946.99 
 10 KSP preconditioned resid norm 2.594904096175e+03 true resid norm 7.983337243771e+02 ||r(i)||/||b|| 5.855032815380e-03
 10 KSP Residual norm 2594.9 
 11 KSP preconditioned resid norm 2.311968790169e+03 true resid norm 6.870824933131e+02 ||r(i)||/||b|| 5.039108861849e-03
 11 KSP Residual norm 2311.97 
 12 KSP preconditioned resid norm 2.079971665605e+03 true resid norm 6.318318176146e+02 ||r(i)||/||b|| 4.633896718846e-03
 12 KSP Residual norm 2079.97 
 13 KSP preconditioned resid norm 1.886761216402e+03 true resid norm 5.568184911310e+02 ||r(i)||/||b|| 4.083743976025e-03
 13 KSP Residual norm 1886.76 
 14 KSP preconditioned resid norm 1.723585875997e+03 true resid norm 5.186529765921e+02 ||r(i)||/||b|| 3.803835545230e-03
 14 KSP Residual norm 1723.59 
 15 KSP preconditioned resid norm 1.584135487528e+03 true resid norm 4.650731464720e+02 ||r(i)||/||b|| 3.410877495211e-03
 15 KSP Residual norm 1584.14 
 16 KSP preconditioned resid norm 1.463755818509e+03 true resid norm 4.372612193339e+02 ||r(i)||/||b|| 3.206902965412e-03
 16 KSP Residual norm 1463.76 
 17 KSP preconditioned resid norm 1.358866173679e+03 true resid norm 3.973337643902e+02 ||r(i)||/||b|| 2.914072346096e-03
 17 KSP Residual norm 1358.87 
 18 KSP preconditioned resid norm 1.266778761650e+03 true resid norm 3.762590528147e+02 ||r(i)||/||b|| 2.759509004875e-03
 18 KSP Residual norm 1266.78 
 19 KSP preconditioned resid norm 1.185305963597e+03 true resid norm 3.455095285668e+02 ||r(i)||/||b|| 2.533989941817e-03
 19 KSP Residual norm 1185.31 
 20 KSP preconditioned resid norm 1.112713644522e+03 true resid norm 3.290695809019e+02 ||r(i)||/||b|| 2.413418268441e-03
 20 KSP Residual norm 1112.71 
 21 KSP preconditioned resid norm 1.047330743723e+03 true resid norm 3.046859841789e+02 ||r(i)||/||b|| 2.234587342713e-03
 21 KSP Residual norm 1047.33 
 22 KSP preconditioned resid norm 9.872190361077e+02 true resid norm 2.915418293043e+02 ||r(i)||/||b|| 2.138187233622e-03
 22 KSP Residual norm 987.219 
 23 KSP preconditioned resid norm 9.297158200274e+02 true resid norm 2.707299670068e+02 ||r(i)||/||b|| 1.985551646548e-03
 23 KSP Residual norm 929.716 
 24 KSP preconditioned resid norm 8.711168180798e+02 true resid norm 2.585284824328e+02 ||r(i)||/||b|| 1.896065144355e-03
 24 KSP Residual norm 871.117 
 25 KSP preconditioned resid norm 8.071257590769e+02 true resid norm 2.362350597501e+02 ||r(i)||/||b|| 1.732563694537e-03
 25 KSP Residual norm 807.126 
 26 KSP preconditioned resid norm 7.337148004310e+02 true resid norm 2.198162674440e+02 ||r(i)||/||b|| 1.612147175974e-03
 26 KSP Residual norm 733.715 
 27 KSP preconditioned resid norm 6.481079316784e+02 true resid norm 1.907931213373e+02 ||r(i)||/||b|| 1.399289485422e-03
 27 KSP Residual norm 648.108 
 28 KSP preconditioned resid norm 5.498721728382e+02 true resid norm 1.657972791351e+02 ||r(i)||/||b|| 1.215968310489e-03
 28 KSP Residual norm 549.872 
 29 KSP preconditioned resid norm 4.405443763836e+02 true resid norm 1.299168725341e+02 ||r(i)||/||b|| 9.528190138183e-04
 29 KSP Residual norm 440.544 
 30 KSP preconditioned resid norm 3.240236086790e+02 true resid norm 9.742416896340e+01 ||r(i)||/||b|| 7.145153572674e-04
 30 KSP Residual norm 324.024 
 31 KSP preconditioned resid norm 2.059166803676e+02 true resid norm 6.038871504357e+01 ||r(i)||/||b|| 4.428948664728e-04
 31 KSP Residual norm 205.917 
 32 KSP preconditioned resid norm 9.928747101615e+01 true resid norm 2.942859533885e+01 ||r(i)||/||b|| 2.158312822798e-04
 32 KSP Residual norm 99.2875 
 33 KSP preconditioned resid norm 3.081144236888e+01 true resid norm 9.001159617176e+00 ||r(i)||/||b|| 6.601510536983e-05
 33 KSP Residual norm 30.8114 
 34 KSP preconditioned resid norm 9.499307885981e+00 true resid norm 3.197499073879e+00 ||r(i)||/||b|| 2.345067160894e-05
 34 KSP Residual norm 9.49931 
 35 KSP preconditioned resid norm 6.301154502299e+00 true resid norm 2.131038916952e+00 ||r(i)||/||b|| 1.562918164248e-05
 35 KSP Residual norm 6.30115 
 36 KSP preconditioned resid norm 5.232116757360e+00 true resid norm 1.680863714690e+00 ||r(i)||/||b|| 1.232756666440e-05
 36 KSP Residual norm 5.23212 
 37 KSP preconditioned resid norm 4.383648958169e+00 true resid norm 1.503916846559e+00 ||r(i)||/||b|| 1.102982652408e-05
 37 KSP Residual norm 4.38365 
 38 KSP preconditioned resid norm 3.496300521941e+00 true resid norm 1.019488875098e+00 ||r(i)||/||b|| 7.476999450667e-06
 38 KSP Residual norm 3.4963 
 39 KSP preconditioned resid norm 1.329417775526e+00 true resid norm 6.305255432772e-01 ||r(i)||/||b|| 4.624316415674e-06
 39 KSP Residual norm 1.32942 
 40 KSP preconditioned resid norm 1.173715165878e+00 true resid norm 3.391746525136e-01 ||r(i)||/||b|| 2.487529538054e-06
 40 KSP Residual norm 1.17372 
 41 KSP preconditioned resid norm 1.010471503016e+00 true resid norm 2.898013854885e-01 ||r(i)||/||b|| 2.125422702519e-06
 41 KSP Residual norm 1.01047 
 42 KSP preconditioned resid norm 5.986676220588e-01 true resid norm 3.726978951592e-01 ||r(i)||/||b|| 2.733391236958e-06
 42 KSP Residual norm 0.598668 
 43 KSP preconditioned resid norm 3.865662538772e-01 true resid norm 1.104258914088e-01 ||r(i)||/||b|| 8.098708574169e-07
 43 KSP Residual norm 0.386566 
 44 KSP preconditioned resid norm 3.873809044313e-01 true resid norm 1.199838094365e-01 ||r(i)||/||b|| 8.799692661280e-07
 44 KSP Residual norm 0.387381 
 45 KSP preconditioned resid norm 2.363433250281e-01 true resid norm 1.144972173805e-01 ||r(i)||/||b|| 8.397302338134e-07
 45 KSP Residual norm 0.236343 
 46 KSP preconditioned resid norm 1.677043408823e-01 true resid norm 4.861518916415e-02 ||r(i)||/||b|| 3.565470419080e-07
 46 KSP Residual norm 0.167704 
 47 KSP preconditioned resid norm 1.210646694945e-01 true resid norm 4.991913563286e-02 ||r(i)||/||b|| 3.661102723349e-07
 47 KSP Residual norm 0.121065 
 48 KSP preconditioned resid norm 1.178117227582e-01 true resid norm 3.372010481766e-02 ||r(i)||/||b|| 2.473054992128e-07
 48 KSP Residual norm 0.117812 
 49 KSP preconditioned resid norm 1.101626826323e-01 true resid norm 3.628562416903e-02 ||r(i)||/||b|| 2.661211893585e-07
 49 KSP Residual norm 0.110163 
 50 KSP preconditioned resid norm 8.417129350461e-02 true resid norm 2.581841783845e-02 ||r(i)||/||b|| 1.893539995486e-07
 50 KSP Residual norm 0.0841713 
 51 KSP preconditioned resid norm 7.799903830399e-02 true resid norm 2.243536442089e-02 ||r(i)||/||b|| 1.645424599992e-07
 51 KSP Residual norm 0.077999 
 52 KSP preconditioned resid norm 1.317723328638e-01 true resid norm 4.126139787619e-02 ||r(i)||/||b|| 3.026138458100e-07
 52 KSP Residual norm 0.131772 
 53 KSP preconditioned resid norm 6.069947269632e-02 true resid norm 2.216505265229e-02 ||r(i)||/||b|| 1.625599754477e-07
 53 KSP Residual norm 0.0606995 
 54 KSP preconditioned resid norm 5.435441953157e-02 true resid norm 1.552684149554e-02 ||r(i)||/||b|| 1.138748917898e-07
 54 KSP Residual norm 0.0543544 
 55 KSP preconditioned resid norm 2.547161131414e-02 true resid norm 1.577103638976e-02 ||r(i)||/||b|| 1.156658334416e-07
 55 KSP Residual norm 0.0254716 
 56 KSP preconditioned resid norm 2.001933827413e-02 true resid norm 5.694615940615e-03 ||r(i)||/||b|| 4.176469336718e-08
 56 KSP Residual norm 0.0200193 
 57 KSP preconditioned resid norm 1.876429465485e-02 true resid norm 5.519797333920e-03 ||r(i)||/||b|| 4.048256203828e-08
 57 KSP Residual norm 0.0187643 
 58 KSP preconditioned resid norm 1.692036543945e-02 true resid norm 5.453246578101e-03 ||r(i)||/||b|| 3.999447435351e-08
 58 KSP Residual norm 0.0169204 
 59 KSP preconditioned resid norm 1.582371180583e-02 true resid norm 4.496211187034e-03 ||r(i)||/||b|| 3.297551292288e-08
 59 KSP Residual norm 0.0158237 
 60 KSP preconditioned resid norm 1.190913027763e-02 true resid norm 3.914110514439e-03 ||r(i)||/||b|| 2.870634774066e-08
 60 KSP Residual norm 0.0119091 
 61 KSP preconditioned resid norm 1.146442223272e-02 true resid norm 3.271054008401e-03 ||r(i)||/||b|| 2.399012840778e-08
 61 KSP Residual norm 0.0114644 
 62 KSP preconditioned resid norm 1.067803229789e-02 true resid norm 6.183719226703e-03 ||r(i)||/||b|| 4.535180951011e-08
 62 KSP Residual norm 0.010678 
 63 KSP preconditioned resid norm 8.643055835024e-03 true resid norm 2.460641084429e-03 ||r(i)||/||b|| 1.804650593640e-08
 63 KSP Residual norm 0.00864306 
 64 KSP preconditioned resid norm 8.464933444833e-03 true resid norm 2.410422764398e-03 ||r(i)||/||b|| 1.767820142573e-08
 64 KSP Residual norm 0.00846493 
 65 KSP preconditioned resid norm 5.841843565636e-03 true resid norm 3.286620231803e-03 ||r(i)||/||b|| 2.410429212910e-08
 65 KSP Residual norm 0.00584184 
 66 KSP preconditioned resid norm 4.549278271976e-03 true resid norm 1.294612772402e-03 ||r(i)||/||b|| 9.494776475261e-09
 66 KSP Residual norm 0.00454928 
 67 KSP preconditioned resid norm 3.514982562057e-03 true resid norm 1.010411827086e-03 ||r(i)||/||b|| 7.410427774743e-09
 67 KSP Residual norm 0.00351498 
 68 KSP preconditioned resid norm 3.522198184965e-03 true resid norm 1.011773142531e-03 ||r(i)||/||b|| 7.420411753067e-09
 68 KSP Residual norm 0.0035222 
 69 KSP preconditioned resid norm 3.073114581057e-03 true resid norm 8.751758033987e-04 ||r(i)||/||b|| 6.418597751365e-09
 69 KSP Residual norm 0.00307311 
 70 KSP preconditioned resid norm 3.114994556465e-03 true resid norm 8.879564167853e-04 ||r(i)||/||b|| 6.512331622921e-09
 70 KSP Residual norm 0.00311499 
 71 KSP preconditioned resid norm 3.006653595406e-03 true resid norm 1.042656194287e-03 ||r(i)||/||b|| 7.646910115783e-09
 71 KSP Residual norm 0.00300665 
 72 KSP preconditioned resid norm 3.374944103191e-03 true resid norm 9.584399807989e-04 ||r(i)||/||b|| 7.029262785471e-09
 72 KSP Residual norm 0.00337494 
 73 KSP preconditioned resid norm 2.583871653685e-03 true resid norm 7.512827414514e-04 ||r(i)||/||b|| 5.509957766421e-09
 73 KSP Residual norm 0.00258387 
 74 KSP preconditioned resid norm 1.954747455191e-03 true resid norm 5.834519242594e-04 ||r(i)||/||b|| 4.279075352104e-09
 74 KSP Residual norm 0.00195475 
 75 KSP preconditioned resid norm 2.481888252857e-03 true resid norm 7.091861543258e-04 ||r(i)||/||b|| 5.201218586915e-09
 75 KSP Residual norm 0.00248189 
 76 KSP preconditioned resid norm 3.219625627106e-03 true resid norm 9.215795139409e-04 ||r(i)||/||b|| 6.758925661466e-09
 76 KSP Residual norm 0.00321963 
 77 KSP preconditioned resid norm 2.168313805953e-03 true resid norm 6.494908996548e-04 ||r(i)||/||b|| 4.763409605096e-09
 77 KSP Residual norm 0.00216831 
 78 KSP preconditioned resid norm 5.430344606505e-04 true resid norm 1.555158829080e-04 ||r(i)||/||b|| 1.140563864378e-09
 78 KSP Residual norm 0.000543034 
 79 KSP preconditioned resid norm 1.180633568496e-04 true resid norm 3.412969440973e-05 ||r(i)||/||b|| 2.503094566170e-10
 79 KSP Residual norm 0.000118063 
 80 KSP preconditioned resid norm 1.489292655512e-04 true resid norm 5.291253375816e-05 ||r(i)||/||b|| 3.880640539652e-10
 80 KSP Residual norm 0.000148929 
 81 KSP preconditioned resid norm 1.725342222348e-05 true resid norm 5.489604690976e-06 ||r(i)||/||b|| 4.026112717988e-11
 81 KSP Residual norm 1.72534e-05 
Linear solve converged due to CONVERGED_RTOL iterations 81
KSP Object: 1 MPI processes
  type: bcgs
  maximum iterations=10000
  tolerances:  relative=1e-10, absolute=1e-50, divergence=10000
  left preconditioning
  using nonzero initial guess
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: mg
    MG: type is MULTIPLICATIVE, levels=1 cycles=v
      Cycles per PCApply=1
      Using Galerkin computed coarse grid matrices
  Coarse grid solver -- level -------------------------------
    KSP Object:    (mg_levels_0_)     1 MPI processes
      type: chebyshev
        Chebyshev: eigenvalue estimates:  min = 0.0968067, max = 1.06487
        Chebyshev: eigenvalues estimated using gmres with translations  [0 0.1; 0 1.1]
        KSP Object:        (mg_levels_0_esteig_)         1 MPI processes
          type: gmres
            GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
            GMRES: happy breakdown tolerance 1e-30
          maximum iterations=10, initial guess is zero
          tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
          left preconditioning
          using NONE norm type for convergence test
      maximum iterations=1, initial guess is zero
      tolerances:  relative=1e-05, absolute=1e-50, divergence=10000
      left preconditioning
      using NONE norm type for convergence test
    PC Object:    (mg_levels_0_)     1 MPI processes
      type: sor
        SOR: type = local_symmetric, iterations = 1, local iterations = 1, omega = 1
      linear system matrix = precond matrix:
      Mat Object:       1 MPI processes
        type: seqaij
        rows=1030301, cols=1030301
        total: nonzeros=6.93209e+06, allocated nonzeros=0
        total number of mallocs used during MatSetValues calls =0
          has attached near null space
          not using I-node routines
  linear system matrix = precond matrix:
  Mat Object:   1 MPI processes
    type: seqaij
    rows=1030301, cols=1030301
    total: nonzeros=6.93209e+06, allocated nonzeros=0
    total number of mallocs used during MatSetValues calls =0
      has attached near null space
      not using I-node routines


-- Ramachandran --
Ran with -pc_type mg -pc_mg_galerkin -ksp_monitor_true_residual -ksp_converged_reason -ksp_view
-----------------------------------------------------------------------------------------------

Calculating Electric Field.....done

Timing Info
     Solve 4.60000000e+01
 TimeSteps 0.00000000e+00
Poisson TimeSteps 0.00000000e+00

From knepley at gmail.com  Mon Oct 12 21:17:19 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 12 Oct 2015 21:17:19 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
Message-ID: <CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>

On Mon, Oct 12, 2015 at 9:11 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Matt,
>
> Actually I felt the boundary conditions were having a role to play and set
> all the boundary conditions to Dirichlet. In this case, convergence was
> almost immediate with the Hypre preconditioner, taking 17 seconds with 3
> iterations. The MG method took about the same time though.
>
> So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to
> diverge. Please find attached the output for the Hypre run using Dirichlet
> and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row.
> Details of the options used before running are in the file. The solver used
> in all cases is bcgs.
>
> Also attached is the MG output for 101^3 grid
> 1) Dirichlet and Neumann
> 2) Dirichlet only
>
> where it seems to take about the same time.
>

Notice that you have no levels of MG here. You need to use -pc_mg_levels <n>

   Matt


> I was thinking if the Null space has something to do with this? From the
> PETSc docs, I understand that the Null space should be removed/attached
> when solving singular systems. My domain will have at least two Dirichlet
> conditions, so it is not singular. But since the Neumann conditions seem to
> affect the convergence with Hypre, perhaps they have a role to play in
> terms of the null space?
>
>
>
> On Mon, Oct 12, 2015 at 3:52 PM, K. N. Ramachandran <knram06 at gmail.com>
> wrote:
>
>> Hello Matt,
>>
>> Thanks for your reply. I am actually retrying against the latest PETSc
>> and will revert once I get the information.
>>
>> On Mon, Oct 12, 2015 at 9:39 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>>> On Sat, Oct 10, 2015 at 7:56 PM, K. N. Ramachandran <knram06 at gmail.com>
>>> wrote:
>>>
>>>> Sorry, some more questions.
>>>>
>>>> 3) Also, for Dirichlet bc, I specify the value through Identity rows,
>>>> i.e. A_ii = 1 and the rhs value would correspond to the Dirichlet
>>>> condition. I am specifying it this way for my convenience. I am aware that
>>>> MatZerosRowColumns might help here, but would keeping it this way be
>>>> detrimental?
>>>>
>>>
>>> That is fine. However you would want to scale these entries to be
>>> approximately the same size as the other diagonal entries.
>>>
>>>
>>>> 4) Can I expect symmetric matrices to perform better, i.e. if I
>>>> eliminate Dirichlet rows? But I would still be left with Neumann boundary
>>>> conditions, where I use the second order formulation. If I used the first
>>>> order formulation and made it symmetric, would that be an advantage? I
>>>> tried the latter, but I didn't see the condition number change much.
>>>>
>>>
>>> Will not matter for MG.
>>>
>>>     Matt
>>>
>>>
>>>>
>>>>
>>
>> Regards,
>> K.N.Ramachandran
>> Ph: 814-441-4279
>>
>
>
>
> Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151012/5f73181c/attachment.html>

From arne.morten.kvarving at sintef.no  Tue Oct 13 08:43:29 2015
From: arne.morten.kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 15:43:29 +0200
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
Message-ID: <561D0A81.6020906@sintef.no>

hi there;

why does the cmake config file set the BUILD_SHARED_LIBS variable? this 
affects all buildsystem where the config file is included, and has 0 
effect on the actual petsc side of things.

i don't see why petsc should decided whether i want to use static 
libraries in my application or not.
i can keep hacking around it, or supply a patch to get rid of it. but i 
figured i'd ask for the reasoning first.

if this should have gone to -dev, i'm sorry. i thought it fit better here.

arnem

From balay at mcs.anl.gov  Tue Oct 13 09:56:25 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 09:56:25 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <561D0A81.6020906@sintef.no>
References: <561D0A81.6020906@sintef.no>
Message-ID: <alpine.LFD.2.20.1510130937160.3547@asterix>

Well the current code has the option of building petsc via cmake [but
this mode is deprecated - and the default build uses straight gnumake]

So if you configure PETSc with --shared-libraries=1 [default] - this
flag is set in conf/PETScConfig.cmake

So you can rebuild petc with --shared-libraries=0 - and this flag will
go away.

BTW: I'm not sure why this affects your application. [as applications
don't build libraries]. Or is it that you have a library that builds
over PETSc?

[in this case - it generally makes sense to build it as shared anyway]

Also - I'm guessing - there must be a way for you to reset this variable in
your cmake config - if thats whats requred by your app.

Jed should be able to confirm..

Satish


On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> hi there;
> 
> why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> affects all buildsystem where the config file is included, and has 0 effect on
> the actual petsc side of things.
> 
> i don't see why petsc should decided whether i want to use static libraries in
> my application or not.
> i can keep hacking around it, or supply a patch to get rid of it. but i
> figured i'd ask for the reasoning first.
> 
> if this should have gone to -dev, i'm sorry. i thought it fit better here.
> 
> arnem
> 
> 


From Arne.Morten.Kvarving at sintef.no  Tue Oct 13 10:38:18 2015
From: Arne.Morten.Kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 15:38:18 +0000
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510130937160.3547@asterix>
References: <561D0A81.6020906@sintef.no>,
	<alpine.LFD.2.20.1510130937160.3547@asterix>
Message-ID: <ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>

Yes I am aware and i can reset and such.
This is about the why, not the how. The config file has nothing to do with the petsc build system, it is for users of petsc, like a .pc file (pkgconfig).

The code in question builds a number of convenience libraries for shared code between multiple apps. I want these static at times but in general shared for installs. This I control using the intended cmake mechanism - namely the BUILD_SHARED_LIBS variable. Petsc breaks this since it overrides the value passed by the user.

I would rather avoid hacking my buildsystem to workaround petsc bugs. Right now i need to deploy a patched petsc everywhere for this reason - to remove the useless variable writing in the cmake config file. Because as you say shared libs should normally be used.


-------- Opprinnelig melding --------
Fra: Satish Balay <balay at mcs.anl.gov>
Dato: 13.10.2015 16.56 (GMT+01:00)
Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
Ko: petsc-users at mcs.anl.gov
Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS

Well the current code has the option of building petsc via cmake [but
this mode is deprecated - and the default build uses straight gnumake]

So if you configure PETSc with --shared-libraries=1 [default] - this
flag is set in conf/PETScConfig.cmake

So you can rebuild petc with --shared-libraries=0 - and this flag will
go away.

BTW: I'm not sure why this affects your application. [as applications
don't build libraries]. Or is it that you have a library that builds
over PETSc?

[in this case - it generally makes sense to build it as shared anyway]

Also - I'm guessing - there must be a way for you to reset this variable in
your cmake config - if thats whats requred by your app.

Jed should be able to confirm..

Satish


On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> hi there;
>
> why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> affects all buildsystem where the config file is included, and has 0 effect on
> the actual petsc side of things.
>
> i don't see why petsc should decided whether i want to use static libraries in
> my application or not.
> i can keep hacking around it, or supply a patch to get rid of it. but i
> figured i'd ask for the reasoning first.
>
> if this should have gone to -dev, i'm sorry. i thought it fit better here.
>
> arnem
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/de9c2490/attachment.html>

From balay at mcs.anl.gov  Tue Oct 13 10:53:09 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 10:53:09 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
References: <561D0A81.6020906@sintef.no>,
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
Message-ID: <alpine.LFD.2.20.1510131041370.3026@asterix>

As mentioned - PETScConfig.cmake is primarily for building PETSc - and
not equivalent to .pc file

It gets used when 'make all-cmake' is used instead of the default
'make all' to build petsc libraries.

[This mode is deprecated - and not properly tested - and I see its currenty broken]

Jed had a tool FindPETSc.cmake - which is a tool for the user to
detect petsc config via cmake.

https://github.com/jedbrown/cmake-modules/

Is this what you are using?

Satish

On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> Yes I am aware and i can reset and such.
> This is about the why, not the how. The config file has nothing to do with the petsc build system, it is for users of petsc, like a .pc file (pkgconfig).
> 
> The code in question builds a number of convenience libraries for shared code between multiple apps. I want these static at times but in general shared for installs. This I control using the intended cmake mechanism - namely the BUILD_SHARED_LIBS variable. Petsc breaks this since it overrides the value passed by the user.
> 
> I would rather avoid hacking my buildsystem to workaround petsc bugs. Right now i need to deploy a patched petsc everywhere for this reason - to remove the useless variable writing in the cmake config file. Because as you say shared libs should normally be used.
> 
> 
> -------- Opprinnelig melding --------
> Fra: Satish Balay <balay at mcs.anl.gov>
> Dato: 13.10.2015 16.56 (GMT+01:00)
> Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> Ko: petsc-users at mcs.anl.gov
> Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
> 
> Well the current code has the option of building petsc via cmake [but
> this mode is deprecated - and the default build uses straight gnumake]
> 
> So if you configure PETSc with --shared-libraries=1 [default] - this
> flag is set in conf/PETScConfig.cmake
> 
> So you can rebuild petc with --shared-libraries=0 - and this flag will
> go away.
> 
> BTW: I'm not sure why this affects your application. [as applications
> don't build libraries]. Or is it that you have a library that builds
> over PETSc?
> 
> [in this case - it generally makes sense to build it as shared anyway]
> 
> Also - I'm guessing - there must be a way for you to reset this variable in
> your cmake config - if thats whats requred by your app.
> 
> Jed should be able to confirm..
> 
> Satish
> 
> 
> On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
> 
> > hi there;
> >
> > why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> > affects all buildsystem where the config file is included, and has 0 effect on
> > the actual petsc side of things.
> >
> > i don't see why petsc should decided whether i want to use static libraries in
> > my application or not.
> > i can keep hacking around it, or supply a patch to get rid of it. but i
> > figured i'd ask for the reasoning first.
> >
> > if this should have gone to -dev, i'm sorry. i thought it fit better here.
> >
> > arnem
> >
> >
> 
> 


From Arne.Morten.Kvarving at sintef.no  Tue Oct 13 11:05:26 2015
From: Arne.Morten.Kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 16:05:26 +0000
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510131041370.3026@asterix>
References: <561D0A81.6020906@sintef.no>,
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>,
	<alpine.LFD.2.20.1510131041370.3026@asterix>
Message-ID: <6s79deqkttduqfby4nyybvl1.1444752324920@email.android.com>

Right. Then I will keep patching as there is no other easy way to use petsc in a portable way. Too many defines used in too many headers for me to do bother writing my own find rule.

Thanks for your answers.


-------- Opprinnelig melding --------
Fra: Satish Balay <balay at mcs.anl.gov>
Dato: 13.10.2015 17.53 (GMT+01:00)
Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
Ko: petsc-users <petsc-users at mcs.anl.gov>
Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS

As mentioned - PETScConfig.cmake is primarily for building PETSc - and
not equivalent to .pc file

It gets used when 'make all-cmake' is used instead of the default
'make all' to build petsc libraries.

[This mode is deprecated - and not properly tested - and I see its currenty broken]

Jed had a tool FindPETSc.cmake - which is a tool for the user to
detect petsc config via cmake.

https://github.com/jedbrown/cmake-modules/

Is this what you are using?

Satish

On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> Yes I am aware and i can reset and such.
> This is about the why, not the how. The config file has nothing to do with the petsc build system, it is for users of petsc, like a .pc file (pkgconfig).
>
> The code in question builds a number of convenience libraries for shared code between multiple apps. I want these static at times but in general shared for installs. This I control using the intended cmake mechanism - namely the BUILD_SHARED_LIBS variable. Petsc breaks this since it overrides the value passed by the user.
>
> I would rather avoid hacking my buildsystem to workaround petsc bugs. Right now i need to deploy a patched petsc everywhere for this reason - to remove the useless variable writing in the cmake config file. Because as you say shared libs should normally be used.
>
>
> -------- Opprinnelig melding --------
> Fra: Satish Balay <balay at mcs.anl.gov>
> Dato: 13.10.2015 16.56 (GMT+01:00)
> Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> Ko: petsc-users at mcs.anl.gov
> Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
>
> Well the current code has the option of building petsc via cmake [but
> this mode is deprecated - and the default build uses straight gnumake]
>
> So if you configure PETSc with --shared-libraries=1 [default] - this
> flag is set in conf/PETScConfig.cmake
>
> So you can rebuild petc with --shared-libraries=0 - and this flag will
> go away.
>
> BTW: I'm not sure why this affects your application. [as applications
> don't build libraries]. Or is it that you have a library that builds
> over PETSc?
>
> [in this case - it generally makes sense to build it as shared anyway]
>
> Also - I'm guessing - there must be a way for you to reset this variable in
> your cmake config - if thats whats requred by your app.
>
> Jed should be able to confirm..
>
> Satish
>
>
> On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
>
> > hi there;
> >
> > why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> > affects all buildsystem where the config file is included, and has 0 effect on
> > the actual petsc side of things.
> >
> > i don't see why petsc should decided whether i want to use static libraries in
> > my application or not.
> > i can keep hacking around it, or supply a patch to get rid of it. but i
> > figured i'd ask for the reasoning first.
> >
> > if this should have gone to -dev, i'm sorry. i thought it fit better here.
> >
> > arnem
> >
> >
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/409a011d/attachment-0001.html>

From balay at mcs.anl.gov  Tue Oct 13 11:13:21 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 11:13:21 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <6s79deqkttduqfby4nyybvl1.1444752324920@email.android.com>
References: <561D0A81.6020906@sintef.no>,
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>,
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<6s79deqkttduqfby4nyybvl1.1444752324920@email.android.com>
Message-ID: <alpine.LFD.2.20.1510131110290.5892@asterix>

Jed might have better suggestion [wrt cmake code and petsc]

You might have to explain why you can't use FindPETSc.cmake - and need
PETScConfig.cmake.

Satish



On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> Right. Then I will keep patching as there is no other easy way to use petsc in a portable way. Too many defines used in too many headers for me to do bother writing my own find rule.
> 
> Thanks for your answers.
> 
> 
> -------- Opprinnelig melding --------
> Fra: Satish Balay <balay at mcs.anl.gov>
> Dato: 13.10.2015 17.53 (GMT+01:00)
> Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> Ko: petsc-users <petsc-users at mcs.anl.gov>
> Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
> 
> As mentioned - PETScConfig.cmake is primarily for building PETSc - and
> not equivalent to .pc file
> 
> It gets used when 'make all-cmake' is used instead of the default
> 'make all' to build petsc libraries.
> 
> [This mode is deprecated - and not properly tested - and I see its currenty broken]
> 
> Jed had a tool FindPETSc.cmake - which is a tool for the user to
> detect petsc config via cmake.
> 
> https://github.com/jedbrown/cmake-modules/
> 
> Is this what you are using?
> 
> Satish
> 
> On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
> 
> > Yes I am aware and i can reset and such.
> > This is about the why, not the how. The config file has nothing to do with the petsc build system, it is for users of petsc, like a .pc file (pkgconfig).
> >
> > The code in question builds a number of convenience libraries for shared code between multiple apps. I want these static at times but in general shared for installs. This I control using the intended cmake mechanism - namely the BUILD_SHARED_LIBS variable. Petsc breaks this since it overrides the value passed by the user.
> >
> > I would rather avoid hacking my buildsystem to workaround petsc bugs. Right now i need to deploy a patched petsc everywhere for this reason - to remove the useless variable writing in the cmake config file. Because as you say shared libs should normally be used.
> >
> >
> > -------- Opprinnelig melding --------
> > Fra: Satish Balay <balay at mcs.anl.gov>
> > Dato: 13.10.2015 16.56 (GMT+01:00)
> > Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> > Ko: petsc-users at mcs.anl.gov
> > Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
> >
> > Well the current code has the option of building petsc via cmake [but
> > this mode is deprecated - and the default build uses straight gnumake]
> >
> > So if you configure PETSc with --shared-libraries=1 [default] - this
> > flag is set in conf/PETScConfig.cmake
> >
> > So you can rebuild petc with --shared-libraries=0 - and this flag will
> > go away.
> >
> > BTW: I'm not sure why this affects your application. [as applications
> > don't build libraries]. Or is it that you have a library that builds
> > over PETSc?
> >
> > [in this case - it generally makes sense to build it as shared anyway]
> >
> > Also - I'm guessing - there must be a way for you to reset this variable in
> > your cmake config - if thats whats requred by your app.
> >
> > Jed should be able to confirm..
> >
> > Satish
> >
> >
> > On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
> >
> > > hi there;
> > >
> > > why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> > > affects all buildsystem where the config file is included, and has 0 effect on
> > > the actual petsc side of things.
> > >
> > > i don't see why petsc should decided whether i want to use static libraries in
> > > my application or not.
> > > i can keep hacking around it, or supply a patch to get rid of it. but i
> > > figured i'd ask for the reasoning first.
> > >
> > > if this should have gone to -dev, i'm sorry. i thought it fit better here.
> > >
> > > arnem
> > >
> > >
> >
> >
> 
> 


From jed at jedbrown.org  Tue Oct 13 11:15:21 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 13 Oct 2015 10:15:21 -0600
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510131041370.3026@asterix>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
Message-ID: <87d1wi4uee.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:

> As mentioned - PETScConfig.cmake is primarily for building PETSc - and
> not equivalent to .pc file

  def dumpCMakeConfig(self):
    '''
    Writes configuration-specific values to ${PETSC_ARCH}/lib/petsc/conf/PETScConfig.cmake.
    This file is private to PETSc and should not be included by third parties
    (a suitable file can be produced later by CMake, but this is not it).
    '''

Arne, is this the file you are talking about?  It is part of the legacy
cmake build and is used to hold the various ARCH-specific variables when
building PETSc.  (Note that this build does not use CMake for configure
since CMake is disgusting for that purpose and also because we needed to
ensure that it makes all the same decisions as our configure.)  It's not
for users and should not be installed.

If you want a pkgconfig file, just use PETSc.pc.  Or use FindPETSc.cmake.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/e20e9ae1/attachment.pgp>

From Arne.Morten.Kvarving at sintef.no  Tue Oct 13 11:20:58 2015
From: Arne.Morten.Kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 16:20:58 +0000
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
Message-ID: <7x3ad188e8sn6kkjyoq2kpgy.1444752861647@email.android.com>

I can certainly can use the find rule but the fact that it isnt part of the petsc distro complicates things more for me than the patch does.

-------- Opprinnelig melding --------
Fra: Satish Balay <balay at mcs.anl.gov>
Dato: 13.10.2015 18.13 (GMT+01:00)
Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
Ko: petsc-users <petsc-users at mcs.anl.gov>
Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS

Jed might have better suggestion [wrt cmake code and petsc]

You might have to explain why you can't use FindPETSc.cmake - and need
PETScConfig.cmake.

Satish



On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:

> Right. Then I will keep patching as there is no other easy way to use petsc in a portable way. Too many defines used in too many headers for me to do bother writing my own find rule.
>
> Thanks for your answers.
>
>
> -------- Opprinnelig melding --------
> Fra: Satish Balay <balay at mcs.anl.gov>
> Dato: 13.10.2015 17.53 (GMT+01:00)
> Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> Ko: petsc-users <petsc-users at mcs.anl.gov>
> Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
>
> As mentioned - PETScConfig.cmake is primarily for building PETSc - and
> not equivalent to .pc file
>
> It gets used when 'make all-cmake' is used instead of the default
> 'make all' to build petsc libraries.
>
> [This mode is deprecated - and not properly tested - and I see its currenty broken]
>
> Jed had a tool FindPETSc.cmake - which is a tool for the user to
> detect petsc config via cmake.
>
> https://github.com/jedbrown/cmake-modules/
>
> Is this what you are using?
>
> Satish
>
> On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
>
> > Yes I am aware and i can reset and such.
> > This is about the why, not the how. The config file has nothing to do with the petsc build system, it is for users of petsc, like a .pc file (pkgconfig).
> >
> > The code in question builds a number of convenience libraries for shared code between multiple apps. I want these static at times but in general shared for installs. This I control using the intended cmake mechanism - namely the BUILD_SHARED_LIBS variable. Petsc breaks this since it overrides the value passed by the user.
> >
> > I would rather avoid hacking my buildsystem to workaround petsc bugs. Right now i need to deploy a patched petsc everywhere for this reason - to remove the useless variable writing in the cmake config file. Because as you say shared libs should normally be used.
> >
> >
> > -------- Opprinnelig melding --------
> > Fra: Satish Balay <balay at mcs.anl.gov>
> > Dato: 13.10.2015 16.56 (GMT+01:00)
> > Til: Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
> > Ko: petsc-users at mcs.anl.gov
> > Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
> >
> > Well the current code has the option of building petsc via cmake [but
> > this mode is deprecated - and the default build uses straight gnumake]
> >
> > So if you configure PETSc with --shared-libraries=1 [default] - this
> > flag is set in conf/PETScConfig.cmake
> >
> > So you can rebuild petc with --shared-libraries=0 - and this flag will
> > go away.
> >
> > BTW: I'm not sure why this affects your application. [as applications
> > don't build libraries]. Or is it that you have a library that builds
> > over PETSc?
> >
> > [in this case - it generally makes sense to build it as shared anyway]
> >
> > Also - I'm guessing - there must be a way for you to reset this variable in
> > your cmake config - if thats whats requred by your app.
> >
> > Jed should be able to confirm..
> >
> > Satish
> >
> >
> > On Tue, 13 Oct 2015, Arne Morten Kvarving wrote:
> >
> > > hi there;
> > >
> > > why does the cmake config file set the BUILD_SHARED_LIBS variable? this
> > > affects all buildsystem where the config file is included, and has 0 effect on
> > > the actual petsc side of things.
> > >
> > > i don't see why petsc should decided whether i want to use static libraries in
> > > my application or not.
> > > i can keep hacking around it, or supply a patch to get rid of it. but i
> > > figured i'd ask for the reasoning first.
> > >
> > > if this should have gone to -dev, i'm sorry. i thought it fit better here.
> > >
> > > arnem
> > >
> > >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/d3ae71b1/attachment.html>

From Arne.Morten.Kvarving at sintef.no  Tue Oct 13 11:26:11 2015
From: Arne.Morten.Kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 16:26:11 +0000
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <87d1wi4uee.fsf@jedbrown.org>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>,
	<87d1wi4uee.fsf@jedbrown.org>
Message-ID: <7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>

Yes that is the file. Cmake is beautiful for configuration, python is a mess - all is in the eye of the beholder. I have my answer. I will patch as long as the file exists and deal with a better way when I have to. The fact that the file is named, and looks like, a cmake config mode file had me confused.

-------- Opprinnelig melding --------
Fra: Jed Brown <jed at jedbrown.org>
Dato: 13.10.2015 18.15 (GMT+01:00)
Til: Satish Balay <balay at mcs.anl.gov>, Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no>
Ko: petsc-users <petsc-users at mcs.anl.gov>
Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS

Satish Balay <balay at mcs.anl.gov> writes:

> As mentioned - PETScConfig.cmake is primarily for building PETSc - and
> not equivalent to .pc file

  def dumpCMakeConfig(self):
    '''
    Writes configuration-specific values to ${PETSC_ARCH}/lib/petsc/conf/PETScConfig.cmake.
    This file is private to PETSc and should not be included by third parties
    (a suitable file can be produced later by CMake, but this is not it).
    '''

Arne, is this the file you are talking about?  It is part of the legacy
cmake build and is used to hold the various ARCH-specific variables when
building PETSc.  (Note that this build does not use CMake for configure
since CMake is disgusting for that purpose and also because we needed to
ensure that it makes all the same decisions as our configure.)  It's not
for users and should not be installed.

If you want a pkgconfig file, just use PETSc.pc.  Or use FindPETSc.cmake.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/e9962be1/attachment.html>

From knepley at gmail.com  Tue Oct 13 11:31:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 13 Oct 2015 11:31:03 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
Message-ID: <CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>

On Tue, Oct 13, 2015 at 11:26 AM, Arne Morten Kvarving <
Arne.Morten.Kvarving at sintef.no> wrote:

> Yes that is the file. Cmake is beautiful for configuration, python is a
> mess - all is in the eye of the beholder. I have my answer. I will patch as
> long as the file exists and deal with a
>

No, some things are actually better than other things. This is why its
engineering and not art appreciation.

  Matt


> better way when I have to. The fact that the file is named, and looks
> like, a cmake config mode file had me confused.
>
> -------- Opprinnelig melding --------
> Fra: Jed Brown <jed at jedbrown.org>
> Dato: 13.10.2015 18.15 (GMT+01:00)
> Til: Satish Balay <balay at mcs.anl.gov>, Arne Morten Kvarving <
> Arne.Morten.Kvarving at sintef.no>
> Ko: petsc-users <petsc-users at mcs.anl.gov>
> Emne: Re: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
>
> Satish Balay <balay at mcs.anl.gov> writes:
>
> > As mentioned - PETScConfig.cmake is primarily for building PETSc - and
> > not equivalent to .pc file
>
>   def dumpCMakeConfig(self):
>     '''
>     Writes configuration-specific values to
> ${PETSC_ARCH}/lib/petsc/conf/PETScConfig.cmake.
>     This file is private to PETSc and should not be included by third
> parties
>     (a suitable file can be produced later by CMake, but this is not it).
>     '''
>
> Arne, is this the file you are talking about?  It is part of the legacy
> cmake build and is used to hold the various ARCH-specific variables when
> building PETSc.  (Note that this build does not use CMake for configure
> since CMake is disgusting for that purpose and also because we needed to
> ensure that it makes all the same decisions as our configure.)  It's not
> for users and should not be installed.
>
> If you want a pkgconfig file, just use PETSc.pc.  Or use FindPETSc.cmake.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/63ccf493/attachment-0001.html>

From balay at mcs.anl.gov  Tue Oct 13 11:31:50 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 11:31:50 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510131041370.3026@asterix>
References: <561D0A81.6020906@sintef.no>,
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
Message-ID: <alpine.LFD.2.20.1510131114030.5892@asterix>

On Tue, 13 Oct 2015, Satish Balay wrote:

> [This mode is deprecated - and not properly tested - and I see its currenty broken]

>>>>>>>>>>>>
Executing: ['/usr/bin/cmake', '--version']
stdout:
cmake version 3.3.2
CMake suite maintained and supported by Kitware (kitware.com/cmake).
              Could not parse CMake version: cmake version 3.3.2
CMake suite maintained and supported by Kitware (kitware.com/cmake)., disabling cmake build option
          CMake configuration was unsuccessful
<<<<<<<<<<<<<

Jed,

If this looks ok - I can push to maint.
[works for me with cmake 2.8.12 and 3.3.2]

Satish

--------
diff --git a/config/cmakeboot.py b/config/cmakeboot.py
index 64accff..5818c14 100755
--- a/config/cmakeboot.py
+++ b/config/cmakeboot.py
@@ -77,7 +77,7 @@ class PETScMaker(script.Script):
 
    output,error,retcode = self.executeShellCommand([self.cmake.cmake, '--version'], checkCommand=noCheck, log=log)
    import re
-   m = re.match(r'cmake version (.+)$', output)
+   m = re.match(r'cmake version (.+)', output)
    if not m:
        self.logPrint('Could not parse CMake version: %s, disabling cmake build option' % output)
        return False

From jed at jedbrown.org  Tue Oct 13 11:37:22 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 13 Oct 2015 10:37:22 -0600
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <7x3ad188e8sn6kkjyoq2kpgy.1444752861647@email.android.com>
References: <7x3ad188e8sn6kkjyoq2kpgy.1444752861647@email.android.com>
Message-ID: <877fmq4tdp.fsf@jedbrown.org>

Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no> writes:

> I can certainly can use the find rule but the fact that it isnt part
> of the petsc distro complicates things more for me than the patch
> does.

PETSC_ARCH/lib/petsc/conf/PETScConfig.cmake is not for you.  It is not
installed (so your code won't work for prefix installs) and we could
delete it from the build tree at any time.  It is a private
implementation detail of PETSc's legacy cmake build, which is not used
any more because gnumake is so much better.

If you want functionality analogous to pkgconfig, use PETSc.pc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/6cc99837/attachment.pgp>

From jed at jedbrown.org  Tue Oct 13 12:05:12 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 13 Oct 2015 11:05:12 -0600
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510131114030.5892@asterix>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<alpine.LFD.2.20.1510131114030.5892@asterix>
Message-ID: <874mhu4s3b.fsf@jedbrown.org>

Satish Balay <balay at mcs.anl.gov> writes:
> If this looks ok - I can push to maint.
> [works for me with cmake 2.8.12 and 3.3.2]

That's fine.  $ only matches at the end of the string unless flags=re.M
is set.

> --------
> diff --git a/config/cmakeboot.py b/config/cmakeboot.py
> index 64accff..5818c14 100755
> --- a/config/cmakeboot.py
> +++ b/config/cmakeboot.py
> @@ -77,7 +77,7 @@ class PETScMaker(script.Script):
>  
>     output,error,retcode = self.executeShellCommand([self.cmake.cmake, '--version'], checkCommand=noCheck, log=log)
>     import re
> -   m = re.match(r'cmake version (.+)$', output)
> +   m = re.match(r'cmake version (.+)', output)
>     if not m:
>         self.logPrint('Could not parse CMake version: %s, disabling cmake build option' % output)
>         return False

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/e2033bfe/attachment.pgp>

From balay at mcs.anl.gov  Tue Oct 13 12:54:22 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 12:54:22 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <874mhu4s3b.fsf@jedbrown.org>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<alpine.LFD.2.20.1510131114030.5892@asterix>
	<874mhu4s3b.fsf@jedbrown.org>
Message-ID: <alpine.LFD.2.20.1510131254160.5892@asterix>

pushed to maint.

thanks,
satish

On Tue, 13 Oct 2015, Jed Brown wrote:

> Satish Balay <balay at mcs.anl.gov> writes:
> > If this looks ok - I can push to maint.
> > [works for me with cmake 2.8.12 and 3.3.2]
> 
> That's fine.  $ only matches at the end of the string unless flags=re.M
> is set.
> 
> > --------
> > diff --git a/config/cmakeboot.py b/config/cmakeboot.py
> > index 64accff..5818c14 100755
> > --- a/config/cmakeboot.py
> > +++ b/config/cmakeboot.py
> > @@ -77,7 +77,7 @@ class PETScMaker(script.Script):
> >  
> >     output,error,retcode = self.executeShellCommand([self.cmake.cmake, '--version'], checkCommand=noCheck, log=log)
> >     import re
> > -   m = re.match(r'cmake version (.+)$', output)
> > +   m = re.match(r'cmake version (.+)', output)
> >     if not m:
> >         self.logPrint('Could not parse CMake version: %s, disabling cmake build option' % output)
> >         return False
> 
> 


From Arne.Morten.Kvarving at sintef.no  Tue Oct 13 13:12:23 2015
From: Arne.Morten.Kvarving at sintef.no (Arne Morten Kvarving)
Date: Tue, 13 Oct 2015 18:12:23 +0000
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>,
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
Message-ID: <12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>

> No, some things are actually better than other things. This is why its engineering and not art appreciation.

let me rephrase that. it depends on your position. i'm sure you have sound, technical reasons for reinventing the wheel. but from a user's point of view that sucks, because now i have to learn how to operate (or more relevant; change) a new wheel. with cmake or automake, it is (well, ideally) learn once, apply everywhere. with custom solutions, you get to spend time learning stuff all over again, stuff which cannot be reapplied elsewhere.

ref the thing here that happened with my misunderstanding of the purpose of that config file. whenever i see a <Package>Config.cmake it is the cmake config mode file for <Package>. it did not occur to me that it could be anything else, because i have never seen it be anything else in any other pieces of software.

not trying to pick a fight. just clarifying what i meant.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/ae91dcf5/attachment.html>

From knepley at gmail.com  Tue Oct 13 13:15:49 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 13 Oct 2015 13:15:49 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
	<12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
Message-ID: <CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>

On Tue, Oct 13, 2015 at 1:12 PM, Arne Morten Kvarving <
Arne.Morten.Kvarving at sintef.no> wrote:

> > No, some things are actually better than other things. This is why its
> engineering and not art appreciation.
>
> let me rephrase that. it depends on your position. i'm sure you have
> sound, technical reasons for reinventing the wheel. but from a user's point
> of view that sucks, because now i have to learn how to operate (or more
> relevant; change) a new wheel. with cmake or automake, it is (well,
> ideally) learn once, apply everywhere. with custom solutions, you get to
> spend time learning stuff all over again, stuff which cannot be reapplied
> elsewhere.
>
> ref the thing here that happened with my misunderstanding of the purpose
> of that config file. whenever i see a <Package>Config.cmake it is the cmake
> config mode file for <Package>. it did not occur to me that it could be
> anything else, because i have never seen it be anything else in any other
> pieces of software.
>

Yes, we need to change the name of that. Its very confusing and no longer
used.

We tried very hard to use CMake, but it was woefully inadequate. We wrote a
long and detailed analysis
of the shortcomings, posted it to the PETSc list, and set it to the
developers. Nothing has been fixed, and
thus we have moved on.

   Matt


> not trying to pick a fight. just clarifying what i meant.
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/b620c90b/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 13 13:25:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 13 Oct 2015 13:25:46 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
	<12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
	<CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>
Message-ID: <B596E0B6-A1B2-4664-AADB-3BD5C569D809@mcs.anl.gov>


> On Oct 13, 2015, at 1:15 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Oct 13, 2015 at 1:12 PM, Arne Morten Kvarving <Arne.Morten.Kvarving at sintef.no> wrote:
> > No, some things are actually better than other things. This is why its engineering and not art appreciation.
> 
> let me rephrase that. it depends on your position. i'm sure you have sound, technical reasons for reinventing the wheel. but from a user's point of view that sucks, because now i have to learn how to operate (or more relevant; change) a new wheel. with cmake or automake, it is (well, ideally) learn once, apply everywhere. with custom solutions, you get to spend time learning stuff all over again, stuff which cannot be reapplied elsewhere. 
> 
> ref the thing here that happened with my misunderstanding of the purpose of that config file. whenever i see a <Package>Config.cmake it is the cmake config mode file for <Package>. it did not occur to me that it could be anything else, because i have never seen it be anything else in any other pieces of software.
> 
> Yes, we need to change the name of that. Its very confusing and no longer used.

   Satish,

    Can you please change the name of this file so it doesn't cause confusion in the future.  Put something in the name to clearly indicate it is not a <Package>Config.cmake for people to use. Perhaps DoNotUseThisInternalToPetscOnlyConfig.cmake


   Thanks

    Barry

> 
> We tried very hard to use CMake, but it was woefully inadequate. We wrote a long and detailed analysis
> of the shortcomings, posted it to the PETSc list, and set it to the developers. Nothing has been fixed, and
> thus we have moved on.
> 
>    Matt
>  
> not trying to pick a fight. just clarifying what i meant.
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From balay at mcs.anl.gov  Tue Oct 13 13:39:10 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 13:39:10 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <B596E0B6-A1B2-4664-AADB-3BD5C569D809@mcs.anl.gov>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
	<12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
	<CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>
	<B596E0B6-A1B2-4664-AADB-3BD5C569D809@mcs.anl.gov>
Message-ID: <alpine.LFD.2.20.1510131336360.5892@asterix>

On Tue, 13 Oct 2015, Barry Smith wrote:

> > ref the thing here that happened with my misunderstanding of the purpose of that config file. whenever i see a <Package>Config.cmake it is the cmake config mode file for <Package>. it did not occur to me that it could be anything else, because i have never seen it be anything else in any other pieces of software.
> > 
> > Yes, we need to change the name of that. Its very confusing and no longer used.
> 
>    Satish,
> 
>     Can you please change the name of this file so it doesn't cause confusion in the future.  Put something in the name to clearly indicate it is not a <Package>Config.cmake for people to use. Perhaps DoNotUseThisInternalToPetscOnlyConfig.cmake

Which do you prefer:

PETScCmakeBuild.conf
or
PETScBuild.cmake

At some point - we might want to completely remove the cmake build related code..

Satish

From knepley at gmail.com  Tue Oct 13 13:42:03 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 13 Oct 2015 13:42:03 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <alpine.LFD.2.20.1510131336360.5892@asterix>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
	<12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
	<CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>
	<B596E0B6-A1B2-4664-AADB-3BD5C569D809@mcs.anl.gov>
	<alpine.LFD.2.20.1510131336360.5892@asterix>
Message-ID: <CAMYG4G=yyzPU3Dh2DVvGkcTS7CC055eOCgO9Gt9W6KJyQqESRQ@mail.gmail.com>

On Tue, Oct 13, 2015 at 1:39 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> On Tue, 13 Oct 2015, Barry Smith wrote:
>
> > > ref the thing here that happened with my misunderstanding of the
> purpose of that config file. whenever i see a <Package>Config.cmake it is
> the cmake config mode file for <Package>. it did not occur to me that it
> could be anything else, because i have never seen it be anything else in
> any other pieces of software.
> > >
> > > Yes, we need to change the name of that. Its very confusing and no
> longer used.
> >
> >    Satish,
> >
> >     Can you please change the name of this file so it doesn't cause
> confusion in the future.  Put something in the name to clearly indicate it
> is not a <Package>Config.cmake for people to use. Perhaps
> DoNotUseThisInternalToPetscOnlyConfig.cmake
>
> Which do you prefer:
>
> PETScCmakeBuild.conf
> or
> PETScBuild.cmake
>

PETScBuildInternal.cmake


> At some point - we might want to completely remove the cmake build related
> code..


I am for that right now. "There can be only one"

   Matt


>
> Satish
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/ce7bf0ea/attachment-0001.html>

From balay at mcs.anl.gov  Tue Oct 13 13:51:09 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 13:51:09 -0500
Subject: [petsc-users] petsc cmake config - BUILD_SHARED_LIBS
In-Reply-To: <CAMYG4G=yyzPU3Dh2DVvGkcTS7CC055eOCgO9Gt9W6KJyQqESRQ@mail.gmail.com>
References: <561D0A81.6020906@sintef.no>
	<alpine.LFD.2.20.1510130937160.3547@asterix>
	<ip4cwcy611gsycogx0566xc0.1444748369227@email.android.com>
	<alpine.LFD.2.20.1510131041370.3026@asterix>
	<87d1wi4uee.fsf@jedbrown.org>
	<7ylwgtp79rrrsp9umrg5rvd5.1444753475045@email.android.com>
	<CAMYG4G=Ru+wcQC0dEtDc7AApc=JNoZPdgFu4TNC63sdzje5jAg@mail.gmail.com>
	<12CC187542689545A118501C0A085553EC7CEF13@SINTEFEXMBX04.sintef.no>
	<CAMYG4GkoeNR2_kYB2RnRJEFg_BE=aAQB6oG77Rkq2io7c=ZFRA@mail.gmail.com>
	<B596E0B6-A1B2-4664-AADB-3BD5C569D809@mcs.anl.gov>
	<alpine.LFD.2.20.1510131336360.5892@asterix>
	<CAMYG4G=yyzPU3Dh2DVvGkcTS7CC055eOCgO9Gt9W6KJyQqESRQ@mail.gmail.com>
Message-ID: <alpine.LFD.2.20.1510131346330.5892@asterix>

On Tue, 13 Oct 2015, Matthew Knepley wrote:

> 
> PETScBuildInternal.cmake

https://bitbucket.org/petsc/petsc/commits/724dfae7c63ee3ea425936afc826da2605377f15

Pushed to master.

thanks,
Satish

From valencia at astro.utoronto.ca  Tue Oct 13 15:14:17 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Tue, 13 Oct 2015 16:14:17 -0400
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <alpine.LFD.2.20.1510081529520.3050@asterix>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
Message-ID: <C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>

Dear Satish and all petsc users
Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
The version we use is old, because apparently is the one that works the best for the third party code.
I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
Thanks for the input.
Diana

_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
> 
> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
> 
> [you should verify the output from the example compile/link commands -
> and the compile/link command for this application.]
> 
> BTW: This is an old version of petsc.
> 
> Satish
> 
> On Thu, 8 Oct 2015, Diana Valencia wrote:
> 
>> Dear PETSc users,
>> 
>> In hopes that one of you can help me solve a compiling problem.
>> 
>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
>> After compiling all the subroutines, I get an error building the code. Here is the stout:
>> 
>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
>> 
>> Thank you in advance for all your help.
>> Diana
>> 
>> 
>> .
>> .
>> .
>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>> ld: cannot find -lpetsc
>> make: *** [stagyy] Error 1
>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>> ld: cannot find -lpetsc
>> make: *** [stagyympi] Error 1
>> make: Target `both' not remade because of errors.
>> 
>> _______________________
>> Diana Valencia
>> Assistant Professor,  Physics & Astrophysics
>> Department of Physical and Environmental Sciences 
>> University of Toronto, Scarborough
>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>> Voice 416 208-2986
>> 
>> 


From knram06 at gmail.com  Tue Oct 13 16:53:35 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Tue, 13 Oct 2015 17:53:35 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
Message-ID: <CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>

Hello Matt,

Ah ok. I had tried specifying -pc_mg_levels earlier, but it asks for
interpolation to be setup. I am currently working on using DMDA from
examples 32 and 34, which sets up the interpolation, using
DMDASetInterpolationType. So this is the recommended way to go forward
right?

Since I supply the sparse matrix format and the rhs values to PETSc, does
it make sense to do a DMDACreate1d instead of 3d? I did a sample
implementation using DMDA 1d and when I call KSPSolve and fetch the
solution, it comes out all zeros, with the solver having converged. I am
trying to understand what I am doing wrong there.

On Mon, Oct 12, 2015 at 10:17 PM, Matthew Knepley <knepley at gmail.com> wrote:

> On Mon, Oct 12, 2015 at 9:11 PM, K. N. Ramachandran <knram06 at gmail.com>
> wrote:
>
>> Hello Matt,
>>
>> Actually I felt the boundary conditions were having a role to play and
>> set all the boundary conditions to Dirichlet. In this case, convergence was
>> almost immediate with the Hypre preconditioner, taking 17 seconds with 3
>> iterations. The MG method took about the same time though.
>>
>> So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to
>> diverge. Please find attached the output for the Hypre run using Dirichlet
>> and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row.
>> Details of the options used before running are in the file. The solver used
>> in all cases is bcgs.
>>
>> Also attached is the MG output for 101^3 grid
>> 1) Dirichlet and Neumann
>> 2) Dirichlet only
>>
>> where it seems to take about the same time.
>>
>
> Notice that you have no levels of MG here. You need to use -pc_mg_levels
> <n>
>
>    Matt
>
>
>>
>>

Thanking You,
Ramachandran K.N.
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/b5775413/attachment.html>

From knepley at gmail.com  Tue Oct 13 16:57:48 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 13 Oct 2015 16:57:48 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
Message-ID: <CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>

On Tue, Oct 13, 2015 at 4:53 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Matt,
>
> Ah ok. I had tried specifying -pc_mg_levels earlier, but it asks for
> interpolation to be setup. I am currently working on using DMDA from
> examples 32 and 34, which sets up the interpolation, using
> DMDASetInterpolationType. So this is the recommended way to go forward
> right?
>
> Since I supply the sparse matrix format and the rhs values to PETSc, does
> it make sense to do a DMDACreate1d instead of 3d? I did a sample
> implementation using DMDA 1d and when I call KSPSolve and fetch the
> solution, it comes out all zeros, with the solver having converged. I am
> trying to understand what I am doing wrong there.
>

My understanding was that you are just solving 3D Poisson with a star
stencil on a regular grid. SNES ex5 solves 2D Poisson with
a star stencil on a regular grid (set lambda = 0.0). Just use that until
you understand the performance, and then change it for 3D
which is only about 10 lines of code.

For this problem, there is no need for the more complicated stuff.

  Thanks,

     Matt


> On Mon, Oct 12, 2015 at 10:17 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>> On Mon, Oct 12, 2015 at 9:11 PM, K. N. Ramachandran <knram06 at gmail.com>
>> wrote:
>>
>>> Hello Matt,
>>>
>>> Actually I felt the boundary conditions were having a role to play and
>>> set all the boundary conditions to Dirichlet. In this case, convergence was
>>> almost immediate with the Hypre preconditioner, taking 17 seconds with 3
>>> iterations. The MG method took about the same time though.
>>>
>>> So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to
>>> diverge. Please find attached the output for the Hypre run using Dirichlet
>>> and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row.
>>> Details of the options used before running are in the file. The solver used
>>> in all cases is bcgs.
>>>
>>> Also attached is the MG output for 101^3 grid
>>> 1) Dirichlet and Neumann
>>> 2) Dirichlet only
>>>
>>> where it seems to take about the same time.
>>>
>>
>> Notice that you have no levels of MG here. You need to use -pc_mg_levels
>> <n>
>>
>>    Matt
>>
>>
>>>
>>>
>
> Thanking You,
> Ramachandran K.N.
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/d8482f28/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 13 16:58:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 13 Oct 2015 16:58:17 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
Message-ID: <39B73FAA-8D9C-43D3-BFD9-1CE1382E68A2@mcs.anl.gov>


   Where is the matrix coming from? If it is from a 1, 2 or 3d structured grid then you can use the DMDA plus PCMG to manage almost everything including the generation of the interpolation. If you are not working with a structured grid then you can us PCGAMG which does algebraic multigrid using the matrix you provide.

  Barry

> On Oct 13, 2015, at 4:53 PM, K. N. Ramachandran <knram06 at gmail.com> wrote:
> 
> Hello Matt,
> 
> Ah ok. I had tried specifying -pc_mg_levels earlier, but it asks for interpolation to be setup. I am currently working on using DMDA from examples 32 and 34, which sets up the interpolation, using DMDASetInterpolationType. So this is the recommended way to go forward right?
> 
> Since I supply the sparse matrix format and the rhs values to PETSc, does it make sense to do a DMDACreate1d instead of 3d? I did a sample implementation using DMDA 1d and when I call KSPSolve and fetch the solution, it comes out all zeros, with the solver having converged. I am trying to understand what I am doing wrong there.
> 
> On Mon, Oct 12, 2015 at 10:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Mon, Oct 12, 2015 at 9:11 PM, K. N. Ramachandran <knram06 at gmail.com> wrote:
> Hello Matt,
> 
> Actually I felt the boundary conditions were having a role to play and set all the boundary conditions to Dirichlet. In this case, convergence was almost immediate with the Hypre preconditioner, taking 17 seconds with 3 iterations. The MG method took about the same time though.
> 
> So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to diverge. Please find attached the output for the Hypre run using Dirichlet and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row. Details of the options used before running are in the file. The solver used in all cases is bcgs.
> 
> Also attached is the MG output for 101^3 grid
> 1) Dirichlet and Neumann
> 2) Dirichlet only
> 
> where it seems to take about the same time.
> 
> Notice that you have no levels of MG here. You need to use -pc_mg_levels <n>
> 
>    Matt
>  
> 
> 
> 
> Thanking You,
> Ramachandran K.N.
> Ph: 814-441-4279


From bsmith at mcs.anl.gov  Tue Oct 13 17:03:25 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 13 Oct 2015 17:03:25 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
Message-ID: <2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>


> On Oct 13, 2015, at 4:57 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Oct 13, 2015 at 4:53 PM, K. N. Ramachandran <knram06 at gmail.com> wrote:
> Hello Matt,
> 
> Ah ok. I had tried specifying -pc_mg_levels earlier, but it asks for interpolation to be setup. I am currently working on using DMDA from examples 32 and 34, which sets up the interpolation, using DMDASetInterpolationType. So this is the recommended way to go forward right?
> 
> Since I supply the sparse matrix format and the rhs values to PETSc, does it make sense to do a DMDACreate1d instead of 3d? I did a sample implementation using DMDA 1d and when I call KSPSolve and fetch the solution, it comes out all zeros, with the solver having converged. I am trying to understand what I am doing wrong there.
> 
> My understanding was that you are just solving 3D Poisson with a star stencil on a regular grid. SNES ex5 solves 2D Poisson with
> a star stencil on a regular grid (set lambda = 0.0). Just use that until you understand the performance, and then change it for 3D
> which is only about 10 lines of code.

   See src/ksp/ksp/examples/tutorials/ex45.c which does exactly the 3d problem.

> 
> For this problem, there is no need for the more complicated stuff.
> 
>   Thanks,
> 
>      Matt
>  
> On Mon, Oct 12, 2015 at 10:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> On Mon, Oct 12, 2015 at 9:11 PM, K. N. Ramachandran <knram06 at gmail.com> wrote:
> Hello Matt,
> 
> Actually I felt the boundary conditions were having a role to play and set all the boundary conditions to Dirichlet. In this case, convergence was almost immediate with the Hypre preconditioner, taking 17 seconds with 3 iterations. The MG method took about the same time though.
> 
> So I reverted to the Dirichlet, Neumann mix of BCs and Hypre starts to diverge. Please find attached the output for the Hypre run using Dirichlet and Neumann for a 21^3 grid (rows), with a max of 7 nonzeros per row. Details of the options used before running are in the file. The solver used in all cases is bcgs.
> 
> Also attached is the MG output for 101^3 grid
> 1) Dirichlet and Neumann
> 2) Dirichlet only
> 
> where it seems to take about the same time.
> 
> Notice that you have no levels of MG here. You need to use -pc_mg_levels <n>
> 
>    Matt
>  
> 
> 
> 
> Thanking You,
> Ramachandran K.N.
> Ph: 814-441-4279
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From balay at mcs.anl.gov  Tue Oct 13 20:57:28 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 20:57:28 -0500
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
Message-ID: <alpine.LFD.2.20.1510132056400.12914@asterix>

"however ex5 in the test examples failed"

Sorry - this message is a bit confusing. Is petsc working fine for
your - or are you still having issues?

Satish

On Tue, 13 Oct 2015, Diana Valencia wrote:

> Dear Satish and all petsc users
> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
> The version we use is old, because apparently is the one that works the best for the third party code.
> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
> Thanks for the input.
> Diana
> 
> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences 
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986
> 
> > On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > 1. Are you able to compile [using petsc makefile] and run PETSc examples?
> > 
> > 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
> > 
> > [you should verify the output from the example compile/link commands -
> > and the compile/link command for this application.]
> > 
> > BTW: This is an old version of petsc.
> > 
> > Satish
> > 
> > On Thu, 8 Oct 2015, Diana Valencia wrote:
> > 
> >> Dear PETSc users,
> >> 
> >> In hopes that one of you can help me solve a compiling problem.
> >> 
> >> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
> >> After compiling all the subroutines, I get an error building the code. Here is the stout:
> >> 
> >> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
> >> 
> >> Thank you in advance for all your help.
> >> Diana
> >> 
> >> 
> >> .
> >> .
> >> .
> >> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
> >> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
> >> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >> ld: cannot find -lpetsc
> >> make: *** [stagyy] Error 1
> >> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >> ld: cannot find -lpetsc
> >> make: *** [stagyympi] Error 1
> >> make: Target `both' not remade because of errors.
> >> 
> >> _______________________
> >> Diana Valencia
> >> Assistant Professor,  Physics & Astrophysics
> >> Department of Physical and Environmental Sciences 
> >> University of Toronto, Scarborough
> >> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >> Voice 416 208-2986
> >> 
> >> 
> 
> 

From valencia at astro.utoronto.ca  Tue Oct 13 21:04:35 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Tue, 13 Oct 2015 22:04:35 -0400
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <alpine.LFD.2.20.1510132056400.12914@asterix>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132056400.12914@asterix>
Message-ID: <F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>

I meant to say that the code is working with PETSC despite the fact that when I tested, petsc did not pass the ex5 test.
cheers
Diana
_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Oct 13, 2015, at 9:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> "however ex5 in the test examples failed"
> 
> Sorry - this message is a bit confusing. Is petsc working fine for
> your - or are you still having issues?
> 
> Satish
> 
> On Tue, 13 Oct 2015, Diana Valencia wrote:
> 
>> Dear Satish and all petsc users
>> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
>> The version we use is old, because apparently is the one that works the best for the third party code.
>> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
>> Thanks for the input.
>> Diana
>> 
>> _______________________
>> Diana Valencia
>> Assistant Professor,  Physics & Astrophysics
>> Department of Physical and Environmental Sciences 
>> University of Toronto, Scarborough
>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>> Voice 416 208-2986
>> 
>>> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
>>> 
>>> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
>>> 
>>> [you should verify the output from the example compile/link commands -
>>> and the compile/link command for this application.]
>>> 
>>> BTW: This is an old version of petsc.
>>> 
>>> Satish
>>> 
>>> On Thu, 8 Oct 2015, Diana Valencia wrote:
>>> 
>>>> Dear PETSc users,
>>>> 
>>>> In hopes that one of you can help me solve a compiling problem.
>>>> 
>>>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
>>>> After compiling all the subroutines, I get an error building the code. Here is the stout:
>>>> 
>>>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
>>>> 
>>>> Thank you in advance for all your help.
>>>> Diana
>>>> 
>>>> 
>>>> .
>>>> .
>>>> .
>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>> ld: cannot find -lpetsc
>>>> make: *** [stagyy] Error 1
>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>> ld: cannot find -lpetsc
>>>> make: *** [stagyympi] Error 1
>>>> make: Target `both' not remade because of errors.
>>>> 
>>>> _______________________
>>>> Diana Valencia
>>>> Assistant Professor,  Physics & Astrophysics
>>>> Department of Physical and Environmental Sciences 
>>>> University of Toronto, Scarborough
>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>> Voice 416 208-2986
>>>> 
>>>> 
>> 
>> 


From balay at mcs.anl.gov  Tue Oct 13 21:17:29 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 21:17:29 -0500
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132056400.12914@asterix>
	<F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>
Message-ID: <alpine.LFD.2.20.1510132114340.12914@asterix>

thats wierd..

perhaps there were warnings in 'make test'? [warnings can be ignored]

alternatively - configure pickedup the wrong mpiexec but you are using
the correct one with your app?

Satish

On Tue, 13 Oct 2015, Diana Valencia wrote:

> I meant to say that the code is working with PETSC despite the fact that when I tested, petsc did not pass the ex5 test.
> cheers
> Diana
> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences 
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986
> 
> > On Oct 13, 2015, at 9:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > "however ex5 in the test examples failed"
> > 
> > Sorry - this message is a bit confusing. Is petsc working fine for
> > your - or are you still having issues?
> > 
> > Satish
> > 
> > On Tue, 13 Oct 2015, Diana Valencia wrote:
> > 
> >> Dear Satish and all petsc users
> >> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
> >> The version we use is old, because apparently is the one that works the best for the third party code.
> >> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
> >> Thanks for the input.
> >> Diana
> >> 
> >> _______________________
> >> Diana Valencia
> >> Assistant Professor,  Physics & Astrophysics
> >> Department of Physical and Environmental Sciences 
> >> University of Toronto, Scarborough
> >> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >> Voice 416 208-2986
> >> 
> >>> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >>> 
> >>> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
> >>> 
> >>> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
> >>> 
> >>> [you should verify the output from the example compile/link commands -
> >>> and the compile/link command for this application.]
> >>> 
> >>> BTW: This is an old version of petsc.
> >>> 
> >>> Satish
> >>> 
> >>> On Thu, 8 Oct 2015, Diana Valencia wrote:
> >>> 
> >>>> Dear PETSc users,
> >>>> 
> >>>> In hopes that one of you can help me solve a compiling problem.
> >>>> 
> >>>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
> >>>> After compiling all the subroutines, I get an error building the code. Here is the stout:
> >>>> 
> >>>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
> >>>> 
> >>>> Thank you in advance for all your help.
> >>>> Diana
> >>>> 
> >>>> 
> >>>> .
> >>>> .
> >>>> .
> >>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
> >>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
> >>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >>>> ld: cannot find -lpetsc
> >>>> make: *** [stagyy] Error 1
> >>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >>>> ld: cannot find -lpetsc
> >>>> make: *** [stagyympi] Error 1
> >>>> make: Target `both' not remade because of errors.
> >>>> 
> >>>> _______________________
> >>>> Diana Valencia
> >>>> Assistant Professor,  Physics & Astrophysics
> >>>> Department of Physical and Environmental Sciences 
> >>>> University of Toronto, Scarborough
> >>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >>>> Voice 416 208-2986
> >>>> 
> >>>> 
> >> 
> >> 
> 
> 

From valencia at astro.utoronto.ca  Tue Oct 13 21:21:13 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Tue, 13 Oct 2015 22:21:13 -0400
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <alpine.LFD.2.20.1510132114340.12914@asterix>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132056400.12914@asterix>
	<F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132114340.12914@asterix>
Message-ID: <BEF7928D-DADA-46EA-978A-5527559E152A@astro.utoronto.ca>

Here is the error message:

[valencia at euler01 petsc-3.2-p7]$ make test
Running test examples to verify correct installation
Using PETSC_DIR=/cluster/home/valencia/PETSC/petsc-3.2-p7 and PETSC_ARCH=arch-linux2-c-opt
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
--------------Error detected during compile or link!-----------------------
See http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html
mpif90 -c  -O3   -I/cluster/home/valencia/PETSC/petsc-3.2-p7/include -I/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include    -o ex5f.o ex5f.F
./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
      DM      da
----------------^
ex5f.F(56): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
      SNES                   snes
---------------------------------^
ex5f.F(57): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
      Vec                    x,r
------------------------------^
ex5f.F(58): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
      Mat                    J,A
------------------------------^
./ex5f.h(60): error #6218: This statement is positioned incorrectly and/or has syntax errors.
      DM      da
----------------^
ex5f.F(59): error #6236: A specification statement cannot appear in the executable section.
      integer(kind=selected_int_kind(5))                its,i1,i4
------^
ex5f.F(60): error #6236: A specification statement cannot appear in the executable section.
      integer(kind=selected_int_kind(5)) ierr
------^
ex5f.F(61): error #6236: A specification statement cannot appear in the executable section.
      real(kind=selected_real_kind(10))       lambda_max,lambda_min
------^
ex5f.F(62): error #6236: A specification statement cannot appear in the executable section.
      logical(kind=4)              flg
------^
ex5f.F(71): error #6236: A specification statement cannot appear in the executable section.
      external FormInitialGuess
------^
ex5f.F(72): error #6236: A specification statement cannot appear in the executable section.
      external FormFunctionLocal,FormJacobianLocal
------^
ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [VECX]
      Vec                    x,r
------^
ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [R]
      Vec                    x,r
-------------------------------^
ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [MATJ]
      Mat                    J,A
------^
ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [A]
      Mat                    J,A
-------------------------------^
ex5f.F(81): error #6404: This name does not have a type, and must have an explicit type.   [IERR]
      call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
------------------------------------------------^
ex5f.F(87): error #6404: This name does not have a type, and must have an explicit type.   [I1]
      i1 = 1
------^
ex5f.F(88): error #6404: This name does not have a type, and must have an explicit type.   [I4]
      i4 = -4
------^
ex5f.F(89): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MAX]
      lambda_max = 6.81
------^
ex5f.F(90): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MIN]
      lambda_min = 0.0
------^
ex5f.F(93): error #6404: This name does not have a type, and must have an explicit type.   [FLG]
     &                           flg,ierr)
---------------------------------^
ex5f.F(103): error #6404: This name does not have a type, and must have an explicit type.   [SNES]
      call SNESCreate(PETSC_COMM_WORLD,snes,ierr)
---------------------------------------^
ex5f.F(116): error #6404: This name does not have a type, and must have an explicit type.   [DA]
     &     PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
-------------------------------------------------^
ex5f.F(121): error #6404: This name does not have a type, and must have an explicit type.   [X]
      call DMCreateGlobalVector(da,x,ierr)
-----------------------------------^
ex5f.F(152): error #6404: This name does not have a type, and must have an explicit type.   [FORMFUNCTIONLOCAL]
      call DMDASetLocalFunction(da,FormFunctionLocal,ierr)
-----------------------------------^
ex5f.F(153): error #6404: This name does not have a type, and must have an explicit type.   [FORMJACOBIANLOCAL]
      call DMDASetLocalJacobian(da,FormJacobianLocal,ierr)
-----------------------------------^
ex5f.F(173): error #6404: This name does not have a type, and must have an explicit type.   [J]
      call DMGetMatrix(da,'aij',J,ierr)
--------------------------------^
ex5f.F(226): error #6404: This name does not have a type, and must have an explicit type.   [ITS]
      call SNESGetIterationNumber(snes,its,ierr)
---------------------------------------^
./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
      DM      da
----------------^
ex5f.F(271): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
      Vec      X
----------------^
/tmp/ifortGXVttY.i(6051): catastrophic error: Too many errors, exiting
compilation aborted for ex5f.F (code 1)
make[3]: [ex5f.o] Error 1 (ignored)
mpif90 -O3   -o ex5f ex5f.o -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
ifort: error #10236: File not found:  'ex5f.o'
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
make[3]: [ex5f] Error 1 (ignored)
/bin/rm -f ex5f.o
Completed test examples

_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Oct 13, 2015, at 10:17 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> thats wierd..
> 
> perhaps there were warnings in 'make test'? [warnings can be ignored]
> 
> alternatively - configure pickedup the wrong mpiexec but you are using
> the correct one with your app?
> 
> Satish
> 
> On Tue, 13 Oct 2015, Diana Valencia wrote:
> 
>> I meant to say that the code is working with PETSC despite the fact that when I tested, petsc did not pass the ex5 test.
>> cheers
>> Diana
>> _______________________
>> Diana Valencia
>> Assistant Professor,  Physics & Astrophysics
>> Department of Physical and Environmental Sciences 
>> University of Toronto, Scarborough
>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>> Voice 416 208-2986
>> 
>>> On Oct 13, 2015, at 9:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> "however ex5 in the test examples failed"
>>> 
>>> Sorry - this message is a bit confusing. Is petsc working fine for
>>> your - or are you still having issues?
>>> 
>>> Satish
>>> 
>>> On Tue, 13 Oct 2015, Diana Valencia wrote:
>>> 
>>>> Dear Satish and all petsc users
>>>> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
>>>> The version we use is old, because apparently is the one that works the best for the third party code.
>>>> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
>>>> Thanks for the input.
>>>> Diana
>>>> 
>>>> _______________________
>>>> Diana Valencia
>>>> Assistant Professor,  Physics & Astrophysics
>>>> Department of Physical and Environmental Sciences 
>>>> University of Toronto, Scarborough
>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>> Voice 416 208-2986
>>>> 
>>>>> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>> 
>>>>> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
>>>>> 
>>>>> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
>>>>> 
>>>>> [you should verify the output from the example compile/link commands -
>>>>> and the compile/link command for this application.]
>>>>> 
>>>>> BTW: This is an old version of petsc.
>>>>> 
>>>>> Satish
>>>>> 
>>>>> On Thu, 8 Oct 2015, Diana Valencia wrote:
>>>>> 
>>>>>> Dear PETSc users,
>>>>>> 
>>>>>> In hopes that one of you can help me solve a compiling problem.
>>>>>> 
>>>>>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
>>>>>> After compiling all the subroutines, I get an error building the code. Here is the stout:
>>>>>> 
>>>>>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
>>>>>> 
>>>>>> Thank you in advance for all your help.
>>>>>> Diana
>>>>>> 
>>>>>> 
>>>>>> .
>>>>>> .
>>>>>> .
>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>>>> ld: cannot find -lpetsc
>>>>>> make: *** [stagyy] Error 1
>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>>>> ld: cannot find -lpetsc
>>>>>> make: *** [stagyympi] Error 1
>>>>>> make: Target `both' not remade because of errors.
>>>>>> 
>>>>>> _______________________
>>>>>> Diana Valencia
>>>>>> Assistant Professor,  Physics & Astrophysics
>>>>>> Department of Physical and Environmental Sciences 
>>>>>> University of Toronto, Scarborough
>>>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>>>> Voice 416 208-2986
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 
>> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/80ed2c4c/attachment-0001.html>

From timothee.nicolas at gmail.com  Tue Oct 13 21:23:17 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 14 Oct 2015 11:23:17 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient ??
Message-ID: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>

Dear all,

I have been playing around with multigrid recently, namely with
/ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
with my own implementation of a laplacian type problem. In all cases, I
have noted no improvement whatsoever in the performance, whether in CPU
time or KSP iteration, by varying the number of levels of the multigrid
solver. As an example, I have attached the log_summary for ex5.c with
nlevels = 2 to 7, launched by

mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine
6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary

where -pc_mg_levels is set to a number between 2 and 7.

So there is a noticeable CPU time improvement from 2 levels to 3 levels
(30%), and then no improvement whatsoever. I am surprised because with 6
levels of refinement of the DMDA the fine grid has more than 1200 points so
with 3 levels the coarse grid still has more than 300 points which is still
pretty large (I assume the ratio between grids is 2). I am wondering how
the coarse solver efficiently solves the problem on the coarse grid with
such a large number of points ? Given the principle of multigrid which is
to erase the smooth part of the error with relaxation methods, which are
usually efficient only for high frequency, I would expect optimal
performance when the coarse grid is basically just a few points in each
direction. Does anyone know why the performance saturates at low number of
levels ? Basically what happens internally seems to be quite different from
what I would expect...

Best

Timothee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_2_levels_of_multigrid.log
Type: application/octet-stream
Size: 13619 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_levels_of_multigrid.log
Type: application/octet-stream
Size: 13616 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_levels_of_multigrid.log
Type: application/octet-stream
Size: 13616 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_levels_of_multigrid.log
Type: application/octet-stream
Size: 13619 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_levels_of_multigrid.log
Type: application/octet-stream
Size: 13616 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_levels_of_multigrid.log
Type: application/octet-stream
Size: 13619 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/46f64bb2/attachment-0011.obj>

From balay at mcs.anl.gov  Tue Oct 13 21:35:28 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Tue, 13 Oct 2015 21:35:28 -0500
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <BEF7928D-DADA-46EA-978A-5527559E152A@astro.utoronto.ca>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132056400.12914@asterix>
	<F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132114340.12914@asterix>
	<BEF7928D-DADA-46EA-978A-5527559E152A@astro.utoronto.ca>
Message-ID: <alpine.LFD.2.20.1510132131260.12914@asterix>


The c examples ran fine - so the libraries are good.

Did you configure PETSc with --with-fortran-datatypes=1

[this is one reason why this compile will fail - esp for old version].

Alternatively - it wold be fixed form vs free form isssue [renaming
the file to ex5.F90 would get the compile going]

Since the libraries are fine - and your app works - you cold ignore this issue.

Satish

On Tue, 13 Oct 2015, Diana Valencia wrote:

> Here is the error message:
> 
> [valencia at euler01 petsc-3.2-p7]$ make test
> Running test examples to verify correct installation
> Using PETSC_DIR=/cluster/home/valencia/PETSC/petsc-3.2-p7 and PETSC_ARCH=arch-linux2-c-opt
> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
> --------------Error detected during compile or link!-----------------------
> See http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html
> mpif90 -c  -O3   -I/cluster/home/valencia/PETSC/petsc-3.2-p7/include -I/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include    -o ex5f.o ex5f.F
> ./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>       DM      da
> ----------------^
> ex5f.F(56): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>       SNES                   snes
> ---------------------------------^
> ex5f.F(57): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
>       Vec                    x,r
> ------------------------------^
> ex5f.F(58): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
>       Mat                    J,A
> ------------------------------^
> ./ex5f.h(60): error #6218: This statement is positioned incorrectly and/or has syntax errors.
>       DM      da
> ----------------^
> ex5f.F(59): error #6236: A specification statement cannot appear in the executable section.
>       integer(kind=selected_int_kind(5))                its,i1,i4
> ------^
> ex5f.F(60): error #6236: A specification statement cannot appear in the executable section.
>       integer(kind=selected_int_kind(5)) ierr
> ------^
> ex5f.F(61): error #6236: A specification statement cannot appear in the executable section.
>       real(kind=selected_real_kind(10))       lambda_max,lambda_min
> ------^
> ex5f.F(62): error #6236: A specification statement cannot appear in the executable section.
>       logical(kind=4)              flg
> ------^
> ex5f.F(71): error #6236: A specification statement cannot appear in the executable section.
>       external FormInitialGuess
> ------^
> ex5f.F(72): error #6236: A specification statement cannot appear in the executable section.
>       external FormFunctionLocal,FormJacobianLocal
> ------^
> ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [VECX]
>       Vec                    x,r
> ------^
> ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [R]
>       Vec                    x,r
> -------------------------------^
> ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [MATJ]
>       Mat                    J,A
> ------^
> ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [A]
>       Mat                    J,A
> -------------------------------^
> ex5f.F(81): error #6404: This name does not have a type, and must have an explicit type.   [IERR]
>       call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
> ------------------------------------------------^
> ex5f.F(87): error #6404: This name does not have a type, and must have an explicit type.   [I1]
>       i1 = 1
> ------^
> ex5f.F(88): error #6404: This name does not have a type, and must have an explicit type.   [I4]
>       i4 = -4
> ------^
> ex5f.F(89): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MAX]
>       lambda_max = 6.81
> ------^
> ex5f.F(90): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MIN]
>       lambda_min = 0.0
> ------^
> ex5f.F(93): error #6404: This name does not have a type, and must have an explicit type.   [FLG]
>      &                           flg,ierr)
> ---------------------------------^
> ex5f.F(103): error #6404: This name does not have a type, and must have an explicit type.   [SNES]
>       call SNESCreate(PETSC_COMM_WORLD,snes,ierr)
> ---------------------------------------^
> ex5f.F(116): error #6404: This name does not have a type, and must have an explicit type.   [DA]
>      &     PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
> -------------------------------------------------^
> ex5f.F(121): error #6404: This name does not have a type, and must have an explicit type.   [X]
>       call DMCreateGlobalVector(da,x,ierr)
> -----------------------------------^
> ex5f.F(152): error #6404: This name does not have a type, and must have an explicit type.   [FORMFUNCTIONLOCAL]
>       call DMDASetLocalFunction(da,FormFunctionLocal,ierr)
> -----------------------------------^
> ex5f.F(153): error #6404: This name does not have a type, and must have an explicit type.   [FORMJACOBIANLOCAL]
>       call DMDASetLocalJacobian(da,FormJacobianLocal,ierr)
> -----------------------------------^
> ex5f.F(173): error #6404: This name does not have a type, and must have an explicit type.   [J]
>       call DMGetMatrix(da,'aij',J,ierr)
> --------------------------------^
> ex5f.F(226): error #6404: This name does not have a type, and must have an explicit type.   [ITS]
>       call SNESGetIterationNumber(snes,its,ierr)
> ---------------------------------------^
> ./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>       DM      da
> ----------------^
> ex5f.F(271): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>       Vec      X
> ----------------^
> /tmp/ifortGXVttY.i(6051): catastrophic error: Too many errors, exiting
> compilation aborted for ex5f.F (code 1)
> make[3]: [ex5f.o] Error 1 (ignored)
> mpif90 -O3   -o ex5f ex5f.o -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> ifort: error #10236: File not found:  'ex5f.o'
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
> make[3]: [ex5f] Error 1 (ignored)
> /bin/rm -f ex5f.o
> Completed test examples
> 
> _______________________
> Diana Valencia
> Assistant Professor,  Physics & Astrophysics
> Department of Physical and Environmental Sciences 
> University of Toronto, Scarborough
> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> Voice 416 208-2986
> 
> > On Oct 13, 2015, at 10:17 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> > 
> > thats wierd..
> > 
> > perhaps there were warnings in 'make test'? [warnings can be ignored]
> > 
> > alternatively - configure pickedup the wrong mpiexec but you are using
> > the correct one with your app?
> > 
> > Satish
> > 
> > On Tue, 13 Oct 2015, Diana Valencia wrote:
> > 
> >> I meant to say that the code is working with PETSC despite the fact that when I tested, petsc did not pass the ex5 test.
> >> cheers
> >> Diana
> >> _______________________
> >> Diana Valencia
> >> Assistant Professor,  Physics & Astrophysics
> >> Department of Physical and Environmental Sciences 
> >> University of Toronto, Scarborough
> >> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >> Voice 416 208-2986
> >> 
> >>> On Oct 13, 2015, at 9:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >>> 
> >>> "however ex5 in the test examples failed"
> >>> 
> >>> Sorry - this message is a bit confusing. Is petsc working fine for
> >>> your - or are you still having issues?
> >>> 
> >>> Satish
> >>> 
> >>> On Tue, 13 Oct 2015, Diana Valencia wrote:
> >>> 
> >>>> Dear Satish and all petsc users
> >>>> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
> >>>> The version we use is old, because apparently is the one that works the best for the third party code.
> >>>> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
> >>>> Thanks for the input.
> >>>> Diana
> >>>> 
> >>>> _______________________
> >>>> Diana Valencia
> >>>> Assistant Professor,  Physics & Astrophysics
> >>>> Department of Physical and Environmental Sciences 
> >>>> University of Toronto, Scarborough
> >>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >>>> Voice 416 208-2986
> >>>> 
> >>>>> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> >>>>> 
> >>>>> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
> >>>>> 
> >>>>> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
> >>>>> 
> >>>>> [you should verify the output from the example compile/link commands -
> >>>>> and the compile/link command for this application.]
> >>>>> 
> >>>>> BTW: This is an old version of petsc.
> >>>>> 
> >>>>> Satish
> >>>>> 
> >>>>> On Thu, 8 Oct 2015, Diana Valencia wrote:
> >>>>> 
> >>>>>> Dear PETSc users,
> >>>>>> 
> >>>>>> In hopes that one of you can help me solve a compiling problem.
> >>>>>> 
> >>>>>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
> >>>>>> After compiling all the subroutines, I get an error building the code. Here is the stout:
> >>>>>> 
> >>>>>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
> >>>>>> 
> >>>>>> Thank you in advance for all your help.
> >>>>>> Diana
> >>>>>> 
> >>>>>> 
> >>>>>> .
> >>>>>> .
> >>>>>> .
> >>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
> >>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
> >>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >>>>>> ld: cannot find -lpetsc
> >>>>>> make: *** [stagyy] Error 1
> >>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
> >>>>>> ld: cannot find -lpetsc
> >>>>>> make: *** [stagyympi] Error 1
> >>>>>> make: Target `both' not remade because of errors.
> >>>>>> 
> >>>>>> _______________________
> >>>>>> Diana Valencia
> >>>>>> Assistant Professor,  Physics & Astrophysics
> >>>>>> Department of Physical and Environmental Sciences 
> >>>>>> University of Toronto, Scarborough
> >>>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
> >>>>>> Voice 416 208-2986
> >>>>>> 
> >>>>>> 
> >>>> 
> >>>> 
> >> 
> >> 
> 
> 

From valencia at astro.utoronto.ca  Tue Oct 13 21:36:42 2015
From: valencia at astro.utoronto.ca (Diana Valencia)
Date: Tue, 13 Oct 2015 22:36:42 -0400
Subject: [petsc-users] ld: cannot find -lpetsc error
In-Reply-To: <alpine.LFD.2.20.1510132131260.12914@asterix>
References: <3C05C428-5DB5-4546-8DD9-061AE9297928@astro.utoronto.ca>
	<alpine.LFD.2.20.1510081529520.3050@asterix>
	<C04927DE-217C-481F-B0A4-DBBC032F0B1D@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132056400.12914@asterix>
	<F85D2987-2D0E-4A1D-858E-DE77F013059A@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132114340.12914@asterix>
	<BEF7928D-DADA-46EA-978A-5527559E152A@astro.utoronto.ca>
	<alpine.LFD.2.20.1510132131260.12914@asterix>
Message-ID: <DC1165C2-CBD0-46A6-9B3D-80D9CD783D29@astro.utoronto.ca>

I think for now, I?ll ignore it. Thank you for all your help
Diana
_______________________
Diana Valencia
Assistant Professor,  Physics & Astrophysics
Department of Physical and Environmental Sciences 
University of Toronto, Scarborough
1265 Military Trail, Toronto, ON, Canada, M1C 1A4
Voice 416 208-2986

> On Oct 13, 2015, at 10:35 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> 
> The c examples ran fine - so the libraries are good.
> 
> Did you configure PETSc with --with-fortran-datatypes=1
> 
> [this is one reason why this compile will fail - esp for old version].
> 
> Alternatively - it wold be fixed form vs free form isssue [renaming
> the file to ex5.F90 would get the compile going]
> 
> Since the libraries are fine - and your app works - you cold ignore this issue.
> 
> Satish
> 
> On Tue, 13 Oct 2015, Diana Valencia wrote:
> 
>> Here is the error message:
>> 
>> [valencia at euler01 petsc-3.2-p7]$ make test
>> Running test examples to verify correct installation
>> Using PETSC_DIR=/cluster/home/valencia/PETSC/petsc-3.2-p7 and PETSC_ARCH=arch-linux2-c-opt
>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 1 MPI process
>> C/C++ example src/snes/examples/tutorials/ex19 run successfully with 2 MPI processes
>> --------------Error detected during compile or link!-----------------------
>> See http://www.mcs.anl.gov/petsc/petsc-2/documentation/faq.html
>> mpif90 -c  -O3   -I/cluster/home/valencia/PETSC/petsc-3.2-p7/include -I/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include    -o ex5f.o ex5f.F
>> ./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>>      DM      da
>> ----------------^
>> ex5f.F(56): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>>      SNES                   snes
>> ---------------------------------^
>> ex5f.F(57): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
>>      Vec                    x,r
>> ------------------------------^
>> ex5f.F(58): error #5082: Syntax error, found ',' when expecting one of: ( % [ : . = =>
>>      Mat                    J,A
>> ------------------------------^
>> ./ex5f.h(60): error #6218: This statement is positioned incorrectly and/or has syntax errors.
>>      DM      da
>> ----------------^
>> ex5f.F(59): error #6236: A specification statement cannot appear in the executable section.
>>      integer(kind=selected_int_kind(5))                its,i1,i4
>> ------^
>> ex5f.F(60): error #6236: A specification statement cannot appear in the executable section.
>>      integer(kind=selected_int_kind(5)) ierr
>> ------^
>> ex5f.F(61): error #6236: A specification statement cannot appear in the executable section.
>>      real(kind=selected_real_kind(10))       lambda_max,lambda_min
>> ------^
>> ex5f.F(62): error #6236: A specification statement cannot appear in the executable section.
>>      logical(kind=4)              flg
>> ------^
>> ex5f.F(71): error #6236: A specification statement cannot appear in the executable section.
>>      external FormInitialGuess
>> ------^
>> ex5f.F(72): error #6236: A specification statement cannot appear in the executable section.
>>      external FormFunctionLocal,FormJacobianLocal
>> ------^
>> ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [VECX]
>>      Vec                    x,r
>> ------^
>> ex5f.F(57): error #6404: This name does not have a type, and must have an explicit type.   [R]
>>      Vec                    x,r
>> -------------------------------^
>> ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [MATJ]
>>      Mat                    J,A
>> ------^
>> ex5f.F(58): error #6404: This name does not have a type, and must have an explicit type.   [A]
>>      Mat                    J,A
>> -------------------------------^
>> ex5f.F(81): error #6404: This name does not have a type, and must have an explicit type.   [IERR]
>>      call PetscInitialize(PETSC_NULL_CHARACTER,ierr)
>> ------------------------------------------------^
>> ex5f.F(87): error #6404: This name does not have a type, and must have an explicit type.   [I1]
>>      i1 = 1
>> ------^
>> ex5f.F(88): error #6404: This name does not have a type, and must have an explicit type.   [I4]
>>      i4 = -4
>> ------^
>> ex5f.F(89): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MAX]
>>      lambda_max = 6.81
>> ------^
>> ex5f.F(90): error #6404: This name does not have a type, and must have an explicit type.   [LAMBDA_MIN]
>>      lambda_min = 0.0
>> ------^
>> ex5f.F(93): error #6404: This name does not have a type, and must have an explicit type.   [FLG]
>>     &                           flg,ierr)
>> ---------------------------------^
>> ex5f.F(103): error #6404: This name does not have a type, and must have an explicit type.   [SNES]
>>      call SNESCreate(PETSC_COMM_WORLD,snes,ierr)
>> ---------------------------------------^
>> ex5f.F(116): error #6404: This name does not have a type, and must have an explicit type.   [DA]
>>     &     PETSC_NULL_INTEGER,PETSC_NULL_INTEGER,da,ierr)
>> -------------------------------------------------^
>> ex5f.F(121): error #6404: This name does not have a type, and must have an explicit type.   [X]
>>      call DMCreateGlobalVector(da,x,ierr)
>> -----------------------------------^
>> ex5f.F(152): error #6404: This name does not have a type, and must have an explicit type.   [FORMFUNCTIONLOCAL]
>>      call DMDASetLocalFunction(da,FormFunctionLocal,ierr)
>> -----------------------------------^
>> ex5f.F(153): error #6404: This name does not have a type, and must have an explicit type.   [FORMJACOBIANLOCAL]
>>      call DMDASetLocalJacobian(da,FormJacobianLocal,ierr)
>> -----------------------------------^
>> ex5f.F(173): error #6404: This name does not have a type, and must have an explicit type.   [J]
>>      call DMGetMatrix(da,'aij',J,ierr)
>> --------------------------------^
>> ex5f.F(226): error #6404: This name does not have a type, and must have an explicit type.   [ITS]
>>      call SNESGetIterationNumber(snes,its,ierr)
>> ---------------------------------------^
>> ./ex5f.h(60): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>>      DM      da
>> ----------------^
>> ex5f.F(271): error #5082: Syntax error, found END-OF-STATEMENT when expecting one of: ( % [ : . = =>
>>      Vec      X
>> ----------------^
>> /tmp/ifortGXVttY.i(6051): catastrophic error: Too many errors, exiting
>> compilation aborted for ex5f.F (code 1)
>> make[3]: [ex5f.o] Error 1 (ignored)
>> mpif90 -O3   -o ex5f ex5f.o -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>> ifort: error #10236: File not found:  'ex5f.o'
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
>> make[3]: [ex5f] Error 1 (ignored)
>> /bin/rm -f ex5f.o
>> Completed test examples
>> 
>> _______________________
>> Diana Valencia
>> Assistant Professor,  Physics & Astrophysics
>> Department of Physical and Environmental Sciences 
>> University of Toronto, Scarborough
>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>> Voice 416 208-2986
>> 
>>> On Oct 13, 2015, at 10:17 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> thats wierd..
>>> 
>>> perhaps there were warnings in 'make test'? [warnings can be ignored]
>>> 
>>> alternatively - configure pickedup the wrong mpiexec but you are using
>>> the correct one with your app?
>>> 
>>> Satish
>>> 
>>> On Tue, 13 Oct 2015, Diana Valencia wrote:
>>> 
>>>> I meant to say that the code is working with PETSC despite the fact that when I tested, petsc did not pass the ex5 test.
>>>> cheers
>>>> Diana
>>>> _______________________
>>>> Diana Valencia
>>>> Assistant Professor,  Physics & Astrophysics
>>>> Department of Physical and Environmental Sciences 
>>>> University of Toronto, Scarborough
>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>> Voice 416 208-2986
>>>> 
>>>>> On Oct 13, 2015, at 9:57 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>> 
>>>>> "however ex5 in the test examples failed"
>>>>> 
>>>>> Sorry - this message is a bit confusing. Is petsc working fine for
>>>>> your - or are you still having issues?
>>>>> 
>>>>> Satish
>>>>> 
>>>>> On Tue, 13 Oct 2015, Diana Valencia wrote:
>>>>> 
>>>>>> Dear Satish and all petsc users
>>>>>> Thank you for your email. Antoine Rozel also from the petsc community was able to help me (he also uses the same third party code I was trying to compile)
>>>>>> The version we use is old, because apparently is the one that works the best for the third party code.
>>>>>> I had to reinstall petsc, and somehow this fixed it.  The code compiled and is now running fine, however ex5 in the test examples failed.
>>>>>> Thanks for the input.
>>>>>> Diana
>>>>>> 
>>>>>> _______________________
>>>>>> Diana Valencia
>>>>>> Assistant Professor,  Physics & Astrophysics
>>>>>> Department of Physical and Environmental Sciences 
>>>>>> University of Toronto, Scarborough
>>>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>>>> Voice 416 208-2986
>>>>>> 
>>>>>>> On Oct 8, 2015, at 4:31 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> 1. Are you able to compile [using petsc makefile] and run PETSc examples?
>>>>>>> 
>>>>>>> 2. You are missing the link option -L/cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/lib
>>>>>>> 
>>>>>>> [you should verify the output from the example compile/link commands -
>>>>>>> and the compile/link command for this application.]
>>>>>>> 
>>>>>>> BTW: This is an old version of petsc.
>>>>>>> 
>>>>>>> Satish
>>>>>>> 
>>>>>>> On Thu, 8 Oct 2015, Diana Valencia wrote:
>>>>>>> 
>>>>>>>> Dear PETSc users,
>>>>>>>> 
>>>>>>>> In hopes that one of you can help me solve a compiling problem.
>>>>>>>> 
>>>>>>>> I am compiling a third party code (stagyy and stagyympi) on a cluster. I load the intel and open_mpi libraries to the cluster before compiling with petsc.
>>>>>>>> After compiling all the subroutines, I get an error building the code. Here is the stout:
>>>>>>>> 
>>>>>>>> I have set the PETSC_DIR and PETSC_ARCH in the .bashrc file, so I don?t think this is the problem.
>>>>>>>> 
>>>>>>>> Thank you in advance for all your help.
>>>>>>>> Diana
>>>>>>>> 
>>>>>>>> 
>>>>>>>> .
>>>>>>>> .
>>>>>>>> .
>>>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c main.f90
>>>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -c advdif.f90
>>>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyy main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_dummy.o  PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>>>>>> ld: cannot find -lpetsc
>>>>>>>> make: *** [stagyy] Error 1
>>>>>>>> mpif90  -w -O2 -assume byterecl -r8 -I /cluster/home/valencia/PETSC/petsc-3.2-p7/arch-linux2-c-opt/include -o stagyympi main.o advdif.o communication.o continents.o evocore.o geometry.o init.o io.o iter.o melting.o miscellaneous.o MPDATA.o multi.o refstate.o phase_change.o poisson.o prolongate_restrict.o relax_all.o relax_cell.o relax_point.o residues.o tracers.o v_advection.o vee_cycle.o viscosity.o  crw.o geoid.o implicit_diffusion.o linpack.o plot.o wrtpng.o blas123.o lapack.o nxs_to_mpi.o PETScimple_mod.o xdmf_hdf5_io_dummy.o bcs_mod.o composition_mod.o control_mod.o continents_mod.o vgrid_mod.o init_mod.o io_mod.o iter_mod.o melting_mod.o meltingfns_mod.o multi_mod.o phase_change_mod.o plot_mod.o precision_mod.o refstat_mod.o stencil_mod.o timestep_mod.o timing_mod.o tracers_mod.o viscosity_mod.o thermochem_mod.o platesLea_mod.o -lpng     -L/lib  -lpetsc -lX11 -lpthread -Wl,-rpath,/lib -lcmumps -ldmumps -lsmumps -lzmumps -lmumps_common -lpord -lparmetis -lmetis -lscalapack -lblacs -lsuperlu_4.2 -lml -lmpi_cxx -lumfpack -lamd -lflapack -lfblas -ldl -L/cluster/apps/openmpi/1.6.5/x86_64/intel_15.0.0/lib -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/compiler/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/ipp/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/mkl/lib/intel64 -L/cluster/apps/intel/composer_xe_2015.0.090/composer_xe_2015.0.090/tbb/lib/intel64 -L/cluster/apps/gcc/4.8.2/lib/gcc/x86_64-unknown-linux-gnu/4.8.2 -L/cluster/apps/gcc/4.8.2/lib64 -L/cluster/apps/gcc/4.8.2/lib -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -lmpi_f90 -lmpi_f77 -lm -lm -lifport -lifcore -lm -lm -lm -lmpi_cxx -ldl -lmpi -lopen-rte -lopen-pal -lnuma -lrt -lnsl -lutil -limf -lsvml -lirng -lipgo -ldecimal -lcilkrts -lstdc++ -lgcc_s -lirc -lpthread -lirc_s -ldl 
>>>>>>>> ld: cannot find -lpetsc
>>>>>>>> make: *** [stagyympi] Error 1
>>>>>>>> make: Target `both' not remade because of errors.
>>>>>>>> 
>>>>>>>> _______________________
>>>>>>>> Diana Valencia
>>>>>>>> Assistant Professor,  Physics & Astrophysics
>>>>>>>> Department of Physical and Environmental Sciences 
>>>>>>>> University of Toronto, Scarborough
>>>>>>>> 1265 Military Trail, Toronto, ON, Canada, M1C 1A4
>>>>>>>> Voice 416 208-2986
>>>>>>>> 
>>>>>>>> 
>>>>>> 
>>>>>> 
>>>> 
>>>> 
>> 


From orxan.shibli at gmail.com  Tue Oct 13 22:28:20 2015
From: orxan.shibli at gmail.com (Orxan Shibliyev)
Date: Tue, 13 Oct 2015 21:28:20 -0600
Subject: [petsc-users] Solving only for certain rows
Message-ID: <CAOYX9gwNLghJjeGntraNmcEo2NYC_Lfi8=Y4quv5BdVw0OM6nw@mail.gmail.com>

I used KSP to solve Ax=b successfully but I would like to solve Ax=b only
for certain rows. How can I do this?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/3a85d411/attachment.html>

From jed at jedbrown.org  Tue Oct 13 22:42:43 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 13 Oct 2015 21:42:43 -0600
Subject: [petsc-users] Solving only for certain rows
In-Reply-To: <CAOYX9gwNLghJjeGntraNmcEo2NYC_Lfi8=Y4quv5BdVw0OM6nw@mail.gmail.com>
References: <CAOYX9gwNLghJjeGntraNmcEo2NYC_Lfi8=Y4quv5BdVw0OM6nw@mail.gmail.com>
Message-ID: <87oag22k0c.fsf@jedbrown.org>

Orxan Shibliyev <orxan.shibli at gmail.com> writes:

> I used KSP to solve Ax=b successfully but I would like to solve Ax=b only
> for certain rows. How can I do this?

It's not actually significantly cheaper no matter what you do.  You can
save a little on non-leading constant factors for some methods
(basically permuting the factorization so you only need a partial
back-solve).  This is usually negligible.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151013/00935422/attachment.pgp>

From yaqiwang at gmail.com  Wed Oct 14 02:08:14 2015
From: yaqiwang at gmail.com (Yaqi Wang)
Date: Wed, 14 Oct 2015 01:08:14 -0600
Subject: [petsc-users] SNESSolve_KSPONLY not working for my linear problem
Message-ID: <CACL+gcPL-qNKfLKRrq6gcLv-4mAQH2TYVueHMGHd-wZ4vE4Ciw@mail.gmail.com>

Hello All,

This is possibly a newbie question but I cannot figure out why. A SNES is
set up in my code but I would like to use it for solving a simple linear
problem.

The sequence of setting up the SNES is as the following:
SNESCreate()
SNESSetFunction()
SNESSetJacobian()
SNESSetFromOptions()

I do not list all calls but just list those I think matters. Note that my
Jacobian is not exact, i.e. it is not consistent with my function.

Then SNESSolve() is called. If I use -snes_type newtonls and
-snes_mf_operator, SNES can solve the linear problem properly, i.e. the
residual norm dropped bellow the tolerance I specified. I know I donot have
to by playing with tolerances, but I tried -snes_type ksponly. I saw my
residual function is only called at the beginning and the end of solve.
Most likely KSP is trying to solve the linear problem with the Jacobian I
specified, which is not exact. I cannot make the solution converge probably
because of this.

My question is why KSPONLY is not using my function for evaluating (b-Ax)
at each linear iteration? I am using the default KSP type, which I guess
should be GMRES. I do not think it matters anyway.
Did I do something wrongly or I had some misunderstanding?

Many thanks,
Yaqi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/a2c60fbc/attachment.html>

From yaqiwang at gmail.com  Wed Oct 14 02:18:06 2015
From: yaqiwang at gmail.com (Yaqi Wang)
Date: Wed, 14 Oct 2015 01:18:06 -0600
Subject: [petsc-users] SNESSolve_KSPONLY not working for my linear
	problem
In-Reply-To: <CACL+gcPL-qNKfLKRrq6gcLv-4mAQH2TYVueHMGHd-wZ4vE4Ciw@mail.gmail.com>
References: <CACL+gcPL-qNKfLKRrq6gcLv-4mAQH2TYVueHMGHd-wZ4vE4Ciw@mail.gmail.com>
Message-ID: <CACL+gcN=2OrVn3Mjs8-R8yj=_J_QJjMwOFwP1fZ145tBzdDE=Q@mail.gmail.com>

While reading my post again, I realized I did not have -snes_mf or
-snes_mf_operator set with -ksp_only. This possibly explains the behavior I
observed. I was not thinking this combination. I will confirm this tomorrow.

Thanks,
Yaqi

On Wed, Oct 14, 2015 at 1:08 AM, Yaqi Wang <yaqiwang at gmail.com> wrote:

> Hello All,
>
> This is possibly a newbie question but I cannot figure out why. A SNES is
> set up in my code but I would like to use it for solving a simple linear
> problem.
>
> The sequence of setting up the SNES is as the following:
> SNESCreate()
> SNESSetFunction()
> SNESSetJacobian()
> SNESSetFromOptions()
>
> I do not list all calls but just list those I think matters. Note that my
> Jacobian is not exact, i.e. it is not consistent with my function.
>
> Then SNESSolve() is called. If I use -snes_type newtonls and
> -snes_mf_operator, SNES can solve the linear problem properly, i.e. the
> residual norm dropped bellow the tolerance I specified. I know I donot have
> to by playing with tolerances, but I tried -snes_type ksponly. I saw my
> residual function is only called at the beginning and the end of solve.
> Most likely KSP is trying to solve the linear problem with the Jacobian I
> specified, which is not exact. I cannot make the solution converge probably
> because of this.
>
> My question is why KSPONLY is not using my function for evaluating (b-Ax)
> at each linear iteration? I am using the default KSP type, which I guess
> should be GMRES. I do not think it matters anyway.
> Did I do something wrongly or I had some misunderstanding?
>
> Many thanks,
> Yaqi
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/dc0b939e/attachment.html>

From vincent.huber at cemosis.fr  Wed Oct 14 02:57:11 2015
From: vincent.huber at cemosis.fr (Vincent Huber)
Date: Wed, 14 Oct 2015 09:57:11 +0200
Subject: [petsc-users] Documentation (small) issue - PCType - petsc-current
Message-ID: <CAK+p5JZ6na6FEKa8zGGcKf5dGKoouAtsHH+NHiDaoVAV_jimsw@mail.gmail.com>

Hello all,

I would like to highlight that the entry PCType is no more on the
documentation for petsc-current
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/>.
We can found it in the petsc-3.5
<http://www.mcs.anl.gov/petsc/petsc-3.5/docs/manualpages/PC/PCType.html>
documentation.

I guess that is a mistake.

?All the best,?
?VH?
?
-- 
Docteur Ing?nieur de recherche
CeMoSiS <http://www.cemosis.fr> - vincent.huber at cemosis.fr
Tel: +33 (0)3 68 8*5 02 06*
IRMA - 7, rue Ren? Descartes
67 000 Strasbourg
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/d52e7dc5/attachment.html>

From knepley at gmail.com  Wed Oct 14 07:22:59 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 14 Oct 2015 07:22:59 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
Message-ID: <CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>

On Tue, Oct 13, 2015 at 9:23 PM, Timoth?e Nicolas <
timothee.nicolas at gmail.com> wrote:

> Dear all,
>
> I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
>
> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine
> 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
>
> where -pc_mg_levels is set to a number between 2 and 7.
>
> So there is a noticeable CPU time improvement from 2 levels to 3 levels
> (30%), and then no improvement whatsoever. I am surprised because with 6
> levels of refinement of the DMDA the fine grid has more than 1200 points so
> with 3 levels the coarse grid still has more than 300 points which is still
> pretty large (I assume the ratio between grids is 2). I am wondering how
> the coarse solver efficiently solves the problem on the coarse grid with
> such a large number of points ? Given the principle of multigrid which is
> to erase the smooth part of the error with relaxation methods, which are
> usually efficient only for high frequency, I would expect optimal
> performance when the coarse grid is basically just a few points in each
> direction. Does anyone know why the performance saturates at low number of
> levels ? Basically what happens internally seems to be quite different from
> what I would expect...
>

A performance model that counts only flops is not sophisticated enough to
understand this effect. Unfortunately, nearly all MG
books/papers use this model. What we need is a model that incorporates
memory bandwidth (for pulling down the values), and
also maybe memory latency. For instance, your relaxation pulls down all the
values and makes a little progress. It does few flops,
but lots of memory access. An LU solve does a little memory access, many
more flops, but makes a lots more progress. If memory
access is more expensive, then we have a tradeoff, and can understand using
a coarse grid which is not just a few points.

  Thanks,

     Matt


> Best
>
> Timothee
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/9377f955/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Oct 14 07:34:04 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Wed, 14 Oct 2015 21:34:04 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>
Message-ID: <CAGi1ndTwvbSoSFKjk44dFiDhH54PW9ufrydcy2yurZHxFMrHbw@mail.gmail.com>

OK, I see. Does it mean that the coarse grid solver is by default set up
with the options -ksp_type preonly -pc_type lu ? What about the
multiprocessor case ?

Thx

Timothee

2015-10-14 21:22 GMT+09:00 Matthew Knepley <knepley at gmail.com>:

> On Tue, Oct 13, 2015 at 9:23 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
>
>> Dear all,
>>
>> I have been playing around with multigrid recently, namely with
>> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
>> with my own implementation of a laplacian type problem. In all cases, I
>> have noted no improvement whatsoever in the performance, whether in CPU
>> time or KSP iteration, by varying the number of levels of the multigrid
>> solver. As an example, I have attached the log_summary for ex5.c with
>> nlevels = 2 to 7, launched by
>>
>> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
>> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
>> -log_summary
>>
>> where -pc_mg_levels is set to a number between 2 and 7.
>>
>> So there is a noticeable CPU time improvement from 2 levels to 3 levels
>> (30%), and then no improvement whatsoever. I am surprised because with 6
>> levels of refinement of the DMDA the fine grid has more than 1200 points so
>> with 3 levels the coarse grid still has more than 300 points which is still
>> pretty large (I assume the ratio between grids is 2). I am wondering how
>> the coarse solver efficiently solves the problem on the coarse grid with
>> such a large number of points ? Given the principle of multigrid which is
>> to erase the smooth part of the error with relaxation methods, which are
>> usually efficient only for high frequency, I would expect optimal
>> performance when the coarse grid is basically just a few points in each
>> direction. Does anyone know why the performance saturates at low number of
>> levels ? Basically what happens internally seems to be quite different from
>> what I would expect...
>>
>
> A performance model that counts only flops is not sophisticated enough to
> understand this effect. Unfortunately, nearly all MG
> books/papers use this model. What we need is a model that incorporates
> memory bandwidth (for pulling down the values), and
> also maybe memory latency. For instance, your relaxation pulls down all
> the values and makes a little progress. It does few flops,
> but lots of memory access. An LU solve does a little memory access, many
> more flops, but makes a lots more progress. If memory
> access is more expensive, then we have a tradeoff, and can understand
> using a coarse grid which is not just a few points.
>
>   Thanks,
>
>      Matt
>
>
>> Best
>>
>> Timothee
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/706f726a/attachment.html>

From knepley at gmail.com  Wed Oct 14 09:50:13 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 14 Oct 2015 09:50:13 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndTwvbSoSFKjk44dFiDhH54PW9ufrydcy2yurZHxFMrHbw@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>
	<CAGi1ndTwvbSoSFKjk44dFiDhH54PW9ufrydcy2yurZHxFMrHbw@mail.gmail.com>
Message-ID: <CAMYG4Gk8bgMsT=Dr9S7P3xth4Or9O4Tizs11k1yZvyo1S4QpGA@mail.gmail.com>

On Wed, Oct 14, 2015 at 7:34 AM, Timoth?e Nicolas <
timothee.nicolas at gmail.com> wrote:

> OK, I see. Does it mean that the coarse grid solver is by default set up
> with the options -ksp_type preonly -pc_type lu ? What about the
> multiprocessor case ?
>

Small scale: We use redundant LU

Large Scale: We use GAMG

   Matt


> Thx
>
> Timothee
>
> 2015-10-14 21:22 GMT+09:00 Matthew Knepley <knepley at gmail.com>:
>
>> On Tue, Oct 13, 2015 at 9:23 PM, Timoth?e Nicolas <
>> timothee.nicolas at gmail.com> wrote:
>>
>>> Dear all,
>>>
>>> I have been playing around with multigrid recently, namely with
>>> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
>>> with my own implementation of a laplacian type problem. In all cases, I
>>> have noted no improvement whatsoever in the performance, whether in CPU
>>> time or KSP iteration, by varying the number of levels of the multigrid
>>> solver. As an example, I have attached the log_summary for ex5.c with
>>> nlevels = 2 to 7, launched by
>>>
>>> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
>>> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
>>> -log_summary
>>>
>>> where -pc_mg_levels is set to a number between 2 and 7.
>>>
>>> So there is a noticeable CPU time improvement from 2 levels to 3 levels
>>> (30%), and then no improvement whatsoever. I am surprised because with 6
>>> levels of refinement of the DMDA the fine grid has more than 1200 points so
>>> with 3 levels the coarse grid still has more than 300 points which is still
>>> pretty large (I assume the ratio between grids is 2). I am wondering how
>>> the coarse solver efficiently solves the problem on the coarse grid with
>>> such a large number of points ? Given the principle of multigrid which is
>>> to erase the smooth part of the error with relaxation methods, which are
>>> usually efficient only for high frequency, I would expect optimal
>>> performance when the coarse grid is basically just a few points in each
>>> direction. Does anyone know why the performance saturates at low number of
>>> levels ? Basically what happens internally seems to be quite different from
>>> what I would expect...
>>>
>>
>> A performance model that counts only flops is not sophisticated enough to
>> understand this effect. Unfortunately, nearly all MG
>> books/papers use this model. What we need is a model that incorporates
>> memory bandwidth (for pulling down the values), and
>> also maybe memory latency. For instance, your relaxation pulls down all
>> the values and makes a little progress. It does few flops,
>> but lots of memory access. An LU solve does a little memory access, many
>> more flops, but makes a lots more progress. If memory
>> access is more expensive, then we have a tradeoff, and can understand
>> using a coarse grid which is not just a few points.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Best
>>>
>>> Timothee
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/794b10ab/attachment.html>

From dave.mayhem23 at gmail.com  Wed Oct 14 10:01:33 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Wed, 14 Oct 2015 17:01:33 +0200
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAMYG4Gk8bgMsT=Dr9S7P3xth4Or9O4Tizs11k1yZvyo1S4QpGA@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>
	<CAGi1ndTwvbSoSFKjk44dFiDhH54PW9ufrydcy2yurZHxFMrHbw@mail.gmail.com>
	<CAMYG4Gk8bgMsT=Dr9S7P3xth4Or9O4Tizs11k1yZvyo1S4QpGA@mail.gmail.com>
Message-ID: <CAJ98EDojz1t+ck=hhN2EgQ9u-V=VNu6GyJPsE7wQLH0LvbGVBA@mail.gmail.com>

On 14 October 2015 at 16:50, Matthew Knepley <knepley at gmail.com> wrote:

> On Wed, Oct 14, 2015 at 7:34 AM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
>
>> OK, I see. Does it mean that the coarse grid solver is by default set up
>> with the options -ksp_type preonly -pc_type lu ? What about the
>> multiprocessor case ?
>>
>
> Small scale: We use redundant LU
>
> Large Scale: We use GAMG
>
>
Is your answer what "you" recommend, or what PETSc does by default?

Your answer gives the impression that PETSc makes a decision regarding the
choice of either redundant/LU or gamg based on something - e.g. the size of
the matrix, the number of cores (or some combination of the two).
Is that really what is happening inside PCMG?





>    Matt
>
>
>> Thx
>>
>> Timothee
>>
>> 2015-10-14 21:22 GMT+09:00 Matthew Knepley <knepley at gmail.com>:
>>
>>> On Tue, Oct 13, 2015 at 9:23 PM, Timoth?e Nicolas <
>>> timothee.nicolas at gmail.com> wrote:
>>>
>>>> Dear all,
>>>>
>>>> I have been playing around with multigrid recently, namely with
>>>> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
>>>> with my own implementation of a laplacian type problem. In all cases, I
>>>> have noted no improvement whatsoever in the performance, whether in CPU
>>>> time or KSP iteration, by varying the number of levels of the multigrid
>>>> solver. As an example, I have attached the log_summary for ex5.c with
>>>> nlevels = 2 to 7, launched by
>>>>
>>>> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
>>>> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
>>>> -log_summary
>>>>
>>>> where -pc_mg_levels is set to a number between 2 and 7.
>>>>
>>>> So there is a noticeable CPU time improvement from 2 levels to 3 levels
>>>> (30%), and then no improvement whatsoever. I am surprised because with 6
>>>> levels of refinement of the DMDA the fine grid has more than 1200 points so
>>>> with 3 levels the coarse grid still has more than 300 points which is still
>>>> pretty large (I assume the ratio between grids is 2). I am wondering how
>>>> the coarse solver efficiently solves the problem on the coarse grid with
>>>> such a large number of points ? Given the principle of multigrid which is
>>>> to erase the smooth part of the error with relaxation methods, which are
>>>> usually efficient only for high frequency, I would expect optimal
>>>> performance when the coarse grid is basically just a few points in each
>>>> direction. Does anyone know why the performance saturates at low number of
>>>> levels ? Basically what happens internally seems to be quite different from
>>>> what I would expect...
>>>>
>>>
>>> A performance model that counts only flops is not sophisticated enough
>>> to understand this effect. Unfortunately, nearly all MG
>>> books/papers use this model. What we need is a model that incorporates
>>> memory bandwidth (for pulling down the values), and
>>> also maybe memory latency. For instance, your relaxation pulls down all
>>> the values and makes a little progress. It does few flops,
>>> but lots of memory access. An LU solve does a little memory access, many
>>> more flops, but makes a lots more progress. If memory
>>> access is more expensive, then we have a tradeoff, and can understand
>>> using a coarse grid which is not just a few points.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Best
>>>>
>>>> Timothee
>>>>
>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/3d81bc1c/attachment.html>

From knepley at gmail.com  Wed Oct 14 10:02:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 14 Oct 2015 10:02:35 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAJ98EDojz1t+ck=hhN2EgQ9u-V=VNu6GyJPsE7wQLH0LvbGVBA@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<CAMYG4Gkyh2njTEEf5eHVqJrno6Fcr=b8Q6wo6rveqDBYT93WDQ@mail.gmail.com>
	<CAGi1ndTwvbSoSFKjk44dFiDhH54PW9ufrydcy2yurZHxFMrHbw@mail.gmail.com>
	<CAMYG4Gk8bgMsT=Dr9S7P3xth4Or9O4Tizs11k1yZvyo1S4QpGA@mail.gmail.com>
	<CAJ98EDojz1t+ck=hhN2EgQ9u-V=VNu6GyJPsE7wQLH0LvbGVBA@mail.gmail.com>
Message-ID: <CAMYG4Gm1FzdqDnDSDZrCrGsNK+F0Fjkf18WYLz1cfTL=0ixt5g@mail.gmail.com>

On Wed, Oct 14, 2015 at 10:01 AM, Dave May <dave.mayhem23 at gmail.com> wrote:

> On 14 October 2015 at 16:50, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Wed, Oct 14, 2015 at 7:34 AM, Timoth?e Nicolas <
>> timothee.nicolas at gmail.com> wrote:
>>
>>> OK, I see. Does it mean that the coarse grid solver is by default set up
>>> with the options -ksp_type preonly -pc_type lu ? What about the
>>> multiprocessor case ?
>>>
>>
>> Small scale: We use redundant LU
>>
>> Large Scale: We use GAMG
>>
>>
> Is your answer what "you" recommend, or what PETSc does by default?
>
> Your answer gives the impression that PETSc makes a decision regarding the
> choice of either redundant/LU or gamg based on something - e.g. the size of
> the matrix, the number of cores (or some combination of the two).
> Is that really what is happening inside PCMG?
>

No, the default is redundant LU, and then you can decide to use GAMG.

   Matt


>
>
>>    Matt
>>
>>
>>> Thx
>>>
>>> Timothee
>>>
>>> 2015-10-14 21:22 GMT+09:00 Matthew Knepley <knepley at gmail.com>:
>>>
>>>> On Tue, Oct 13, 2015 at 9:23 PM, Timoth?e Nicolas <
>>>> timothee.nicolas at gmail.com> wrote:
>>>>
>>>>> Dear all,
>>>>>
>>>>> I have been playing around with multigrid recently, namely with
>>>>> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
>>>>> with my own implementation of a laplacian type problem. In all cases, I
>>>>> have noted no improvement whatsoever in the performance, whether in CPU
>>>>> time or KSP iteration, by varying the number of levels of the multigrid
>>>>> solver. As an example, I have attached the log_summary for ex5.c with
>>>>> nlevels = 2 to 7, launched by
>>>>>
>>>>> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
>>>>> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
>>>>> -log_summary
>>>>>
>>>>> where -pc_mg_levels is set to a number between 2 and 7.
>>>>>
>>>>> So there is a noticeable CPU time improvement from 2 levels to 3
>>>>> levels (30%), and then no improvement whatsoever. I am surprised because
>>>>> with 6 levels of refinement of the DMDA the fine grid has more than 1200
>>>>> points so with 3 levels the coarse grid still has more than 300 points
>>>>> which is still pretty large (I assume the ratio between grids is 2). I am
>>>>> wondering how the coarse solver efficiently solves the problem on the
>>>>> coarse grid with such a large number of points ? Given the principle of
>>>>> multigrid which is to erase the smooth part of the error with relaxation
>>>>> methods, which are usually efficient only for high frequency, I would
>>>>> expect optimal performance when the coarse grid is basically just a few
>>>>> points in each direction. Does anyone know why the performance saturates at
>>>>> low number of levels ? Basically what happens internally seems to be quite
>>>>> different from what I would expect...
>>>>>
>>>>
>>>> A performance model that counts only flops is not sophisticated enough
>>>> to understand this effect. Unfortunately, nearly all MG
>>>> books/papers use this model. What we need is a model that incorporates
>>>> memory bandwidth (for pulling down the values), and
>>>> also maybe memory latency. For instance, your relaxation pulls down all
>>>> the values and makes a little progress. It does few flops,
>>>> but lots of memory access. An LU solve does a little memory access,
>>>> many more flops, but makes a lots more progress. If memory
>>>> access is more expensive, then we have a tradeoff, and can understand
>>>> using a coarse grid which is not just a few points.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>>> Best
>>>>>
>>>>> Timothee
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/5b2a3984/attachment-0001.html>

From yaqiwang at gmail.com  Wed Oct 14 10:23:35 2015
From: yaqiwang at gmail.com (Yaqi Wang)
Date: Wed, 14 Oct 2015 09:23:35 -0600
Subject: [petsc-users] SNESSolve_KSPONLY not working for my linear
	problem
In-Reply-To: <CACL+gcN=2OrVn3Mjs8-R8yj=_J_QJjMwOFwP1fZ145tBzdDE=Q@mail.gmail.com>
References: <CACL+gcPL-qNKfLKRrq6gcLv-4mAQH2TYVueHMGHd-wZ4vE4Ciw@mail.gmail.com>
	<CACL+gcN=2OrVn3Mjs8-R8yj=_J_QJjMwOFwP1fZ145tBzdDE=Q@mail.gmail.com>
Message-ID: <CACL+gcMQm3DGPp5oBfF6KO3x+AU5kDXYSkQ7+oQi0M23ngVYrw@mail.gmail.com>

Sorry for my dumb question. I confirmed that I need to set either -snes_mf
or -snes_mf_operator. Thanks.

On Wed, Oct 14, 2015 at 1:18 AM, Yaqi Wang <yaqiwang at gmail.com> wrote:

> While reading my post again, I realized I did not have -snes_mf or
> -snes_mf_operator set with -ksp_only. This possibly explains the behavior I
> observed. I was not thinking this combination. I will confirm this tomorrow.
>
> Thanks,
> Yaqi
>
> On Wed, Oct 14, 2015 at 1:08 AM, Yaqi Wang <yaqiwang at gmail.com> wrote:
>
>> Hello All,
>>
>> This is possibly a newbie question but I cannot figure out why. A SNES is
>> set up in my code but I would like to use it for solving a simple linear
>> problem.
>>
>> The sequence of setting up the SNES is as the following:
>> SNESCreate()
>> SNESSetFunction()
>> SNESSetJacobian()
>> SNESSetFromOptions()
>>
>> I do not list all calls but just list those I think matters. Note that my
>> Jacobian is not exact, i.e. it is not consistent with my function.
>>
>> Then SNESSolve() is called. If I use -snes_type newtonls and
>> -snes_mf_operator, SNES can solve the linear problem properly, i.e. the
>> residual norm dropped bellow the tolerance I specified. I know I donot have
>> to by playing with tolerances, but I tried -snes_type ksponly. I saw my
>> residual function is only called at the beginning and the end of solve.
>> Most likely KSP is trying to solve the linear problem with the Jacobian I
>> specified, which is not exact. I cannot make the solution converge probably
>> because of this.
>>
>> My question is why KSPONLY is not using my function for evaluating (b-Ax)
>> at each linear iteration? I am using the default KSP type, which I guess
>> should be GMRES. I do not think it matters anyway.
>> Did I do something wrongly or I had some misunderstanding?
>>
>> Many thanks,
>> Yaqi
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/f194e77e/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct 14 13:02:37 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 14 Oct 2015 13:02:37 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
Message-ID: <71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>

1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD! 

  ##########################################################
      #                                                        #
      #                          WARNING!!!                    #
      #                                                        #
      #   This code was compiled with a debugging option,      #
      #   To get timing results run ./configure                #
      #   using --with-debugging=no, the performance will      #
      #   be generally two or three times faster.              #
      #                                                        #
      ##########################################################

2) Please run with -snes_view . 

3) Note that with 7 levels

SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0

with 2 levels

SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0


The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case

  Barry

> On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Dear all,
> 
> I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by 
> 
> mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> 
> where -pc_mg_levels is set to a number between 2 and 7.
> 
> So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> 
> Best
> 
> Timothee
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>


From jianjun.xiao at kit.edu  Wed Oct 14 15:11:38 2015
From: jianjun.xiao at kit.edu (Xiao, Jianjun (IKET))
Date: Wed, 14 Oct 2015 20:11:38 +0000
Subject: [petsc-users] Load balancing problem with DMDA
Message-ID: <1444853498070.73407@kit.edu>

Hello,


We have a finite volume code with structured grid. We use the DMDA data structure.


Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?


Or I have to do it using DMPLEX? Since data structure of the whole code  is based on  DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?


Thank you.


Best regards

Jianjun
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/7958bc2d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pic.JPG
Type: image/jpeg
Size: 12384 bytes
Desc: pic.JPG
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/7958bc2d/attachment.jpe>

From bsmith at mcs.anl.gov  Wed Oct 14 15:32:07 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 14 Oct 2015 15:32:07 -0500
Subject: [petsc-users] Load balancing problem with DMDA
In-Reply-To: <1444853498070.73407@kit.edu>
References: <1444853498070.73407@kit.edu>
Message-ID: <5C930F9A-B2FA-4A87-9046-E61C6ED5B979@mcs.anl.gov>


> On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <jianjun.xiao at kit.edu> wrote:
> 
> Hello,
> 
> We have a finite volume code with structured grid. We use the DMDA data structure.
> 
> Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?

No

> 
> Or I have to do it using DMPLEX? Since data structure of the whole code  is based on  DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?

  No

  DMDA is for simple structured grids.

> 
> Thank you.
> 
> Best regards
> Jianjun
> <pic.JPG>


From balay at mcs.anl.gov  Wed Oct 14 16:17:23 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 14 Oct 2015 16:17:23 -0500
Subject: [petsc-users] Documentation (small) issue - PCType -
 petsc-current
In-Reply-To: <CAK+p5JZ6na6FEKa8zGGcKf5dGKoouAtsHH+NHiDaoVAV_jimsw@mail.gmail.com>
References: <CAK+p5JZ6na6FEKa8zGGcKf5dGKoouAtsHH+NHiDaoVAV_jimsw@mail.gmail.com>
Message-ID: <alpine.LFD.2.20.1510141532350.19003@asterix>

Thanks for the report. Its fixed in repo now.
https://bitbucket.org/petsc/petsc/commits/a21ee325523d0e59c7e5d5a07ac73390be6ac6fe

[website will be updated with the next patch release of 3.6]

Satish

On Wed, 14 Oct 2015, Vincent Huber wrote:

> Hello all,
> 
> I would like to highlight that the entry PCType is no more on the
> documentation for petsc-current
> <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/>.
> We can found it in the petsc-3.5
> <http://www.mcs.anl.gov/petsc/petsc-3.5/docs/manualpages/PC/PCType.html>
> documentation.
> 
> I guess that is a mistake.
> 
> ?All the best,?
> ?VH?
> ?
> 

From coyigg at hotmail.com  Wed Oct 14 18:31:47 2015
From: coyigg at hotmail.com (keguoyi)
Date: Thu, 15 Oct 2015 07:31:47 +0800
Subject: [petsc-users] =?gb2312?Q?RE:_[petsc?= =?gb2312?Q?-users]_Ho?=
 =?gb2312?Q?w_to_imple?= =?gb2312?Q?ment_press?= =?gb2312?Q?ure_convec?=
 =?gb2312?B?dGlvbqhDZGlmZnU=?= =?gb2312?Q?sion_preco?=
 =?gb2312?Q?nditioner_?= =?gb2312?Q?in_petsz?=
In-Reply-To: <CAMYG4Gmy0PpQPmX8k4Mp_PpCdaaPx3DW54O+1ZP5i1kGB7u+iQ@mail.gmail.com>
References: <BLU185-W60C1111F5BF88323F7984AD3310@phx.gbl>,
	<CAMYG4Gmy0PpQPmX8k4Mp_PpCdaaPx3DW54O+1ZP5i1kGB7u+iQ@mail.gmail.com>
Message-ID: <BLU185-W21D4393B657D04D53CD5A1D33F0@phx.gbl>

Dear Matthew,

Thanks for the last Email. For the LSC preconditioner in Petsc, it allows to do inv(S)= inv(A10 A01) A10 A00 A01 inv(A10 A01). Is it also possible to do inv(S)= inv(Mp) Fp inv(Ap), where Mp and Ap are built by ourselves and not the same matrices. Thanks a lot.

Best,
Guoyi

Date: Mon, 12 Oct 2015 08:46:35 -0500
Subject: Re: [petsc-users] How to implement pressure convection?Cdiffusion preconditioner in petsz
From: knepley at gmail.com
To: coyigg at hotmail.com
CC: petsc-users at mcs.anl.gov

On Mon, Oct 12, 2015 at 1:49 AM, keguoyi <coyigg at hotmail.com> wrote:



Dear Petsc developers and users,  

This is Guoyi ke, a graduate student in Texas Tech University.  I have a 2D Navier Stokes problem that has block matrices: J=[F    B^T; B    0]. I want to build a pressure convection?Cdiffusion preconditioner (PCD) P=[F    B^T; 0   Sp]. Here, we let Sp=-Ap(Fp)^(-1)Mp approximate schur complement S=-BF^(-1)B^T, where Ap is pressure Laplacian matrix, Mp is pressure mass matrix, and Fp is convection-diffusion operators on pressure space.

 We use right preconditioner J*P^(-1)=[F    B^T; B    0] * [F    B^T; 0    Sp]^(-1), and is it possible for Petsz to build and implement this precondioner P? Since (Sp)^(-1)=-(Mp)^(-1) Fp(Ap)^(-1), is it possible that we can solve (Mp)^(-1) and (Ap)^(-1) by CG method separately inside preconditioner P. 

Take a look at the PCFIELDSPLIT preconditioner. I think you want the LSC option for that (http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCLSC.html)if I am reading your mail correctly.
  Thanks,
    Matt Any suggestion will be highly appreciated. Thank you so much!

Best,
Guoyi 
 		 	   		  


-- 
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
 		 	   		  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/f22bda13/attachment-0001.html>

From timothee.nicolas at gmail.com  Wed Oct 14 21:32:20 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Thu, 15 Oct 2015 11:32:20 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
Message-ID: <CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>

Thank you Barry for pointing this out. Indeed on a system with no debugging
the Jacobian evaluations no longer dominate the time (less than 10%).
However the rest is similar, except the improvement from 2 to 3 levels is
much better. Still it saturates after levels=3. I understand it in terms of
CPU time thanks to Matthew's explanations, however what surprises me more
is that KSP iterations are not more efficient. At the least, even if it
takes more time to have more levels because of memory issues, I would
expect KSP iterations to converge more rapidly with more levels, but it is
not the case as you can see. Probably there is also a rationale behind this
but I cannot see easily.

I send the new outputs

Best

Timothee

2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:

> 1) Your timings are meaningless! You cannot compare timings when built
> with all debugging on, PERIOD!
>
>   ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
>
> 2) Please run with -snes_view .
>
> 3) Note that with 7 levels
>
> SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
>
> with 2 levels
>
> SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
>
>
> The Jacobian evaluation is dominating the time! Likely if you fix the
> debugging this will be less the case
>
>   Barry
>
> > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> >
> > Dear all,
> >
> > I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
> >
> > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
> -log_summary
> >
> > where -pc_mg_levels is set to a number between 2 and 7.
> >
> > So there is a noticeable CPU time improvement from 2 levels to 3 levels
> (30%), and then no improvement whatsoever. I am surprised because with 6
> levels of refinement of the DMDA the fine grid has more than 1200 points so
> with 3 levels the coarse grid still has more than 300 points which is still
> pretty large (I assume the ratio between grids is 2). I am wondering how
> the coarse solver efficiently solves the problem on the coarse grid with
> such a large number of points ? Given the principle of multigrid which is
> to erase the smooth part of the error with relaxation methods, which are
> usually efficient only for high frequency, I would expect optimal
> performance when the coarse grid is basically just a few points in each
> direction. Does anyone know why the performance saturates at low number of
> levels ? Basically what happens internally seems to be quite different from
> what I would expect...
> >
> > Best
> >
> > Timothee
> >
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_2_multigrid_levels.log
Type: application/octet-stream
Size: 19510 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels.log
Type: application/octet-stream
Size: 21103 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels.log
Type: application/octet-stream
Size: 22682 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels.log
Type: application/octet-stream
Size: 24266 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels.log
Type: application/octet-stream
Size: 25844 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels.log
Type: application/octet-stream
Size: 27422 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/caad9909/attachment-0011.obj>

From bsmith at mcs.anl.gov  Wed Oct 14 22:37:08 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 14 Oct 2015 22:37:08 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
Message-ID: <F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>


  Timothee,

     Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?

   Thanks

  Barry

> On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily. 
> 
> I send the new outputs
> 
> Best
> 
> Timothee
> 
> 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> 
>   ##########################################################
>       #                                                        #
>       #                          WARNING!!!                    #
>       #                                                        #
>       #   This code was compiled with a debugging option,      #
>       #   To get timing results run ./configure                #
>       #   using --with-debugging=no, the performance will      #
>       #   be generally two or three times faster.              #
>       #                                                        #
>       ##########################################################
> 
> 2) Please run with -snes_view .
> 
> 3) Note that with 7 levels
> 
> SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> 
> with 2 levels
> 
> SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> 
> 
> The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> 
>   Barry
> 
> > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > Dear all,
> >
> > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> >
> > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> >
> > where -pc_mg_levels is set to a number between 2 and 7.
> >
> > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> >
> > Best
> >
> > Timothee
> > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> 
> 
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>


From timothee.nicolas at gmail.com  Wed Oct 14 22:53:29 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Thu, 15 Oct 2015 12:53:29 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
Message-ID: <CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>

OK,

Richardson is 30-70% faster for these tests, but other than this I don't
see any change.

Timothee



2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:

>
>   Timothee,
>
>      Thank you for reporting this issue, it is indeed disturbing and could
> be due to a performance regression we may have introduced by being too
> clever for our own good. Could you please rerun with the additional option
> -mg_levels_ksp_type richardson and send the same output?
>
>    Thanks
>
>   Barry
>
> > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> >
> > Thank you Barry for pointing this out. Indeed on a system with no
> debugging the Jacobian evaluations no longer dominate the time (less than
> 10%). However the rest is similar, except the improvement from 2 to 3
> levels is much better. Still it saturates after levels=3. I understand it
> in terms of CPU time thanks to Matthew's explanations, however what
> surprises me more is that KSP iterations are not more efficient. At the
> least, even if it takes more time to have more levels because of memory
> issues, I would expect KSP iterations to converge more rapidly with more
> levels, but it is not the case as you can see. Probably there is also a
> rationale behind this but I cannot see easily.
> >
> > I send the new outputs
> >
> > Best
> >
> > Timothee
> >
> > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > 1) Your timings are meaningless! You cannot compare timings when built
> with all debugging on, PERIOD!
> >
> >   ##########################################################
> >       #                                                        #
> >       #                          WARNING!!!                    #
> >       #                                                        #
> >       #   This code was compiled with a debugging option,      #
> >       #   To get timing results run ./configure                #
> >       #   using --with-debugging=no, the performance will      #
> >       #   be generally two or three times faster.              #
> >       #                                                        #
> >       ##########################################################
> >
> > 2) Please run with -snes_view .
> >
> > 3) Note that with 7 levels
> >
> > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> >
> > with 2 levels
> >
> > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
> 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> >
> >
> > The Jacobian evaluation is dominating the time! Likely if you fix the
> debugging this will be less the case
> >
> >   Barry
> >
> > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > >
> > > Dear all,
> > >
> > > I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
> > >
> > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
> -log_summary
> > >
> > > where -pc_mg_levels is set to a number between 2 and 7.
> > >
> > > So there is a noticeable CPU time improvement from 2 levels to 3
> levels (30%), and then no improvement whatsoever. I am surprised because
> with 6 levels of refinement of the DMDA the fine grid has more than 1200
> points so with 3 levels the coarse grid still has more than 300 points
> which is still pretty large (I assume the ratio between grids is 2). I am
> wondering how the coarse solver efficiently solves the problem on the
> coarse grid with such a large number of points ? Given the principle of
> multigrid which is to erase the smooth part of the error with relaxation
> methods, which are usually efficient only for high frequency, I would
> expect optimal performance when the coarse grid is basically just a few
> points in each direction. Does anyone know why the performance saturates at
> low number of levels ? Basically what happens internally seems to be quite
> different from what I would expect...
> > >
> > > Best
> > >
> > > Timothee
> > >
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> >
> >
> >
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_2_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 18647 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 19692 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 20688 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 21680 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0009.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 22626 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0010.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels_richardson.log
Type: application/octet-stream
Size: 23612 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/615ce2db/attachment-0011.obj>

From bsmith at mcs.anl.gov  Wed Oct 14 23:15:04 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 14 Oct 2015 23:15:04 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
	<CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
Message-ID: <E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>


  Wow, quick response! Yes the times still indicate that after 4 levels you get no improvement in time.

t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00, 5.7643e+00 ]

I'll look more specifically at the numbers to see where the time is being transformed tomorrow when I am less drunk. It is a trade off between the work saved in the direct solve vs the work needed for the coarser levels in the multigrid cycle. 

Try refining the grid a couple more times, likely more levels will still help in that case

 Ahh, you should also try -pc_mg_type full


  Barry



> On Oct 14, 2015, at 10:53 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> OK,
> 
> Richardson is 30-70% faster for these tests, but other than this I don't see any change.
> 
> Timothee
> 
> 
> 
> 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 
>   Timothee,
> 
>      Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?
> 
>    Thanks
> 
>   Barry
> 
> > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily.
> >
> > I send the new outputs
> >
> > Best
> >
> > Timothee
> >
> > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> >
> >   ##########################################################
> >       #                                                        #
> >       #                          WARNING!!!                    #
> >       #                                                        #
> >       #   This code was compiled with a debugging option,      #
> >       #   To get timing results run ./configure                #
> >       #   using --with-debugging=no, the performance will      #
> >       #   be generally two or three times faster.              #
> >       #                                                        #
> >       ##########################################################
> >
> > 2) Please run with -snes_view .
> >
> > 3) Note that with 7 levels
> >
> > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> >
> > with 2 levels
> >
> > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> >
> >
> > The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> >
> >   Barry
> >
> > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> > >
> > > Dear all,
> > >
> > > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> > >
> > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> > >
> > > where -pc_mg_levels is set to a number between 2 and 7.
> > >
> > > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> > >
> > > Best
> > >
> > > Timothee
> > > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> >
> >
> > <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> 
> 
> <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>


From fdkong.jd at gmail.com  Wed Oct 14 23:29:04 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Wed, 14 Oct 2015 22:29:04 -0600
Subject: [petsc-users] how to run snes/examples/tutorials/ex12
Message-ID: <CAN5Wd-+nMDNW8Lf_L_bWAjj0MNG2LHjqyHHZcrTKeqZBroCX+w@mail.gmail.com>

Hi all,

Anyone knows how to run snes/examples/tutorials/ex12? I can compile the
example, but I did not find any runscript in the makefile. Do I need to go
through the entire source code to figure out how to run it?

Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/65a63382/attachment.html>

From balay at mcs.anl.gov  Wed Oct 14 23:42:04 2015
From: balay at mcs.anl.gov (Satish Balay)
Date: Wed, 14 Oct 2015 23:42:04 -0500
Subject: [petsc-users] how to run snes/examples/tutorials/ex12
In-Reply-To: <CAN5Wd-+nMDNW8Lf_L_bWAjj0MNG2LHjqyHHZcrTKeqZBroCX+w@mail.gmail.com>
References: <CAN5Wd-+nMDNW8Lf_L_bWAjj0MNG2LHjqyHHZcrTKeqZBroCX+w@mail.gmail.com>
Message-ID: <alpine.LFD.2.20.1510142341080.19003@asterix>

looks like the run commands for this example are in config/builder.py

It might also need packages listed in config/examples/arch-c-exodus-dbg-builder.py

Satish

On Wed, 14 Oct 2015, Fande Kong wrote:

> Hi all,
> 
> Anyone knows how to run snes/examples/tutorials/ex12? I can compile the
> example, but I did not find any runscript in the makefile. Do I need to go
> through the entire source code to figure out how to run it?
> 
> Thanks,
> 
> Fande,
> 


From knram06 at gmail.com  Wed Oct 14 23:44:31 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Thu, 15 Oct 2015 00:44:31 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
Message-ID: <CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>

Hello all,

@Matt: Actually I can't fully adopt the SNES ex5 since I think I am much
closer to getting the DMDA stuff to work, even if it is just serial. So I
am currently focussing on that.

@Barry: The matrix is coming from the Finite difference discretization of a
3D regular grid. I had a look at KSP ex45.c and I am struggling with what
looks to be a simple problem.

I currently supply the matrix to PETSc in a CSR form. I have specified
ComputeRHS and ComputeMatrix functions, analogous to ex45.c.

If my 3D grid is regular cube with 5 nodes on each side, I create DMDA
using DMDACreate3d((comm world), boundary types, STAR_STENCIL, etc., M = 5,
N = 5, P = 5, dof=1, s=1, PETSC_NULL on remaining).

In the ComputeMatrix function, I try to reuse my CSR form by doing:

#undef __FUNCT__
#define __FUNCT__ "ComputeMatrix"
PetscErrorCode ComputeMatrix(KSP ksp, Mat J, Mat A, void* ctx)
{
    PetscFunctionBeginUser;

    Mat temp;
    // build up the matrix
    MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, numRows, numRows, rows,
cols, values, &temp);

    MatCopy(temp, A, DIFFERENT_NONZERO_PATTERN);

    MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
    MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);

    PetscFunctionReturn(0);

} // end of ComputeMatrix

Where rows, cols, values are stored globally, for now. Also I am keeping
the code serial, so I am just trying to use DMDA as a convenient way to
take care of interpolation between MG levels, if my understanding is
correct.

But on the MatCopy line, I get an error

DM Object: 1 MPI processes
  type: da
Processor [0] M 5 N 5 P 5 m 1 n 1 p 1 w 1 s 1
X range of indices: 0 5, Y range of indices: 0 5, Z range of indices: 0 5
[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Inserting a new nonzero at (25,27) in the matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015

I have been trying but I have not been able to resolve this quickly. I feel
like I am creating the DMDA correctly. Is there a better way of reusing a
CSR form to setup a DM Matrix? Should I be doing DMCreateMatrix?

Thanks a lot for your help.


Regards,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/052f3732/attachment.html>

From bsmith at mcs.anl.gov  Wed Oct 14 23:54:45 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Wed, 14 Oct 2015 23:54:45 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
Message-ID: <FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>


  we need the full error message, what function is causing the error?  In theory what you are doing sounds like it should work, it is all the little details that are likely causing the issues. In parallel you will not be able to reuse some old code to "compute the Finite difference discretization of a 3D regular grid" so you might consider just discarding that old code and using what PETSc provides for computing the "Finite difference discretization of a 3D regular grid."

  Barry


> On Oct 14, 2015, at 11:44 PM, K. N. Ramachandran <knram06 at gmail.com> wrote:
> 
> Hello all,
> 
> @Matt: Actually I can't fully adopt the SNES ex5 since I think I am much closer to getting the DMDA stuff to work, even if it is just serial. So I am currently focussing on that.
> 
> @Barry: The matrix is coming from the Finite difference discretization of a 3D regular grid. I had a look at KSP ex45.c and I am struggling with what looks to be a simple problem.
> 
> I currently supply the matrix to PETSc in a CSR form. I have specified ComputeRHS and ComputeMatrix functions, analogous to ex45.c.
> 
> If my 3D grid is regular cube with 5 nodes on each side, I create DMDA using DMDACreate3d((comm world), boundary types, STAR_STENCIL, etc., M = 5, N = 5, P = 5, dof=1, s=1, PETSC_NULL on remaining).
> 
> In the ComputeMatrix function, I try to reuse my CSR form by doing:
> 
> #undef __FUNCT__
> #define __FUNCT__ "ComputeMatrix"
> PetscErrorCode ComputeMatrix(KSP ksp, Mat J, Mat A, void* ctx)
> {
>     PetscFunctionBeginUser;
> 
>     Mat temp;
>     // build up the matrix
>     MatCreateSeqAIJWithArrays(PETSC_COMM_WORLD, numRows, numRows, rows, cols, values, &temp);
> 
>     MatCopy(temp, A, DIFFERENT_NONZERO_PATTERN);
> 
>     MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
>     MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
> 
>     PetscFunctionReturn(0);
> 
> } // end of ComputeMatrix
> 
> Where rows, cols, values are stored globally, for now. Also I am keeping the code serial, so I am just trying to use DMDA as a convenient way to take care of interpolation between MG levels, if my understanding is correct.
> 
> But on the MatCopy line, I get an error
> 
> DM Object: 1 MPI processes
>   type: da
> Processor [0] M 5 N 5 P 5 m 1 n 1 p 1 w 1 s 1
> X range of indices: 0 5, Y range of indices: 0 5, Z range of indices: 0 5
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at (25,27) in the matrix
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
> 
> I have been trying but I have not been able to resolve this quickly. I feel like I am creating the DMDA correctly. Is there a better way of reusing a CSR form to setup a DM Matrix? Should I be doing DMCreateMatrix?
> 
> Thanks a lot for your help.
> 
> 
> Regards,
> K.N.Ramachandran
> Ph: 814-441-4279


From fdkong.jd at gmail.com  Thu Oct 15 00:13:30 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Wed, 14 Oct 2015 23:13:30 -0600
Subject: [petsc-users] how to run snes/examples/tutorials/ex12
In-Reply-To: <alpine.LFD.2.20.1510142341080.19003@asterix>
References: <CAN5Wd-+nMDNW8Lf_L_bWAjj0MNG2LHjqyHHZcrTKeqZBroCX+w@mail.gmail.com>
	<alpine.LFD.2.20.1510142341080.19003@asterix>
Message-ID: <CAN5Wd-LSCdjZaWhk0gkhkB2maAOSyrxsoCVdiViPBVdHu-qhhg@mail.gmail.com>

Satish,

Thanks. I found it. Any reason to put run commands in config/builder.py not
in the local makefile. How users could figure out?

Thanks,

Fande,

On Wed, Oct 14, 2015 at 10:42 PM, Satish Balay <balay at mcs.anl.gov> wrote:

> looks like the run commands for this example are in config/builder.py
>
> It might also need packages listed in
> config/examples/arch-c-exodus-dbg-builder.py
>
> Satish
>
> On Wed, 14 Oct 2015, Fande Kong wrote:
>
> > Hi all,
> >
> > Anyone knows how to run snes/examples/tutorials/ex12? I can compile the
> > example, but I did not find any runscript in the makefile. Do I need to
> go
> > through the entire source code to figure out how to run it?
> >
> > Thanks,
> >
> > Fande,
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151014/e75b7c43/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 15 00:17:48 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 15 Oct 2015 00:17:48 -0500
Subject: [petsc-users] how to run snes/examples/tutorials/ex12
In-Reply-To: <CAN5Wd-LSCdjZaWhk0gkhkB2maAOSyrxsoCVdiViPBVdHu-qhhg@mail.gmail.com>
References: <CAN5Wd-+nMDNW8Lf_L_bWAjj0MNG2LHjqyHHZcrTKeqZBroCX+w@mail.gmail.com>
	<alpine.LFD.2.20.1510142341080.19003@asterix>
	<CAN5Wd-LSCdjZaWhk0gkhkB2maAOSyrxsoCVdiViPBVdHu-qhhg@mail.gmail.com>
Message-ID: <2123B2B2-7402-4066-9369-66ECCC41D815@mcs.anl.gov>


> On Oct 15, 2015, at 12:13 AM, Fande Kong <fdkong.jd at gmail.com> wrote:
> 
> Satish,
> 
> Thanks. I found it. Any reason to put run commands in config/builder.py not in the local makefile.

  no

> How users could figure out? 

  They can't

> 
> Thanks,
> 
> Fande,
> 
> On Wed, Oct 14, 2015 at 10:42 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> looks like the run commands for this example are in config/builder.py
> 
> It might also need packages listed in config/examples/arch-c-exodus-dbg-builder.py
> 
> Satish
> 
> On Wed, 14 Oct 2015, Fande Kong wrote:
> 
> > Hi all,
> >
> > Anyone knows how to run snes/examples/tutorials/ex12? I can compile the
> > example, but I did not find any runscript in the makefile. Do I need to go
> > through the entire source code to figure out how to run it?
> >
> > Thanks,
> >
> > Fande,
> >
> 
> 


From timothee.nicolas at gmail.com  Thu Oct 15 00:26:51 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Thu, 15 Oct 2015 14:26:51 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
	<CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
	<E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>
Message-ID: <CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>

OK,

I ran an other battery of tests, here are the outputs. It seems to get a
bit better when refining more as you suggested. For instance, for one more
level of refinement, the CPU time saturation occurs for 5 levels instead of
3 previously. However the number of KSP iterations always tends to
(marginally) increase with the number of levels. But in the same time, it
always remain pretty low (less than 5 with extremely good convergence) so
maybe it is not really surprising ?

Timothee

2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:

>
>   Wow, quick response! Yes the times still indicate that after 4 levels
> you get no improvement in time.
>
> t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00,
> 5.7643e+00 ]
>
> I'll look more specifically at the numbers to see where the time is being
> transformed tomorrow when I am less drunk. It is a trade off between the
> work saved in the direct solve vs the work needed for the coarser levels in
> the multigrid cycle.
>
> Try refining the grid a couple more times, likely more levels will still
> help in that case
>
>  Ahh, you should also try -pc_mg_type full
>
>
>   Barry
>
>
>
> > On Oct 14, 2015, at 10:53 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > Richardson is 30-70% faster for these tests, but other than this I don't
> see any change.
> >
> > Timothee
> >
> >
> >
> > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> >   Timothee,
> >
> >      Thank you for reporting this issue, it is indeed disturbing and
> could be due to a performance regression we may have introduced by being
> too clever for our own good. Could you please rerun with the additional
> option -mg_levels_ksp_type richardson and send the same output?
> >
> >    Thanks
> >
> >   Barry
> >
> > > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > >
> > > Thank you Barry for pointing this out. Indeed on a system with no
> debugging the Jacobian evaluations no longer dominate the time (less than
> 10%). However the rest is similar, except the improvement from 2 to 3
> levels is much better. Still it saturates after levels=3. I understand it
> in terms of CPU time thanks to Matthew's explanations, however what
> surprises me more is that KSP iterations are not more efficient. At the
> least, even if it takes more time to have more levels because of memory
> issues, I would expect KSP iterations to converge more rapidly with more
> levels, but it is not the case as you can see. Probably there is also a
> rationale behind this but I cannot see easily.
> > >
> > > I send the new outputs
> > >
> > > Best
> > >
> > > Timothee
> > >
> > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > 1) Your timings are meaningless! You cannot compare timings when built
> with all debugging on, PERIOD!
> > >
> > >   ##########################################################
> > >       #                                                        #
> > >       #                          WARNING!!!                    #
> > >       #                                                        #
> > >       #   This code was compiled with a debugging option,      #
> > >       #   To get timing results run ./configure                #
> > >       #   using --with-debugging=no, the performance will      #
> > >       #   be generally two or three times faster.              #
> > >       #                                                        #
> > >       ##########################################################
> > >
> > > 2) Please run with -snes_view .
> > >
> > > 3) Note that with 7 levels
> > >
> > > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> > >
> > > with 2 levels
> > >
> > > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> > >
> > >
> > > The Jacobian evaluation is dominating the time! Likely if you fix the
> debugging this will be less the case
> > >
> > >   Barry
> > >
> > > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
> > > >
> > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
> -log_summary
> > > >
> > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > >
> > > > So there is a noticeable CPU time improvement from 2 levels to 3
> levels (30%), and then no improvement whatsoever. I am surprised because
> with 6 levels of refinement of the DMDA the fine grid has more than 1200
> points so with 3 levels the coarse grid still has more than 300 points
> which is still pretty large (I assume the ratio between grids is 2). I am
> wondering how the coarse solver efficiently solves the problem on the
> coarse grid with such a large number of points ? Given the principle of
> multigrid which is to erase the smooth part of the error with relaxation
> methods, which are usually efficient only for high frequency, I would
> expect optimal performance when the coarse grid is basically just a few
> points in each direction. Does anyone know why the performance saturates at
> low number of levels ? Basically what happens internally seems to be quite
> different from what I would expect...
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > >
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > >
> > >
> > >
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> >
> >
> >
> <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_2_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20100 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0020.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 21027 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0021.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_3_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 21027 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0022.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22071 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0023.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_4_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 21998 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0024.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22986 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0025.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_5_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 22997 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0026.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 23963 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0027.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_6_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 23923 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0028.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 24882 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0029.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_7_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 24899 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0030.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_8_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 25830 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0031.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_8_multigrid_levels_richardson_8_refine.log
Type: application/octet-stream
Size: 25819 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0032.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20024 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0033.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 20961 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0034.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 21872 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0035.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 22787 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0036.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 23686 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0037.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 24595 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0038.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log
Type: application/octet-stream
Size: 25490 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/9f16ed89/attachment-0039.obj>

From bsmith at mcs.anl.gov  Thu Oct 15 00:34:43 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 15 Oct 2015 00:34:43 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
	<CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
	<E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>
	<CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>
Message-ID: <6611ED6A-4AC4-4F96-B96B-08D3D991B2A8@mcs.anl.gov>


> On Oct 15, 2015, at 12:26 AM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> OK,
> 
> I ran an other battery of tests, here are the outputs. It seems to get a bit better when refining more as you suggested. For instance, for one more level of refinement, the CPU time saturation occurs for 5 levels instead of 3 previously. However the number of KSP iterations always tends to (marginally) increase with the number of levels. But in the same time, it always remain pretty low (less than 5 with extremely good convergence) so maybe it is not really surprising ?

  Yes, this is normal. You should not expect the number of outter KSP iterations to decrease with the number of levels, at best it will be the same. It will not decrease.

  Barry

> 
> Timothee
> 
> 2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 
>   Wow, quick response! Yes the times still indicate that after 4 levels you get no improvement in time.
> 
> t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00, 5.7643e+00 ]
> 
> I'll look more specifically at the numbers to see where the time is being transformed tomorrow when I am less drunk. It is a trade off between the work saved in the direct solve vs the work needed for the coarser levels in the multigrid cycle.
> 
> Try refining the grid a couple more times, likely more levels will still help in that case
> 
>  Ahh, you should also try -pc_mg_type full
> 
> 
>   Barry
> 
> 
> 
> > On Oct 14, 2015, at 10:53 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > Richardson is 30-70% faster for these tests, but other than this I don't see any change.
> >
> > Timothee
> >
> >
> >
> > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> >   Timothee,
> >
> >      Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?
> >
> >    Thanks
> >
> >   Barry
> >
> > > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> > >
> > > Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily.
> > >
> > > I send the new outputs
> > >
> > > Best
> > >
> > > Timothee
> > >
> > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> > >
> > >   ##########################################################
> > >       #                                                        #
> > >       #                          WARNING!!!                    #
> > >       #                                                        #
> > >       #   This code was compiled with a debugging option,      #
> > >       #   To get timing results run ./configure                #
> > >       #   using --with-debugging=no, the performance will      #
> > >       #   be generally two or three times faster.              #
> > >       #                                                        #
> > >       ##########################################################
> > >
> > > 2) Please run with -snes_view .
> > >
> > > 3) Note that with 7 levels
> > >
> > > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> > >
> > > with 2 levels
> > >
> > > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> > >
> > >
> > > The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> > >
> > >   Barry
> > >
> > > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> > > >
> > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> > > >
> > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > >
> > > > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > >
> > >
> > > <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> >
> >
> > <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
> 
> 
> <ex5_2_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_8_refine.log><ex5_4_multigrid_levels_richardson_7_refine.log><ex5_4_multigrid_levels_richardson_8_refine.log><ex5_5_multigrid_levels_richardson_7_refine.log><ex5_5_multigrid_levels_richardson_8_refine.log><ex5_6_multigrid_levels_richardson_7_refine.log><ex5_6_multigrid_levels_richardson_8_refine.log><ex5_7_multigrid_levels_richardson_7_refine.log><ex5_7_multigrid_levels_richardson_8_refine.log><ex5_8_multigrid_levels_richardson_7_refine.log><ex5_8_multigrid_levels_richardson_8_refine.log><ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log>


From knepley at gmail.com  Thu Oct 15 06:07:24 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 06:07:24 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
Message-ID: <CAMYG4GnemFSvA7MAWypbTbP3DNADSCkjxQY77j2FufKtE6dQAA@mail.gmail.com>

On Wed, Oct 14, 2015 at 9:32 PM, Timoth?e Nicolas <
timothee.nicolas at gmail.com> wrote:

> Thank you Barry for pointing this out. Indeed on a system with no
> debugging the Jacobian evaluations no longer dominate the time (less than
> 10%). However the rest is similar, except the improvement from 2 to 3
> levels is much better. Still it saturates after levels=3. I understand it
> in terms of CPU time thanks to Matthew's explanations, however what
> surprises me more is that KSP iterations are not more efficient. At the
> least, even if it takes more time to have more levels because of memory
> issues, I would expect KSP iterations to converge more rapidly with more
> levels, but it is not the case as you can see. Probably there is also a
> rationale behind this but I cannot see easily.
>

That conclusion makes no sense to me. Thought experiment:

  I have K levels, which means that on the coarsest level K I do a direct
solve. Now I add a level
  so that I have K+1. On level K-1, now instead using the result of a
direct solve as a starting guess,
  I use some iterative result. I cannot imagine that the iterates would go
down.

   Matt


> I send the new outputs
>
> Best
>
> Timothee
>
> 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
>
>> 1) Your timings are meaningless! You cannot compare timings when built
>> with all debugging on, PERIOD!
>>
>>   ##########################################################
>>       #                                                        #
>>       #                          WARNING!!!                    #
>>       #                                                        #
>>       #   This code was compiled with a debugging option,      #
>>       #   To get timing results run ./configure                #
>>       #   using --with-debugging=no, the performance will      #
>>       #   be generally two or three times faster.              #
>>       #                                                        #
>>       ##########################################################
>>
>> 2) Please run with -snes_view .
>>
>> 3) Note that with 7 levels
>>
>> SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
>>
>> with 2 levels
>>
>> SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00
>> 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
>>
>>
>> The Jacobian evaluation is dominating the time! Likely if you fix the
>> debugging this will be less the case
>>
>>   Barry
>>
>> > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <
>> timothee.nicolas at gmail.com> wrote:
>> >
>> > Dear all,
>> >
>> > I have been playing around with multigrid recently, namely with
>> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
>> with my own implementation of a laplacian type problem. In all cases, I
>> have noted no improvement whatsoever in the performance, whether in CPU
>> time or KSP iteration, by varying the number of levels of the multigrid
>> solver. As an example, I have attached the log_summary for ex5.c with
>> nlevels = 2 to 7, launched by
>> >
>> > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
>> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
>> -log_summary
>> >
>> > where -pc_mg_levels is set to a number between 2 and 7.
>> >
>> > So there is a noticeable CPU time improvement from 2 levels to 3 levels
>> (30%), and then no improvement whatsoever. I am surprised because with 6
>> levels of refinement of the DMDA the fine grid has more than 1200 points so
>> with 3 levels the coarse grid still has more than 300 points which is still
>> pretty large (I assume the ratio between grids is 2). I am wondering how
>> the coarse solver efficiently solves the problem on the coarse grid with
>> such a large number of points ? Given the principle of multigrid which is
>> to erase the smooth part of the error with relaxation methods, which are
>> usually efficient only for high frequency, I would expect optimal
>> performance when the coarse grid is basically just a few points in each
>> direction. Does anyone know why the performance saturates at low number of
>> levels ? Basically what happens internally seems to be quite different from
>> what I would expect...
>> >
>> > Best
>> >
>> > Timothee
>> >
>> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/226bb2b4/attachment.html>

From bikash at umich.edu  Thu Oct 15 08:20:38 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Thu, 15 Oct 2015 09:20:38 -0400
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
Message-ID: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>

HI,

I'm trying to evaluate B = P^T * A * P, where P is an MPIDENSE matrix and A
is an MPIAIJ matrix. Neither MatTransposeMatMult nor MatPtAP support such
an operation for the given type of matrices. I tried to look into
MATELEMENTAL class and didn't find any MatTransposeMatMult function for a
pair of Elemental matrices.

What would be a possible and efficient way to perform such the above
operation?

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/3b6d0b3e/attachment.html>

From knepley at gmail.com  Thu Oct 15 08:40:55 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 08:40:55 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
Message-ID: <CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>

On Thu, Oct 15, 2015 at 8:20 AM, Bikash Kanungo <bikash at umich.edu> wrote:

> HI,
>
> I'm trying to evaluate B = P^T * A * P, where P is an MPIDENSE matrix and
> A is an MPIAIJ matrix. Neither MatTransposeMatMult nor MatPtAP support such
> an operation for the given type of matrices. I tried to look into
> MATELEMENTAL class and didn't find any MatTransposeMatMult function for a
> pair of Elemental matrices.
>
> What would be a possible and efficient way to perform such the above
> operation?
>

Jed, is TAIJ the way to do this?

  Matt


> Thanks,
> Bikash
>
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/257353e4/attachment.html>

From knepley at gmail.com  Thu Oct 15 08:43:28 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 08:43:28 -0500
Subject: [petsc-users] Load balancing problem with DMDA
In-Reply-To: <5C930F9A-B2FA-4A87-9046-E61C6ED5B979@mcs.anl.gov>
References: <1444853498070.73407@kit.edu>
	<5C930F9A-B2FA-4A87-9046-E61C6ED5B979@mcs.anl.gov>
Message-ID: <CAMYG4G=bjnQQjREzLFFS=vE7LBNB4xCdMa5xKGRG06d2hcao3g@mail.gmail.com>

On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <jianjun.xiao at kit.edu>
> wrote:
> >
> > Hello,
> >
> > We have a finite volume code with structured grid. We use the DMDA data
> structure.
> >
> > Now we have the load balancing problem as shown in the attached picture.
> The domain is decomposed into 4 partitions. The sub-domain in black (proc
> 2) is solid which is of no interest. We would like to calculate only the
> fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted.
> Does DMDA now support multi-block with communication between blocks? I mean
> I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc
> 4. So the solid sub-domain does not exist. Any suggestion?
>
> No
>
> >
> > Or I have to do it using DMPLEX? Since data structure of the whole code
> is based on  DMDA, is there any "simple way" to switch from DMDA to DMPLEX
> without changing the loop structure in the code?
>
>   No
>
>   DMDA is for simple structured grids.


Using Plex would be some work since no one has ever used it to do FD. In
fact, now I think the way to do this
is to only represent the vertices and edges. I would help you if you want
to experiment with that.

  Thanks,

     Matt


>
> >
> > Thank you.
> >
> > Best regards
> > Jianjun
> > <pic.JPG>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/3f46fba2/attachment-0001.html>

From jianjun.xiao at kit.edu  Thu Oct 15 08:55:24 2015
From: jianjun.xiao at kit.edu (Xiao, Jianjun (IKET))
Date: Thu, 15 Oct 2015 13:55:24 +0000
Subject: [petsc-users] Load balancing problem with DMDA
In-Reply-To: <CAMYG4G=bjnQQjREzLFFS=vE7LBNB4xCdMa5xKGRG06d2hcao3g@mail.gmail.com>
References: <1444853498070.73407@kit.edu>
	<5C930F9A-B2FA-4A87-9046-E61C6ED5B979@mcs.anl.gov>,
	<CAMYG4G=bjnQQjREzLFFS=vE7LBNB4xCdMa5xKGRG06d2hcao3g@mail.gmail.com>
Message-ID: <1444917324695.60917@kit.edu>

Matt,


Yes. I would like to try. Where should I start?


________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Thursday, October 15, 2015 3:43 PM
To: Barry Smith
Cc: Xiao, Jianjun (IKET); petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Load balancing problem with DMDA

On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <bsmith at mcs.anl.gov<mailto:bsmith at mcs.anl.gov>> wrote:

> On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <jianjun.xiao at kit.edu<mailto:jianjun.xiao at kit.edu>> wrote:
>
> Hello,
>
> We have a finite volume code with structured grid. We use the DMDA data structure.
>
> Now we have the load balancing problem as shown in the attached picture. The domain is decomposed into 4 partitions. The sub-domain in black (proc 2) is solid which is of no interest. We would like to calculate only the fluid part in proc 1, 3 and 4 which means the resource in proc 2 is wasted. Does DMDA now support multi-block with communication between blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?

No

>
> Or I have to do it using DMPLEX? Since data structure of the whole code  is based on  DMDA, is there any "simple way" to switch from DMDA to DMPLEX without changing the loop structure in the code?

  No

  DMDA is for simple structured grids.

Using Plex would be some work since no one has ever used it to do FD. In fact, now I think the way to do this
is to only represent the vertices and edges. I would help you if you want to experiment with that.

  Thanks,

     Matt


>
> Thank you.
>
> Best regards
> Jianjun
> <pic.JPG>




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/bfd8009e/attachment.html>

From knepley at gmail.com  Thu Oct 15 09:45:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 09:45:01 -0500
Subject: [petsc-users] Load balancing problem with DMDA
In-Reply-To: <1444917324695.60917@kit.edu>
References: <1444853498070.73407@kit.edu>
	<5C930F9A-B2FA-4A87-9046-E61C6ED5B979@mcs.anl.gov>
	<CAMYG4G=bjnQQjREzLFFS=vE7LBNB4xCdMa5xKGRG06d2hcao3g@mail.gmail.com>
	<1444917324695.60917@kit.edu>
Message-ID: <CAMYG4GkkACja5zPJRQQA53uMmmMxO0DCd+MPQCTfR7yeTiSjug@mail.gmail.com>

On Thu, Oct 15, 2015 at 8:55 AM, Xiao, Jianjun (IKET) <jianjun.xiao at kit.edu>
wrote:

> Matt,
>
>
> Yes. I would like to try. Where should I start?
>
I would take a small 2D problem that you care about and try to make an
equivalent Plex code to the DMDA code.

I think it will be slower than DMDA, but trying a simple thing does not
hurt.

I can make you a simple 2D lattice in Plex.

  Thanks,

    Matt



> ------------------------------
> *From:* Matthew Knepley <knepley at gmail.com>
> *Sent:* Thursday, October 15, 2015 3:43 PM
> *To:* Barry Smith
> *Cc:* Xiao, Jianjun (IKET); petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Load balancing problem with DMDA
>
> On Wed, Oct 14, 2015 at 3:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> > On Oct 14, 2015, at 3:11 PM, Xiao, Jianjun (IKET) <jianjun.xiao at kit.edu>
>> wrote:
>> >
>> > Hello,
>> >
>> > We have a finite volume code with structured grid. We use the DMDA data
>> structure.
>> >
>> > Now we have the load balancing problem as shown in the attached
>> picture. The domain is decomposed into 4 partitions. The sub-domain in
>> black (proc 2) is solid which is of no interest. We would like to calculate
>> only the fluid part in proc 1, 3 and 4 which means the resource in proc 2
>> is wasted. Does DMDA now support multi-block with communication between
>> blocks? I mean I build block 1 and assign to proc1, block 2 to proc 3 and
>> block 3 to proc 4. So the solid sub-domain does not exist. Any suggestion?
>>
>> No
>>
>> >
>> > Or I have to do it using DMPLEX? Since data structure of the whole
>> code  is based on  DMDA, is there any "simple way" to switch from DMDA to
>> DMPLEX without changing the loop structure in the code?
>>
>>   No
>>
>>   DMDA is for simple structured grids.
>
>
> Using Plex would be some work since no one has ever used it to do FD. In
> fact, now I think the way to do this
> is to only represent the vertices and edges. I would help you if you want
> to experiment with that.
>
>   Thanks,
>
>      Matt
>
>
>>
>> >
>> > Thank you.
>> >
>> > Best regards
>> > Jianjun
>> > <pic.JPG>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/b5a250c7/attachment.html>

From knram06 at gmail.com  Thu Oct 15 10:43:15 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Thu, 15 Oct 2015 11:43:15 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
Message-ID: <CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>

Hello Barry,

The full error message is

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Argument out of range
[0]PETSC ERROR: Inserting a new nonzero at (25,27) in the matrix
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
[0]PETSC ERROR: /gpfs/work/(user name)/progs/PICC_Code/piccode on a
linux-debug named (host) by (user) Thu Oct 15 11:22:41 2015
[0]PETSC ERROR: Configure options --with-fc=mpif90 PETSC_ARCH=linux-debug
--with-debugging=1 --with-clanguage=cxx --with-cxx=mpicxx --with-cc=mpicc
--with-blas-lib="-L/usr/global/intel/mkl/11.0.0.079/lib/intel64 -lmkl_rt"
--with-lapack-lib="-L/usr/global/intel/mkl/11.0.0.079/lib/intel64 -lmkl_rt"
--with-shared-libraries --with-x=0 --download-hypre=1
[0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 484 in /gpfs/scratch/(user
name)/petsc-3.6.2/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #2 MatSetValues() line 1173 in /gpfs/scratch/(user
name)/petsc-3.6.2/src/mat/interface/matrix.c
[0]PETSC ERROR: #3 MatCopy_Basic() line 3722 in /gpfs/scratch/(user
name)/petsc-3.6.2/src/mat/interface/matrix.c
[0]PETSC ERROR: #4 MatCopy_SeqAIJ() line 2630 in /gpfs/scratch/(user
name)/petsc-3.6.2/src/mat/impls/aij/seq/aij.c
[0]PETSC ERROR: #5 MatCopy() line 3779 in /gpfs/scratch/(user
name)/petsc-3.6.2/src/mat/interface/matrix.c
48        MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
(gdb) n
49        MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);


I tried a DMDACreate1D where I specify the dimension as n^3, where n is the
number of nodes on one side of the cube, but I still get the same error.
The error is generated at the MatCopy line as mentioned earlier.

In parallel you will not be able to reuse some old code to "compute the
> Finite difference discretization of a 3D regular grid" so you might
> consider just discarding that old code and using what PETSc provides for
> computing the "Finite difference discretization of a 3D regular grid."
>

Ah ok, I had meant to say Finite difference discretization of the Poisson
equation.

If my parallel code could generate the sparse matrices formats for the
portion owned by each ranks, would there be a convenient way to generate
the PETSc matrices to be part of the DM? I had noticed that
CreateMatSeqAIJWithArrays changes the location of the Mat pointer passed in
for ComputeMatrix, so the actual matrix in DM remains as all zeros, so my
final solution comes out as zeros too.


Thanking You,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/4a5a4181/attachment.html>

From knepley at gmail.com  Thu Oct 15 10:46:39 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 10:46:39 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
Message-ID: <CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>

On Thu, Oct 15, 2015 at 10:43 AM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Barry,
>
> The full error message is
>
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Argument out of range
> [0]PETSC ERROR: Inserting a new nonzero at (25,27) in the matrix
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
> [0]PETSC ERROR: /gpfs/work/(user name)/progs/PICC_Code/piccode on a
> linux-debug named (host) by (user) Thu Oct 15 11:22:41 2015
> [0]PETSC ERROR: Configure options --with-fc=mpif90 PETSC_ARCH=linux-debug
> --with-debugging=1 --with-clanguage=cxx --with-cxx=mpicxx --with-cc=mpicc
> --with-blas-lib="-L/usr/global/intel/mkl/11.0.0.079/lib/intel64 -lmkl_rt"
> --with-lapack-lib="-L/usr/global/intel/mkl/11.0.0.079/lib/intel64
> -lmkl_rt" --with-shared-libraries --with-x=0 --download-hypre=1
> [0]PETSC ERROR: #1 MatSetValues_SeqAIJ() line 484 in /gpfs/scratch/(user
> name)/petsc-3.6.2/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #2 MatSetValues() line 1173 in /gpfs/scratch/(user
> name)/petsc-3.6.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: #3 MatCopy_Basic() line 3722 in /gpfs/scratch/(user
> name)/petsc-3.6.2/src/mat/interface/matrix.c
> [0]PETSC ERROR: #4 MatCopy_SeqAIJ() line 2630 in /gpfs/scratch/(user
> name)/petsc-3.6.2/src/mat/impls/aij/seq/aij.c
> [0]PETSC ERROR: #5 MatCopy() line 3779 in /gpfs/scratch/(user
> name)/petsc-3.6.2/src/mat/interface/matrix.c
> 48        MatAssemblyBegin(A, MAT_FINAL_ASSEMBLY);
> (gdb) n
> 49        MatAssemblyEnd(A, MAT_FINAL_ASSEMBLY);
>
>
> I tried a DMDACreate1D where I specify the dimension as n^3, where n is
> the number of nodes on one side of the cube, but I still get the same
> error. The error is generated at the MatCopy line as mentioned earlier.
>

This will not give you what you want since its got 1D connectivity.


> In parallel you will not be able to reuse some old code to "compute the
>> Finite difference discretization of a 3D regular grid" so you might
>> consider just discarding that old code and using what PETSc provides for
>> computing the "Finite difference discretization of a 3D regular grid."
>>
>
> Ah ok, I had meant to say Finite difference discretization of the Poisson
> equation.
>
> If my parallel code could generate the sparse matrices formats for the
> portion owned by each ranks, would there be a convenient way to generate
> the PETSc matrices to be part of the DM? I had noticed that
> CreateMatSeqAIJWithArrays changes the location of the Mat pointer passed in
> for ComputeMatrix, so the actual matrix in DM remains as all zeros, so my
> final solution comes out as zeros too.
>

This seems like the hardest way to do this. We have running examples, that
scale well, and produce exactly the matrix
you are using. In addition, they create the matrix in parallel, so the
whole thing is scalable.

In order to do it the way you want, you would need to match the
partitioning, which is hard.

  Matt

Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/0d0b07ed/attachment-0001.html>

From knram06 at gmail.com  Thu Oct 15 11:15:13 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Thu, 15 Oct 2015 12:15:13 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
	<CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
Message-ID: <CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>

Hello Matt,

On Thu, Oct 15, 2015 at 11:46 AM, Matthew Knepley <knepley at gmail.com> wrote:

>
> This seems like the hardest way to do this. We have running examples, that
> scale well, and produce exactly the matrix
> you are using. In addition, they create the matrix in parallel, so the
> whole thing is scalable.
>
> In order to do it the way you want, you would need to match the
> partitioning, which is hard.
>

Yes, I understand. I am trying to keep the solver module separate from the
problem physics (like boundary conditions), i.e. my code has knowledge of
the physics and I generate the matrix in sparse format. I then hand it over
to some Solver module (say, PETSc) which can efficiently solve this.

Examples like snes/ex5.c, ksp/ex45.c, etc. use knowledge of the boundary
conditions to fill up the matrix and the RHS and this makes sense, if the
entire code is written using PETSc's native data structures. But from a
modularity point of view, if my problem needs a different partitioning
based on the physics, I feel like my code should generate the relevant
matrices on each rank and send it to PETSc.

I am trying to simulate the movement of charged particles in a domain, and
these particles become collimated along a particular direction. So one way
to partition might be 1D slabs perpendicular to the collimated direction,
so each rank can calculate the field and move the particles on each
timestep. Of course communication across the slabs would be inevitable. But
this way, the partitioning is based on the physics of the problem and the
solver module just efficiently solves the system without having to know the
boundary conditions, which makes sense to me overall.

Hope this description helps.


Thanking You,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/b760589f/attachment.html>

From knepley at gmail.com  Thu Oct 15 11:18:44 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 11:18:44 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
	<CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
	<CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
Message-ID: <CAMYG4Gnbtp_CW0+DcbfxGGuTWsT5VHm7afdbmCMyixQOJ1q6Rg@mail.gmail.com>

On Thu, Oct 15, 2015 at 11:15 AM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Matt,
>
> On Thu, Oct 15, 2015 at 11:46 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>>
>> This seems like the hardest way to do this. We have running examples,
>> that scale well, and produce exactly the matrix
>> you are using. In addition, they create the matrix in parallel, so the
>> whole thing is scalable.
>>
>> In order to do it the way you want, you would need to match the
>> partitioning, which is hard.
>>
>
> Yes, I understand. I am trying to keep the solver module separate from the
> problem physics (like boundary conditions), i.e. my code has knowledge of
> the physics and I generate the matrix in sparse format. I then hand it over
> to some Solver module (say, PETSc) which can efficiently solve this.
>
> Examples like snes/ex5.c, ksp/ex45.c, etc. use knowledge of the boundary
> conditions to fill up the matrix and the RHS and this makes sense, if the
> entire code is written using PETSc's native data structures. But from a
> modularity point of view, if my problem needs a different partitioning
> based on the physics, I feel like my code should generate the relevant
> matrices on each rank and send it to PETSc.
>
> I am trying to simulate the movement of charged particles in a domain, and
> these particles become collimated along a particular direction. So one way
> to partition might be 1D slabs perpendicular to the collimated direction,
> so each rank can calculate the field and move the particles on each
> timestep. Of course communication across the slabs would be inevitable. But
> this way, the partitioning is based on the physics of the problem and the
> solver module just efficiently solves the system without having to know the
> boundary conditions, which makes sense to me overall.
>
> Hope this description helps.
>

The bad design decision is using a storage format designed for high speed
MatMult() to use for data interchange. You should be
using the API for value insertion rather than some storage format.

  Matt


>
> Thanking You,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/74b19894/attachment.html>

From knram06 at gmail.com  Thu Oct 15 12:10:01 2015
From: knram06 at gmail.com (K. N. Ramachandran)
Date: Thu, 15 Oct 2015 13:10:01 -0400
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAMYG4Gnbtp_CW0+DcbfxGGuTWsT5VHm7afdbmCMyixQOJ1q6Rg@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
	<CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
	<CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
	<CAMYG4Gnbtp_CW0+DcbfxGGuTWsT5VHm7afdbmCMyixQOJ1q6Rg@mail.gmail.com>
Message-ID: <CAP8=CDZckGxe5TyapiRgCzF0FXkt1EsTDyKeohZjd6Yt4RbGcQ@mail.gmail.com>

Hello Matt,

On Thu, Oct 15, 2015 at 12:18 PM, Matthew Knepley <knepley at gmail.com> wrote:

>
> The bad design decision is using a storage format designed for high speed
> MatMult() to use for data interchange. You should be
> using the API for value insertion rather than some storage format.
>

Ok, noted. Instead of using CreateMatSeqAIJWithArrays, I have also tried
setting up the Mat passed into the ComputeMatrix function, using
MatSetValue where the cols and values are referred to values in the CSR
format, but I still get the same error. It looks like the Mat is not
preallocated properly in the DM, even when I use DMDACreate3d.

Could you suggest what I can do to resolve this?

Thanks,
K.N.Ramachandran
Ph: 814-441-4279
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/f7312e00/attachment.html>

From knepley at gmail.com  Thu Oct 15 12:12:45 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 12:12:45 -0500
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDZckGxe5TyapiRgCzF0FXkt1EsTDyKeohZjd6Yt4RbGcQ@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
	<CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
	<CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
	<CAMYG4Gnbtp_CW0+DcbfxGGuTWsT5VHm7afdbmCMyixQOJ1q6Rg@mail.gmail.com>
	<CAP8=CDZckGxe5TyapiRgCzF0FXkt1EsTDyKeohZjd6Yt4RbGcQ@mail.gmail.com>
Message-ID: <CAMYG4GkB28_ZLk+hQi8sz-w=tW+5syxgLZkad4FG6DG9YBJaWA@mail.gmail.com>

On Thu, Oct 15, 2015 at 12:10 PM, K. N. Ramachandran <knram06 at gmail.com>
wrote:

> Hello Matt,
>
> On Thu, Oct 15, 2015 at 12:18 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
>>
>> The bad design decision is using a storage format designed for high speed
>> MatMult() to use for data interchange. You should be
>> using the API for value insertion rather than some storage format.
>>
>
> Ok, noted. Instead of using CreateMatSeqAIJWithArrays, I have also tried
> setting up the Mat passed into the ComputeMatrix function, using
> MatSetValue where the cols and values are referred to values in the CSR
> format, but I still get the same error. It looks like the Mat is not
> preallocated properly in the DM, even when I use DMDACreate3d.
>

Another explanation is that the matrix is preallocated correctly, but you
are mapping the row and columns indices differently than the
DM. This what I was refering to when I said that getting the parallel
partition correct would be challenging.

  Matt


> Could you suggest what I can do to resolve this?
>
> Thanks,
> K.N.Ramachandran
> Ph: 814-441-4279
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/6ce30452/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 15 13:34:46 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 15 Oct 2015 13:34:46 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
Message-ID: <EF48F511-A8BD-4D6F-9AC8-1E9B5A9B1A56@mcs.anl.gov>


> On Oct 15, 2015, at 8:40 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Oct 15, 2015 at 8:20 AM, Bikash Kanungo <bikash at umich.edu> wrote:
> HI,
> 
> I'm trying to evaluate B = P^T * A * P, where P is an MPIDENSE matrix and A is an MPIAIJ matrix. Neither MatTransposeMatMult nor MatPtAP support such an operation for the given type of matrices. I tried to look into MATELEMENTAL class and didn't find any MatTransposeMatMult function for a pair of Elemental matrices. 
> 
> What would be a possible and efficient way to perform such the above operation?

   Hong will see about providing the missing piece.  What are you sizes of the matrix P? Presumably it has many rows but few columns?

  Barry

> 
> Jed, is TAIJ the way to do this?
> 
>   Matt
>  
> Thanks,
> Bikash 
> 
> -- 
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering 
> University of Michigan
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener


From jed at jedbrown.org  Thu Oct 15 13:36:43 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 15 Oct 2015 12:36:43 -0600
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
Message-ID: <87eggwashw.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> Jed, is TAIJ the way to do this?

No, TAIJ is unrelated.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/e87383ab/attachment.pgp>

From knepley at gmail.com  Thu Oct 15 13:41:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 13:41:56 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <87eggwashw.fsf@jedbrown.org>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
Message-ID: <CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>

On Thu, Oct 15, 2015 at 1:36 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > Jed, is TAIJ the way to do this?
>
> No, TAIJ is unrelated.
>

I do not understand "unrelated". My understanding was that TAIJ could be
used (with T = I)
to get the action of A on a set of vectors, which I think would be A P. Why
specifically would
you not use it?

  Thanks,

     Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/b0c2c747/attachment.html>

From jed at jedbrown.org  Thu Oct 15 13:49:42 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 15 Oct 2015 12:49:42 -0600
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
Message-ID: <876128arw9.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:
> I do not understand "unrelated". My understanding was that TAIJ could be
> used (with T = I)
> to get the action of A on a set of vectors, which I think would be A P. Why
> specifically would
> you not use it?

MAIJ is enough for that; it would involve packing P into a Vec, but
Bikash might as well use MatMatMult to perform the same operation in a
more direct way (MatMatMult_MPIAIJ_MPIDense exists).  The problem is
that MatTransposeMatMult_MPIDense_MPIDense needs to be implemented.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/86c6c000/attachment.pgp>

From jed at jedbrown.org  Thu Oct 15 13:56:13 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 15 Oct 2015 12:56:13 -0600
Subject: [petsc-users] Diagnosing Poisson Solver Behavior
In-Reply-To: <CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
References: <CAP8=CDbgZvf112D6S0SRnVATthg3U27ZwAnD8mnyjJ+OpAQbSQ@mail.gmail.com>
	<CAP8=CDbGmKtdEJZqN9H8ZHsR3+57jJLkJKkuufouniBwFvSSbw@mail.gmail.com>
	<CAMYG4Gk6x9cNL-S7-NznmcuBS9f2KMNQmmWLEoVW2paXKdoqeQ@mail.gmail.com>
	<CAP8=CDYEh1sq4TxqMz_5R1_61YNWJ76aGrrXVLr9+LFh1euDzA@mail.gmail.com>
	<CAP8=CDY8g=55CHEtPK7=nMUgQvemXcG_3+3+7MAxW7jWpoZ7mg@mail.gmail.com>
	<CAMYG4GkfonpWdaH3Naou_2KOVZBRFTYYeXTuA2p-DWk=_kJ-3A@mail.gmail.com>
	<CAP8=CDYJ7gnPXNnUQzEb82vkdfV=O_HXV5UburFnL-GjoBfzuw@mail.gmail.com>
	<CAMYG4Gn7-vA7jwpv1mn9qxiQ3XsLbbm0wjKbjsPr_Y0vkcNzwA@mail.gmail.com>
	<2F55BB27-1208-4CBF-B065-C2A0697DB7EC@mcs.anl.gov>
	<CAP8=CDYUwLeaCHPWr6Thht_d+VgFg6q93opm1Zr6hzUx19eYKQ@mail.gmail.com>
	<FF20CBFE-D64E-4B1B-96AA-4242E41E0AA3@mcs.anl.gov>
	<CAP8=CDaOV1JKxtcA_4jM31K=x1bCDN3-JbDcT536oCv+WV-erA@mail.gmail.com>
	<CAMYG4GnoXys0sqAok+yKe3vJm-kvbZV=b5KJVWLiWcTQS-OK+w@mail.gmail.com>
	<CAP8=CDYdcPKvO7Z8u7Aypjh=nrdPK=-Vtm=TX=w6c7vMy9Bzxw@mail.gmail.com>
Message-ID: <8737xcarle.fsf@jedbrown.org>

"K. N. Ramachandran" <knram06 at gmail.com> writes:
> I am trying to simulate the movement of charged particles in a domain, and
> these particles become collimated along a particular direction. So one way
> to partition might be 1D slabs perpendicular to the collimated direction,
> so each rank can calculate the field and move the particles on each
> timestep. Of course communication across the slabs would be inevitable. But
> this way, the partitioning is based on the physics of the problem and the
> solver module just efficiently solves the system without having to know the
> boundary conditions, which makes sense to me overall.

I don't understand what you mean by "without having to know the boundary
conditions".  The boundary conditions are essential to solving
equations.  Of course if you're assembling a matrix, they need to be
part of that matrix.  (Usually the matrix will be singular "without"
boundary conditions.)

As for using a 1D partition, that may be very inefficient for the
solver, both due to limited scalability of the partition and more
interestingly if the directions of strong coupling do not lie entirely
within your planes and completely swamp the out-of-plane coupling
strength, then your solver can be expected to converge more slowly.

Note that DMDA can be instructed to produce a 1D partition if you like.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/e3a41da6/attachment-0001.pgp>

From hzhang at mcs.anl.gov  Thu Oct 15 14:14:07 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Thu, 15 Oct 2015 14:14:07 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <876128arw9.fsf@jedbrown.org>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
Message-ID: <CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>

I plan to implement MatTransposeMatMult_MPIDense_MPIDense via

1. add MatTransposeMatMult_elemental_elemental()
2. C_dense = P_dense^T * B_dense
    via MatConvert_dense_elemental() and MatConvert_elemental_dense()

Let me know if you have better suggestions.

Hong

On Thu, Oct 15, 2015 at 1:49 PM, Jed Brown <jed at jedbrown.org> wrote:

> Matthew Knepley <knepley at gmail.com> writes:
> > I do not understand "unrelated". My understanding was that TAIJ could be
> > used (with T = I)
> > to get the action of A on a set of vectors, which I think would be A P.
> Why
> > specifically would
> > you not use it?
>
> MAIJ is enough for that; it would involve packing P into a Vec, but
> Bikash might as well use MatMatMult to perform the same operation in a
> more direct way (MatMatMult_MPIAIJ_MPIDense exists).  The problem is
> that MatTransposeMatMult_MPIDense_MPIDense needs to be implemented.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/65dd92a6/attachment.html>

From jed at jedbrown.org  Thu Oct 15 14:57:41 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 15 Oct 2015 13:57:41 -0600
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
	<CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
Message-ID: <87r3kvaoqy.fsf@jedbrown.org>

r<#secure method=pgpmime mode=sign>
Hong <hzhang at mcs.anl.gov> writes:

> I plan to implement MatTransposeMatMult_MPIDense_MPIDense via
>
> 1. add MatTransposeMatMult_elemental_elemental()
> 2. C_dense = P_dense^T * B_dense
>     via MatConvert_dense_elemental() and MatConvert_elemental_dense()

The above involves a ton of data movement and MPIDense is a logical
distribution for matrices with a modest number of columns.  I think I
would just do the local GEMM and then MPI_Reduce_scatter it.

From bsmith at mcs.anl.gov  Thu Oct 15 16:01:31 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 15 Oct 2015 16:01:31 -0500
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
	<CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
	<E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>
	<CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>
Message-ID: <B63C79FE-2AB7-4AD0-9492-E5C7F8108F8E@mcs.anl.gov>


  I guess the -pc_mg_type full is the best you are going to get. In parallel the coarser grids are problematic because they have little local work but still communication.

  Barry


> On Oct 15, 2015, at 12:26 AM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> 
> OK,
> 
> I ran an other battery of tests, here are the outputs. It seems to get a bit better when refining more as you suggested. For instance, for one more level of refinement, the CPU time saturation occurs for 5 levels instead of 3 previously. However the number of KSP iterations always tends to (marginally) increase with the number of levels. But in the same time, it always remain pretty low (less than 5 with extremely good convergence) so maybe it is not really surprising ?
> 
> Timothee
> 
> 2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> 
>   Wow, quick response! Yes the times still indicate that after 4 levels you get no improvement in time.
> 
> t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00, 5.7643e+00 ]
> 
> I'll look more specifically at the numbers to see where the time is being transformed tomorrow when I am less drunk. It is a trade off between the work saved in the direct solve vs the work needed for the coarser levels in the multigrid cycle.
> 
> Try refining the grid a couple more times, likely more levels will still help in that case
> 
>  Ahh, you should also try -pc_mg_type full
> 
> 
>   Barry
> 
> 
> 
> > On Oct 14, 2015, at 10:53 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > Richardson is 30-70% faster for these tests, but other than this I don't see any change.
> >
> > Timothee
> >
> >
> >
> > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> >   Timothee,
> >
> >      Thank you for reporting this issue, it is indeed disturbing and could be due to a performance regression we may have introduced by being too clever for our own good. Could you please rerun with the additional option -mg_levels_ksp_type richardson and send the same output?
> >
> >    Thanks
> >
> >   Barry
> >
> > > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> > >
> > > Thank you Barry for pointing this out. Indeed on a system with no debugging the Jacobian evaluations no longer dominate the time (less than 10%). However the rest is similar, except the improvement from 2 to 3 levels is much better. Still it saturates after levels=3. I understand it in terms of CPU time thanks to Matthew's explanations, however what surprises me more is that KSP iterations are not more efficient. At the least, even if it takes more time to have more levels because of memory issues, I would expect KSP iterations to converge more rapidly with more levels, but it is not the case as you can see. Probably there is also a rationale behind this but I cannot see easily.
> > >
> > > I send the new outputs
> > >
> > > Best
> > >
> > > Timothee
> > >
> > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > 1) Your timings are meaningless! You cannot compare timings when built with all debugging on, PERIOD!
> > >
> > >   ##########################################################
> > >       #                                                        #
> > >       #                          WARNING!!!                    #
> > >       #                                                        #
> > >       #   This code was compiled with a debugging option,      #
> > >       #   To get timing results run ./configure                #
> > >       #   using --with-debugging=no, the performance will      #
> > >       #   be generally two or three times faster.              #
> > >       #                                                        #
> > >       ##########################################################
> > >
> > > 2) Please run with -snes_view .
> > >
> > > 3) Note that with 7 levels
> > >
> > > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> > >
> > > with 2 levels
> > >
> > > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> > >
> > >
> > > The Jacobian evaluation is dominating the time! Likely if you fix the debugging this will be less the case
> > >
> > >   Barry
> > >
> > > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Dear all,
> > > >
> > > > I have been playing around with multigrid recently, namely with /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and with my own implementation of a laplacian type problem. In all cases, I have noted no improvement whatsoever in the performance, whether in CPU time or KSP iteration, by varying the number of levels of the multigrid solver. As an example, I have attached the log_summary for ex5.c with nlevels = 2 to 7, launched by
> > > >
> > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9 -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor -log_summary
> > > >
> > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > >
> > > > So there is a noticeable CPU time improvement from 2 levels to 3 levels (30%), and then no improvement whatsoever. I am surprised because with 6 levels of refinement of the DMDA the fine grid has more than 1200 points so with 3 levels the coarse grid still has more than 300 points which is still pretty large (I assume the ratio between grids is 2). I am wondering how the coarse solver efficiently solves the problem on the coarse grid with such a large number of points ? Given the principle of multigrid which is to erase the smooth part of the error with relaxation methods, which are usually efficient only for high frequency, I would expect optimal performance when the coarse grid is basically just a few points in each direction. Does anyone know why the performance saturates at low number of levels ? Basically what happens internally seems to be quite different from what I would expect...
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > > <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > >
> > >
> > > <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> >
> >
> > <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
> 
> 
> <ex5_2_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_8_refine.log><ex5_4_multigrid_levels_richardson_7_refine.log><ex5_4_multigrid_levels_richardson_8_refine.log><ex5_5_multigrid_levels_richardson_7_refine.log><ex5_5_multigrid_levels_richardson_8_refine.log><ex5_6_multigrid_levels_richardson_7_refine.log><ex5_6_multigrid_levels_richardson_8_refine.log><ex5_7_multigrid_levels_richardson_7_refine.log><ex5_7_multigrid_levels_richardson_8_refine.log><ex5_8_multigrid_levels_richardson_7_refine.log><ex5_8_multigrid_levels_richardson_8_refine.log><ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log>


From timothee.nicolas at gmail.com  Thu Oct 15 19:44:04 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Fri, 16 Oct 2015 09:44:04 +0900
Subject: [petsc-users] Number of levels of multigrid : 2-3 is sufficient
	??
In-Reply-To: <B63C79FE-2AB7-4AD0-9492-E5C7F8108F8E@mcs.anl.gov>
References: <CAGi1ndQyj82YgZ2-Dvr67UvemTs9iL_xqH-4q4PNgf8Eu3KD0Q@mail.gmail.com>
	<71A2EB98-FB16-49E9-9F77-5E7F32943AE3@mcs.anl.gov>
	<CAGi1ndRTtQQZwu1+wg5Dhw53k51EkmB1zNvHGDSdDMON_XJL_g@mail.gmail.com>
	<F2245717-E028-4132-8552-00BD3EC14C89@mcs.anl.gov>
	<CAGi1ndTXdrdeAhCrOOXu74z6e8Mpq5Ni=CCmFsRskqEfiXwxwA@mail.gmail.com>
	<E74F45F8-CC86-4A1B-B038-8E8F0C6720DC@mcs.anl.gov>
	<CAGi1ndSUY15uNJVBCGjMf9LXYi8=kXvNrRqXHcYs8P3TyRT9aQ@mail.gmail.com>
	<B63C79FE-2AB7-4AD0-9492-E5C7F8108F8E@mcs.anl.gov>
Message-ID: <CAGi1ndTVte2KccJxpU3vqK+VEFPQ6fsDznRUaKHG2fjeWoc02w@mail.gmail.com>

OK, perfect, thank you.

2015-10-16 6:01 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:

>
>   I guess the -pc_mg_type full is the best you are going to get. In
> parallel the coarser grids are problematic because they have little local
> work but still communication.
>
>   Barry
>
>
> > On Oct 15, 2015, at 12:26 AM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> >
> > OK,
> >
> > I ran an other battery of tests, here are the outputs. It seems to get a
> bit better when refining more as you suggested. For instance, for one more
> level of refinement, the CPU time saturation occurs for 5 levels instead of
> 3 previously. However the number of KSP iterations always tends to
> (marginally) increase with the number of levels. But in the same time, it
> always remain pretty low (less than 5 with extremely good convergence) so
> maybe it is not really surprising ?
> >
> > Timothee
> >
> > 2015-10-15 13:15 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> >
> >   Wow, quick response! Yes the times still indicate that after 4 levels
> you get no improvement in time.
> >
> > t = [1.5629e+01 , 6.2692e+00, 5.3451e+00, 5.4948e+00, 5.4940e+00,
> 5.7643e+00 ]
> >
> > I'll look more specifically at the numbers to see where the time is
> being transformed tomorrow when I am less drunk. It is a trade off between
> the work saved in the direct solve vs the work needed for the coarser
> levels in the multigrid cycle.
> >
> > Try refining the grid a couple more times, likely more levels will still
> help in that case
> >
> >  Ahh, you should also try -pc_mg_type full
> >
> >
> >   Barry
> >
> >
> >
> > > On Oct 14, 2015, at 10:53 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > >
> > > OK,
> > >
> > > Richardson is 30-70% faster for these tests, but other than this I
> don't see any change.
> > >
> > > Timothee
> > >
> > >
> > >
> > > 2015-10-15 12:37 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > >
> > >   Timothee,
> > >
> > >      Thank you for reporting this issue, it is indeed disturbing and
> could be due to a performance regression we may have introduced by being
> too clever for our own good. Could you please rerun with the additional
> option -mg_levels_ksp_type richardson and send the same output?
> > >
> > >    Thanks
> > >
> > >   Barry
> > >
> > > > On Oct 14, 2015, at 9:32 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > > >
> > > > Thank you Barry for pointing this out. Indeed on a system with no
> debugging the Jacobian evaluations no longer dominate the time (less than
> 10%). However the rest is similar, except the improvement from 2 to 3
> levels is much better. Still it saturates after levels=3. I understand it
> in terms of CPU time thanks to Matthew's explanations, however what
> surprises me more is that KSP iterations are not more efficient. At the
> least, even if it takes more time to have more levels because of memory
> issues, I would expect KSP iterations to converge more rapidly with more
> levels, but it is not the case as you can see. Probably there is also a
> rationale behind this but I cannot see easily.
> > > >
> > > > I send the new outputs
> > > >
> > > > Best
> > > >
> > > > Timothee
> > > >
> > > > 2015-10-15 3:02 GMT+09:00 Barry Smith <bsmith at mcs.anl.gov>:
> > > > 1) Your timings are meaningless! You cannot compare timings when
> built with all debugging on, PERIOD!
> > > >
> > > >   ##########################################################
> > > >       #                                                        #
> > > >       #                          WARNING!!!                    #
> > > >       #                                                        #
> > > >       #   This code was compiled with a debugging option,      #
> > > >       #   To get timing results run ./configure                #
> > > >       #   using --with-debugging=no, the performance will      #
> > > >       #   be generally two or three times faster.              #
> > > >       #                                                        #
> > > >       ##########################################################
> > > >
> > > > 2) Please run with -snes_view .
> > > >
> > > > 3) Note that with 7 levels
> > > >
> > > > SNESJacobianEval      21 1.0 2.4364e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 54  0  0  0  0  54  0  0  0  0     0
> > > >
> > > > with 2 levels
> > > >
> > > > SNESJacobianEval       6 1.0 2.2441e+01 1.0 0.00e+00 0.0 0.0e+00
> 0.0e+00 0.0e+00 34  0  0  0  0  34  0  0  0  0     0
> > > >
> > > >
> > > > The Jacobian evaluation is dominating the time! Likely if you fix
> the debugging this will be less the case
> > > >
> > > >   Barry
> > > >
> > > > > On Oct 13, 2015, at 9:23 PM, Timoth?e Nicolas <
> timothee.nicolas at gmail.com> wrote:
> > > > >
> > > > > Dear all,
> > > > >
> > > > > I have been playing around with multigrid recently, namely with
> /ksp/ksp/examples/tutorials/ex42.c, with /snes/examples/tutorial/ex5.c and
> with my own implementation of a laplacian type problem. In all cases, I
> have noted no improvement whatsoever in the performance, whether in CPU
> time or KSP iteration, by varying the number of levels of the multigrid
> solver. As an example, I have attached the log_summary for ex5.c with
> nlevels = 2 to 7, launched by
> > > > >
> > > > > mpiexec -n 1 ./ex5 -da_grid_x 21 -da_grid_y 21 -ksp_rtol 1.0e-9
> -da_refine 6 -pc_type mg -pc_mg_levels # -snes_monitor -ksp_monitor
> -log_summary
> > > > >
> > > > > where -pc_mg_levels is set to a number between 2 and 7.
> > > > >
> > > > > So there is a noticeable CPU time improvement from 2 levels to 3
> levels (30%), and then no improvement whatsoever. I am surprised because
> with 6 levels of refinement of the DMDA the fine grid has more than 1200
> points so with 3 levels the coarse grid still has more than 300 points
> which is still pretty large (I assume the ratio between grids is 2). I am
> wondering how the coarse solver efficiently solves the problem on the
> coarse grid with such a large number of points ? Given the principle of
> multigrid which is to erase the smooth part of the error with relaxation
> methods, which are usually efficient only for high frequency, I would
> expect optimal performance when the coarse grid is basically just a few
> points in each direction. Does anyone know why the performance saturates at
> low number of levels ? Basically what happens internally seems to be quite
> different from what I would expect...
> > > > >
> > > > > Best
> > > > >
> > > > > Timothee
> > > > >
> <ex5_2_levels_of_multigrid.log><ex5_3_levels_of_multigrid.log><ex5_4_levels_of_multigrid.log><ex5_5_levels_of_multigrid.log><ex5_6_levels_of_multigrid.log><ex5_7_levels_of_multigrid.log>
> > > >
> > > >
> > > >
> <ex5_2_multigrid_levels.log><ex5_3_multigrid_levels.log><ex5_4_multigrid_levels.log><ex5_5_multigrid_levels.log><ex5_6_multigrid_levels.log><ex5_7_multigrid_levels.log>
> > >
> > >
> > >
> <ex5_2_multigrid_levels_richardson.log><ex5_3_multigrid_levels_richardson.log><ex5_4_multigrid_levels_richardson.log><ex5_5_multigrid_levels_richardson.log><ex5_6_multigrid_levels_richardson.log><ex5_7_multigrid_levels_richardson.log>
> >
> >
> >
> <ex5_2_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_7_refine.log><ex5_3_multigrid_levels_richardson_8_refine.log><ex5_4_multigrid_levels_richardson_7_refine.log><ex5_4_multigrid_levels_richardson_8_refine.log><ex5_5_multigrid_levels_richardson_7_refine.log><ex5_5_multigrid_levels_richardson_8_refine.log><ex5_6_multigrid_levels_richardson_7_refine.log><ex5_6_multigrid_levels_richardson_8_refine.log><ex5_7_multigrid_levels_richardson_7_refine.log><ex5_7_multigrid_levels_richardson_8_refine.log><ex5_8_multigrid_levels_richardson_7_refine.log><ex5_8_multigrid_levels_richardson_8_refine.log><ex5_pc_mg_full_2_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_3_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_4_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_5_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_6_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_7_multigrid_levels_richardson_7_refine.log><ex5_pc_mg_full_8_multigrid_levels_richardson_7_refine.log>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/5ffbf25a/attachment-0001.html>

From a.croucher at auckland.ac.nz  Thu Oct 15 22:51:30 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Fri, 16 Oct 2015 16:51:30 +1300
Subject: [petsc-users] HDF5 vec output with DMPlex
Message-ID: <56207442.8090304@auckland.ac.nz>

hi,

I am wanting to output time-stepped simulation results to an HDF5 file, 
from a PETSc Vec created on a DMPlex using DMCreateGlobalVector().

I tried doing it the way it's done in 
src/vec/vec/examples/tutorials/ex19.c, using PetscViewerHDF5Open(), 
PetscViewerHDF5PushGroup() and PetscViewerHDF5SetTimestep(), but the 
resulting HDF5 file only ever seems to have results for the last time 
step in it.

It works OK for a Vec created simply using VecCreate(), as in the ex19 
example. Is there any reason it shouldn't work with DMPlex?

Cheers, Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611


From knepley at gmail.com  Thu Oct 15 22:54:54 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 15 Oct 2015 22:54:54 -0500
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <56207442.8090304@auckland.ac.nz>
References: <56207442.8090304@auckland.ac.nz>
Message-ID: <CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>

On Thu, Oct 15, 2015 at 10:51 PM, Adrian Croucher <a.croucher at auckland.ac.nz
> wrote:

> hi,
>
> I am wanting to output time-stepped simulation results to an HDF5 file,
> from a PETSc Vec created on a DMPlex using DMCreateGlobalVector().
>
> I tried doing it the way it's done in
> src/vec/vec/examples/tutorials/ex19.c, using PetscViewerHDF5Open(),
> PetscViewerHDF5PushGroup() and PetscViewerHDF5SetTimestep(), but the
> resulting HDF5 file only ever seems to have results for the last time step
> in it.
>
> It works OK for a Vec created simply using VecCreate(), as in the ex19
> example. Is there any reason it shouldn't work with DMPlex?
>

No, that should work. This is how we do it in PyLith. There must be some
setup problem. When you look at the HDF5 file. Does
it have a dimension for timestep?

  Thanks,

    Matt


> Cheers, Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151015/bd156052/attachment.html>

From a.croucher at auckland.ac.nz  Thu Oct 15 23:14:53 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Fri, 16 Oct 2015 17:14:53 +1300
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>
References: <56207442.8090304@auckland.ac.nz>
	<CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>
Message-ID: <562079BD.4090700@auckland.ac.nz>

hi

On 16/10/15 16:54, Matthew Knepley wrote:
>
> No, that should work. This is how we do it in PyLith. There must be 
> some setup problem. When you look at the HDF5 file. Does
> it have a dimension for timestep?

No, it doesn't.

Is there any extra setup I might need to do to get it to work? I've 
attached a minimal example code, mesh and output.

Cheers, Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/84978acb/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: testview.F90
Type: text/x-fortran
Size: 1746 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/84978acb/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: column.exo
Type: application/octet-stream
Size: 14676 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/84978acb/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: view.h5
Type: application/x-hdf
Size: 11048 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/84978acb/attachment-0001.hdf>

From bikash at umich.edu  Fri Oct 16 07:52:20 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Fri, 16 Oct 2015 08:52:20 -0400
Subject: [petsc-users] MatGetSubMatrices or VecCreateGhost
Message-ID: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>

Hi,

I need to multiply a matrix A with several vectors v_i's and use the
resultant vectors in my calculation which involves access to non-local
nodes. I can think of two possible ways to do it:

1. Store the vectors v_i's into an MPIDENSE matrix and perform MatMatMult
with A to get a resultant dense matrix B. I can now use MatGetSubMatrices
on B to access the non-local part required in my subsequent calculations.

2. Use MatMult to evaluate A*v_i for each of the vectors and provide ghost
padding through VecCreateGhost for the resultant vector.

Which of the two is a more efficient way to perform the task? Also, if
there is a better way possible to achieve the same, then kindly let me know.

Thanks,
Bikash

-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/c9c8b6a8/attachment.html>

From knepley at gmail.com  Fri Oct 16 09:01:57 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 16 Oct 2015 09:01:57 -0500
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <562079BD.4090700@auckland.ac.nz>
References: <56207442.8090304@auckland.ac.nz>
	<CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>
	<562079BD.4090700@auckland.ac.nz>
Message-ID: <CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>

On Thu, Oct 15, 2015 at 11:14 PM, Adrian Croucher <a.croucher at auckland.ac.nz
> wrote:

> hi
>
> On 16/10/15 16:54, Matthew Knepley wrote:
>
>
> No, that should work. This is how we do it in PyLith. There must be some
> setup problem. When you look at the HDF5 file. Does
> it have a dimension for timestep?
>
>
> No, it doesn't.
>
> Is there any extra setup I might need to do to get it to work? I've
> attached a minimal example code, mesh and output.
>

Now I remember. I did not want the output to depend on the viewer.

Does your example work if you replace PetscViewerHDF5SetTimestep() with
DMSetOutputSequenceNumber()?

  Thanks,

    Matt


> Cheers, Adrian
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/227c95cb/attachment.html>

From hzhang at mcs.anl.gov  Fri Oct 16 10:12:47 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Fri, 16 Oct 2015 10:12:47 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <87r3kvaoqy.fsf@jedbrown.org>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
	<CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
	<87r3kvaoqy.fsf@jedbrown.org>
Message-ID: <CAGCphBtnN3iUGfOhnkvH9kg9bwyabpUThw1BAWM7+hui3J15vw@mail.gmail.com>

Jed:
>
>
> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via
> >
> > 1. add MatTransposeMatMult_elemental_elemental()
> > 2. C_dense = P_dense^T * B_dense
> >     via MatConvert_dense_elemental() and MatConvert_elemental_dense()
>
> The above involves a ton of data movement and MPIDense is a logical
> distribution for matrices with a modest number of columns.  I think I
> would just do the local GEMM and then MPI_Reduce_scatter it.
>
This would work.

However, I recall that you did a smart ordering which allows
MatConvert_mpidense_elemental() uses same physical matrix storage for petsc
and elemental, but logically in the layout of elemental. As an example,
petsc/src/mat/examples/tests/ex103.c:
mpiexec -n 6 ./ex103
 Outplace MatConvert, A_elemental:
Mat Object: 6 MPI processes
  type: elemental
Elemental matrix (cyclic ordering)
0 0 0 0 0
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3
4 4 4 4 4
5 5 5 5 5
0 0 0 0 0
1 1 1 1 1
2 2 2 2 2
3 3 3 3 3

Elemental matrix (explicit ordering)
  Mat Object:   6 MPI processes
    type: mpidense
0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
0.0000000000000000e+00 0.0000000000000000e+00
0.0000000000000000e+00 0.0000000000000000e+00 0.0000000000000000e+00
0.0000000000000000e+00 0.0000000000000000e+00
1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00
1.0000000000000000e+00 1.0000000000000000e+00
1.0000000000000000e+00 1.0000000000000000e+00 1.0000000000000000e+00
1.0000000000000000e+00 1.0000000000000000e+00
2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00
2.0000000000000000e+00 2.0000000000000000e+00
2.0000000000000000e+00 2.0000000000000000e+00 2.0000000000000000e+00
2.0000000000000000e+00 2.0000000000000000e+00
3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00
3.0000000000000000e+00 3.0000000000000000e+00
3.0000000000000000e+00 3.0000000000000000e+00 3.0000000000000000e+00
3.0000000000000000e+00 3.0000000000000000e+00
4.0000000000000000e+00 4.0000000000000000e+00 4.0000000000000000e+00
4.0000000000000000e+00 4.0000000000000000e+00
5.0000000000000000e+00 5.0000000000000000e+00 5.0000000000000000e+00
5.0000000000000000e+00 5.0000000000000000e+00

i.e., elemental and petsc dense matrices have same ownership.
If there is no data movement for MatConvert(), then it would be easier to
use elemental.

Hong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/4cd3d58d/attachment.html>

From jed at jedbrown.org  Fri Oct 16 10:25:29 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 16 Oct 2015 09:25:29 -0600
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAGCphBtnN3iUGfOhnkvH9kg9bwyabpUThw1BAWM7+hui3J15vw@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
	<CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
	<87r3kvaoqy.fsf@jedbrown.org>
	<CAGCphBtnN3iUGfOhnkvH9kg9bwyabpUThw1BAWM7+hui3J15vw@mail.gmail.com>
Message-ID: <878u7296om.fsf@jedbrown.org>

Hong <hzhang at mcs.anl.gov> writes:

> Jed:
>>
>>
>> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via
>> >
>> > 1. add MatTransposeMatMult_elemental_elemental()
>> > 2. C_dense = P_dense^T * B_dense
>> >     via MatConvert_dense_elemental() and MatConvert_elemental_dense()
>>
>> The above involves a ton of data movement and MPIDense is a logical
>> distribution for matrices with a modest number of columns.  I think I
>> would just do the local GEMM and then MPI_Reduce_scatter it.
>>
> This would work.
>
> However, I recall that you did a smart ordering which allows
> MatConvert_mpidense_elemental() uses same physical matrix storage for petsc
> and elemental, 

Same storage for vectors.  This is your code, but I think you'll find
that it moves the matrix entries around.  (Note that Elemental [MC,MR]
is a 2D distribution while MPIDense is 1D.)  Also, I think it would be
better if this particular operation did not depend on Elemental.

You could write a conversion to Elemental [VC,*], which would then match
the MPIDense distribution and thus not need to move any matrix entries.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/19d9cc7f/attachment.pgp>

From bsmith at mcs.anl.gov  Fri Oct 16 10:48:55 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Fri, 16 Oct 2015 10:48:55 -0500
Subject: [petsc-users] MatGetSubMatrices or VecCreateGhost
In-Reply-To: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>
References: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>
Message-ID: <DBEE2C79-D603-49D5-995B-5EABC421BC1C@mcs.anl.gov>


> On Oct 16, 2015, at 7:52 AM, Bikash Kanungo <bikash at umich.edu> wrote:
> 
> Hi,
> 
> I need to multiply a matrix A with several vectors v_i's and use the resultant vectors in my calculation which involves access to non-local nodes.

   What percentage of the "non-local" results do you need in "your calculation", a few percent, all of them? How do you know which "non-local" results you will need for "your calculation"? Are they ghost points on a grid? Something else?

    What is "several vectors"? 5? 100? 1000?

  Barry

> I can think of two possible ways to do it:
> 
> 1. Store the vectors v_i's into an MPIDENSE matrix and perform MatMatMult with A to get a resultant dense matrix B. I can now use MatGetSubMatrices on B to access the non-local part required in my subsequent calculations.
> 
> 2. Use MatMult to evaluate A*v_i for each of the vectors and provide ghost padding through VecCreateGhost for the resultant vector. 
> 
> Which of the two is a more efficient way to perform the task? Also, if there is a better way possible to achieve the same, then kindly let me know.
> 
> Thanks,
> Bikash   
> 
> -- 
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering 
> University of Michigan
> 


From bikash at umich.edu  Fri Oct 16 11:33:09 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Fri, 16 Oct 2015 12:33:09 -0400
Subject: [petsc-users] MatGetSubMatrices or VecCreateGhost
In-Reply-To: <DBEE2C79-D603-49D5-995B-5EABC421BC1C@mcs.anl.gov>
References: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>
	<DBEE2C79-D603-49D5-995B-5EABC421BC1C@mcs.anl.gov>
Message-ID: <CA+H+vq+YzNVhJZWk5b=V1rUamVhfCUrZ+cLyoLw-UOv_+iwhAg@mail.gmail.com>

Hi Barry,

I'm sorry, I should have provided more details.

Matrix A defined in my problem is of size NxN and I have N vectors v_i,
each of length N, which are to be multiplied by A. N is fairly small
10000-50000. Ideally speaking, it doesn't require parallel storage. But I
will soon exhaust memory by storing both A and v_i's sequentially. So I
planned to implement it through a sub-communiator as this Mat-Vec(s)
operation(s) is part of a bigger problem which requires a greater degree of
parallelization.

My ghost points are physical grid points, FEM nodal points to be exact. So
I have a list of nodes whose data is required on a given processor to
perform evaluation of any physical quantity. It is through this list that I
decide the ghost points (I check if the global ids of these nodes are
within the ownership range for the PETSc partioning or not).

Now depending upon the number of processors I put into my sub-communicator,
the percentage of non-local data I'll require will range from 5-10%.

I hope I'm able to put forth my problem properly and adequately.

Thanks,
Bikash
On Oct 16, 2015 11:49 AM, "Barry Smith" <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 16, 2015, at 7:52 AM, Bikash Kanungo <bikash at umich.edu> wrote:
> >
> > Hi,
> >
> > I need to multiply a matrix A with several vectors v_i's and use the
> resultant vectors in my calculation which involves access to non-local
> nodes.
>
>    What percentage of the "non-local" results do you need in "your
> calculation", a few percent, all of them? How do you know which "non-local"
> results you will need for "your calculation"? Are they ghost points on a
> grid? Something else?
>
>     What is "several vectors"? 5? 100? 1000?
>
>   Barry
>
> > I can think of two possible ways to do it:
> >
> > 1. Store the vectors v_i's into an MPIDENSE matrix and perform
> MatMatMult with A to get a resultant dense matrix B. I can now use
> MatGetSubMatrices on B to access the non-local part required in my
> subsequent calculations.
> >
> > 2. Use MatMult to evaluate A*v_i for each of the vectors and provide
> ghost padding through VecCreateGhost for the resultant vector.
> >
> > Which of the two is a more efficient way to perform the task? Also, if
> there is a better way possible to achieve the same, then kindly let me know.
> >
> > Thanks,
> > Bikash
> >
> > --
> > Bikash S. Kanungo
> > PhD Student
> > Computational Materials Physics Group
> > Mechanical Engineering
> > University of Michigan
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/6e1f0494/attachment-0001.html>

From jed at jedbrown.org  Fri Oct 16 11:42:11 2015
From: jed at jedbrown.org (Jed Brown)
Date: Fri, 16 Oct 2015 10:42:11 -0600
Subject: [petsc-users] MatGetSubMatrices or VecCreateGhost
In-Reply-To: <CA+H+vq+YzNVhJZWk5b=V1rUamVhfCUrZ+cLyoLw-UOv_+iwhAg@mail.gmail.com>
References: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>
	<DBEE2C79-D603-49D5-995B-5EABC421BC1C@mcs.anl.gov>
	<CA+H+vq+YzNVhJZWk5b=V1rUamVhfCUrZ+cLyoLw-UOv_+iwhAg@mail.gmail.com>
Message-ID: <876126934s.fsf@jedbrown.org>

Bikash Kanungo <bikash at umich.edu> writes:
> Matrix A defined in my problem is of size NxN and I have N vectors v_i,
> each of length N, which are to be multiplied by A. N is fairly small
> 10000-50000.

Your vectors form a square matrix, so I would use MatMatMult.  If the
number of vectors was much smaller than their size (e.g., 10), then
alternatives might make sense.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/d970bd16/attachment.pgp>

From bikash at umich.edu  Fri Oct 16 19:25:00 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Fri, 16 Oct 2015 20:25:00 -0400
Subject: [petsc-users] MatGetSubMatrices or VecCreateGhost
In-Reply-To: <876126934s.fsf@jedbrown.org>
References: <CA+H+vqKe6b9vU0dct3e7AgRip29jhiqyKdrsuWc3cX9ka_Muvw@mail.gmail.com>
	<DBEE2C79-D603-49D5-995B-5EABC421BC1C@mcs.anl.gov>
	<CA+H+vq+YzNVhJZWk5b=V1rUamVhfCUrZ+cLyoLw-UOv_+iwhAg@mail.gmail.com>
	<876126934s.fsf@jedbrown.org>
Message-ID: <CA+H+vqJZ22OoH=j7mFvXDKRc6wj5-PnJCohQTSp68fm=GZkY0Q@mail.gmail.com>

Thanks for the input Jed.
On Oct 16, 2015 12:42 PM, "Jed Brown" <jed at jedbrown.org> wrote:

> Bikash Kanungo <bikash at umich.edu> writes:
> > Matrix A defined in my problem is of size NxN and I have N vectors v_i,
> > each of length N, which are to be multiplied by A. N is fairly small
> > 10000-50000.
>
> Your vectors form a square matrix, so I would use MatMatMult.  If the
> number of vectors was much smaller than their size (e.g., 10), then
> alternatives might make sense.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151016/0af62bde/attachment.html>

From zonexo at gmail.com  Sun Oct 18 11:28:16 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Mon, 19 Oct 2015 00:28:16 +0800
Subject: [petsc-users] Suuden problem compiling
Message-ID: <5623C8A0.5080203@gmail.com>

Hi,

I suddenly have problem compiling my CFD code. The error is:

/_*PETSc_solvers.F90(303): error #8284: If the actual argument is 
scalar, the dummy argument shall be scalar unless the actual argument is 
of type character or is an element of an array that is not assumed 
shape, pointer, or polymorphic. [C]*_//_*
*_//_*    call 
MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)*_//_*
*_//_*---------^*_//_*
*_//_*PETSc_solvers.F90(525): error #8284: If the actual argument is 
scalar, the dummy argument shall be scalar unless the actual argument is 
of type character or is an element of an array that is not assumed 
shape, pointer, or polymorphic.   [C]*_//_*
*_//_*                        call 
MatSetValues(A_mat,1,II,1,JJ,big_A(ijk,kk),ADD_VALUES,ierr)*_//_*
*_//_*-----------------------------^*_//_*
*_//_*PETSc_solvers.F90(525): error #8284: If the actual argument is 
scalar, the dummy argument shall be scalar unless the actual argument is 
of type character or is an element of an array that is not assumed 
shape, pointer, or polymorphic.   [E]*_//_*
*_//_*                        call 
MatSetValues(A_mat,1,II,1,JJ,big_A(ijk,kk),ADD_VALUES,ierr)*_//_*
*_//_*-----------------------------^*_//_*
*_//_*compilation aborted for PETSc_solvers.F90 (code 1)*_//_*
*_//_*make: [PETSc_solvers.o] Error 1 (ignored)*_/

I never have this error before and it was compiling ok with gnu and 
v3.6.2. Now with intel, it can't work. I wonder why...

-- 
Thank you.

Yours sincerely,

TAY wee-beng

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151019/94432e63/attachment.html>

From bsmith at mcs.anl.gov  Sun Oct 18 12:47:52 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 18 Oct 2015 12:47:52 -0500
Subject: [petsc-users] Suuden problem compiling
In-Reply-To: <5623C8A0.5080203@gmail.com>
References: <5623C8A0.5080203@gmail.com>
Message-ID: <3C9B12B4-6499-45CE-88D0-5AE734345FB9@mcs.anl.gov>


  Gotta love Intel Fortran error messages; and the fact that they cannot point to the exact argument that is wrong.

   The reason you are getting this is because Fortran compilers are becoming more picky and complaining about passing a scalar when an array is expected.

  I believe in the first call II is a scalar while MatSetValues expects a 1d array of values so you can "fix" the problem by doing

   PetscInt   IIA(1)
   IIA(1) = II

    call MatSetValues(A_semi_xyz,1,IIA,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)

   The second case has the same problem with both II and also JJ 



> On Oct 18, 2015, at 11:28 AM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> Hi,
> 
> I suddenly have problem compiling my CFD code. The error is:
> 
> PETSc_solvers.F90(303): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic.   [C]
>     call MatSetValues(A_semi_xyz,1,II,7,int_semi_xyz(ijk,1:7),semi_mat_xyz(ijk,1:7),INSERT_VALUES,ierr)
> ---------^
> PETSc_solvers.F90(525): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic.   [C]
>                         call MatSetValues(A_mat,1,II,1,JJ,big_A(ijk,kk),ADD_VALUES,ierr)
> -----------------------------^
> PETSc_solvers.F90(525): error #8284: If the actual argument is scalar, the dummy argument shall be scalar unless the actual argument is of type character or is an element of an array that is not assumed shape, pointer, or polymorphic.   [E]
>                         call MatSetValues(A_mat,1,II,1,JJ,big_A(ijk,kk),ADD_VALUES,ierr)
> -----------------------------^
> compilation aborted for PETSc_solvers.F90 (code 1)
> make: [PETSc_solvers.o] Error 1 (ignored)
> 
> I never have this error before and it was compiling ok with gnu and v3.6.2. Now with intel, it can't work. I wonder why...
> -- 
> Thank you.
> 
> Yours sincerely,
> 
> TAY wee-beng
> 


From a.croucher at auckland.ac.nz  Sun Oct 18 17:06:11 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Mon, 19 Oct 2015 11:06:11 +1300
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>
References: <56207442.8090304@auckland.ac.nz>	<CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>	<562079BD.4090700@auckland.ac.nz>
	<CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>
Message-ID: <562417D3.5040404@auckland.ac.nz>

hi,

On 17/10/15 03:01, Matthew Knepley wrote:
>
> Now I remember. I did not want the output to depend on the viewer.
>
> Does your example work if you replace PetscViewerHDF5SetTimestep() 
> with DMSetOutputSequenceNumber()?

Aha, yes, that appears to do the trick. It gives me the time values in 
the output as well which was the next thing I was going to try and 
figure out how to do.

Thanks...

Cheers, Adrian

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151019/5fd9a3c5/attachment.html>

From sahai2 at illinois.edu  Tue Oct 20 02:10:06 2015
From: sahai2 at illinois.edu (Sahai, Amal)
Date: Tue, 20 Oct 2015 07:10:06 +0000
Subject: [petsc-users] Parallel ODE solver
Message-ID: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>

I am trying to solve a very large system of ODEs for modeling the time-varying composition of a homogeneous (no spatial variation) reaction mixture composed of 50000 species. I was wondering if there is parallel ode solver included in PETSc? I have been using the sundials package for solving a smaller system in serial and would like to move on to parallel framework to speed up my calculations.

Regards
Amal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/3b259adf/attachment.html>

From juan at tf.uni-kiel.de  Tue Oct 20 02:21:04 2015
From: juan at tf.uni-kiel.de (Julian Andrej)
Date: Tue, 20 Oct 2015 09:21:04 +0200
Subject: [petsc-users] Parallel ODE solver
In-Reply-To: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>
References: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>
Message-ID: <CABFzUT3Czvi4ddnw1Vy=YbF=3QmUA+zKftX9nOmgoixUgU5HAA@mail.gmail.com>

There are a variety of algorithms for ODE/DAEs in the PetscTS
implementation. For examples see e.g.
http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/index.html

Sundials is also capable of parallel solving, even from the Matlab
sundialsTB interface, as far as i know.

On Tue, Oct 20, 2015 at 9:10 AM, Sahai, Amal <sahai2 at illinois.edu> wrote:
> I am trying to solve a very large system of ODEs for modeling the
> time-varying composition of a homogeneous (no spatial variation) reaction
> mixture composed of 50000 species. I was wondering if there is parallel ode
> solver included in PETSc? I have been using the sundials package for solving
> a smaller system in serial and would like to move on to parallel framework
> to speed up my calculations.
>
> Regards
> Amal

From francesco.caimmi at polimi.it  Tue Oct 20 02:53:27 2015
From: francesco.caimmi at polimi.it (Francesco Caimmi)
Date: Tue, 20 Oct 2015 09:53:27 +0200
Subject: [petsc-users] Points in a ball
Message-ID: <1838734.5T1Yd9CUtY@pc-fcaimmi>

Dear all,

for an application I have in mind I have a domain divided in cells (more or less) and 
for each cell centroid, say ci, I need  to know which cell centroids are inside a 
spherical neighbourhood of radius r  centred at ci.

Were the centroids placed on regular grid, a typical value for the ratio of r over 
the grid spacing would be something like 3 or 4.

However, since the discrete domain is actually not structured, I am using a DMPlex 
object to manage the domain. 

It is not very clear to me *if* and *how* I can tell the DMPlex to add ghost 
points for all the points in such a spherical neighbourhood. If I were using  DMDA 
object I think I could just set the stencil width, but I wasn't able to figure out a 
corresponding option/procedure for DMPlex (if it exists).

Any hint would be really really appreciated.

Best Regards,
-- 
Francesco Caimmi
 
Laboratorio di Ingegneria dei Polimeri
http://www.chem.polimi.it/polyenglab/

Politecnico di Milano - Dipartimento di Chimica,
Materiali e Ingegneria Chimica ?Giulio Natta?

P.zza Leonardo da Vinci, 32
I-20133 Milano
Tel. +39.02.2399.4711
Fax +39.02.7063.8173

francesco.caimmi at polimi.it
Skype: fmglcaimmi (please arrange meetings by e-mail)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/619a5fac/attachment.html>

From patrick.sanan at gmail.com  Tue Oct 20 03:34:32 2015
From: patrick.sanan at gmail.com (Patrick Sanan)
Date: Tue, 20 Oct 2015 10:34:32 +0200
Subject: [petsc-users] Parallel ODE solver
In-Reply-To: <CABFzUT3Czvi4ddnw1Vy=YbF=3QmUA+zKftX9nOmgoixUgU5HAA@mail.gmail.com>
References: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>
	<CABFzUT3Czvi4ddnw1Vy=YbF=3QmUA+zKftX9nOmgoixUgU5HAA@mail.gmail.com>
Message-ID: <20151020083432.GA27353@Patricks-MacBook-Pro-4.local>

Further note that you can use sundials from within PETSc, without any
more hassle than supplying a configure flags (--download-sundials, or
see the other entries under SUNDIALS: with ./configure --help).
That could potentially be very helpful in moving to TS.

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/TS/TSSUNDIALS.html


On Tue, Oct 20, 2015 at 09:21:04AM +0200, Julian Andrej wrote:
> There are a variety of algorithms for ODE/DAEs in the PetscTS
> implementation. For examples see e.g.
> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/index.html
> 
> Sundials is also capable of parallel solving, even from the Matlab
> sundialsTB interface, as far as i know.
> 
> On Tue, Oct 20, 2015 at 9:10 AM, Sahai, Amal <sahai2 at illinois.edu> wrote:
> > I am trying to solve a very large system of ODEs for modeling the
> > time-varying composition of a homogeneous (no spatial variation) reaction
> > mixture composed of 50000 species. I was wondering if there is parallel ode
> > solver included in PETSc? I have been using the sundials package for solving
> > a smaller system in serial and would like to move on to parallel framework
> > to speed up my calculations.
> >
> > Regards
> > Amal
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 473 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/e7bd74e0/attachment.pgp>

From natacha.bereux at gmail.com  Tue Oct 20 03:36:42 2015
From: natacha.bereux at gmail.com (Natacha BEREUX)
Date: Tue, 20 Oct 2015 10:36:42 +0200
Subject: [petsc-users] Call KSPConvergedDefault from Fortran
Message-ID: <CAFdHYfuwN1DvbRJSFwMGcEL0QU_iSwvNHdup=2YCLn8tdk_ALA@mail.gmail.com>

Dear PETSc users,


I am using PETSc  Fortran API.
I try to solve a linear system, and would like to define my own convergence
test.
More precisely, my convergence test calls the default convergence test
and then checks the true residual norm :
     subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)

      implicit none

#include <petsc/finclude/petscsys.h>
#include <petsc/finclude/petscvec.h>
#include <petsc/finclude/petscksp.h>

      KSP              ksp
      PetscErrorCode ierr
      PetscInt n,dummy
      KSPConvergedReason flag
      PetscReal rnorm, true_rnorm
      Vec true_res
      !
      call KSPConvergedDefault(ksp,n,rnorm,flag,PETSC_NULL_OBJECT,ierr)
      !
      ! If convergence test succeeds
       if ( (flag == KSP_CONVERGED_ATOL) .or.                           &
     &  (flag == KSP_CONVERGED_RTOL)) then
        ! Compute true residual
        call KSPBuildResidual( ksp , PETSC_NULL_OBJECT,                 &
     & PETSC_NULL_OBJECT, true_res, ierr )
        ! Compute true residual norm
        call VecNorm(true_res,NORM_2,true_rnorm,ierr)
        ! And check again convergence with respect to the true residual norm
        call KSPConvergedDefault(ksp,n,true_rnorm,flag,                 &
     & PETSC_NULL_OBJECT, ierr)
        ! Free memory
        call VecDestroy(true_res, ierr)
      endif
      !
      ierr = 0

      end

I get the following error message

[0]PETSC ERROR: --------------------- Error Message
--------------------------------------------------------------
[0]PETSC ERROR: Null argument, when expecting valid pointer
[0]PETSC ERROR: Convergence context must have been created with
KSPConvergedDefaultCreate()
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for
trouble shooting.
[0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015

I understand that I should have  call KSPConvergedDefaultCreate() before
calling KSConvergedDefault( ....)

Am I right ?

If so what is the Fortran calling sequence for  KSPConvergedDefaultCreate ?
It is supposed to return a valid pointer and I don't succeed in doing so
from Fortran.

If you have any idea, I'd greatly appreciate it
Thank you!
Natacha


PS I have attached my code (it's a slightly modified version of ex2f.F
showing  the convergence test code )
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/7cbbc115/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ex2f.F
Type: text/x-fortran
Size: 14822 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/7cbbc115/attachment-0001.bin>

From Lukasz.Kaczmarczyk at glasgow.ac.uk  Tue Oct 20 08:46:41 2015
From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk)
Date: Tue, 20 Oct 2015 13:46:41 +0000
Subject: [petsc-users] problem with BT linesearcher,
Message-ID: <01146B08-DC90-4EC8-AD70-252FA99B8443@glasgow.ac.uk>

Hello,

I have problem with BT line-searcher. I can do calculations with L2, CP line-searchers, however I get error for BT. Error is at first step. This is plasticity which is initially linear after some time nonlinearity kicks in. Could you advise where to search fro problem,

For L2  or CP I get,
Step 1 Time 0.05 final time   3
  0 SNES Function norm 2.282177322938e-03 
      Line search: lambdas = [1, 0.5, 0], fnorms = [8.50089e-17, 0.00114109, 0.00228218]
      Line search terminated: lambda = 1, fnorms = 8.23981e-17
  1 SNES Function norm 8.239809253872e-17 
number of Newton iterations = 1
reduction delta_time = 1.0000e+00 delta_time = 5.0000e-02
out file out_1.h5m
Step 2 Time 0.1 final time   3
  0 SNES Function norm 2.282177322938e-03 
      Line search: lambdas = [1, 0.5, 0], fnorms = [9.15228e-17, 0.00114109, 0.00228218]
      Line search terminated: lambda = 1, fnorms = 9.15228e-17
  1 SNES Function norm 9.152279314292e-17 
number of Newton iterations = 1
reduction delta_time = 1.0000e+00 delta_time = 5.0000e-02

Fro BT I get,
Step 1 Time 0.05 final time   3
  0 SNES Function norm 2.282177322938e-03 
      Line search: gnorm after quadratic fit 1.141088661469e-03
      Line search: Quadratically determined step, lambda=4.9999999999999706e-01
  1 SNES Function norm 1.141088661469e-03 
[0]PETSC ERROR: --------------------- MoFEM Error Message---------------------------------------------------------------------------
[0]PETSC ERROR: MoFEM version 0.3.0
[0]PETSC ERROR: MoFEM git commit id a0b356adadb599d17aa11ae2e6d354355111ee7d
[0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html for bug reporting.
[0]PETSC ERROR: See http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html for trouble shooting.
[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
[0]PETSC ERROR: Object is in wrong state
[0]PETSC ERROR: Not for unassembled vector
[0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
[0]PETSC ERROR: Petsc Development GIT revision: v3.6-41-g8f9ce63  GIT Date: 2015-06-20 12:33:17 -0500
[0]PETSC ERROR: ./small_strain_plasticity_problem on a arch-darwin-c-opt named dh23-177.eng.gla.ac.uk by likask Tue Oct 20 14:21:15 2015
[0]PETSC ERROR: Configure options --download-blacs=1 --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1 --download-metis=1 --download-moab=1 --download-mumps=1 --download-netcdf=netcdf-4.3.3.1.tar.gz --download-openmpi=1 --download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1 --with-debugging=0 PETSC_ARCH=arch-darwin-c-opt
[0]PETSC ERROR: #1 VecCopy() line 1630 in /opt/petsc/src/vec/vec/interface/vector.c
[0]PETSC ERROR: #2 KSPSolve_FGMRES() line 290 in /opt/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
[0]PETSC ERROR: #3 KSPSolve() line 604 in /opt/petsc/src/ksp/ksp/interface/itfunc.c
[0]PETSC ERROR: #4 SNESSolve_NEWTONLS() line 233 in /opt/petsc/src/snes/impls/ls/ls.c
[0]PETSC ERROR: #5 SNESSolve() line 3894 in /opt/petsc/src/snes/interface/snes.c
[0]PETSC ERROR: #6 main() line 421 in /Users/likask/MyBuild/mofem-cephas/mofem/users_modules/small_strain_plasticity/small_strain_plasticity_problem.cpp
[0]PETSC ERROR: PETSc Option Table entries:
[0]PETSC ERROR: -ksp_type fgmres
[0]PETSC ERROR: -my_H 0.1
[0]PETSC ERROR: -my_delta_time 0.05
[0]PETSC ERROR: -my_file plate_with_hole.h5m
[0]PETSC ERROR: -my_final_time 3
[0]PETSC ERROR: -my_load_history plate_with_hole_history.txt
[0]PETSC ERROR: -my_order 1
[0]PETSC ERROR: -my_poisson_ratio 0.25

Kind regards,
Lukasz Kaczmarczyk

"Prediction is very difficult, especially about the future." -- Niels Bohr
?? there are dark corners in the Bourne shell, and people use all of them." --Chet Ramey

HomePage:
http://www.gla.ac.uk/schools/engineering/staff/lukaszkaczmarczyk/

MoFEM @mofemjoseph 
http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/index.html





From knepley at gmail.com  Tue Oct 20 08:57:22 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 20 Oct 2015 08:57:22 -0500
Subject: [petsc-users] Points in a ball
In-Reply-To: <1838734.5T1Yd9CUtY@pc-fcaimmi>
References: <1838734.5T1Yd9CUtY@pc-fcaimmi>
Message-ID: <CAMYG4GkemSWkd7W+K43UtK3i4Tr_9YZjOzWptDADK9G7xgokDw@mail.gmail.com>

On Tue, Oct 20, 2015 at 2:53 AM, Francesco Caimmi <
francesco.caimmi at polimi.it> wrote:

> Dear all,
>
>
>
> for an application I have in mind I have a domain divided in cells (more
> or less) and for each cell centroid, say ci, I need to know which cell
> centroids are inside a spherical neighbourhood of radius r centred at ci.
>
>
>
> Were the centroids placed on regular grid, a typical value for the ratio
> of r over the grid spacing would be something like 3 or 4.
>
>
>
> However, since the discrete domain is actually not structured, I am using
> a DMPlex object to manage the domain.
>
>
>
> It is not very clear to me *if* and *how* I can tell the DMPlex to add
> ghost points for all the points in such a spherical neighbourhood. If I
> were using DMDA object I think I could just set the stencil width, but I
> wasn't able to figure out a corresponding option/procedure for DMPlex (if
> it exists).
>
>
>
> Any hint would be really really appreciated.
>

The function that defines grid overlap is here:


https://bitbucket.org/petsc/petsc/src/fc309a34c451569cb3041ed38231443f9c022d73/src/dm/impls/plex/plexdistribute.c?at=master&fileviewer=file-view-default#plexdistribute.c-546

It calls a routine to define this overlap topologically


https://bitbucket.org/petsc/petsc/src/fc309a34c451569cb3041ed38231443f9c022d73/src/dm/impls/plex/plexdistribute.c?at=master&fileviewer=file-view-default#plexdistribute.c-612

You would have to modify this function to use a geometric criterion


https://bitbucket.org/petsc/petsc/src/fc309a34c451569cb3041ed38231443f9c022d73/src/dm/impls/plex/plexpartition.c?at=master&fileviewer=file-view-default#plexpartition.c-1470

If you do this, I can put it in as an option.

  Thanks,

    Matt


> Best Regards,
>
> --
>
> Francesco Caimmi
>
> Laboratorio di Ingegneria dei Polimeri
>
> http://www.chem.polimi.it/polyenglab/
>
>
>
> Politecnico di Milano - Dipartimento di Chimica,
>
> Materiali e Ingegneria Chimica ?Giulio Natta?
>
>
>
> P.zza Leonardo da Vinci, 32
>
> I-20133 Milano
>
> Tel. +39.02.2399.4711
>
> Fax +39.02.7063.8173
>
>
>
> francesco.caimmi at polimi.it
>
> Skype: fmglcaimmi (please arrange meetings by e-mail)
>
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/1a625fd2/attachment.html>

From knepley at gmail.com  Tue Oct 20 09:46:28 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 20 Oct 2015 09:46:28 -0500
Subject: [petsc-users] problem with BT linesearcher,
In-Reply-To: <01146B08-DC90-4EC8-AD70-252FA99B8443@glasgow.ac.uk>
References: <01146B08-DC90-4EC8-AD70-252FA99B8443@glasgow.ac.uk>
Message-ID: <CAMYG4Gnj_oAVAQz4YPb-cwEhZeS_DRfhq8x=C0KU0Y98gbCi2A@mail.gmail.com>

On Tue, Oct 20, 2015 at 8:46 AM, Lukasz Kaczmarczyk <
Lukasz.Kaczmarczyk at glasgow.ac.uk> wrote:

> Hello,
>
> I have problem with BT line-searcher. I can do calculations with L2, CP
> line-searchers, however I get error for BT. Error is at first step. This is
> plasticity which is initially linear after some time nonlinearity kicks in.
> Could you advise where to search fro problem,
>
> For L2  or CP I get,
> Step 1 Time 0.05 final time   3
>   0 SNES Function norm 2.282177322938e-03
>       Line search: lambdas = [1, 0.5, 0], fnorms = [8.50089e-17,
> 0.00114109, 0.00228218]
>       Line search terminated: lambda = 1, fnorms = 8.23981e-17
>   1 SNES Function norm 8.239809253872e-17
> number of Newton iterations = 1
> reduction delta_time = 1.0000e+00 delta_time = 5.0000e-02
> out file out_1.h5m
> Step 2 Time 0.1 final time   3
>   0 SNES Function norm 2.282177322938e-03
>       Line search: lambdas = [1, 0.5, 0], fnorms = [9.15228e-17,
> 0.00114109, 0.00228218]
>       Line search terminated: lambda = 1, fnorms = 9.15228e-17
>   1 SNES Function norm 9.152279314292e-17
> number of Newton iterations = 1
> reduction delta_time = 1.0000e+00 delta_time = 5.0000e-02
>
> Fro BT I get,
> Step 1 Time 0.05 final time   3
>   0 SNES Function norm 2.282177322938e-03
>       Line search: gnorm after quadratic fit 1.141088661469e-03
>       Line search: Quadratically determined step,
> lambda=4.9999999999999706e-01
>   1 SNES Function norm 1.141088661469e-03
> [0]PETSC ERROR: --------------------- MoFEM Error
> Message---------------------------------------------------------------------------
> [0]PETSC ERROR: MoFEM version 0.3.0
> [0]PETSC ERROR: MoFEM git commit id
> a0b356adadb599d17aa11ae2e6d354355111ee7d
> [0]PETSC ERROR: See
> http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/guidelines_bug_reporting.html
> for bug reporting.
> [0]PETSC ERROR: See
> http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/faq_and_bugs.html
> for trouble shooting.
> [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> [0]PETSC ERROR: Object is in wrong state
> [0]PETSC ERROR: Not for unassembled vector
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> [0]PETSC ERROR: Petsc Development GIT revision: v3.6-41-g8f9ce63  GIT
> Date: 2015-06-20 12:33:17 -0500
> [0]PETSC ERROR: ./small_strain_plasticity_problem on a arch-darwin-c-opt
> named dh23-177.eng.gla.ac.uk by likask Tue Oct 20 14:21:15 2015
> [0]PETSC ERROR: Configure options --download-blacs=1
> --download-fblaslapack=1 --download-hdf5=1 --download-hypre=1
> --download-metis=1 --download-moab=1 --download-mumps=1
> --download-netcdf=netcdf-4.3.3.1.tar.gz --download-openmpi=1
> --download-parmetis=1 --download-scalapack=1 --download-superlu_dist=1
> --with-debugging=0 PETSC_ARCH=arch-darwin-c-opt
> [0]PETSC ERROR: #1 VecCopy() line 1630 in
> /opt/petsc/src/vec/vec/interface/vector.c
> [0]PETSC ERROR: #2 KSPSolve_FGMRES() line 290 in
> /opt/petsc/src/ksp/ksp/impls/gmres/fgmres/fgmres.c
> [0]PETSC ERROR: #3 KSPSolve() line 604 in
> /opt/petsc/src/ksp/ksp/interface/itfunc.c
>

Right here it says that the vector F, your residual, is unassembled. Do you
assemble it at the end
of your routine?

  Thanks,

     Matt


> [0]PETSC ERROR: #4 SNESSolve_NEWTONLS() line 233 in
> /opt/petsc/src/snes/impls/ls/ls.c
> [0]PETSC ERROR: #5 SNESSolve() line 3894 in
> /opt/petsc/src/snes/interface/snes.c
> [0]PETSC ERROR: #6 main() line 421 in
> /Users/likask/MyBuild/mofem-cephas/mofem/users_modules/small_strain_plasticity/small_strain_plasticity_problem.cpp
> [0]PETSC ERROR: PETSc Option Table entries:
> [0]PETSC ERROR: -ksp_type fgmres
> [0]PETSC ERROR: -my_H 0.1
> [0]PETSC ERROR: -my_delta_time 0.05
> [0]PETSC ERROR: -my_file plate_with_hole.h5m
> [0]PETSC ERROR: -my_final_time 3
> [0]PETSC ERROR: -my_load_history plate_with_hole_history.txt
> [0]PETSC ERROR: -my_order 1
> [0]PETSC ERROR: -my_poisson_ratio 0.25
>
> Kind regards,
> Lukasz Kaczmarczyk
>
> "Prediction is very difficult, especially about the future." -- Niels Bohr
> ?? there are dark corners in the Bourne shell, and people use all of
> them." --Chet Ramey
>
> HomePage:
> http://www.gla.ac.uk/schools/engineering/staff/lukaszkaczmarczyk/
>
> MoFEM @mofemjoseph
> http://userweb.eng.gla.ac.uk/lukasz.kaczmarczyk/MoFem/html/index.html
>
>
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/b8e3a7cc/attachment.html>

From thronesf at gmail.com  Tue Oct 20 10:13:10 2015
From: thronesf at gmail.com (Sharp Stone)
Date: Tue, 20 Oct 2015 11:13:10 -0400
Subject: [petsc-users] how to run petsc in Xcode
Message-ID: <CABJpBAycd7qDEotaJHi_UinBCKQOjynE367TtPjPk2PaxOTVQg@mail.gmail.com>

Hi All,

I'm trying to find out how to run PETSc in Xcode. Now I'm able to run the
MPI in Xcode but still failed in running PETSc. Searched for the web but no
luck at all. Does anyone have any instructions? Thanks very much in advance!

-- 
Best regards,

Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/e2dd4130/attachment-0001.html>

From bikash at umich.edu  Tue Oct 20 10:53:44 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Tue, 20 Oct 2015 11:53:44 -0400
Subject: [petsc-users] Testing MatGetSubMatrices
Message-ID: <CA+H+vq+aHb1bN6VEwA8Lsgxn8-h0YX98M9_+Aiy==Y_o5k4xzA@mail.gmail.com>

Hi,

I was trying to perform a simple test on MatGetSubMatrices. I have an
MPIDENSE matrix X which has been populated and assembled and another matrix
XTemp of the same type and size. Since X is MPIDENSE in order to extract a
sub-matrix from it, the IS must be sorted. This can be done easily through
ISSort. Once the sub-matrix is obtained, I'm trying to set the values in
XTemp by using the sorted IS as the global indices and the sub-matrix as
the values to be set. Once XTemp is set and assembled, I evaluate XTemp - X
and see if the norm of the resultant matrix is zero or not. The norm turned
out to be as high as 1.0e6.

I'm providing a simplified version of the code that performs the above
test. Kindly let me know if I have done any mistake is designing the test.


      //
      // Given an initialized and assembled MPIDENSE Mat X and
      // an uninitialized MPIDENSE Mat XTemp of the same size as that of X
      //
      IS  * ISRowIds, * ISColIds;
      PetscMalloc(1*sizeof(IS), &ISRowIds);
      PetscMalloc(1*sizeof(IS), &ISColIds);

      //
      // globalRowIds and globalColIds are STL vectors containing
      // the global node ids for which the sub-matrix from X need
      // to be extracted
      //
      const PetscInt * rowIdsPtr = globalRowIds.data();
      const PetscInt * colIdsPtr = globalColIds.data();

      ISCreateGeneral(PETSC_COMM_WORLD, globalRowIds.size(), rowIdsPtr,
PETSC_COPY_VALUES, &ISRowIds[0]);
      ISCreateGeneral(PETSC_COMM_WORLD, globalColIds.size(), colIdsPtr,
PETSC_COPY_VALUES, &ISColIds[0]);
      ISSort(ISRowIds[0]);
      ISSort(ISColIds[0]);

      Mat * subMat;
      MatGetSubMatrices(X, 1, ISRowIds, ISColIds, MAT_INITIAL_MATRIX,
&subMat);

      PetscScalar * subMatValues;
      MatDenseGetArray(*subMat, &subMatValues);

      const PetscInt * rowIndicesSorted, * colIndicesSorted;
      ISGetIndices(ISRowIds[0], &rowIndicesSorted);
      ISGetIndices(ISColIds[0], &colIndicesSorted);

      MatSetValues(XTemp,
                   globalRowIds.size(),
                   rowIndicesSorted,
                   globalColIds.size(),
                   colIndicesSorted,
                   subMatValues,
                   INSERT_VALUES);

      //
      // commencing assembly
      //
      MatAssemblyBegin(XTemp,
                         MAT_FINAL_ASSEMBLY);

      //
      // concluding assembly
      //
      MatAssemblyEnd(XTemp,
               MAT_FINAL_ASSEMBLY);

      MatAXPY(XTemp, -1.0, X, DIFFERENT_NONZERO_PATTERN);
      double diffNorm;
      MatNorm(XTemp, NORM_1, &diffNorm);



Thanks,
Bikash
-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/f4db60e1/attachment.html>

From knepley at gmail.com  Tue Oct 20 10:58:46 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 20 Oct 2015 10:58:46 -0500
Subject: [petsc-users] Testing MatGetSubMatrices
In-Reply-To: <CA+H+vq+aHb1bN6VEwA8Lsgxn8-h0YX98M9_+Aiy==Y_o5k4xzA@mail.gmail.com>
References: <CA+H+vq+aHb1bN6VEwA8Lsgxn8-h0YX98M9_+Aiy==Y_o5k4xzA@mail.gmail.com>
Message-ID: <CAMYG4Gk8ZP2K5JLdxGpq6ZhNLyYgXidJmLj1m7QNLp9oQRfDMA@mail.gmail.com>

On Tue, Oct 20, 2015 at 10:53 AM, Bikash Kanungo <bikash at umich.edu> wrote:

> Hi,
>
> I was trying to perform a simple test on MatGetSubMatrices. I have an
> MPIDENSE matrix X which has been populated and assembled and another matrix
> XTemp of the same type and size. Since X is MPIDENSE in order to extract a
> sub-matrix from it, the IS must be sorted. This can be done easily through
> ISSort. Once the sub-matrix is obtained, I'm trying to set the values in
> XTemp by using the sorted IS as the global indices and the sub-matrix as
> the values to be set. Once XTemp is set and assembled, I evaluate XTemp - X
> and see if the norm of the resultant matrix is zero or not. The norm turned
> out to be as high as 1.0e6.
>

Make the problem very small (10 rows maybe) and starting printing things
out.

   Matt


>
> I'm providing a simplified version of the code that performs the above
> test. Kindly let me know if I have done any mistake is designing the test.
>
>
>       //
>       // Given an initialized and assembled MPIDENSE Mat X and
>       // an uninitialized MPIDENSE Mat XTemp of the same size as that of X
>       //
>       IS  * ISRowIds, * ISColIds;
>       PetscMalloc(1*sizeof(IS), &ISRowIds);
>       PetscMalloc(1*sizeof(IS), &ISColIds);
>
>       //
>       // globalRowIds and globalColIds are STL vectors containing
>       // the global node ids for which the sub-matrix from X need
>       // to be extracted
>       //
>       const PetscInt * rowIdsPtr = globalRowIds.data();
>       const PetscInt * colIdsPtr = globalColIds.data();
>
>       ISCreateGeneral(PETSC_COMM_WORLD, globalRowIds.size(), rowIdsPtr,
> PETSC_COPY_VALUES, &ISRowIds[0]);
>       ISCreateGeneral(PETSC_COMM_WORLD, globalColIds.size(), colIdsPtr,
> PETSC_COPY_VALUES, &ISColIds[0]);
>       ISSort(ISRowIds[0]);
>       ISSort(ISColIds[0]);
>
>       Mat * subMat;
>       MatGetSubMatrices(X, 1, ISRowIds, ISColIds, MAT_INITIAL_MATRIX,
> &subMat);
>
>       PetscScalar * subMatValues;
>       MatDenseGetArray(*subMat, &subMatValues);
>
>       const PetscInt * rowIndicesSorted, * colIndicesSorted;
>       ISGetIndices(ISRowIds[0], &rowIndicesSorted);
>       ISGetIndices(ISColIds[0], &colIndicesSorted);
>
>       MatSetValues(XTemp,
>                    globalRowIds.size(),
>                    rowIndicesSorted,
>                    globalColIds.size(),
>                    colIndicesSorted,
>                    subMatValues,
>                    INSERT_VALUES);
>
>       //
>       // commencing assembly
>       //
>       MatAssemblyBegin(XTemp,
>                          MAT_FINAL_ASSEMBLY);
>
>       //
>       // concluding assembly
>       //
>       MatAssemblyEnd(XTemp,
>                MAT_FINAL_ASSEMBLY);
>
>       MatAXPY(XTemp, -1.0, X, DIFFERENT_NONZERO_PATTERN);
>       double diffNorm;
>       MatNorm(XTemp, NORM_1, &diffNorm);
>
>
>
> Thanks,
> Bikash
> --
> Bikash S. Kanungo
> PhD Student
> Computational Materials Physics Group
> Mechanical Engineering
> University of Michigan
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/4831e582/attachment.html>

From fdkong.jd at gmail.com  Tue Oct 20 12:44:41 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 20 Oct 2015 11:44:41 -0600
Subject: [petsc-users] mathematics formula of DMPlexProjectFunction
Message-ID: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>

Hi all,

Any body knows the mathematics formula corresponding to the function
DMPlexProjectFunction? I already went through the code, but I do not
understand quite well. I will appreciate any help.


Thanks,

Fande,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/a76bd1d6/attachment.html>

From bikash at umich.edu  Tue Oct 20 12:52:42 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Tue, 20 Oct 2015 13:52:42 -0400
Subject: [petsc-users] Testing MatGetSubMatrices
In-Reply-To: <CAMYG4Gk8ZP2K5JLdxGpq6ZhNLyYgXidJmLj1m7QNLp9oQRfDMA@mail.gmail.com>
References: <CA+H+vq+aHb1bN6VEwA8Lsgxn8-h0YX98M9_+Aiy==Y_o5k4xzA@mail.gmail.com>
	<CAMYG4Gk8ZP2K5JLdxGpq6ZhNLyYgXidJmLj1m7QNLp9oQRfDMA@mail.gmail.com>
Message-ID: <CA+H+vqKp8uK=x+544RV-cvSyqhD503DeCCNHJT9ofZ1ZzxgJAA@mail.gmail.com>

Thanks for the suggestion Matthew. The issue was that MatDenseGetArray
stores the value in a compressed column format rather than a row format. So
while setting the values into XTemp I must carefully swap between row and
column indices.

Regards,
Bikash


On Tue, Oct 20, 2015 at 11:58 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 20, 2015 at 10:53 AM, Bikash Kanungo <bikash at umich.edu> wrote:
>
>> Hi,
>>
>> I was trying to perform a simple test on MatGetSubMatrices. I have an
>> MPIDENSE matrix X which has been populated and assembled and another matrix
>> XTemp of the same type and size. Since X is MPIDENSE in order to extract a
>> sub-matrix from it, the IS must be sorted. This can be done easily through
>> ISSort. Once the sub-matrix is obtained, I'm trying to set the values in
>> XTemp by using the sorted IS as the global indices and the sub-matrix as
>> the values to be set. Once XTemp is set and assembled, I evaluate XTemp - X
>> and see if the norm of the resultant matrix is zero or not. The norm turned
>> out to be as high as 1.0e6.
>>
>
> Make the problem very small (10 rows maybe) and starting printing things
> out.
>
>    Matt
>
>
>>
>> I'm providing a simplified version of the code that performs the above
>> test. Kindly let me know if I have done any mistake is designing the test.
>>
>>
>>       //
>>       // Given an initialized and assembled MPIDENSE Mat X and
>>       // an uninitialized MPIDENSE Mat XTemp of the same size as that of
>> X
>>       //
>>       IS  * ISRowIds, * ISColIds;
>>       PetscMalloc(1*sizeof(IS), &ISRowIds);
>>       PetscMalloc(1*sizeof(IS), &ISColIds);
>>
>>       //
>>       // globalRowIds and globalColIds are STL vectors containing
>>       // the global node ids for which the sub-matrix from X need
>>       // to be extracted
>>       //
>>       const PetscInt * rowIdsPtr = globalRowIds.data();
>>       const PetscInt * colIdsPtr = globalColIds.data();
>>
>>       ISCreateGeneral(PETSC_COMM_WORLD, globalRowIds.size(), rowIdsPtr,
>> PETSC_COPY_VALUES, &ISRowIds[0]);
>>       ISCreateGeneral(PETSC_COMM_WORLD, globalColIds.size(), colIdsPtr,
>> PETSC_COPY_VALUES, &ISColIds[0]);
>>       ISSort(ISRowIds[0]);
>>       ISSort(ISColIds[0]);
>>
>>       Mat * subMat;
>>       MatGetSubMatrices(X, 1, ISRowIds, ISColIds, MAT_INITIAL_MATRIX,
>> &subMat);
>>
>>       PetscScalar * subMatValues;
>>       MatDenseGetArray(*subMat, &subMatValues);
>>
>>       const PetscInt * rowIndicesSorted, * colIndicesSorted;
>>       ISGetIndices(ISRowIds[0], &rowIndicesSorted);
>>       ISGetIndices(ISColIds[0], &colIndicesSorted);
>>
>>       MatSetValues(XTemp,
>>                    globalRowIds.size(),
>>                    rowIndicesSorted,
>>                    globalColIds.size(),
>>                    colIndicesSorted,
>>                    subMatValues,
>>                    INSERT_VALUES);
>>
>>       //
>>       // commencing assembly
>>       //
>>       MatAssemblyBegin(XTemp,
>>                          MAT_FINAL_ASSEMBLY);
>>
>>       //
>>       // concluding assembly
>>       //
>>       MatAssemblyEnd(XTemp,
>>                MAT_FINAL_ASSEMBLY);
>>
>>       MatAXPY(XTemp, -1.0, X, DIFFERENT_NONZERO_PATTERN);
>>       double diffNorm;
>>       MatNorm(XTemp, NORM_1, &diffNorm);
>>
>>
>>
>> Thanks,
>> Bikash
>> --
>> Bikash S. Kanungo
>> PhD Student
>> Computational Materials Physics Group
>> Mechanical Engineering
>> University of Michigan
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
Bikash S. Kanungo
PhD Student
Computational Materials Physics Group
Mechanical Engineering
University of Michigan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/d6a43187/attachment.html>

From jed at jedbrown.org  Tue Oct 20 13:03:09 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 20 Oct 2015 12:03:09 -0600
Subject: [petsc-users] mathematics formula of DMPlexProjectFunction
In-Reply-To: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>
References: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>
Message-ID: <87vba1zaci.fsf@jedbrown.org>

Fande Kong <fdkong.jd at gmail.com> writes:
> Any body knows the mathematics formula corresponding to the function
> DMPlexProjectFunction? I already went through the code, but I do not
> understand quite well. I will appreciate any help.

The definition of a finite element involves a dual space (the basis for
which is sometimes called the "nodes").  For a typical finite element
space, the "nodes" are Dirac delta functions at vertices.  Consequently,
the inner product

  (n_i, f) = \int \delta(x - x_i) f(x) = f(x_i)

if f is sufficiently regular.  For more general dual basis functions,
this inner product needs to be evaluated by quadrature.
DMPlexProjectFunction does this projection.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/63c4f164/attachment.pgp>

From bsmith at mcs.anl.gov  Tue Oct 20 13:14:55 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 20 Oct 2015 13:14:55 -0500
Subject: [petsc-users] how to run petsc in Xcode
In-Reply-To: <CABJpBAycd7qDEotaJHi_UinBCKQOjynE367TtPjPk2PaxOTVQg@mail.gmail.com>
References: <CABJpBAycd7qDEotaJHi_UinBCKQOjynE367TtPjPk2PaxOTVQg@mail.gmail.com>
Message-ID: <150C60E1-D60A-4925-8F49-E6FA2A533F16@mcs.anl.gov>


 http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf  section 15.13


> On Oct 20, 2015, at 10:13 AM, Sharp Stone <thronesf at gmail.com> wrote:
> 
> Hi All,
> 
> I'm trying to find out how to run PETSc in Xcode. Now I'm able to run the MPI in Xcode but still failed in running PETSc. Searched for the web but no luck at all. Does anyone have any instructions? Thanks very much in advance!
> 
> -- 
> Best regards,
> 
> Feng


From bsmith at mcs.anl.gov  Tue Oct 20 13:25:41 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 20 Oct 2015 13:25:41 -0500
Subject: [petsc-users] Call KSPConvergedDefault from Fortran
In-Reply-To: <CAFdHYfuwN1DvbRJSFwMGcEL0QU_iSwvNHdup=2YCLn8tdk_ALA@mail.gmail.com>
References: <CAFdHYfuwN1DvbRJSFwMGcEL0QU_iSwvNHdup=2YCLn8tdk_ALA@mail.gmail.com>
Message-ID: <07089B3C-88C2-4E95-9D8F-53E8D2FA2F14@mcs.anl.gov>


   It will be very similar to usage from C. You should be able to do something like

   PetscFortranAddress ctx

   call KSPConvergedDefaultCreate(ctx,ierr)

   call KSPMonitorSet(ksp, MyKSPConverged,ctx,kspconvergeddefaultdestroy,ierr)

   then in your MyKSPConverged pass the argument called "dummy" to the default monitor.

  Barry





> On Oct 20, 2015, at 3:36 AM, Natacha BEREUX <natacha.bereux at gmail.com> wrote:
> 
> Dear PETSc users, 
> 
> 
> I am using PETSc  Fortran API. 
> I try to solve a linear system, and would like to define my own convergence test. 
> More precisely, my convergence test calls the default convergence test 
> and then checks the true residual norm : 
>      subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)
> 
>       implicit none
> 
> #include <petsc/finclude/petscsys.h>
> #include <petsc/finclude/petscvec.h>
> #include <petsc/finclude/petscksp.h>
> 
>       KSP              ksp
>       PetscErrorCode ierr
>       PetscInt n,dummy
>       KSPConvergedReason flag
>       PetscReal rnorm, true_rnorm
>       Vec true_res
>       !
>       call KSPConvergedDefault(ksp,n,rnorm,flag,PETSC_NULL_OBJECT,ierr)
>       !
>       ! If convergence test succeeds 
>        if ( (flag == KSP_CONVERGED_ATOL) .or.                           &
>      &  (flag == KSP_CONVERGED_RTOL)) then 
>         ! Compute true residual
>         call KSPBuildResidual( ksp , PETSC_NULL_OBJECT,                 &
>      & PETSC_NULL_OBJECT, true_res, ierr )
>         ! Compute true residual norm 
>         call VecNorm(true_res,NORM_2,true_rnorm,ierr)
>         ! And check again convergence with respect to the true residual norm
>         call KSPConvergedDefault(ksp,n,true_rnorm,flag,                 &
>      & PETSC_NULL_OBJECT, ierr)
>         ! Free memory
>         call VecDestroy(true_res, ierr) 
>       endif 
>       !       
>       ierr = 0
> 
>       end
> 
> I get the following error message
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Null argument, when expecting valid pointer
> [0]PETSC ERROR: Convergence context must have been created with KSPConvergedDefaultCreate()
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015 
> 
> I understand that I should have  call KSPConvergedDefaultCreate() before calling KSConvergedDefault( ....) 
> 
> Am I right ? 
> 
> If so what is the Fortran calling sequence for  KSPConvergedDefaultCreate ? It is supposed to return a valid pointer and I don't succeed in doing so from Fortran. 
> 
> If you have any idea, I'd greatly appreciate it
> Thank you!
> Natacha 
> 
> 
> PS I have attached my code (it's a slightly modified version of ex2f.F showing  the convergence test code )
> <ex2f.F>


From fdkong.jd at gmail.com  Tue Oct 20 13:32:46 2015
From: fdkong.jd at gmail.com (Fande Kong)
Date: Tue, 20 Oct 2015 12:32:46 -0600
Subject: [petsc-users] mathematics formula of DMPlexProjectFunction
In-Reply-To: <87vba1zaci.fsf@jedbrown.org>
References: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>
	<87vba1zaci.fsf@jedbrown.org>
Message-ID: <CAN5Wd-+xWAX2Z3=Lm4kehj4PPoN2wt04V45yXNeo-S3OmTB4oQ@mail.gmail.com>

Jed, Thanks.

It means that the inner product against basis "n_i" is just the function
value at the point "x_i" if the function is sufficiently regular, e.g.,
sin(x) or cos(x).

The basis function at x_i is "x-x_i" which is not the one we use
to discretize the equations?


On Tue, Oct 20, 2015 at 12:03 PM, Jed Brown <jed at jedbrown.org> wrote:

> Fande Kong <fdkong.jd at gmail.com> writes:
> > Any body knows the mathematics formula corresponding to the function
> > DMPlexProjectFunction? I already went through the code, but I do not
> > understand quite well. I will appreciate any help.
>
> The definition of a finite element involves a dual space (the basis for
> which is sometimes called the "nodes").  For a typical finite element
> space, the "nodes" are Dirac delta functions at vertices.  Consequently,
> the inner product
>
>   (n_i, f) = \int \delta(x - x_i) f(x) = f(x_i)
>
> if f is sufficiently regular.  For more general dual basis functions,
> this inner product needs to be evaluated by quadrature.
> DMPlexProjectFunction does this projection.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/583f9986/attachment-0001.html>

From knepley at gmail.com  Tue Oct 20 13:44:22 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 20 Oct 2015 13:44:22 -0500
Subject: [petsc-users] mathematics formula of DMPlexProjectFunction
In-Reply-To: <CAN5Wd-+xWAX2Z3=Lm4kehj4PPoN2wt04V45yXNeo-S3OmTB4oQ@mail.gmail.com>
References: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>
	<87vba1zaci.fsf@jedbrown.org>
	<CAN5Wd-+xWAX2Z3=Lm4kehj4PPoN2wt04V45yXNeo-S3OmTB4oQ@mail.gmail.com>
Message-ID: <CAMYG4GmviPPY=-LDzN_FhjhYUDSadJyESnf4kzh4eSh_dCtFTQ@mail.gmail.com>

On Tue, Oct 20, 2015 at 1:32 PM, Fande Kong <fdkong.jd at gmail.com> wrote:

> Jed, Thanks.
>
> It means that the inner product against basis "n_i" is just the function
> value at the point "x_i" if the function is sufficiently regular, e.g.,
> sin(x) or cos(x).
>

No, n_i is not part of the basis for the approximation space, its a basis
vector for the dual space. Thus its
not an inner product, it the dual pairing. Here we can always use an
integral since the dual elements can
be represented by measures

  Riesz-Markov-Kakutani Representation Theorem (
https://en.wikipedia.org/wiki/Riesz%E2%80%93Markov%E2%80%93Kakutani_representation_theorem
)

    Matt


> The basis function at x_i is "x-x_i" which is not the one we use
> to discretize the equations?
>
>
> On Tue, Oct 20, 2015 at 12:03 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Fande Kong <fdkong.jd at gmail.com> writes:
>> > Any body knows the mathematics formula corresponding to the function
>> > DMPlexProjectFunction? I already went through the code, but I do not
>> > understand quite well. I will appreciate any help.
>>
>> The definition of a finite element involves a dual space (the basis for
>> which is sometimes called the "nodes").  For a typical finite element
>> space, the "nodes" are Dirac delta functions at vertices.  Consequently,
>> the inner product
>>
>>   (n_i, f) = \int \delta(x - x_i) f(x) = f(x_i)
>>
>> if f is sufficiently regular.  For more general dual basis functions,
>> this inner product needs to be evaluated by quadrature.
>> DMPlexProjectFunction does this projection.
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/bb3ac399/attachment.html>

From jed at jedbrown.org  Tue Oct 20 13:45:26 2015
From: jed at jedbrown.org (Jed Brown)
Date: Tue, 20 Oct 2015 12:45:26 -0600
Subject: [petsc-users] mathematics formula of DMPlexProjectFunction
In-Reply-To: <CAN5Wd-+xWAX2Z3=Lm4kehj4PPoN2wt04V45yXNeo-S3OmTB4oQ@mail.gmail.com>
References: <CAN5Wd-JCsnsP3d7OMCX1hS6WqNB5vLmXmzkSv1gyer0tUV5VJA@mail.gmail.com>
	<87vba1zaci.fsf@jedbrown.org>
	<CAN5Wd-+xWAX2Z3=Lm4kehj4PPoN2wt04V45yXNeo-S3OmTB4oQ@mail.gmail.com>
Message-ID: <87pp09z8e1.fsf@jedbrown.org>

Fande Kong <fdkong.jd at gmail.com> writes:

> Jed, Thanks.
>
> It means that the inner product against basis "n_i" is just the function
> value at the point "x_i" if the function is sufficiently regular, e.g.,
> sin(x) or cos(x).

Yes.

> The basis function at x_i is "x-x_i" which is not the one we use
> to discretize the equations?

The *dual basis* function in that case is \delta(x-x_i), the Dirac delta
function.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/8c35d879/attachment.pgp>

From thronesf at gmail.com  Tue Oct 20 15:21:10 2015
From: thronesf at gmail.com (Sharp Stone)
Date: Tue, 20 Oct 2015 16:21:10 -0400
Subject: [petsc-users] how to run petsc in Xcode
In-Reply-To: <150C60E1-D60A-4925-8F49-E6FA2A533F16@mcs.anl.gov>
References: <CABJpBAycd7qDEotaJHi_UinBCKQOjynE367TtPjPk2PaxOTVQg@mail.gmail.com>
	<150C60E1-D60A-4925-8F49-E6FA2A533F16@mcs.anl.gov>
Message-ID: <CABJpBAwLGRdTTOzCbxt26MUnjYMJWo0kh6TGsd_0GTmsm5fxiQ@mail.gmail.com>

Hi Barry,

Thank you for your reply.

I saw the manual instructions. But still do not know exactly what the
manual, actually 6 lines of statements, means. I'm sorry to ask is there
any further info or examples for stupid users such as me to build the Petsc
framework?

Thanks!

On Tue, Oct 20, 2015 at 2:14 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>  http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf  section 15.13
>
>
> > On Oct 20, 2015, at 10:13 AM, Sharp Stone <thronesf at gmail.com> wrote:
> >
> > Hi All,
> >
> > I'm trying to find out how to run PETSc in Xcode. Now I'm able to run
> the MPI in Xcode but still failed in running PETSc. Searched for the web
> but no luck at all. Does anyone have any instructions? Thanks very much in
> advance!
> >
> > --
> > Best regards,
> >
> > Feng
>
>


-- 
Best regards,

Feng
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/8b470bbd/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 20 20:06:17 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 20 Oct 2015 20:06:17 -0500
Subject: [petsc-users] how to run petsc in Xcode
In-Reply-To: <CABJpBAwLGRdTTOzCbxt26MUnjYMJWo0kh6TGsd_0GTmsm5fxiQ@mail.gmail.com>
References: <CABJpBAycd7qDEotaJHi_UinBCKQOjynE367TtPjPk2PaxOTVQg@mail.gmail.com>
	<150C60E1-D60A-4925-8F49-E6FA2A533F16@mcs.anl.gov>
	<CABJpBAwLGRdTTOzCbxt26MUnjYMJWo0kh6TGsd_0GTmsm5fxiQ@mail.gmail.com>
Message-ID: <8288DAC5-C74A-48B9-BA9D-4CB46A025E71@mcs.anl.gov>


Follow the instructions in systems/Apple/OSX/bin/makeall to build the PETSc framework and documenta- tion suitable for use in Xcode.

                      
199

15.14. PARALLELCOMMUNICATION PETSc3.6October2,2015


You can then use the PETSc framework in ${PETSC DIR}/arch-osx/PETSc.framework in the usual manner for Apple frameworks, see the examples in systems/Apple/OSX/examples. When working in xcode things like function name completion should work for all PETSc functions as well as MPI functions. You must also link against the Apple Accelerate.framework.

The instructions in makeall say 

#!/bin/sh
#
# This script makes a Apple Mac OS X installer for PETSc, it uses arch-osx-release.py, arch-osx-debug.py, makeframework, makedocs, makedmg
#
# Run from the root PETSc directory
#
# See ./makeframework on how to use the framework:
#
#

So just run this script and follow the directions at the end. So run 

./systems/Apple/OSX/bin/makeall



> On Oct 20, 2015, at 3:21 PM, Sharp Stone <thronesf at gmail.com> wrote:
> 
> Hi Barry,
> 
> Thank you for your reply. 
> 
> I saw the manual instructions. But still do not know exactly what the manual, actually 6 lines of statements, means. I'm sorry to ask is there any further info or examples for stupid users such as me to build the Petsc framework?
> 
> Thanks!
> 
> On Tue, Oct 20, 2015 at 2:14 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>  http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf  section 15.13
> 
> 
> > On Oct 20, 2015, at 10:13 AM, Sharp Stone <thronesf at gmail.com> wrote:
> >
> > Hi All,
> >
> > I'm trying to find out how to run PETSc in Xcode. Now I'm able to run the MPI in Xcode but still failed in running PETSc. Searched for the web but no luck at all. Does anyone have any instructions? Thanks very much in advance!
> >
> > --
> > Best regards,
> >
> > Feng
> 
> 
> 
> 
> -- 
> Best regards,
> 
> Feng

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image24496.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image24656.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image24816.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image24976.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25136.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25296.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25456.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25616.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25776.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image25936.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0009.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image26096.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0010.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page199image26256.png
Type: image/png
Size: 110 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0011.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: page200image992.png
Type: image/png
Size: 128 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151020/24171d04/attachment-0012.png>

From sahai2 at illinois.edu  Tue Oct 20 22:06:08 2015
From: sahai2 at illinois.edu (Sahai, Amal)
Date: Wed, 21 Oct 2015 03:06:08 +0000
Subject: [petsc-users] Parallel ODE solver
In-Reply-To: <20151020083432.GA27353@Patricks-MacBook-Pro-4.local>
References: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>
	<CABFzUT3Czvi4ddnw1Vy=YbF=3QmUA+zKftX9nOmgoixUgU5HAA@mail.gmail.com>,
	<20151020083432.GA27353@Patricks-MacBook-Pro-4.local>
Message-ID: <54F02FA4CAB5E646927484F9ADCEFBD126E68876@chimbx4.ad.uillinois.edu>

I was able to get a simple time dependent ODE running in parallel using the time-stepper in Petsc. I had a follow up question regarding this: For my original problem I can provide the RHS function but have no way of computing the Jacobian analytically and then storing it using DMTSSetIJacobianLocal. How can I compute the Jacobian numerically by just supplying the RHS function to Petsc without having to manually perturb and generate individual elements in the Jacobian myself?

Thanks!

Regards
Amal
________________________________________
From: Patrick Sanan [patrick.sanan at gmail.com]
Sent: Tuesday, October 20, 2015 4:34 AM
To: Julian Andrej
Cc: Sahai, Amal; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Parallel ODE solver

Further note that you can use sundials from within PETSc, without any
more hassle than supplying a configure flags (--download-sundials, or
see the other entries under SUNDIALS: with ./configure --help).
That could potentially be very helpful in moving to TS.

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/TS/TSSUNDIALS.html


On Tue, Oct 20, 2015 at 09:21:04AM +0200, Julian Andrej wrote:
> There are a variety of algorithms for ODE/DAEs in the PetscTS
> implementation. For examples see e.g.
> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/index.html
>
> Sundials is also capable of parallel solving, even from the Matlab
> sundialsTB interface, as far as i know.
>
> On Tue, Oct 20, 2015 at 9:10 AM, Sahai, Amal <sahai2 at illinois.edu> wrote:
> > I am trying to solve a very large system of ODEs for modeling the
> > time-varying composition of a homogeneous (no spatial variation) reaction
> > mixture composed of 50000 species. I was wondering if there is parallel ode
> > solver included in PETSc? I have been using the sundials package for solving
> > a smaller system in serial and would like to move on to parallel framework
> > to speed up my calculations.
> >
> > Regards
> > Amal

From bsmith at mcs.anl.gov  Tue Oct 20 22:32:43 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 20 Oct 2015 22:32:43 -0500
Subject: [petsc-users] Parallel ODE solver
In-Reply-To: <54F02FA4CAB5E646927484F9ADCEFBD126E68876@chimbx4.ad.uillinois.edu>
References: <54F02FA4CAB5E646927484F9ADCEFBD126E6871E@chimbx4.ad.uillinois.edu>
	<CABFzUT3Czvi4ddnw1Vy=YbF=3QmUA+zKftX9nOmgoixUgU5HAA@mail.gmail.com>
	<20151020083432.GA27353@Patricks-MacBook-Pro-4.local>
	<54F02FA4CAB5E646927484F9ADCEFBD126E68876@chimbx4.ad.uillinois.edu>
Message-ID: <D3FAD1CB-88DF-46CC-A95C-4AFD74E3059D@mcs.anl.gov>


  If you know (or mostly know) the nonzero structure of the Jacobian (and hopefully it is very sparse) then PETSc can automatically and pretty efficiently compute the Jacobian for you.

  See   http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/MatFD/MatFDColoringCreate.html  and the example http://www.mcs.anl.gov/petsc/petsc-current/src/snes/examples/tutorials/ex14.c.html

   If you do not know the nonzero structure of the Jacobian you'll need to tell us a bit more about your problem and we may have some ideas

   Barry



> On Oct 20, 2015, at 10:06 PM, Sahai, Amal <sahai2 at illinois.edu> wrote:
> 
> I was able to get a simple time dependent ODE running in parallel using the time-stepper in Petsc. I had a follow up question regarding this: For my original problem I can provide the RHS function but have no way of computing the Jacobian analytically and then storing it using DMTSSetIJacobianLocal. How can I compute the Jacobian numerically by just supplying the RHS function to Petsc without having to manually perturb and generate individual elements in the Jacobian myself?
> 
> Thanks!
> 
> Regards
> Amal
> ________________________________________
> From: Patrick Sanan [patrick.sanan at gmail.com]
> Sent: Tuesday, October 20, 2015 4:34 AM
> To: Julian Andrej
> Cc: Sahai, Amal; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Parallel ODE solver
> 
> Further note that you can use sundials from within PETSc, without any
> more hassle than supplying a configure flags (--download-sundials, or
> see the other entries under SUNDIALS: with ./configure --help).
> That could potentially be very helpful in moving to TS.
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/TS/TSSUNDIALS.html
> 
> 
> On Tue, Oct 20, 2015 at 09:21:04AM +0200, Julian Andrej wrote:
>> There are a variety of algorithms for ODE/DAEs in the PetscTS
>> implementation. For examples see e.g.
>> http://www.mcs.anl.gov/petsc/petsc-current/src/ts/examples/tutorials/index.html
>> 
>> Sundials is also capable of parallel solving, even from the Matlab
>> sundialsTB interface, as far as i know.
>> 
>> On Tue, Oct 20, 2015 at 9:10 AM, Sahai, Amal <sahai2 at illinois.edu> wrote:
>>> I am trying to solve a very large system of ODEs for modeling the
>>> time-varying composition of a homogeneous (no spatial variation) reaction
>>> mixture composed of 50000 species. I was wondering if there is parallel ode
>>> solver included in PETSc? I have been using the sundials package for solving
>>> a smaller system in serial and would like to move on to parallel framework
>>> to speed up my calculations.
>>> 
>>> Regards
>>> Amal


From hzhang at mcs.anl.gov  Wed Oct 21 10:42:33 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Wed, 21 Oct 2015 10:42:33 -0500
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <878u7296om.fsf@jedbrown.org>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
	<CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
	<87r3kvaoqy.fsf@jedbrown.org>
	<CAGCphBtnN3iUGfOhnkvH9kg9bwyabpUThw1BAWM7+hui3J15vw@mail.gmail.com>
	<878u7296om.fsf@jedbrown.org>
Message-ID: <CAGCphBsnXR=B4e8Sxa-AVqHEyMb0iKt8bKLPW2HiPm1XPn_XSA@mail.gmail.com>

Bikash,
I implemented MatTransposeMatMult_MPIDense_MPIDense()
in the branch hzhang/mattransmatmult_dense.
Once it passes our nightly tests, I'll merge it to petsc master branch.

See petsc/src/mat/examples/tests/ex104.c, in which I added a simple
example doing
B = P^T * A * P, where P is an MPIDENSE matrix and A is an MPIAIJ matrix.

Let us know if you see any bug or performance issues.

Hong

On Fri, Oct 16, 2015 at 10:25 AM, Jed Brown <jed at jedbrown.org> wrote:

> Hong <hzhang at mcs.anl.gov> writes:
>
> > Jed:
> >>
> >>
> >> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via
> >> >
> >> > 1. add MatTransposeMatMult_elemental_elemental()
> >> > 2. C_dense = P_dense^T * B_dense
> >> >     via MatConvert_dense_elemental() and MatConvert_elemental_dense()
> >>
> >> The above involves a ton of data movement and MPIDense is a logical
> >> distribution for matrices with a modest number of columns.  I think I
> >> would just do the local GEMM and then MPI_Reduce_scatter it.
> >>
> > This would work.
> >
> > However, I recall that you did a smart ordering which allows
> > MatConvert_mpidense_elemental() uses same physical matrix storage for
> petsc
> > and elemental,
>
> Same storage for vectors.  This is your code, but I think you'll find
> that it moves the matrix entries around.  (Note that Elemental [MC,MR]
> is a 2D distribution while MPIDense is 1D.)  Also, I think it would be
> better if this particular operation did not depend on Elemental.
>
> You could write a conversion to Elemental [VC,*], which would then match
> the MPIDense distribution and thus not need to move any matrix entries.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151021/eadd0fd4/attachment.html>

From bikash at umich.edu  Wed Oct 21 11:29:37 2015
From: bikash at umich.edu (Bikash Kanungo)
Date: Wed, 21 Oct 2015 12:29:37 -0400
Subject: [petsc-users] MatPtAP for involving MPIDENSE and MPIAIJ matrices
In-Reply-To: <CAGCphBsnXR=B4e8Sxa-AVqHEyMb0iKt8bKLPW2HiPm1XPn_XSA@mail.gmail.com>
References: <CA+H+vqJhcKce8+H0qfYB=AqwoBPPPSRWU9K+QN8EVUCfO1vZHw@mail.gmail.com>
	<CAMYG4GkoAgHvUuv2HR0usy0o4H4qGSYccakY6ctCTRw2DsBikg@mail.gmail.com>
	<87eggwashw.fsf@jedbrown.org>
	<CAMYG4G=6FqxCvndi+=zo7PWXO-XxYknXqF-poQUPE7_+v1TazQ@mail.gmail.com>
	<876128arw9.fsf@jedbrown.org>
	<CAGCphBtYi=yzsukTbJPvj35FnkXPtz4QMFjgraQ_NLR-g8qtXg@mail.gmail.com>
	<87r3kvaoqy.fsf@jedbrown.org>
	<CAGCphBtnN3iUGfOhnkvH9kg9bwyabpUThw1BAWM7+hui3J15vw@mail.gmail.com>
	<878u7296om.fsf@jedbrown.org>
	<CAGCphBsnXR=B4e8Sxa-AVqHEyMb0iKt8bKLPW2HiPm1XPn_XSA@mail.gmail.com>
Message-ID: <CA+H+vqJFHC8Rih4Knhs4kUN01BrVQQ3-Yf+FTm=Di76dcPxF0g@mail.gmail.com>

Thanks a ton, Hong. You guys are amazing! I'll report any bug or
performance issue if I find any.
On Oct 21, 2015 11:43 AM, "Hong" <hzhang at mcs.anl.gov> wrote:

> Bikash,
> I implemented MatTransposeMatMult_MPIDense_MPIDense()
> in the branch hzhang/mattransmatmult_dense.
> Once it passes our nightly tests, I'll merge it to petsc master branch.
>
> See petsc/src/mat/examples/tests/ex104.c, in which I added a simple
> example doing
> B = P^T * A * P, where P is an MPIDENSE matrix and A is an MPIAIJ matrix.
>
> Let us know if you see any bug or performance issues.
>
> Hong
>
> On Fri, Oct 16, 2015 at 10:25 AM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Hong <hzhang at mcs.anl.gov> writes:
>>
>> > Jed:
>> >>
>> >>
>> >> > I plan to implement MatTransposeMatMult_MPIDense_MPIDense via
>> >> >
>> >> > 1. add MatTransposeMatMult_elemental_elemental()
>> >> > 2. C_dense = P_dense^T * B_dense
>> >> >     via MatConvert_dense_elemental() and MatConvert_elemental_dense()
>> >>
>> >> The above involves a ton of data movement and MPIDense is a logical
>> >> distribution for matrices with a modest number of columns.  I think I
>> >> would just do the local GEMM and then MPI_Reduce_scatter it.
>> >>
>> > This would work.
>> >
>> > However, I recall that you did a smart ordering which allows
>> > MatConvert_mpidense_elemental() uses same physical matrix storage for
>> petsc
>> > and elemental,
>>
>> Same storage for vectors.  This is your code, but I think you'll find
>> that it moves the matrix entries around.  (Note that Elemental [MC,MR]
>> is a 2D distribution while MPIDense is 1D.)  Also, I think it would be
>> better if this particular operation did not depend on Elemental.
>>
>> You could write a conversion to Elemental [VC,*], which would then match
>> the MPIDense distribution and thus not need to move any matrix entries.
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151021/24b0404f/attachment.html>

From dave.may at erdw.ethz.ch  Wed Oct 21 15:07:22 2015
From: dave.may at erdw.ethz.ch (Dave May)
Date: Wed, 21 Oct 2015 21:07:22 +0100
Subject: [petsc-users] DM question.
In-Reply-To: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
Message-ID: <CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>

Hey Mike,



On 21 October 2015 at 18:01, Afanasiev Michael <
michael.afanasiev at erdw.ethz.ch> wrote:

> Hey Dave,
>
> So I?ve got a couple of days where there?s nothing pressing to work on?
> was thinking of ripping out the parallel routines (ugly) in my wave
> propagation code and replacing them with Petsc DM routines. I can read in
> my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a
> nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.
>
> But here I?m stuck, and am having a whale of a time with the
> documentation. All I *think* I need is a way to modify the exodus-created
> DM, and add to it the degrees of freedom that are introduced by my
> quadrature rule. This would be really neat. I can just treat each
> sub-domain as its own mesh, with its own global numbering. Then whenever
> necessary I can scatter stuff the the *real* global degrees of freedom
> with something like VecLocToGlob. Most of the things I like about the code
> could stay the same (element-wise, matrix-free nature), just these parallel
> broadcasts would be infinitely nicer.
>
>
First off - I don't use DMPLEX.


> But I just can?t figure out how to set this up. The main problem really
> boils down to: what?s the best way to add my quadrature points to an
> already-created DM, which was constructed with an exodus file? I guess I
> could do this after the file is read, but before the partitioning. In this
> case though, what?s stopping the partitioner from cutting an element in
> half? It seems like it would be a lot cleaner to do this post-partitioning.
>
>
Presumably what is read from exodus is just the vertices of the hexes, and
what you want to do is define the function space (given by your GLL
locations) on top of element geometry read in. Is that what you are asking
about?


> Any hints here?
>

Actually I have no experience with this object.
I would just send an email to
  petsc-users at mcs.anl.gov
asking for help.

The developer of DMPLEX (Matt Knepley) will definitely answer within in 1
day.

Cheers,
  Dave


> Best,
> Mike.
> --
> Michael Afanasiev
> Ph.D. Candidate
> Computational Seismology
> Institut f?r Geophysik
> ETH Z?rich
>
> Sonneggstrasse 5, NO H 39.2
> CH 8092 Z?rich
> michael.afanasiev at erdw.ethz.ch
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151021/ec209c27/attachment.html>

From knepley at gmail.com  Wed Oct 21 19:32:35 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Wed, 21 Oct 2015 19:32:35 -0500
Subject: [petsc-users] DM question.
In-Reply-To: <CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
Message-ID: <CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>

On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch> wrote:

> Hey Mike,
>
>
>
> On 21 October 2015 at 18:01, Afanasiev Michael <
> michael.afanasiev at erdw.ethz.ch> wrote:
>
>> Hey Dave,
>>
>> So I?ve got a couple of days where there?s nothing pressing to work on?
>> was thinking of ripping out the parallel routines (ugly) in my wave
>> propagation code and replacing them with Petsc DM routines. I can read in
>> my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a
>> nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.
>>
>> But here I?m stuck, and am having a whale of a time with the
>> documentation. All I *think* I need is a way to modify the
>> exodus-created DM, and add to it the degrees of freedom that are introduced
>> by my quadrature rule. This would be really neat. I can just treat each
>> sub-domain as its own mesh, with its own global numbering. Then whenever
>> necessary I can scatter stuff the the *real* global degrees of freedom
>> with something like VecLocToGlob. Most of the things I like about the code
>> could stay the same (element-wise, matrix-free nature), just these parallel
>> broadcasts would be infinitely nicer.
>>
>>
> First off - I don't use DMPLEX.
>

Dave is refreshingly candid about his shortcomings ;)


>
>
But I just can?t figure out how to set this up. The main problem really
>> boils down to: what?s the best way to add my quadrature points to an
>> already-created DM, which was constructed with an exodus file? I guess I
>> could do this after the file is read, but before the partitioning. In this
>> case though, what?s stopping the partitioner from cutting an element in
>> half? It seems like it would be a lot cleaner to do this post-partitioning.
>>
>>
> Presumably what is read from exodus is just the vertices of the hexes, and
> what you want to do is define the function space (given by your GLL
> locations) on top of element geometry read in. Is that what you are asking
> about?
>

So Dave is right. We read in topology and geometry from ExodusII. Then you
define a function space on top. How
exactly are you discretizing? In order to create vectors, do local to
global, etc. Petsc really only need to know the
amount of data associated with each mesh piece. You can define this with
PetscSection. If you give me an idea
what you want I can help you write the code easily I think.

  Thanks,

     Matt


> Any hints here?
>>
>
> Actually I have no experience with this object.
> I would just send an email to
>   petsc-users at mcs.anl.gov
> asking for help.
>
> The developer of DMPLEX (Matt Knepley) will definitely answer within in 1
> day.
>
> Cheers,
>   Dave
>
>
>> Best,
>> Mike.
>> --
>> Michael Afanasiev
>> Ph.D. Candidate
>> Computational Seismology
>> Institut f?r Geophysik
>> ETH Z?rich
>>
>> Sonneggstrasse 5, NO H 39.2
>> CH 8092 Z?rich
>> michael.afanasiev at erdw.ethz.ch
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151021/1894023c/attachment.html>

From Lukasz.Kaczmarczyk at glasgow.ac.uk  Thu Oct 22 04:22:49 2015
From: Lukasz.Kaczmarczyk at glasgow.ac.uk (Lukasz Kaczmarczyk)
Date: Thu, 22 Oct 2015 09:22:49 +0000
Subject: [petsc-users] lecture about dynamics
Message-ID: <CAE9C3C4-A6BE-4861-96BA-CC61824CC4CA@glasgow.ac.uk>

Peter,

At the end, about 40 minutes is about amplification factor. This is for oscillatory excitation,  however you can get something from this.

http://ocw.mit.edu/resources/res-2-002-finite-element-procedures-for-solids-and-structures-spring-2010/linear/lecture-10/


Regards,
L.

From natacha.bereux at gmail.com  Thu Oct 22 04:24:48 2015
From: natacha.bereux at gmail.com (Natacha BEREUX)
Date: Thu, 22 Oct 2015 11:24:48 +0200
Subject: [petsc-users] Call KSPConvergedDefault from Fortran
In-Reply-To: <07089B3C-88C2-4E95-9D8F-53E8D2FA2F14@mcs.anl.gov>
References: <CAFdHYfuwN1DvbRJSFwMGcEL0QU_iSwvNHdup=2YCLn8tdk_ALA@mail.gmail.com>
	<07089B3C-88C2-4E95-9D8F-53E8D2FA2F14@mcs.anl.gov>
Message-ID: <CAFdHYfs_NsU=bZtew=9GFHhAqMDJn-69BCS7Qk+=AGtKMGeL=g@mail.gmail.com>

Thanks a lot Barry : it works perfectly for me !
Best regards
Natacha


On Tue, Oct 20, 2015 at 8:25 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>    It will be very similar to usage from C. You should be able to do
> something like
>
>    PetscFortranAddress ctx
>
>    call KSPConvergedDefaultCreate(ctx,ierr)
>
>    call KSPMonitorSet(ksp,
> MyKSPConverged,ctx,kspconvergeddefaultdestroy,ierr)
>
>    then in your MyKSPConverged pass the argument called "dummy" to the
> default monitor.
>
>   Barry
>
>
>
>
>
> > On Oct 20, 2015, at 3:36 AM, Natacha BEREUX <natacha.bereux at gmail.com>
> wrote:
> >
> > Dear PETSc users,
> >
> >
> > I am using PETSc  Fortran API.
> > I try to solve a linear system, and would like to define my own
> convergence test.
> > More precisely, my convergence test calls the default convergence test
> > and then checks the true residual norm :
> >      subroutine MyKSPConverged(ksp,n,rnorm,flag,dummy,ierr)
> >
> >       implicit none
> >
> > #include <petsc/finclude/petscsys.h>
> > #include <petsc/finclude/petscvec.h>
> > #include <petsc/finclude/petscksp.h>
> >
> >       KSP              ksp
> >       PetscErrorCode ierr
> >       PetscInt n,dummy
> >       KSPConvergedReason flag
> >       PetscReal rnorm, true_rnorm
> >       Vec true_res
> >       !
> >       call KSPConvergedDefault(ksp,n,rnorm,flag,PETSC_NULL_OBJECT,ierr)
> >       !
> >       ! If convergence test succeeds
> >        if ( (flag == KSP_CONVERGED_ATOL) .or.                           &
> >      &  (flag == KSP_CONVERGED_RTOL)) then
> >         ! Compute true residual
> >         call KSPBuildResidual( ksp , PETSC_NULL_OBJECT,                 &
> >      & PETSC_NULL_OBJECT, true_res, ierr )
> >         ! Compute true residual norm
> >         call VecNorm(true_res,NORM_2,true_rnorm,ierr)
> >         ! And check again convergence with respect to the true residual
> norm
> >         call KSPConvergedDefault(ksp,n,true_rnorm,flag,                 &
> >      & PETSC_NULL_OBJECT, ierr)
> >         ! Free memory
> >         call VecDestroy(true_res, ierr)
> >       endif
> >       !
> >       ierr = 0
> >
> >       end
> >
> > I get the following error message
> >
> > [0]PETSC ERROR: --------------------- Error Message
> --------------------------------------------------------------
> > [0]PETSC ERROR: Null argument, when expecting valid pointer
> > [0]PETSC ERROR: Convergence context must have been created with
> KSPConvergedDefaultCreate()
> > [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
> for trouble shooting.
> > [0]PETSC ERROR: Petsc Release Version 3.6.2, Oct, 02, 2015
> >
> > I understand that I should have  call KSPConvergedDefaultCreate() before
> calling KSConvergedDefault( ....)
> >
> > Am I right ?
> >
> > If so what is the Fortran calling sequence for
> KSPConvergedDefaultCreate ? It is supposed to return a valid pointer and I
> don't succeed in doing so from Fortran.
> >
> > If you have any idea, I'd greatly appreciate it
> > Thank you!
> > Natacha
> >
> >
> > PS I have attached my code (it's a slightly modified version of ex2f.F
> showing  the convergence test code )
> > <ex2f.F>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/644787c0/attachment-0001.html>

From zocca.marco at gmail.com  Thu Oct 22 09:20:35 2015
From: zocca.marco at gmail.com (Marco Zocca)
Date: Thu, 22 Oct 2015 16:20:35 +0200
Subject: [petsc-users] on SNESSetJacobian semantics
Message-ID: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>

The signature of SNESSetJacobian is

PetscErrorCode  SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode
(*J)(SNES,Vec,Mat,Mat,void*),void *ctx)

however it would seem redundant to supply both a constant matrix and a
function to compute the Jacobian.
The manual says of J "(if NULL then SNES retains any previously set value":
this would only apply to linear functions, is this correct?

Why then are there not two monomorphic "SetJacobian" functions, one for
linear maps that wouldn't require recomputation and one for every other
case ?

Coming from a functional background, I find reasoning with "NULL" to be
very error-prone.

Thank you,
Marco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/a81ab14c/attachment.html>

From knepley at gmail.com  Thu Oct 22 09:30:59 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 22 Oct 2015 09:30:59 -0500
Subject: [petsc-users] on SNESSetJacobian semantics
In-Reply-To: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
References: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
Message-ID: <CAMYG4G=g09cYre-MeRaq+hcETe4XMTqk2OpZr9pbNRLmRuGBDQ@mail.gmail.com>

On Thu, Oct 22, 2015 at 9:20 AM, Marco Zocca <zocca.marco at gmail.com> wrote:

> The signature of SNESSetJacobian is
>
> PetscErrorCode  SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode
> (*J)(SNES,Vec,Mat,Mat,void*),void *ctx)
>
> however it would seem redundant to supply both a constant matrix and a
> function to compute the Jacobian.
> The manual says of J "(if NULL then SNES retains any previously set
> value": this would only apply to linear functions, is this correct?
>

It means, if you give NULL, but have previously called SetJacobian() then
it uses the function pointer previously passed
rather than setting it to NULL.

  Matt


> Why then are there not two monomorphic "SetJacobian" functions, one for
> linear maps that wouldn't require recomputation and one for every other
> case ?
>
> Coming from a functional background, I find reasoning with "NULL" to be
> very error-prone.
>
> Thank you,
> Marco
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/c493a3ee/attachment.html>

From zocca.marco at gmail.com  Thu Oct 22 09:38:29 2015
From: zocca.marco at gmail.com (Marco Zocca)
Date: Thu, 22 Oct 2015 16:38:29 +0200
Subject: [petsc-users] on SNESSetJacobian semantics
In-Reply-To: <CAMYG4G=g09cYre-MeRaq+hcETe4XMTqk2OpZr9pbNRLmRuGBDQ@mail.gmail.com>
References: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
	<CAMYG4G=g09cYre-MeRaq+hcETe4XMTqk2OpZr9pbNRLmRuGBDQ@mail.gmail.com>
Message-ID: <CAKE6T0Tzhk25ZRHnsQpAsRsW0sUccoB5aBUVqS_oLxDSZ4VmEQ@mail.gmail.com>

Thank you, but this is not really helping;

why are there two slots to supply the Jacobian of the same function, i.e.
Amat and J ?





On 22 October 2015 at 16:30, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Oct 22, 2015 at 9:20 AM, Marco Zocca <zocca.marco at gmail.com>
> wrote:
>
>> The signature of SNESSetJacobian is
>>
>> PetscErrorCode  SNESSetJacobian(SNES snes,Mat Amat,Mat
>> Pmat,PetscErrorCode (*J)(SNES,Vec,Mat,Mat,void*),void *ctx)
>>
>> however it would seem redundant to supply both a constant matrix and a
>> function to compute the Jacobian.
>> The manual says of J "(if NULL then SNES retains any previously set
>> value": this would only apply to linear functions, is this correct?
>>
>
> It means, if you give NULL, but have previously called SetJacobian() then
> it uses the function pointer previously passed
> rather than setting it to NULL.
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/452024b8/attachment.html>

From knepley at gmail.com  Thu Oct 22 09:40:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 22 Oct 2015 09:40:04 -0500
Subject: [petsc-users] on SNESSetJacobian semantics
In-Reply-To: <CAKE6T0SKnAbXpR3p2_SqBJvrfDTTyotf79pA6eZvqnPWGh-HyQ@mail.gmail.com>
References: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
	<CAMYG4G=g09cYre-MeRaq+hcETe4XMTqk2OpZr9pbNRLmRuGBDQ@mail.gmail.com>
	<CAKE6T0SKnAbXpR3p2_SqBJvrfDTTyotf79pA6eZvqnPWGh-HyQ@mail.gmail.com>
Message-ID: <CAMYG4GnkuV=11fYn6HbhgaPuXdObZwD6Y4Pug9Dz0bv_H3ytiw@mail.gmail.com>

On Thu, Oct 22, 2015 at 9:37 AM, Marco Zocca <zocca.marco at gmail.com> wrote:

> Thank you, but this is not really helping;
>
> why are there two slots to supply the Jacobian of the same function, i.e.
> Amat and J ?
>

Amat is just the storage, not the values. J computes the values.

   Matt


> Marco
>
>
>
>
> On 22 October 2015 at 16:30, Matthew Knepley <knepley at gmail.com> wrote:
>
>> On Thu, Oct 22, 2015 at 9:20 AM, Marco Zocca <zocca.marco at gmail.com>
>> wrote:
>>
>>> The signature of SNESSetJacobian is
>>>
>>> PetscErrorCode  SNESSetJacobian(SNES snes,Mat Amat,Mat
>>> Pmat,PetscErrorCode (*J)(SNES,Vec,Mat,Mat,void*),void *ctx)
>>>
>>> however it would seem redundant to supply both a constant matrix and a
>>> function to compute the Jacobian.
>>> The manual says of J "(if NULL then SNES retains any previously set
>>> value": this would only apply to linear functions, is this correct?
>>>
>>
>> It means, if you give NULL, but have previously called SetJacobian() then
>> it uses the function pointer previously passed
>> rather than setting it to NULL.
>>
>>   Matt
>>
>>
>>> Why then are there not two monomorphic "SetJacobian" functions, one for
>>> linear maps that wouldn't require recomputation and one for every other
>>> case ?
>>>
>>> Coming from a functional background, I find reasoning with "NULL" to be
>>> very error-prone.
>>>
>>> Thank you,
>>> Marco
>>>
>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/d22dd167/attachment.html>

From jed at jedbrown.org  Thu Oct 22 10:46:24 2015
From: jed at jedbrown.org (Jed Brown)
Date: Thu, 22 Oct 2015 09:46:24 -0600
Subject: [petsc-users] on SNESSetJacobian semantics
In-Reply-To: <CAMYG4GnkuV=11fYn6HbhgaPuXdObZwD6Y4Pug9Dz0bv_H3ytiw@mail.gmail.com>
References: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
	<CAMYG4G=g09cYre-MeRaq+hcETe4XMTqk2OpZr9pbNRLmRuGBDQ@mail.gmail.com>
	<CAKE6T0SKnAbXpR3p2_SqBJvrfDTTyotf79pA6eZvqnPWGh-HyQ@mail.gmail.com>
	<CAMYG4GnkuV=11fYn6HbhgaPuXdObZwD6Y4Pug9Dz0bv_H3ytiw@mail.gmail.com>
Message-ID: <87611yykhb.fsf@jedbrown.org>

Matthew Knepley <knepley at gmail.com> writes:

> On Thu, Oct 22, 2015 at 9:37 AM, Marco Zocca <zocca.marco at gmail.com> wrote:
>
>> Thank you, but this is not really helping;
>>
>> why are there two slots to supply the Jacobian of the same function, i.e.
>> Amat and J ?
>>
>
> Amat is just the storage, not the values. J computes the values.

More precisely, Amat provides the action of the Jacobian (possibly
matrix-free).

Pmat provides the data from which to construct the preconditioner.  This
is most frequently an assembled matrix (possibly from a simpler
discretization, skipping some terms, etc.), but could hold additional
information or eschew matrix entries entirely.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/b4e41087/attachment.pgp>

From bsmith at mcs.anl.gov  Thu Oct 22 13:33:45 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 22 Oct 2015 13:33:45 -0500
Subject: [petsc-users] on SNESSetJacobian semantics
In-Reply-To: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
References: <CAKE6T0Qe7HM3vFPLM48A=qabdpWDf+pQyJA2v3Nj-hxQVMhcEw@mail.gmail.com>
Message-ID: <A1E3DEC2-EA79-4AD9-BE96-0C28BBA42B98@mcs.anl.gov>


> On Oct 22, 2015, at 9:20 AM, Marco Zocca <zocca.marco at gmail.com> wrote:
> 
> The signature of SNESSetJacobian is 
> 
> PetscErrorCode  SNESSetJacobian(SNES snes,Mat Amat,Mat Pmat,PetscErrorCode (*J)(SNES,Vec,Mat,Mat,void*),void *ctx)         
> 
> however it would seem redundant to supply both a constant matrix and a function to compute the Jacobian.
> The manual says of J "(if NULL then SNES retains any previously set value": this would only apply to linear functions, is this correct?
> 
> Why then are there not two monomorphic "SetJacobian" functions, one for linear maps that wouldn't require recomputation and one for every other case ? 

  For the linear case you can just pass in a function J that simply returns without changing the matrices.

  Barry

> 
> Coming from a functional background, I find reasoning with "NULL" to be very error-prone.

   Agreed. Limitation of C and the style of API we selected of trying to minimize the number of different functions in the interface.

> 
> Thank you,
> Marco


From daveliu at mit.edu  Thu Oct 22 13:42:56 2015
From: daveliu at mit.edu (David Liu)
Date: Thu, 22 Oct 2015 14:42:56 -0400
Subject: [petsc-users] configuring petsc without downloading packages
Message-ID: <CAMSHvJGkSALGs1Z4MgyRyQ0A7s7HdQ6EeMO2dytrWdpPaDEZRw@mail.gmail.com>

Hi, I have a simple question that I could not find a comprehensive answer
anywhere on the website or manual, though I found examples of what I'm
trying to do. Say I want to build a version of PETSc with some external
package, such as MUMPS or SuperLU_DIST. I can do this by using flags such
as "--download-superlu_dist" when I am configuring. However, what if I
already have SuperLU_DIST, say either by installing it by myself from the
SuperLU_DIST source, or from a previous configuration of PETSc, i.e. with
downloaded files sitting in ${PETSC_DIR}/${PETSC_ARCH}/externalpackages?
How can I save the trouble of re-downloading a package that I already have?
For MPI, I can specify using "--with-mpi-dir", or I can simply have mpirun
and mpicc, etc in my path. But is there a standard way to do this for
external packages that you usually download? I vaguely recall reading
somewhere that you can simply do

--download-superlu_dist=${PATH_TO_SUPERLU_DIST}

but that notation obviously seems a little weird.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/86ed2e15/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 22 13:49:23 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 22 Oct 2015 13:49:23 -0500
Subject: [petsc-users] configuring petsc without downloading packages
In-Reply-To: <CAMSHvJGkSALGs1Z4MgyRyQ0A7s7HdQ6EeMO2dytrWdpPaDEZRw@mail.gmail.com>
References: <CAMSHvJGkSALGs1Z4MgyRyQ0A7s7HdQ6EeMO2dytrWdpPaDEZRw@mail.gmail.com>
Message-ID: <D066576D-B391-4934-93C5-B7ADB5C796FD@mcs.anl.gov>


> On Oct 22, 2015, at 1:42 PM, David Liu <daveliu at mit.edu> wrote:
> 
> Hi, I have a simple question that I could not find a comprehensive answer anywhere on the website or manual, though I found examples of what I'm trying to do. Say I want to build a version of PETSc with some external package, such as MUMPS or SuperLU_DIST. I can do this by using flags such as "--download-superlu_dist" when I am configuring. However, what if I already have SuperLU_DIST, say either by installing it by myself from the SuperLU_DIST source, or from a previous configuration of PETSc, i.e. with downloaded files sitting in ${PETSC_DIR}/${PETSC_ARCH}/externalpackages? How can I save the trouble of re-downloading a package that I already have? For MPI, I can specify using "--with-mpi-dir", or I can simply have mpirun and mpicc, etc in my path. But is there a standard way to do this for external packages that you usually download?


   It is the same. --with-mumps-dir etc.

   Note that each release of PETSc only works with one release of external packages so if you don't use the --download option you need to verify that you are using the correct release of the package. You can look in config/BuildSystem/config/packages/xxx.py to find what version of the external package xxx we support.

  Barry

> I vaguely recall reading somewhere that you can simply do
> 
> --download-superlu_dist=${PATH_TO_SUPERLU_DIST}
> 
> but that notation obviously seems a little weird. 


From knepley at gmail.com  Thu Oct 22 13:50:01 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Thu, 22 Oct 2015 13:50:01 -0500
Subject: [petsc-users] configuring petsc without downloading packages
In-Reply-To: <CAMSHvJGkSALGs1Z4MgyRyQ0A7s7HdQ6EeMO2dytrWdpPaDEZRw@mail.gmail.com>
References: <CAMSHvJGkSALGs1Z4MgyRyQ0A7s7HdQ6EeMO2dytrWdpPaDEZRw@mail.gmail.com>
Message-ID: <CAMYG4Gn_tp88K_V_FKGZ-9WymX08BErrt5_TZj6hgkcwRZdCMw@mail.gmail.com>

On Thu, Oct 22, 2015 at 1:42 PM, David Liu <daveliu at mit.edu> wrote:

> Hi, I have a simple question that I could not find a comprehensive answer
> anywhere on the website or manual, though I found examples of what I'm
> trying to do. Say I want to build a version of PETSc with some external
> package, such as MUMPS or SuperLU_DIST. I can do this by using flags such
> as "--download-superlu_dist" when I am configuring. However, what if I
> already have SuperLU_DIST, say either by installing it by myself from the
> SuperLU_DIST source, or from a previous configuration of PETSc, i.e. with
> downloaded files sitting in ${PETSC_DIR}/${PETSC_ARCH}/externalpackages?
> How can I save the trouble of re-downloading a package that I already have?
> For MPI, I can specify using "--with-mpi-dir", or I can simply have mpirun
> and mpicc, etc in my path. But is there a standard way to do this for
> external packages that you usually download? I vaguely recall reading
> somewhere that you can simply do
>
> --download-superlu_dist=${PATH_TO_SUPERLU_DIST}
>
> but that notation obviously seems a little weird.
>

1) If you already have it in externalpackages/ it will not download it again

2) If you download the tarball yourself, then
--download-<package>=<path/to/tarball> will use it

3) If you installed the package yourself,
--with-<package>-dir=</path/to/package> will use that

   Matt

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151022/add3e048/attachment.html>

From michael.afanasiev at erdw.ethz.ch  Fri Oct 23 03:01:03 2015
From: michael.afanasiev at erdw.ethz.ch (Afanasiev  Michael)
Date: Fri, 23 Oct 2015 08:01:03 +0000
Subject: [petsc-users] DM question.
In-Reply-To: <CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
Message-ID: <453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>

Hi Matthew,

So I?m discretizing via a tensor product of Lagrange polynomials co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order might range from 4 to 10 or something like that. The current problem is solved on 2D hexes.

I had found DMPlexCreateSection, and followed plex/ex1to get things set up. You can see my attempt below. Basically I?ve got 4 fields (displacement, velocity, acceleration, and strain) over each element. Here I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange polynomials (line 11). This seemed to somewhat achieve what I wanted ? I created a global vector and wrote it to a vtk file with VecView, and the numbering seemed to make sense. You can also see my attempt at defining a boundary condition (it looked like DMPlexCreateFromExodus labeled side sets as ?Face Sets?, seems to have worked).

Does this seem to be headed in the right direction?

Cheers,
Mike.

DM
mesh::createSection(const DM &dm)
{

01        // displacement, velocity, acceleration, strain
02        IS bcPointIs[1];
03        PetscInt numBc = 1;
04        PetscInt bcField[1];
05        PetscInt numFields = 4;
06        PetscInt dim; DMGetDimension(dm, &dim);
07        PetscInt numComp[numFields];
08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
09        PetscInt numDof[numFields*(dim+1)];
10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
12        bcField[0] = 0;
13        PetscSection section;
14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof, numBc, bcField,
16                                                NULL, bcPointIs, NULL, &section);
17        ISDestroy(&bcPointIs[0]);
18        PetscSectionSetFieldName(section, 0, "u");
19        PetscSectionSetFieldName(section, 1, "v");
20        PetscSectionSetFieldName(section, 2, "a");
21        PetscSectionSetFieldName(section, 3, "e");
22        DMSetDefaultSection(dm, section);
23        return dm;
}

--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch<mailto:dave.may at erdw.ethz.ch>> wrote:
Hey Mike,



On 21 October 2015 at 18:01, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hey Dave,

So I?ve got a couple of days where there?s nothing pressing to work on? was thinking of ripping out the parallel routines (ugly) in my wave propagation code and replacing them with Petsc DM routines. I can read in my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.

But here I?m stuck, and am having a whale of a time with the documentation. All I think I need is a way to modify the exodus-created DM, and add to it the degrees of freedom that are introduced by my quadrature rule. This would be really neat. I can just treat each sub-domain as its own mesh, with its own global numbering. Then whenever necessary I can scatter stuff the the real global degrees of freedom with something like VecLocToGlob. Most of the things I like about the code could stay the same (element-wise, matrix-free nature), just these parallel broadcasts would be infinitely nicer.


First off - I don't use DMPLEX.

Dave is refreshingly candid about his shortcomings ;)


But I just can?t figure out how to set this up. The main problem really boils down to: what?s the best way to add my quadrature points to an already-created DM, which was constructed with an exodus file? I guess I could do this after the file is read, but before the partitioning. In this case though, what?s stopping the partitioner from cutting an element in half? It seems like it would be a lot cleaner to do this post-partitioning.


Presumably what is read from exodus is just the vertices of the hexes, and what you want to do is define the function space (given by your GLL locations) on top of element geometry read in. Is that what you are asking about?

So Dave is right. We read in topology and geometry from ExodusII. Then you define a function space on top. How
exactly are you discretizing? In order to create vectors, do local to global, etc. Petsc really only need to know the
amount of data associated with each mesh piece. You can define this with PetscSection. If you give me an idea
what you want I can help you write the code easily I think.

  Thanks,

     Matt

Any hints here?

Actually I have no experience with this object.
I would just send an email to
  petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
asking for help.

The developer of DMPLEX (Matt Knepley) will definitely answer within in 1 day.

Cheers,
  Dave


Best,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>





--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151023/be3fe575/attachment-0001.html>

From lawrence.mitchell at imperial.ac.uk  Fri Oct 23 05:52:23 2015
From: lawrence.mitchell at imperial.ac.uk (Lawrence Mitchell)
Date: Fri, 23 Oct 2015 11:52:23 +0100
Subject: [petsc-users] DMPlex natural to global mappings
Message-ID: <3B30D93A-5982-4C2A-94A0-AA3217720C19@imperial.ac.uk>

Dear all,

I'm trying to understand how to use the recent natural-global mappings that have made it into dmplex.  The goal is to do checkpoint restart (onto, perhaps, differing numbers of processes) with DMs with a non-zero overlap (I can see that this is currently unsupported, but let's try the zero overlap case first).

I'm running into a number of problems that may be misunderstandings, but are possibly bugs.

Let's try the simplest thing first, build a DMPlex on two processes, dump it to disk and then try to reload.

#include <petsc.h>


int main(int argc, char **argv)
{
    PetscErrorCode ierr;
    MPI_Comm       comm;
    DM             dm, newdm, pardm;
    PetscViewer    vwr;
    const char    *name = "dump.h5";

    PetscInitialize(&argc, &argv, NULL, NULL);

    comm = PETSC_COMM_WORLD;
    ierr = DMPlexCreateBoxMesh(comm, 2, PETSC_TRUE, &dm); CHKERRQ(ierr);
    ierr = DMPlexDistribute(dm, 0, NULL, &pardm); CHKERRQ(ierr);
    if (pardm) {
        ierr = DMDestroy(&dm); CHKERRQ(ierr);
        dm = pardm;
    }

    ierr = PetscViewerCreate(comm, &vwr); CHKERRQ(ierr);
    ierr = PetscViewerSetType(vwr, PETSCVIEWERHDF5); CHKERRQ(ierr);
    ierr = PetscViewerFileSetMode(vwr, FILE_MODE_WRITE); CHKERRQ(ierr);
    ierr = PetscViewerSetFormat(vwr, PETSC_VIEWER_NATIVE); CHKERRQ(ierr);
    ierr = PetscViewerFileSetName(vwr, name); CHKERRQ(ierr);
    ierr = DMView(dm, vwr); CHKERRQ(ierr);

    ierr = PetscViewerDestroy(&vwr); CHKERRQ(ierr);

    ierr = DMCreate(comm, &newdm); CHKERRQ(ierr);
    ierr = DMSetType(newdm, DMPLEX); CHKERRQ(ierr);

    ierr = PetscViewerCreate(comm, &vwr);
    ierr = PetscViewerSetType(vwr, PETSCVIEWERHDF5); CHKERRQ(ierr);
    ierr = PetscViewerFileSetMode(vwr, FILE_MODE_READ); CHKERRQ(ierr);
    ierr = PetscViewerSetFormat(vwr, PETSC_VIEWER_NATIVE); CHKERRQ(ierr);
    ierr = PetscViewerFileSetName(vwr, name); CHKERRQ(ierr);

    ierr = DMLoad(newdm, vwr); CHKERRQ(ierr);

    ierr = PetscViewerDestroy(&vwr); CHKERRQ(ierr);

    ierr = DMPlexDistribute(newdm, 0, NULL, &pardm); CHKERRQ(ierr);

    if (pardm) {
        ierr = DMDestroy(&newdm); CHKERRQ(ierr);
        newdm = pardm;
    }

    ierr = DMDestroy(&dm); CHKERRQ(ierr);
    ierr = DMDestroy(&newdm); CHKERRQ(ierr);
    PetscFinalize();

    return 0;
}


This fails with the following error in DMPlexDistributeCoordinates:

VecSetBlockSize:

[0]PETSC ERROR: Invalid argument
[0]PETSC ERROR: Int value must be same on all processes, argument # 2

Digging a bit, I think this is because in DMCreateLocalVector the block size is determined by checking the block size of the section.  But on rank 1, the section is empty (and so the created Vec does not have a block size set).  We then go on to copy this block size over to the distributed coordinates, and get an error because on rank-0 the block size is 2, not 1.

I fix this in the following way:

commit aef309e0b49bf6e55e6f970bc0bdabf021b346f6
Author: Lawrence Mitchell <lawrence.mitchell at imperial.ac.uk>
Date:   Fri Oct 23 09:32:36 2015 +0100

    plexdistribute: Use better guess for local coordinate block size

    If we create local coordinates on a proces with an empty section, we end
    up setting a block size of 1 on the newly distributed local
    coordinates (since the local block size is unset).  In this case, check
    if we have global coordinates in place and try using the block size
    they provide instead.

diff --git a/src/dm/impls/plex/plexdistribute.c b/src/dm/impls/plex/plexdistribu
index 24d39b8..8049019 100644
--- a/src/dm/impls/plex/plexdistribute.c
+++ b/src/dm/impls/plex/plexdistribute.c
@@ -1040,7 +1040,7 @@ PetscErrorCode DMPlexDistributeCoordinates(DM dm, PetscSF
 {
   MPI_Comm         comm;
   PetscSection     originalCoordSection, newCoordSection;
-  Vec              originalCoordinates, newCoordinates;
+  Vec              originalCoordinates, newCoordinates, globalCoordinates;
   PetscInt         bs;
   const char      *name;
   const PetscReal *maxCell, *L;
@@ -1055,6 +1055,7 @@ PetscErrorCode DMPlexDistributeCoordinates(DM dm, PetscSF
   ierr = DMGetCoordinateSection(dm, &originalCoordSection);CHKERRQ(ierr);
   ierr = DMGetCoordinateSection(dmParallel, &newCoordSection);CHKERRQ(ierr);
   ierr = DMGetCoordinatesLocal(dm, &originalCoordinates);CHKERRQ(ierr);
+  ierr = DMGetCoordinates(dm, &globalCoordinates);CHKERRQ(ierr);
   if (originalCoordinates) {
     ierr = VecCreate(comm, &newCoordinates);CHKERRQ(ierr);
     ierr = PetscObjectGetName((PetscObject) originalCoordinates, &name);CHKERRQ
@@ -1063,6 +1064,15 @@ PetscErrorCode DMPlexDistributeCoordinates(DM dm, PetscSF
     ierr = DMPlexDistributeField(dm, migrationSF, originalCoordSection, origina
     ierr = DMSetCoordinatesLocal(dmParallel, newCoordinates);CHKERRQ(ierr);
     ierr = VecGetBlockSize(originalCoordinates, &bs);CHKERRQ(ierr);
+    if (globalCoordinates) {
+      PetscLayout map, gmap;
+      ierr = VecGetLayout(originalCoordinates, &map); CHKERRQ(ierr);
+      ierr = VecGetLayout(globalCoordinates, &gmap); CHKERRQ(ierr);
+      /* If local block size not set but global size is, pick global size */
+      if (map->bs < 0 && gmap->bs > 0) {
+        ierr = PetscLayoutGetBlockSize(gmap, &bs); CHKERRQ(ierr);
+      }
+    }
     ierr = VecSetBlockSize(newCoordinates, bs);CHKERRQ(ierr);
     ierr = VecDestroy(&newCoordinates);CHKERRQ(ierr);
   }

Now at least I can dump and reload my DM.

OK, now let's think about the natural-to-global stuff.  If I understand this correctly, I need to set a section on the serial DM (before distribution) then use DMSetUseNatural and now the natural-to-global SF will be created during distribution.

This is pretty awkward for me to deal with, since I don't have a section at all before distributing, and would rather not build it then.  Is this a hard limitation?  Naively it seems like I should be able to derive a natural numbering from the distributed DM by using the point migration SF.  Thoughts?


Cheers,

Lawrence
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 455 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151023/9585a560/attachment.pgp>

From knepley at gmail.com  Fri Oct 23 06:14:18 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Fri, 23 Oct 2015 06:14:18 -0500
Subject: [petsc-users] DM question.
In-Reply-To: <453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
	<453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
Message-ID: <CAMYG4G=AH5dr1y03ppurs1tTpz7w+X=wSEEsma1YWu1uzWCgqw@mail.gmail.com>

On Fri, Oct 23, 2015 at 3:01 AM, Afanasiev Michael <
michael.afanasiev at erdw.ethz.ch> wrote:

> Hi Matthew,
>
> So I?m discretizing via a tensor product of Lagrange polynomials
> co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order
> might range from 4 to 10 or something like that. The current problem is
> solved on 2D hexes.
>
> I had found DMPlexCreateSection, and followed plex/ex1to get things set
> up. You can see my attempt below. Basically I?ve got 4 fields
> (displacement, velocity, acceleration, and strain) over each element. Here
> I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange
> polynomials (line 11). This seemed to * somewhat* achieve what I wanted ?
> I created a global vector and wrote it to a vtk file with VecView, and the
> numbering seemed to make sense. You can also see my attempt at defining a
> boundary condition (it looked like DMPlexCreateFromExodus labeled side sets
> as ?Face Sets?, seems to have worked).
>
> Does this seem to be headed in the right direction?
>

Yes, however I have some questions. Starting out, I think the GLL points
include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1
dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below
you have numDof[] for
a 1D mesh with DG element.

The "Face Sets" is the right label to use for boundary conditions. This
will eliminate those variables
from the global system, but they will be present in the local spaces.

With elements like these, it is common (I think) to eliminate the cell
unknown explicitly, since the
system is dense and not connected to other cells. For this, you would
retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so
I do not know if there
are any pitfalls.

You can see an example of a similar implementation specifically for the
kind of spectral elements
you are considering here: https://github.com/jedbrown/dohp. It would
probably be useful to understand
what is done there as you implement.

  Thanks,

     Matt


> Cheers,
> Mike.
>
> DM
> mesh::createSection(const DM &dm)
> {
>
> 01        // displacement, velocity, acceleration, strain
> 02        IS bcPointIs[1];
> 03        PetscInt numBc = 1;
> 04        PetscInt bcField[1];
> 05        PetscInt numFields = 4;
> 06        PetscInt dim; DMGetDimension(dm, &dim);
> 07        PetscInt numComp[numFields];
> 08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
> 09        PetscInt numDof[numFields*(dim+1)];
> 10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
> 11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
> 12        bcField[0] = 0;
> 13        PetscSection section;
> 14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
> 15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof, numBc,
> bcField,
> 16                                                NULL, bcPointIs, NULL,
> &section);
> 17        ISDestroy(&bcPointIs[0]);
> 18        PetscSectionSetFieldName(section, 0, "u");
> 19        PetscSectionSetFieldName(section, 1, "v");
> 20        PetscSectionSetFieldName(section, 2, "a");
> 21        PetscSectionSetFieldName(section, 3, "e");
> 22        DMSetDefaultSection(dm, section);
> 23        return dm;
> }
>
> --
> Michael Afanasiev
> Ph.D. Candidate
> Computational Seismology
> Institut f?r Geophysik
> ETH Z?rich
>
> Sonneggstrasse 5, NO H 39.2
> CH 8092 Z?rich
> michael.afanasiev at erdw.ethz.ch
>
> On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch> wrote:
>
>> Hey Mike,
>>
>>
>>
>> On 21 October 2015 at 18:01, Afanasiev Michael <
>> michael.afanasiev at erdw.ethz.ch> wrote:
>>
>>> Hey Dave,
>>>
>>> So I?ve got a couple of days where there?s nothing pressing to work on?
>>> was thinking of ripping out the parallel routines (ugly) in my wave
>>> propagation code and replacing them with Petsc DM routines. I can read in
>>> my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a
>>> nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.
>>>
>>> But here I?m stuck, and am having a whale of a time with the
>>> documentation. All I *think* I need is a way to modify the
>>> exodus-created DM, and add to it the degrees of freedom that are introduced
>>> by my quadrature rule. This would be really neat. I can just treat each
>>> sub-domain as its own mesh, with its own global numbering. Then whenever
>>> necessary I can scatter stuff the the *real* global degrees of freedom
>>> with something like VecLocToGlob. Most of the things I like about the code
>>> could stay the same (element-wise, matrix-free nature), just these parallel
>>> broadcasts would be infinitely nicer.
>>>
>>>
>> First off - I don't use DMPLEX.
>>
>
> Dave is refreshingly candid about his shortcomings ;)
>
>
>>
>>
> But I just can?t figure out how to set this up. The main problem really
>>> boils down to: what?s the best way to add my quadrature points to an
>>> already-created DM, which was constructed with an exodus file? I guess I
>>> could do this after the file is read, but before the partitioning. In this
>>> case though, what?s stopping the partitioner from cutting an element in
>>> half? It seems like it would be a lot cleaner to do this post-partitioning.
>>>
>>>
>> Presumably what is read from exodus is just the vertices of the hexes,
>> and what you want to do is define the function space (given by your GLL
>> locations) on top of element geometry read in. Is that what you are asking
>> about?
>>
>
> So Dave is right. We read in topology and geometry from ExodusII. Then you
> define a function space on top. How
> exactly are you discretizing? In order to create vectors, do local to
> global, etc. Petsc really only need to know the
> amount of data associated with each mesh piece. You can define this with
> PetscSection. If you give me an idea
> what you want I can help you write the code easily I think.
>
>   Thanks,
>
>      Matt
>
>
>> Any hints here?
>>>
>>
>> Actually I have no experience with this object.
>> I would just send an email to
>>   petsc-users at mcs.anl.gov
>> asking for help.
>>
>> The developer of DMPLEX (Matt Knepley) will definitely answer within in 1
>> day.
>>
>> Cheers,
>>   Dave
>>
>>
>>> Best,
>>> Mike.
>>> --
>>> Michael Afanasiev
>>> Ph.D. Candidate
>>> Computational Seismology
>>> Institut f?r Geophysik
>>> ETH Z?rich
>>>
>>> Sonneggstrasse 5, NO H 39.2
>>> CH 8092 Z?rich
>>> michael.afanasiev at erdw.ethz.ch
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151023/c132cd11/attachment.html>

From zhaopin at ilingtong.com  Fri Oct 23 09:22:25 2015
From: zhaopin at ilingtong.com (=?gb2312?B?pbc/?=)
Date: Fri, 23 Oct 2015 22:22:25 +0800
Subject: [petsc-users] =?gb2312?b?cGV0c2MtdXNlcnPE47rDo7ooQUQp?=
Message-ID: <201510232222398172616@ilingtong.com>


petsc-users

????2015??????5000??????24??0????????????????????????????????????

????????????????


























































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































































?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????R?????????????????????????????????????????????????????~???????r???????????????????????????????Y???????????????????????????????????m?????????????????????????????????????????N????????????????????????
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151023/60c7a527/attachment-0001.html>

From m.guterres at gmail.com  Fri Oct 23 14:21:44 2015
From: m.guterres at gmail.com (Marcelo Xavier Guterres)
Date: Fri, 23 Oct 2015 17:21:44 -0200
Subject: [petsc-users] Question - read text file
Message-ID: <CAKaLTH2+va34bRbgBOaMZUPQ79CMq1gtrnyK=M3xbwQtLA=bjA@mail.gmail.com>

Dear friends,

I need to read a matrix from a text file (txt).

This matrix has a high dimension (2000 X 2000) .

Example: date.txt - provision in the text file:

1 2 3 5
5 5 7 8
5 6 8 9


What's the proper way to perform the action?

thanks for help


-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- --
Marcelo Xavier Guterres
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
-- -- -- -- -- -- -- -
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151023/660def32/attachment.html>

From pvsang002 at gmail.com  Fri Oct 23 23:04:54 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Sat, 24 Oct 2015 11:04:54 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
Message-ID: <CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>

Hi Barry,

The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da, Mat
J), the function pointer in DMDASetGetMatrix() only accept function with
that two arguments.
As you suggested, I am writing a routine (based on
DMCreateMatrix_DA_3d_MPIAIJ())
to preallocate the matrix in the way I wish, to do that I think It needs to
have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J,
void* my_context). But DMDASetGetMatrix() will not accept pointer of this
function. Please give a suggestion to overcome this.

Thank you.

Pham


On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
>     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in
> 3d)   You need to copy this routine and add whatever additional
> preallocation information you need.   Then call DMDASetGetMatrix()  so that
> the DM will use your routine to create the matrix for you.
>
>    Barry
>
>
>
>
> > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Hi,
> >
> > I am using DMCreateMatrix to create matrix from a existed DM object and
> defined stencil.
> > In my code, boundary nodes need to involve many inner nodes, thus matrix
> rows corresponding to boundary nodes are almost dense. How can I  tell
> petsc that those rows need to be preallocated with more entries? I don't
> want to use MatMPIAIJSetPreallocation() since advantages of DM might be
> lost.
> >
> > Many thanks.
> >
> > Sam.
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151024/09be0e6f/attachment.html>

From dave.mayhem23 at gmail.com  Sat Oct 24 01:33:26 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Sat, 24 Oct 2015 08:33:26 +0200
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
Message-ID: <CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>

If you need to access a user defined context from within your CreateMatrix
function, you can attach it to the DM via PetscObjectCompose

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html

If your context is not a petsc object, you can use PetscContainerCreate(),

http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate

You would then set your user context pointer inside the container and then
use PetscObjectCompose() to attach the container to the DM

Thanks,
  Dave


On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:

> Hi Barry,
>
> The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da, Mat
> J), the function pointer in DMDASetGetMatrix() only accept function with
> that two arguments.
> As you suggested, I am writing a routine (based on DMCreateMatrix_DA_3d_MPIAIJ())
> to preallocate the matrix in the way I wish, to do that I think It needs to
> have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat
> J, void* my_context). But DMDASetGetMatrix() will not accept pointer of
> this function. Please give a suggestion to overcome this.
>
> Thank you.
>
> Pham
>
>
> On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in
>> 3d)   You need to copy this routine and add whatever additional
>> preallocation information you need.   Then call DMDASetGetMatrix()  so that
>> the DM will use your routine to create the matrix for you.
>>
>>    Barry
>>
>>
>>
>>
>> > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > I am using DMCreateMatrix to create matrix from a existed DM object and
>> defined stencil.
>> > In my code, boundary nodes need to involve many inner nodes, thus
>> matrix rows corresponding to boundary nodes are almost dense. How can I
>> tell petsc that those rows need to be preallocated with more entries? I
>> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
>> be lost.
>> >
>> > Many thanks.
>> >
>> > Sam.
>> >
>> >
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151024/854db5c8/attachment.html>

From pvsang002 at gmail.com  Sat Oct 24 01:43:25 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Sat, 24 Oct 2015 13:43:25 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
Message-ID: <CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>

Thank you, Dave,

Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
my_context) when I need to create my matrix, and forget about
DMDASetGetMatrix()?

Thank you.

Pham

On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com> wrote:

> If you need to access a user defined context from within your CreateMatrix
> function, you can attach it to the DM via PetscObjectCompose
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
>
> If your context is not a petsc object, you can use PetscContainerCreate(),
>
>
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
>
> You would then set your user context pointer inside the container and then
> use PetscObjectCompose() to attach the container to the DM
>
> Thanks,
>   Dave
>
>
> On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
>
>> Hi Barry,
>>
>> The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
>> Mat J), the function pointer in DMDASetGetMatrix() only accept function
>> with that two arguments.
>> As you suggested, I am writing a routine (based on DMCreateMatrix_DA_3d_MPIAIJ())
>> to preallocate the matrix in the way I wish, to do that I think It needs to
>> have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat
>> J, void* my_context). But DMDASetGetMatrix() will not accept pointer of
>> this function. Please give a suggestion to overcome this.
>>
>> Thank you.
>>
>> Pham
>>
>>
>> On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>>     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
>>> in 3d)   You need to copy this routine and add whatever additional
>>> preallocation information you need.   Then call DMDASetGetMatrix()  so that
>>> the DM will use your routine to create the matrix for you.
>>>
>>>    Barry
>>>
>>>
>>>
>>>
>>> > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > I am using DMCreateMatrix to create matrix from a existed DM object
>>> and defined stencil.
>>> > In my code, boundary nodes need to involve many inner nodes, thus
>>> matrix rows corresponding to boundary nodes are almost dense. How can I
>>> tell petsc that those rows need to be preallocated with more entries? I
>>> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
>>> be lost.
>>> >
>>> > Many thanks.
>>> >
>>> > Sam.
>>> >
>>> >
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151024/47e5fe14/attachment.html>

From bsmith at mcs.anl.gov  Sat Oct 24 12:46:39 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 24 Oct 2015 12:46:39 -0500
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
Message-ID: <FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>


> On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> 
> Thank you, Dave,
> 
> Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context) when I need to create my matrix, and forget about DMDASetGetMatrix()?

  You can do that if that is all you need. In some contexts when something else in PETSc needs to create a specific matrix (like with geometric multigrid) then that code calls DMCreateMatrix() which would then use your creation routine. 

   So, in summary you can start by just calling your own routine and convert it to use DMSetGetMatrix() later if you need to.

  Barry

> 
> Thank you.
> 
> Pham
> 
> On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com> wrote:
> If you need to access a user defined context from within your CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> 
> If your context is not a petsc object, you can use PetscContainerCreate(), 
> 
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> 
> You would then set your user context pointer inside the container and then use PetscObjectCompose() to attach the container to the DM
> 
> Thanks,
>   Dave
> 
> 
> On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
> Hi Barry,
> 
> The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da, Mat J), the function pointer in DMDASetGetMatrix() only accept function with that two arguments. 
> As you suggested, I am writing a routine (based on DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish, to do that I think It needs to have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But DMDASetGetMatrix() will not accept pointer of this function. Please give a suggestion to overcome this.
> 
> Thank you.
> 
> Pham
> 
> 
> On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in 3d)   You need to copy this routine and add whatever additional preallocation information you need.   Then call DMDASetGetMatrix()  so that the DM will use your routine to create the matrix for you.
> 
>    Barry
> 
> 
> 
> 
> > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Hi,
> >
> > I am using DMCreateMatrix to create matrix from a existed DM object and defined stencil.
> > In my code, boundary nodes need to involve many inner nodes, thus matrix rows corresponding to boundary nodes are almost dense. How can I  tell petsc that those rows need to be preallocated with more entries? I don't want to use MatMPIAIJSetPreallocation() since advantages of DM might be lost.
> >
> > Many thanks.
> >
> > Sam.
> >
> >
> 
> 
> 
> 


From pvsang002 at gmail.com  Sat Oct 24 16:01:47 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Sun, 25 Oct 2015 04:01:47 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
Message-ID: <CAOx7dQmeBLUXq=pd2RC8mFQh2STSyTKauxSViTqAbTxKFAHqrQ@mail.gmail.com>

Many thanks, Dave!

Best,
Pham

On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Thank you, Dave,
> >
> > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
> my_context) when I need to create my matrix, and forget about
> DMDASetGetMatrix()?
>
>   You can do that if that is all you need. In some contexts when something
> else in PETSc needs to create a specific matrix (like with geometric
> multigrid) then that code calls DMCreateMatrix() which would then use your
> creation routine.
>
>    So, in summary you can start by just calling your own routine and
> convert it to use DMSetGetMatrix() later if you need to.
>
>   Barry
>
> >
> > Thank you.
> >
> > Pham
> >
> > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com>
> wrote:
> > If you need to access a user defined context from within your
> CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> >
> > If your context is not a petsc object, you can use
> PetscContainerCreate(),
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> >
> > You would then set your user context pointer inside the container and
> then use PetscObjectCompose() to attach the container to the DM
> >
> > Thanks,
> >   Dave
> >
> >
> > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
> > Hi Barry,
> >
> > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
> Mat J), the function pointer in DMDASetGetMatrix() only accept function
> with that two arguments.
> > As you suggested, I am writing a routine (based on
> DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish,
> to do that I think It needs to have one more input argument:
> My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But
> DMDASetGetMatrix() will not accept pointer of this function. Please give a
> suggestion to overcome this.
> >
> > Thank you.
> >
> > Pham
> >
> >
> > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
> in 3d)   You need to copy this routine and add whatever additional
> preallocation information you need.   Then call DMDASetGetMatrix()  so that
> the DM will use your routine to create the matrix for you.
> >
> >    Barry
> >
> >
> >
> >
> > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > I am using DMCreateMatrix to create matrix from a existed DM object
> and defined stencil.
> > > In my code, boundary nodes need to involve many inner nodes, thus
> matrix rows corresponding to boundary nodes are almost dense. How can I
> tell petsc that those rows need to be preallocated with more entries? I
> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
> be lost.
> > >
> > > Many thanks.
> > >
> > > Sam.
> > >
> > >
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151025/508d8d87/attachment.html>

From zonexo at gmail.com  Sun Oct 25 04:08:02 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 25 Oct 2015 17:08:02 +0800
Subject: [petsc-users] Recommended processors according to problem size for
	good scaling
Message-ID: <562C9BF2.8000009@gmail.com>

Hi,

Is there a recommended no. of unknowns for each processor for good 
scaling? Supposed I have 1 million unknowns (cells), how many processors 
should I use?

I remember reading that 50k unknowns per procs is a good choice. Is that 
true?

-- Thank you. Yours sincerely, TAY wee-beng

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151025/a8259c46/attachment.html>

From rupp at iue.tuwien.ac.at  Sun Oct 25 04:21:46 2015
From: rupp at iue.tuwien.ac.at (Karl Rupp)
Date: Sun, 25 Oct 2015 10:21:46 +0100
Subject: [petsc-users] Recommended processors according to problem size
 for good scaling
In-Reply-To: <562C9BF2.8000009@gmail.com>
References: <562C9BF2.8000009@gmail.com>
Message-ID: <562C9F2A.10101@iue.tuwien.ac.at>

Hi,

I assume you have read
  http://www.mcs.anl.gov/petsc/documentation/faq.html#computers

As a rule of thumb, 50k cells is a reasonable lower limit. It really 
depends a lot on your interconnect, the dimensionality of your problem 
(2d vs. 3d, ratio of 'surface' to 'volume' of the cell batch), etc. 
Also, most likely your speed-up will level off after about 4 threads per 
socket, as then the memory channels become saturated.

In either case, it's best to run your own tiny scaling study to get 
actual numbers. -log_summary is really handy for that purpose :-)

Best regards,
Karli


On 10/25/2015 10:08 AM, TAY wee-beng wrote:
> Hi,
>
> Is there a recommended no. of unknowns for each processor for good
> scaling? Supposed I have 1 million unknowns (cells), how many processors
> should I use?
>
> I remember reading that 50k unknowns per procs is a good choice. Is that
> true?
>
> -- Thank you. Yours sincerely, TAY wee-beng
>


From mailinglists at xgm.de  Sun Oct 25 11:40:36 2015
From: mailinglists at xgm.de (Florian Lindner)
Date: Sun, 25 Oct 2015 17:40:36 +0100
Subject: [petsc-users] PetscBool in PETSc 3.1? (Ubuntu Precice)
Message-ID: <7232803.c5actVMB60@horus>

Hello,

our build system Travis uses the old Ubuntu Precice Pangolin version that comes with PETSc 3.1.

While trying to activate petsc for our CI tests, I get the message that the type PetscBool wasn't found. I downloaded http://packages.ubuntu.com/precise/libpetsc3.1-dev and grepped for PetscBool, found non.

Just to make sure I didn't screw it up, is it true, that there is no PetscBool in version 3.1?

Is there a PPA known that provides newer PETSc versions to Precice Pangolin

Thanks,
Florian


From knepley at gmail.com  Sun Oct 25 11:52:56 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sun, 25 Oct 2015 11:52:56 -0500
Subject: [petsc-users] PetscBool in PETSc 3.1? (Ubuntu Precice)
In-Reply-To: <7232803.c5actVMB60@horus>
References: <7232803.c5actVMB60@horus>
Message-ID: <CAMYG4GmwsTJcED0sUcsP=xQ3+HwgPLwUEV+EiQMGyjPEQ_KHsA@mail.gmail.com>

On Sun, Oct 25, 2015 at 11:40 AM, Florian Lindner <mailinglists at xgm.de>
wrote:

> Hello,
>
> our build system Travis uses the old Ubuntu Precice Pangolin version that
> comes with PETSc 3.1.
>
> While trying to activate petsc for our CI tests, I get the message that
> the type PetscBool wasn't found. I downloaded
> http://packages.ubuntu.com/precise/libpetsc3.1-dev and grepped for
> PetscBool, found non.
>
> Just to make sure I didn't screw it up, is it true, that there is no
> PetscBool in version 3.1?
>
> Is there a PPA known that provides newer PETSc versions to Precice Pangolin
>

It used to be called PetscTruth, but we could not handle the truth, so we
changed it. I think
you can just add

typedef PetscTruth PetscBool

  Thanks,

    Matt


> Thanks,
> Florian
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151025/869ec663/attachment.html>

From bsmith at mcs.anl.gov  Sun Oct 25 11:56:03 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sun, 25 Oct 2015 11:56:03 -0500
Subject: [petsc-users] PetscBool in PETSc 3.1? (Ubuntu Precice)
In-Reply-To: <7232803.c5actVMB60@horus>
References: <7232803.c5actVMB60@horus>
Message-ID: <1658EC90-1398-4905-A677-49D4848B5681@mcs.anl.gov>


> On Oct 25, 2015, at 11:40 AM, Florian Lindner <mailinglists at xgm.de> wrote:
> 
> Hello,
> 
> our build system Travis uses the old Ubuntu Precice Pangolin version that comes with PETSc 3.1.
> 
> While trying to activate petsc for our CI tests, I get the message that the type PetscBool wasn't found. I downloaded http://packages.ubuntu.com/precise/libpetsc3.1-dev and grepped for PetscBool, found non.
> 
> Just to make sure I didn't screw it up, is it true, that there is no PetscBool in version 3.1?

   In version 3.1 it is called PetscTruth

   PETSc version 3.1 is really, really old, we don't recommend you use it for any purpose since we don't support it.

  Barry


> 
> Is there a PPA known that provides newer PETSc versions to Precice Pangolin
> 
> Thanks,
> Florian
> 


From michael.afanasiev at erdw.ethz.ch  Mon Oct 26 06:44:59 2015
From: michael.afanasiev at erdw.ethz.ch (Afanasiev  Michael)
Date: Mon, 26 Oct 2015 11:44:59 +0000
Subject: [petsc-users] DM question.
In-Reply-To: <CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
	<453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
	<CAMYG4G=AH5dr1y03ppurs1tTpz7w+X=wSEEsma1YWu1uzWCgqw@mail.gmail.com>
	<4AD13183-D52E-4B8D-A3A1-5BFB79D777E5@erdw.ethz.ch>
	<CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
Message-ID: <524266A8-C08E-462B-92CF-B1ACF36AA3CA@erdw.ethz.ch>

Hi Matthew,

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

Right, makes sense. With some insight from Dave, I?m getting what I expect for a simple 4-cell mesh distributed amongst 4 processors (length of 91 for the global vector, length of 25 for the local vectors, for 4th order polynomial and the GLL basis). Now I imagine what?s left is to figure out how these vectors are globally numbered.

I usually do this by having a consistent way of numbering the dofs on the reference element (i.e. given a (u, v)-space loop through v, then u), and then getting the local->global map via finding coincident points in a kd-tree. My question is: in which order are the local vectors now defined? If I knew this, I could create my local->global map as before. From the way the section was created, I guess it might be something like loop vertices, edges, and then interior?

Thanks for all your help,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 5:11 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 9:03 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

For the GLL basis we have (k+1) points in each dimension, including the endpoints. So for example a 2D element with 4-order polynomials will have 25 locally numbered points per element. I should also mention that the velocity, displacement, and acceleration fields are vectors, with d=dim components at each integration point, whereas strain has (2^dim)-1 components. In what you?ve written above, does this then become (sum over the 4 fields):

(sum(compPerField*dofPerField)) -> vertex
(sum((compPerField*dofPerField)*(k+1)) -> edge
(sum((compPerField*dofPerField)*(k+1)^2) -> quad

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

I believe I don?t worry about this. Everything is solved in a matrix-free sense, on each element. The relevant physical quantities are calculated on each element, and then summed into the global degrees of freedom. The summed global dof are then brought back to the element level, where updates to the acceleration, velocity, and displacement are calculated via a Newmark time step. So no global system of equations is ever formed.

You accumulate into the global dofs, so you would need more storage if you do not do static condensation. It just good to know this,
but you do not have to do it.

This being said, all the routines to a) integrate the element matrices, b) calculate the gll point locations, c) construct the local->global dof mapping, and d) do the element-wise matrix free time stepping are already written.  My problem is really that I do my mesh decomposition by hand (poorly), and I?d like to transfer this over to Petsc. Of course if I do this, I might as well use DM's LocalToGlobal vector routines to sum my field vectors across mesh partitions.

Yes.

Perhaps a better question to ask would be in the form of a workflow:

1) Read in exodus mesh and partition (done)
2) Set up local and global degrees of freedom on each mesh partition (done)

Here you just have to setup PetscSections that mirror your local and global layout. Then the LocalToGlobal and its inverse will work.

   Matt

3) Integrate element matrices (done)

for i 1, nTimeSteps
3) Step fields forward on each partition (done)
4) Sum to global degrees of freedom on each partition (done)
5) Sum to global degrees of freedom across partitions (????)
6) Retrieve summed global degrees of freedom across partitions (????)
7) Continue

So really what I hope Petsc will help with is just steps 5 and 6. I guess this involves, given a partitioned DMPlex object, which has been partitioned according to the geometry and topology defined in an exodus file, adding the degrees of freedom to each partition-local vector (via a DMPlexSection?). Then, ensuring that the dof added along the partition boundaries sum together properly when a LocalToGlobal vector operation is defined.

Does this make sense? If so, is this something that DMPlex is designed for?
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 1:14 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 3:01 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

So I?m discretizing via a tensor product of Lagrange polynomials co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order might range from 4 to 10 or something like that. The current problem is solved on 2D hexes.

I had found DMPlexCreateSection, and followed plex/ex1to get things set up. You can see my attempt below. Basically I?ve got 4 fields (displacement, velocity, acceleration, and strain) over each element. Here I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange polynomials (line 11). This seemed to somewhat achieve what I wanted ? I created a global vector and wrote it to a vtk file with VecView, and the numbering seemed to make sense. You can also see my attempt at defining a boundary condition (it looked like DMPlexCreateFromExodus labeled side sets as ?Face Sets?, seems to have worked).

Does this seem to be headed in the right direction?

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

The "Face Sets" is the right label to use for boundary conditions. This will eliminate those variables
from the global system, but they will be present in the local spaces.

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

You can see an example of a similar implementation specifically for the kind of spectral elements
you are considering here: https://github.com/jedbrown/dohp. It would probably be useful to understand
what is done there as you implement.

  Thanks,

     Matt

Cheers,
Mike.

DM
mesh::createSection(const DM &dm)
{

01        // displacement, velocity, acceleration, strain
02        IS bcPointIs[1];
03        PetscInt numBc = 1;
04        PetscInt bcField[1];
05        PetscInt numFields = 4;
06        PetscInt dim; DMGetDimension(dm, &dim);
07        PetscInt numComp[numFields];
08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
09        PetscInt numDof[numFields*(dim+1)];
10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
12        bcField[0] = 0;
13        PetscSection section;
14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof, numBc, bcField,
16                                                NULL, bcPointIs, NULL, &section);
17        ISDestroy(&bcPointIs[0]);
18        PetscSectionSetFieldName(section, 0, "u");
19        PetscSectionSetFieldName(section, 1, "v");
20        PetscSectionSetFieldName(section, 2, "a");
21        PetscSectionSetFieldName(section, 3, "e");
22        DMSetDefaultSection(dm, section);
23        return dm;
}

--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch<mailto:dave.may at erdw.ethz.ch>> wrote:
Hey Mike,



On 21 October 2015 at 18:01, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hey Dave,

So I?ve got a couple of days where there?s nothing pressing to work on? was thinking of ripping out the parallel routines (ugly) in my wave propagation code and replacing them with Petsc DM routines. I can read in my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.

But here I?m stuck, and am having a whale of a time with the documentation. All I think I need is a way to modify the exodus-created DM, and add to it the degrees of freedom that are introduced by my quadrature rule. This would be really neat. I can just treat each sub-domain as its own mesh, with its own global numbering. Then whenever necessary I can scatter stuff the the real global degrees of freedom with something like VecLocToGlob. Most of the things I like about the code could stay the same (element-wise, matrix-free nature), just these parallel broadcasts would be infinitely nicer.


First off - I don't use DMPLEX.

Dave is refreshingly candid about his shortcomings ;)


But I just can?t figure out how to set this up. The main problem really boils down to: what?s the best way to add my quadrature points to an already-created DM, which was constructed with an exodus file? I guess I could do this after the file is read, but before the partitioning. In this case though, what?s stopping the partitioner from cutting an element in half? It seems like it would be a lot cleaner to do this post-partitioning.


Presumably what is read from exodus is just the vertices of the hexes, and what you want to do is define the function space (given by your GLL locations) on top of element geometry read in. Is that what you are asking about?

So Dave is right. We read in topology and geometry from ExodusII. Then you define a function space on top. How
exactly are you discretizing? In order to create vectors, do local to global, etc. Petsc really only need to know the
amount of data associated with each mesh piece. You can define this with PetscSection. If you give me an idea
what you want I can help you write the code easily I think.

  Thanks,

     Matt

Any hints here?

Actually I have no experience with this object.
I would just send an email to
  petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
asking for help.

The developer of DMPLEX (Matt Knepley) will definitely answer within in 1 day.

Cheers,
  Dave


Best,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>





--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151026/4eec9a28/attachment-0001.html>

From michael.afanasiev at erdw.ethz.ch  Mon Oct 26 06:45:01 2015
From: michael.afanasiev at erdw.ethz.ch (Afanasiev  Michael)
Date: Mon, 26 Oct 2015 11:45:01 +0000
Subject: [petsc-users] DM question.
In-Reply-To: <CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
	<453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
	<CAMYG4G=AH5dr1y03ppurs1tTpz7w+X=wSEEsma1YWu1uzWCgqw@mail.gmail.com>
	<4AD13183-D52E-4B8D-A3A1-5BFB79D777E5@erdw.ethz.ch>
	<CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
Message-ID: <68481225-D8CE-443A-A69F-4BDD6C19980D@erdw.ethz.ch>

Hi Matthew,

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

Right, makes sense. With some insight from Dave, I?m getting what I expect for a simple 4-cell mesh distributed amongst 4 processors (length of 91 for the global vector, length of 25 for the local vectors, for 4th order polynomial and the GLL basis). Now I imagine what?s left is to figure out how these vectors are globally numbered.

I usually do this by having a consistent way of numbering the dofs on the reference element (i.e. given a (u, v)-space loop through v, then u), and then getting the local->global map via finding coincident points in a kd-tree. My question is: in which order are the local vectors now defined? If I knew this, I could create my local->global map as before. From the way the section was created, I guess it might be something like loop vertices, edges, and then interior?

Thanks for all your help,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 5:11 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 9:03 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

For the GLL basis we have (k+1) points in each dimension, including the endpoints. So for example a 2D element with 4-order polynomials will have 25 locally numbered points per element. I should also mention that the velocity, displacement, and acceleration fields are vectors, with d=dim components at each integration point, whereas strain has (2^dim)-1 components. In what you?ve written above, does this then become (sum over the 4 fields):

(sum(compPerField*dofPerField)) -> vertex
(sum((compPerField*dofPerField)*(k+1)) -> edge
(sum((compPerField*dofPerField)*(k+1)^2) -> quad

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

I believe I don?t worry about this. Everything is solved in a matrix-free sense, on each element. The relevant physical quantities are calculated on each element, and then summed into the global degrees of freedom. The summed global dof are then brought back to the element level, where updates to the acceleration, velocity, and displacement are calculated via a Newmark time step. So no global system of equations is ever formed.

You accumulate into the global dofs, so you would need more storage if you do not do static condensation. It just good to know this,
but you do not have to do it.

This being said, all the routines to a) integrate the element matrices, b) calculate the gll point locations, c) construct the local->global dof mapping, and d) do the element-wise matrix free time stepping are already written.  My problem is really that I do my mesh decomposition by hand (poorly), and I?d like to transfer this over to Petsc. Of course if I do this, I might as well use DM's LocalToGlobal vector routines to sum my field vectors across mesh partitions.

Yes.

Perhaps a better question to ask would be in the form of a workflow:

1) Read in exodus mesh and partition (done)
2) Set up local and global degrees of freedom on each mesh partition (done)

Here you just have to setup PetscSections that mirror your local and global layout. Then the LocalToGlobal and its inverse will work.

   Matt

3) Integrate element matrices (done)

for i 1, nTimeSteps
3) Step fields forward on each partition (done)
4) Sum to global degrees of freedom on each partition (done)
5) Sum to global degrees of freedom across partitions (????)
6) Retrieve summed global degrees of freedom across partitions (????)
7) Continue

So really what I hope Petsc will help with is just steps 5 and 6. I guess this involves, given a partitioned DMPlex object, which has been partitioned according to the geometry and topology defined in an exodus file, adding the degrees of freedom to each partition-local vector (via a DMPlexSection?). Then, ensuring that the dof added along the partition boundaries sum together properly when a LocalToGlobal vector operation is defined.

Does this make sense? If so, is this something that DMPlex is designed for?
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 1:14 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 3:01 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

So I?m discretizing via a tensor product of Lagrange polynomials co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order might range from 4 to 10 or something like that. The current problem is solved on 2D hexes.

I had found DMPlexCreateSection, and followed plex/ex1to get things set up. You can see my attempt below. Basically I?ve got 4 fields (displacement, velocity, acceleration, and strain) over each element. Here I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange polynomials (line 11). This seemed to somewhat achieve what I wanted ? I created a global vector and wrote it to a vtk file with VecView, and the numbering seemed to make sense. You can also see my attempt at defining a boundary condition (it looked like DMPlexCreateFromExodus labeled side sets as ?Face Sets?, seems to have worked).

Does this seem to be headed in the right direction?

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

The "Face Sets" is the right label to use for boundary conditions. This will eliminate those variables
from the global system, but they will be present in the local spaces.

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

You can see an example of a similar implementation specifically for the kind of spectral elements
you are considering here: https://github.com/jedbrown/dohp. It would probably be useful to understand
what is done there as you implement.

  Thanks,

     Matt

Cheers,
Mike.

DM
mesh::createSection(const DM &dm)
{

01        // displacement, velocity, acceleration, strain
02        IS bcPointIs[1];
03        PetscInt numBc = 1;
04        PetscInt bcField[1];
05        PetscInt numFields = 4;
06        PetscInt dim; DMGetDimension(dm, &dim);
07        PetscInt numComp[numFields];
08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
09        PetscInt numDof[numFields*(dim+1)];
10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
12        bcField[0] = 0;
13        PetscSection section;
14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof, numBc, bcField,
16                                                NULL, bcPointIs, NULL, &section);
17        ISDestroy(&bcPointIs[0]);
18        PetscSectionSetFieldName(section, 0, "u");
19        PetscSectionSetFieldName(section, 1, "v");
20        PetscSectionSetFieldName(section, 2, "a");
21        PetscSectionSetFieldName(section, 3, "e");
22        DMSetDefaultSection(dm, section);
23        return dm;
}

--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch<mailto:dave.may at erdw.ethz.ch>> wrote:
Hey Mike,



On 21 October 2015 at 18:01, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hey Dave,

So I?ve got a couple of days where there?s nothing pressing to work on? was thinking of ripping out the parallel routines (ugly) in my wave propagation code and replacing them with Petsc DM routines. I can read in my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.

But here I?m stuck, and am having a whale of a time with the documentation. All I think I need is a way to modify the exodus-created DM, and add to it the degrees of freedom that are introduced by my quadrature rule. This would be really neat. I can just treat each sub-domain as its own mesh, with its own global numbering. Then whenever necessary I can scatter stuff the the real global degrees of freedom with something like VecLocToGlob. Most of the things I like about the code could stay the same (element-wise, matrix-free nature), just these parallel broadcasts would be infinitely nicer.


First off - I don't use DMPLEX.

Dave is refreshingly candid about his shortcomings ;)


But I just can?t figure out how to set this up. The main problem really boils down to: what?s the best way to add my quadrature points to an already-created DM, which was constructed with an exodus file? I guess I could do this after the file is read, but before the partitioning. In this case though, what?s stopping the partitioner from cutting an element in half? It seems like it would be a lot cleaner to do this post-partitioning.


Presumably what is read from exodus is just the vertices of the hexes, and what you want to do is define the function space (given by your GLL locations) on top of element geometry read in. Is that what you are asking about?

So Dave is right. We read in topology and geometry from ExodusII. Then you define a function space on top. How
exactly are you discretizing? In order to create vectors, do local to global, etc. Petsc really only need to know the
amount of data associated with each mesh piece. You can define this with PetscSection. If you give me an idea
what you want I can help you write the code easily I think.

  Thanks,

     Matt

Any hints here?

Actually I have no experience with this object.
I would just send an email to
  petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
asking for help.

The developer of DMPLEX (Matt Knepley) will definitely answer within in 1 day.

Cheers,
  Dave


Best,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>





--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151026/fb410b8b/attachment-0001.html>

From knepley at gmail.com  Mon Oct 26 06:52:00 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Mon, 26 Oct 2015 06:52:00 -0500
Subject: [petsc-users] DM question.
In-Reply-To: <68481225-D8CE-443A-A69F-4BDD6C19980D@erdw.ethz.ch>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
	<453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
	<CAMYG4G=AH5dr1y03ppurs1tTpz7w+X=wSEEsma1YWu1uzWCgqw@mail.gmail.com>
	<4AD13183-D52E-4B8D-A3A1-5BFB79D777E5@erdw.ethz.ch>
	<CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
	<68481225-D8CE-443A-A69F-4BDD6C19980D@erdw.ethz.ch>
Message-ID: <CAMYG4G=u=atuSgotEPVX0TG=fBtN+mJUeMa9JBDgWMJKshxfQA@mail.gmail.com>

On Mon, Oct 26, 2015 at 6:45 AM, Afanasiev Michael <
michael.afanasiev at erdw.ethz.ch> wrote:

> Hi Matthew,
>
> No. There are (k-1)^2 unknowns in the interior of the cell, so that we have
>
>   4 + 4 * (k-1) + (k-1)^2 = (k+1)^2
>
>
> Right, makes sense. With some insight from Dave, I?m getting what I expect
> for a simple 4-cell mesh distributed amongst 4 processors (length of 91 for
> the global vector, length of 25 for the local vectors, for 4th order
> polynomial and the GLL basis). Now I imagine what?s left is to figure out
> how these vectors are globally numbered.
>
> I usually do this by having a consistent way of numbering the dofs on the
> reference element (i.e. given a (u, v)-space loop through v, then u), and
> then getting the local->global map via finding coincident points in a
> kd-tree. My question is: in which order are the local vectors now defined?
> If I knew this, I could create my local->global map as before. From the way
> the section was created, I guess it might be something like loop vertices,
> edges, and then interior?
>

The numbering is cell unknowns, vertex unknowns, face unknowns, and then
edge unknowns. This is, of course, arbitrary and therefore
you can change the order using


http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/PetscSectionSetPermutation.html#PetscSectionSetPermutation

which renumbers all the mesh points in the PetscSection defining the space.
You should be able to reproduce your old ordering using
this.

  Thanks,

    Matt


> Thanks for all your help,
> Mike.
> --
> Michael Afanasiev
> Ph.D. Candidate
> Computational Seismology
> Institut f?r Geophysik
> ETH Z?rich
>
> Sonneggstrasse 5, NO H 39.2
> CH 8092 Z?rich
> michael.afanasiev at erdw.ethz.ch
>
> On Oct 23, 2015, at 5:11 PM, Matthew Knepley <knepley at gmail.com> wrote:
>
> On Fri, Oct 23, 2015 at 9:03 AM, Afanasiev Michael <
> michael.afanasiev at erdw.ethz.ch> wrote:
>
>> Hi Matthew,
>>
>> Yes, however I have some questions. Starting out, I think the GLL points
>> include the endpoints, so
>> that means for polynomial degree k that numDof[] should really have 4*1
>> dofs on each vertex,
>> 4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like
>> below you have numDof[] for
>> a 1D mesh with DG element.
>>
>>
>> For the GLL basis we have (k+1) points in each dimension, including the
>> endpoints. So for example a 2D element with 4-order polynomials will have
>> 25 locally numbered points per element. I should also mention that the
>> velocity, displacement, and acceleration fields are vectors, with d=dim
>> components at each integration point, whereas strain has (2^dim)-1
>> components. In what you?ve written above, does this then become (sum over
>> the 4 fields):
>>
>> (sum(compPerField*dofPerField)) -> vertex
>> (sum((compPerField*dofPerField)*(k+1)) -> edge
>> (sum((compPerField*dofPerField)*(k+1)^2) -> quad
>>
>
> No. There are (k-1)^2 unknowns in the interior of the cell, so that we have
>
>   4 + 4 * (k-1) + (k-1)^2 = (k+1)^2
>
>
>> With elements like these, it is common (I think) to eliminate the cell
>> unknown explicitly, since the
>> system is dense and not connected to other cells. For this, you would
>> retain the vertex and edge
>> unknowns, but not the cell unknowns. I have not tried to do this myself,
>> so I do not know if there
>> are any pitfalls.
>>
>>
>> I believe I don?t worry about this. Everything is solved in a matrix-free
>> sense, on each element. The relevant physical quantities are calculated on
>> each element, and then summed into the global degrees of freedom. The
>> summed global dof are then brought back to the element level, where updates
>> to the acceleration, velocity, and displacement are calculated via a
>> Newmark time step. So no global system of equations is ever formed.
>>
>
> You accumulate into the global dofs, so you would need more storage if you
> do not do static condensation. It just good to know this,
> but you do not have to do it.
>
>
>> This being said, all the routines to a) integrate the element matrices,
>> b) calculate the gll point locations, c) construct the local->global dof
>> mapping, and d) do the element-wise matrix free time stepping are already
>> written.  My problem is really that I do my mesh decomposition by hand
>> (poorly), and I?d like to transfer this over to Petsc. Of course if I do
>> this, I might as well use DM's LocalToGlobal vector routines to sum my
>> field vectors across mesh partitions.
>>
>
> Yes.
>
>
>> Perhaps a better question to ask would be in the form of a workflow:
>>
>> 1) Read in exodus mesh and partition (done)
>> 2) Set up local and global degrees of freedom on each mesh partition
>> (done)
>>
>
> Here you just have to setup PetscSections that mirror your local and
> global layout. Then the LocalToGlobal and its inverse will work.
>
>    Matt
>
>
>> 3) Integrate element matrices (done)
>>
>> for i 1, nTimeSteps
>> 3) Step fields forward on each partition (done)
>> 4) Sum to global degrees of freedom on each partition (done)
>> 5) Sum to global degrees of freedom across partitions (????)
>> 6) Retrieve summed global degrees of freedom across partitions (????)
>> 7) Continue
>>
>> So really what I hope Petsc will help with is just steps 5 and 6. I guess
>> this involves, given a partitioned DMPlex object, which has been
>> partitioned according to the geometry and topology defined in an exodus
>> file, adding the degrees of freedom to each partition-local vector (via a
>> DMPlexSection?). Then, ensuring that the dof added along the partition
>> boundaries sum together properly when a LocalToGlobal vector operation is
>> defined.
>>
>> Does this make sense? If so, is this something that DMPlex is designed
>> for?
>> --
>> Michael Afanasiev
>> Ph.D. Candidate
>> Computational Seismology
>> Institut f?r Geophysik
>> ETH Z?rich
>>
>> Sonneggstrasse 5, NO H 39.2
>> CH 8092 Z?rich
>> michael.afanasiev at erdw.ethz.ch
>>
>> On Oct 23, 2015, at 1:14 PM, Matthew Knepley <knepley at gmail.com> wrote:
>>
>> On Fri, Oct 23, 2015 at 3:01 AM, Afanasiev Michael <
>> michael.afanasiev at erdw.ethz.ch> wrote:
>>
>>> Hi Matthew,
>>>
>>> So I?m discretizing via a tensor product of Lagrange polynomials
>>> co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order
>>> might range from 4 to 10 or something like that. The current problem is
>>> solved on 2D hexes.
>>>
>>> I had found DMPlexCreateSection, and followed plex/ex1to get things set
>>> up. You can see my attempt below. Basically I?ve got 4 fields
>>> (displacement, velocity, acceleration, and strain) over each element. Here
>>> I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange
>>> polynomials (line 11). This seemed to *somewhat* achieve what I wanted
>>> ? I created a global vector and wrote it to a vtk file with VecView, and
>>> the numbering seemed to make sense. You can also see my attempt at defining
>>> a boundary condition (it looked like DMPlexCreateFromExodus labeled side
>>> sets as ?Face Sets?, seems to have worked).
>>>
>>> Does this seem to be headed in the right direction?
>>>
>>
>> Yes, however I have some questions. Starting out, I think the GLL points
>> include the endpoints, so
>> that means for polynomial degree k that numDof[] should really have 4*1
>> dofs on each vertex,
>> 4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like
>> below you have numDof[] for
>> a 1D mesh with DG element.
>>
>> The "Face Sets" is the right label to use for boundary conditions. This
>> will eliminate those variables
>> from the global system, but they will be present in the local spaces.
>>
>> With elements like these, it is common (I think) to eliminate the cell
>> unknown explicitly, since the
>> system is dense and not connected to other cells. For this, you would
>> retain the vertex and edge
>> unknowns, but not the cell unknowns. I have not tried to do this myself,
>> so I do not know if there
>> are any pitfalls.
>>
>> You can see an example of a similar implementation specifically for the
>> kind of spectral elements
>> you are considering here: https://github.com/jedbrown/dohp. It would
>> probably be useful to understand
>> what is done there as you implement.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>>> Cheers,
>>> Mike.
>>>
>>> DM
>>> mesh::createSection(const DM &dm)
>>> {
>>>
>>> 01        // displacement, velocity, acceleration, strain
>>> 02        IS bcPointIs[1];
>>> 03        PetscInt numBc = 1;
>>> 04        PetscInt bcField[1];
>>> 05        PetscInt numFields = 4;
>>> 06        PetscInt dim; DMGetDimension(dm, &dim);
>>> 07        PetscInt numComp[numFields];
>>> 08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
>>> 09        PetscInt numDof[numFields*(dim+1)];
>>> 10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
>>> 11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
>>> 12        bcField[0] = 0;
>>> 13        PetscSection section;
>>> 14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
>>> 15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof,
>>> numBc, bcField,
>>> 16                                                NULL, bcPointIs, NULL,
>>> &section);
>>> 17        ISDestroy(&bcPointIs[0]);
>>> 18        PetscSectionSetFieldName(section, 0, "u");
>>> 19        PetscSectionSetFieldName(section, 1, "v");
>>> 20        PetscSectionSetFieldName(section, 2, "a");
>>> 21        PetscSectionSetFieldName(section, 3, "e");
>>> 22        DMSetDefaultSection(dm, section);
>>> 23        return dm;
>>> }
>>>
>>> --
>>> Michael Afanasiev
>>> Ph.D. Candidate
>>> Computational Seismology
>>> Institut f?r Geophysik
>>> ETH Z?rich
>>>
>>> Sonneggstrasse 5, NO H 39.2
>>> CH 8092 Z?rich
>>> michael.afanasiev at erdw.ethz.ch
>>>
>>> On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com> wrote:
>>>
>>> On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch> wrote:
>>>
>>>> Hey Mike,
>>>>
>>>>
>>>>
>>>> On 21 October 2015 at 18:01, Afanasiev Michael <
>>>> michael.afanasiev at erdw.ethz.ch> wrote:
>>>>
>>>>> Hey Dave,
>>>>>
>>>>> So I?ve got a couple of days where there?s nothing pressing to work
>>>>> on? was thinking of ripping out the parallel routines (ugly) in my wave
>>>>> propagation code and replacing them with Petsc DM routines. I can read in
>>>>> my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a
>>>>> nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.
>>>>>
>>>>> But here I?m stuck, and am having a whale of a time with the
>>>>> documentation. All I *think* I need is a way to modify the
>>>>> exodus-created DM, and add to it the degrees of freedom that are introduced
>>>>> by my quadrature rule. This would be really neat. I can just treat each
>>>>> sub-domain as its own mesh, with its own global numbering. Then whenever
>>>>> necessary I can scatter stuff the the *real* global degrees of
>>>>> freedom with something like VecLocToGlob. Most of the things I like about
>>>>> the code could stay the same (element-wise, matrix-free nature), just these
>>>>> parallel broadcasts would be infinitely nicer.
>>>>>
>>>>>
>>>> First off - I don't use DMPLEX.
>>>>
>>>
>>> Dave is refreshingly candid about his shortcomings ;)
>>>
>>>
>>>>
>>>>
>>> But I just can?t figure out how to set this up. The main problem really
>>>>> boils down to: what?s the best way to add my quadrature points to an
>>>>> already-created DM, which was constructed with an exodus file? I guess I
>>>>> could do this after the file is read, but before the partitioning. In this
>>>>> case though, what?s stopping the partitioner from cutting an element in
>>>>> half? It seems like it would be a lot cleaner to do this post-partitioning.
>>>>>
>>>>>
>>>> Presumably what is read from exodus is just the vertices of the hexes,
>>>> and what you want to do is define the function space (given by your GLL
>>>> locations) on top of element geometry read in. Is that what you are asking
>>>> about?
>>>>
>>>
>>> So Dave is right. We read in topology and geometry from ExodusII. Then
>>> you define a function space on top. How
>>> exactly are you discretizing? In order to create vectors, do local to
>>> global, etc. Petsc really only need to know the
>>> amount of data associated with each mesh piece. You can define this with
>>> PetscSection. If you give me an idea
>>> what you want I can help you write the code easily I think.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>>> Any hints here?
>>>>>
>>>>
>>>> Actually I have no experience with this object.
>>>> I would just send an email to
>>>>   petsc-users at mcs.anl.gov
>>>> asking for help.
>>>>
>>>> The developer of DMPLEX (Matt Knepley) will definitely answer within in
>>>> 1 day.
>>>>
>>>> Cheers,
>>>>   Dave
>>>>
>>>>
>>>>> Best,
>>>>> Mike.
>>>>> --
>>>>> Michael Afanasiev
>>>>> Ph.D. Candidate
>>>>> Computational Seismology
>>>>> Institut f?r Geophysik
>>>>> ETH Z?rich
>>>>>
>>>>> Sonneggstrasse 5, NO H 39.2
>>>>> CH 8092 Z?rich
>>>>> michael.afanasiev at erdw.ethz.ch
>>>>>
>>>>>
>>>>
>>>
>>>
>>> --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151026/cf16c6e4/attachment-0001.html>

From dmeiser at txcorp.com  Mon Oct 26 10:04:24 2015
From: dmeiser at txcorp.com (Dominic Meiser)
Date: Mon, 26 Oct 2015 09:04:24 -0600
Subject: [petsc-users] PetscBool in PETSc 3.1? (Ubuntu Precice)
In-Reply-To: <7232803.c5actVMB60@horus>
References: <7232803.c5actVMB60@horus>
Message-ID: <562E40F8.1010900@txcorp.com>



On 10/25/2015 10:40 AM, Florian Lindner wrote:
> Hello,
>
> our build system Travis uses the old Ubuntu Precice Pangolin version that comes with PETSc 3.1.

FWIW petsc master or maint build relatively quickly on travis' build 
machines (2-5 minutes depending on how many third party packages you 
need). If you're able to use travis' containerized infrastructure you 
can cache the PETSc builds reducing the PETSc build times to near 0. 
I've done that for a couple of my projects and it turned out to be 
really easy, see for example here:

https://github.com/d-meiser/SuperContinuum

Have a look at .travis.yml and utilities/get_petsc.sh.

Cheers,
Dominic

>
> While trying to activate petsc for our CI tests, I get the message that the type PetscBool wasn't found. I downloaded http://packages.ubuntu.com/precise/libpetsc3.1-dev and grepped for PetscBool, found non.
>
> Just to make sure I didn't screw it up, is it true, that there is no PetscBool in version 3.1?
>
> Is there a PPA known that provides newer PETSc versions to Precice Pangolin
>
> Thanks,
> Florian
>

-- 
Dominic Meiser
Tech-X Corporation
5621 Arapahoe Avenue
Boulder, CO 80303
USA
Telephone: 303-996-2036
Fax: 303-448-7756
www.txcorp.com

From michael.afanasiev at erdw.ethz.ch  Mon Oct 26 12:15:13 2015
From: michael.afanasiev at erdw.ethz.ch (Afanasiev  Michael)
Date: Mon, 26 Oct 2015 17:15:13 +0000
Subject: [petsc-users] DM question.
In-Reply-To: <CAMYG4G=u=atuSgotEPVX0TG=fBtN+mJUeMa9JBDgWMJKshxfQA@mail.gmail.com>
References: <4F167B1C-9CC0-40DB-83F0-E89EA09D5341@erdw.ethz.ch>
	<CAJ98EDpFiwZeTnC0zav27cvwYShbaD67QuH=VO4+gvg3+7a46Q@mail.gmail.com>
	<CAMYG4G=0jWrGU8JQZHup_G-y5okiZRJong3vBJcrwafeQpZagQ@mail.gmail.com>
	<453C9AC2-8EC3-4D76-B6CE-75D4C496A5D9@erdw.ethz.ch>
	<CAMYG4G=AH5dr1y03ppurs1tTpz7w+X=wSEEsma1YWu1uzWCgqw@mail.gmail.com>
	<4AD13183-D52E-4B8D-A3A1-5BFB79D777E5@erdw.ethz.ch>
	<CAMYG4Gn7B6Gnyu1HnfGcDzxX-tG5PTGkoMUchSrQJZrpD1zUng@mail.gmail.com>
	<68481225-D8CE-443A-A69F-4BDD6C19980D@erdw.ethz.ch>
	<CAMYG4G=u=atuSgotEPVX0TG=fBtN+mJUeMa9JBDgWMJKshxfQA@mail.gmail.com>
Message-ID: <E9AB51D0-955F-4284-B030-992A1B1B7A90@erdw.ethz.ch>

Hi Matt,

Re: a chat with Dave, I?ve constructed a minimal working example of the problem here: https://github.com/michael-afanasiev/salvus2. The relevant stuff is in example/main.c. There?s a little note in there detailing what we?d like to accomplish. It should compile fine with a quick modification of the CMakeLists to tell the makefile where your petsc is. This is all detailed in the Readme, and there?s a small test exodus file hanging out in there as well. A hint would be great.

Best,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 26, 2015, at 12:52 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Mon, Oct 26, 2015 at 6:45 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

Right, makes sense. With some insight from Dave, I?m getting what I expect for a simple 4-cell mesh distributed amongst 4 processors (length of 91 for the global vector, length of 25 for the local vectors, for 4th order polynomial and the GLL basis). Now I imagine what?s left is to figure out how these vectors are globally numbered.

I usually do this by having a consistent way of numbering the dofs on the reference element (i.e. given a (u, v)-space loop through v, then u), and then getting the local->global map via finding coincident points in a kd-tree. My question is: in which order are the local vectors now defined? If I knew this, I could create my local->global map as before. From the way the section was created, I guess it might be something like loop vertices, edges, and then interior?

The numbering is cell unknowns, vertex unknowns, face unknowns, and then edge unknowns. This is, of course, arbitrary and therefore
you can change the order using

  http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/IS/PetscSectionSetPermutation.html#PetscSectionSetPermutation

which renumbers all the mesh points in the PetscSection defining the space. You should be able to reproduce your old ordering using
this.

  Thanks,

    Matt

Thanks for all your help,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 5:11 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 9:03 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

For the GLL basis we have (k+1) points in each dimension, including the endpoints. So for example a 2D element with 4-order polynomials will have 25 locally numbered points per element. I should also mention that the velocity, displacement, and acceleration fields are vectors, with d=dim components at each integration point, whereas strain has (2^dim)-1 components. In what you?ve written above, does this then become (sum over the 4 fields):

(sum(compPerField*dofPerField)) -> vertex
(sum((compPerField*dofPerField)*(k+1)) -> edge
(sum((compPerField*dofPerField)*(k+1)^2) -> quad

No. There are (k-1)^2 unknowns in the interior of the cell, so that we have

  4 + 4 * (k-1) + (k-1)^2 = (k+1)^2

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

I believe I don?t worry about this. Everything is solved in a matrix-free sense, on each element. The relevant physical quantities are calculated on each element, and then summed into the global degrees of freedom. The summed global dof are then brought back to the element level, where updates to the acceleration, velocity, and displacement are calculated via a Newmark time step. So no global system of equations is ever formed.

You accumulate into the global dofs, so you would need more storage if you do not do static condensation. It just good to know this,
but you do not have to do it.

This being said, all the routines to a) integrate the element matrices, b) calculate the gll point locations, c) construct the local->global dof mapping, and d) do the element-wise matrix free time stepping are already written.  My problem is really that I do my mesh decomposition by hand (poorly), and I?d like to transfer this over to Petsc. Of course if I do this, I might as well use DM's LocalToGlobal vector routines to sum my field vectors across mesh partitions.

Yes.

Perhaps a better question to ask would be in the form of a workflow:

1) Read in exodus mesh and partition (done)
2) Set up local and global degrees of freedom on each mesh partition (done)

Here you just have to setup PetscSections that mirror your local and global layout. Then the LocalToGlobal and its inverse will work.

   Matt

3) Integrate element matrices (done)

for i 1, nTimeSteps
3) Step fields forward on each partition (done)
4) Sum to global degrees of freedom on each partition (done)
5) Sum to global degrees of freedom across partitions (????)
6) Retrieve summed global degrees of freedom across partitions (????)
7) Continue

So really what I hope Petsc will help with is just steps 5 and 6. I guess this involves, given a partitioned DMPlex object, which has been partitioned according to the geometry and topology defined in an exodus file, adding the degrees of freedom to each partition-local vector (via a DMPlexSection?). Then, ensuring that the dof added along the partition boundaries sum together properly when a LocalToGlobal vector operation is defined.

Does this make sense? If so, is this something that DMPlex is designed for?
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 23, 2015, at 1:14 PM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Fri, Oct 23, 2015 at 3:01 AM, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hi Matthew,

So I?m discretizing via a tensor product of Lagrange polynomials co-located at the Gauss-Lobatto-Legendre (GLL) points. The polynomial order might range from 4 to 10 or something like that. The current problem is solved on 2D hexes.

I had found DMPlexCreateSection, and followed plex/ex1to get things set up. You can see my attempt below. Basically I?ve got 4 fields (displacement, velocity, acceleration, and strain) over each element. Here I?ve tried to setup a tensor product of 4th order (5 GLL points) Lagrange polynomials (line 11). This seemed to somewhat achieve what I wanted ? I created a global vector and wrote it to a vtk file with VecView, and the numbering seemed to make sense. You can also see my attempt at defining a boundary condition (it looked like DMPlexCreateFromExodus labeled side sets as ?Face Sets?, seems to have worked).

Does this seem to be headed in the right direction?

Yes, however I have some questions. Starting out, I think the GLL points include the endpoints, so
that means for polynomial degree k that numDof[] should really have 4*1 dofs on each vertex,
4*(k-1) dofs on each edge, and 4*(k-1)^2 on each quad. It looks like below you have numDof[] for
a 1D mesh with DG element.

The "Face Sets" is the right label to use for boundary conditions. This will eliminate those variables
from the global system, but they will be present in the local spaces.

With elements like these, it is common (I think) to eliminate the cell unknown explicitly, since the
system is dense and not connected to other cells. For this, you would retain the vertex and edge
unknowns, but not the cell unknowns. I have not tried to do this myself, so I do not know if there
are any pitfalls.

You can see an example of a similar implementation specifically for the kind of spectral elements
you are considering here: https://github.com/jedbrown/dohp. It would probably be useful to understand
what is done there as you implement.

  Thanks,

     Matt

Cheers,
Mike.

DM
mesh::createSection(const DM &dm)
{

01        // displacement, velocity, acceleration, strain
02        IS bcPointIs[1];
03        PetscInt numBc = 1;
04        PetscInt bcField[1];
05        PetscInt numFields = 4;
06        PetscInt dim; DMGetDimension(dm, &dim);
07        PetscInt numComp[numFields];
08        for (auto i=0; i<numFields; i++) {numComp[i] = dim;}
09        PetscInt numDof[numFields*(dim+1)];
10        for (auto i=0; i<numFields*(dim+1); i++) {numDof[i] = 0;}
11        for (auto i=0; i<numFields; i++) {numDof[i*(dim+1)+dim] = 5;}
12        bcField[0] = 0;
13        PetscSection section;
14        DMPlexGetStratumIS(dm, "Face Sets", 1, &bcPointIs[0]);
15        DMPlexCreateSection(dm, dim, numFields, numComp, numDof, numBc, bcField,
16                                                NULL, bcPointIs, NULL, &section);
17        ISDestroy(&bcPointIs[0]);
18        PetscSectionSetFieldName(section, 0, "u");
19        PetscSectionSetFieldName(section, 1, "v");
20        PetscSectionSetFieldName(section, 2, "a");
21        PetscSectionSetFieldName(section, 3, "e");
22        DMSetDefaultSection(dm, section);
23        return dm;
}

--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>

On Oct 22, 2015, at 2:32 AM, Matthew Knepley <knepley at gmail.com<mailto:knepley at gmail.com>> wrote:

On Wed, Oct 21, 2015 at 3:07 PM, Dave May <dave.may at erdw.ethz.ch<mailto:dave.may at erdw.ethz.ch>> wrote:
Hey Mike,



On 21 October 2015 at 18:01, Afanasiev Michael <michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>> wrote:
Hey Dave,

So I?ve got a couple of days where there?s nothing pressing to work on? was thinking of ripping out the parallel routines (ugly) in my wave propagation code and replacing them with Petsc DM routines. I can read in my exodusii mesh with DMPLEX, and partition it with chaco, which gives me a nicely partitioned DM. This takes me like 5 lines of code.  That?s amazing.

But here I?m stuck, and am having a whale of a time with the documentation. All I think I need is a way to modify the exodus-created DM, and add to it the degrees of freedom that are introduced by my quadrature rule. This would be really neat. I can just treat each sub-domain as its own mesh, with its own global numbering. Then whenever necessary I can scatter stuff the the real global degrees of freedom with something like VecLocToGlob. Most of the things I like about the code could stay the same (element-wise, matrix-free nature), just these parallel broadcasts would be infinitely nicer.


First off - I don't use DMPLEX.

Dave is refreshingly candid about his shortcomings ;)


But I just can?t figure out how to set this up. The main problem really boils down to: what?s the best way to add my quadrature points to an already-created DM, which was constructed with an exodus file? I guess I could do this after the file is read, but before the partitioning. In this case though, what?s stopping the partitioner from cutting an element in half? It seems like it would be a lot cleaner to do this post-partitioning.


Presumably what is read from exodus is just the vertices of the hexes, and what you want to do is define the function space (given by your GLL locations) on top of element geometry read in. Is that what you are asking about?

So Dave is right. We read in topology and geometry from ExodusII. Then you define a function space on top. How
exactly are you discretizing? In order to create vectors, do local to global, etc. Petsc really only need to know the
amount of data associated with each mesh piece. You can define this with PetscSection. If you give me an idea
what you want I can help you write the code easily I think.

  Thanks,

     Matt

Any hints here?

Actually I have no experience with this object.
I would just send an email to
  petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>
asking for help.

The developer of DMPLEX (Matt Knepley) will definitely answer within in 1 day.

Cheers,
  Dave


Best,
Mike.
--
Michael Afanasiev
Ph.D. Candidate
Computational Seismology
Institut f?r Geophysik
ETH Z?rich

Sonneggstrasse 5, NO H 39.2
CH 8092 Z?rich
michael.afanasiev at erdw.ethz.ch<mailto:michael.afanasiev at erdw.ethz.ch>





--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener




--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151026/f05d0863/attachment-0001.html>

From a.croucher at auckland.ac.nz  Mon Oct 26 22:03:55 2015
From: a.croucher at auckland.ac.nz (Adrian Croucher)
Date: Tue, 27 Oct 2015 16:03:55 +1300
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>
References: <56207442.8090304@auckland.ac.nz>	<CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>	<562079BD.4090700@auckland.ac.nz>
	<CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>
Message-ID: <562EE99B.1000504@auckland.ac.nz>

hi

When I read the HDF5 file back in again using PetscViewerHDF5Open() and 
VecLoad(), is there any easy way to navigate to the last time step in 
the results?

Or, equivalently, to find how many timesteps there are in there, so I 
could then use PetscViewerHDF5SetTimestep() before doing VecLoad() ?

Cheers, Adrian

On 17/10/15 03:01, Matthew Knepley wrote:
>
> Now I remember. I did not want the output to depend on the viewer.
>
> Does your example work if you replace PetscViewerHDF5SetTimestep() 
> with DMSetOutputSequenceNumber()?
>
>   Thanks,
>
>     Matt

-- 
Dr Adrian Croucher
Senior Research Fellow
Department of Engineering Science
University of Auckland, New Zealand
email: a.croucher at auckland.ac.nz
tel: +64 (0)9 923 84611

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/010521f0/attachment.html>

From knepley at gmail.com  Tue Oct 27 07:01:09 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 07:01:09 -0500
Subject: [petsc-users] HDF5 vec output with DMPlex
In-Reply-To: <562EE99B.1000504@auckland.ac.nz>
References: <56207442.8090304@auckland.ac.nz>
	<CAMYG4G==d+9G93L4n0VYiX+=S2wA+j7Ep4y11Le54zsGPs_r=Q@mail.gmail.com>
	<562079BD.4090700@auckland.ac.nz>
	<CAMYG4G=o0oOahT1bZpO6UbcMrJuWrrm6c9F31trvkASjP_5+Tg@mail.gmail.com>
	<562EE99B.1000504@auckland.ac.nz>
Message-ID: <CAMYG4G=qf27S170+5SUX+G0yUL1DsjBbW-ryrB7Bv4QSQ9ctJw@mail.gmail.com>

On Mon, Oct 26, 2015 at 10:03 PM, Adrian Croucher <a.croucher at auckland.ac.nz
> wrote:

> hi
>
> When I read the HDF5 file back in again using PetscViewerHDF5Open() and
> VecLoad(), is there any easy way to navigate to the last time step in the
> results?
>
> Or, equivalently, to find how many timesteps there are in there, so I
> could then use PetscViewerHDF5SetTimestep() before doing VecLoad() ?
>

I am using

      ierr = PetscViewerHDF5PushGroup(user.checkpoint,
"/fields");CHKERRQ(ierr);
      ierr = DMSetOutputSequenceNumber(dm, user.restart, 0.0);CHKERRQ(ierr);
      ierr = VecLoad(u, user.checkpoint);CHKERRQ(ierr);
      ierr = DMOutputSequenceLoad(dm, user.checkpoint, "time",
user.restart, &time);CHKERRQ(ierr);
      ierr = TSSetTime(ts, (PetscReal) time);CHKERRQ(ierr);
      ierr = DMSetOutputSequenceNumber(dm, 0, 0.0);CHKERRQ(ierr);
      ierr = PetscViewerHDF5PopGroup(user.checkpoint);CHKERRQ(ierr);
      ierr = PetscViewerDestroy(&user.checkpoint);CHKERRQ(ierr);

but right now I do not have a query for the last timestep. We will have to
add an HDF5 query for this. I
am putting it on the list.

  Thanks,

    Matt


> Cheers, Adrian
>
> On 17/10/15 03:01, Matthew Knepley wrote:
>
>
> Now I remember. I did not want the output to depend on the viewer.
>
> Does your example work if you replace PetscViewerHDF5SetTimestep() with
> DMSetOutputSequenceNumber()?
>
>   Thanks,
>
>     Matt
>
>
> --
> Dr Adrian Croucher
> Senior Research Fellow
> Department of Engineering Science
> University of Auckland, New Zealand
> email: a.croucher at auckland.ac.nz
> tel: +64 (0)9 923 84611
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/a3141450/attachment.html>

From davydden at gmail.com  Tue Oct 27 08:15:11 2015
From: davydden at gmail.com (Denis Davydov)
Date: Tue, 27 Oct 2015 14:15:11 +0100
Subject: [petsc-users] [SLEPc] any restriction on the calling order of EPS
	functions?
Message-ID: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>

Dear developers,

I wonder if there are any restriction (apart from obvious) on the calling order of EPS functions?
Is the following logic correct:

once I create EPS object (and specified it?s type)
   ierr = EPSCreate (mpi_communicator, eps);
   ierr = EPSSetType (eps, const_cast<char *>(EPSKRYLOVSCHUR)); // some solver, doesn?t matter which
   ierr = EPSGetST(eps, st);
   ierr = STSetType (st, const_cast<char *>(STSHIFT));

I can execute the following in arbitrary order:
    ierr = EPSSetOperators (eps, *opA, PETSC_NULL);
    ierr = EPSSetProblemType (eps, GHEP);
    ierr = EPSSetInitialVector (eps, *initial_vector);
    int ierr = EPSSetTarget (eps, *target_eigenvalue );
    ierr = EPSSetWhichEigenpairs (eps, set_which);
    ierr = EPSSetConvergenceTest (eps, EPS_CONV_ABS);
    ierr = EPSSetDimensions (eps, n_eigenpairs, PETSC_DECIDE, PETSC_DECIDE);
    int ierr = STSetMatMode(st,SHELL );

and solve:
    ierr = EPSSetFromOptions (eps);
    ierr = EPSSolve (eps);


Kind regards,
Denis 


From gary.rebt at gmx.ch  Tue Oct 27 09:06:16 2015
From: gary.rebt at gmx.ch (Gary Rebt)
Date: Tue, 27 Oct 2015 15:06:16 +0100
Subject: [petsc-users] ILU preconditioner hangs with some zero elements on
	the diagonal
Message-ID: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>

An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/59602a39/attachment.html>

From gary.rebt at gmx.ch  Tue Oct 27 09:09:49 2015
From: gary.rebt at gmx.ch (Gary Rebt)
Date: Tue, 27 Oct 2015 15:09:49 +0100
Subject: [petsc-users] ILU preconditioner hangs with some zero elements on
	the diagonal
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
Message-ID: <trinity-226c93b5-4bff-4803-bd09-53542938d39f-1445954988927@3capp-gmx-bs10>

Dear petsc-users,
?
While using the FEniCS package to Solve a simple Stokes' flow problem, I have run into problems with PETSc preconditioners. In particular, I would like to use ILU (no parallel version) along with GMRES to solve my linear system but the solver just hangs indefinitely at?MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage is at 100% but even for a tiny system (59x59 for minimal test case), the solver does not seem to manage to push through it after 30 mins.
?
PETSc version is 3.6 and the matrix for the minimal test case is as follows :
http://pastebin.com/t3fvdkaS
?
It contains zero diagonal entries, has a condition number of around 1e3 but is definitely non-singular. Direct solvers manage to solve the system as well as GMRES without preconditioner (although after many iterations for a 59x59 system..).
?
Playing with the available options here?http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html] did not seem to solve the issue (even after activating diagonal_fill and/or nonzeros_along_diagonal) although sometimes error 71 is returned which stands for zero pivot detected. Are there yet other options that I have not considered? The default ILU factorization in MATLAB returns satisfactory problems without errors so surely it must be possible with PETSc?
?
As for the choice of ILU, I agree it might be suboptimal in this setting but I do need it for benchmarking purposes.
?
Best regards,
?
Gary

PS : My mistake for the previous e-mail in HTML, you can delete it.

From knepley at gmail.com  Tue Oct 27 09:10:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 09:10:04 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
Message-ID: <CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>

On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch> wrote:

> Dear petsc-users,
>
> While using the FEniCS package to Solve a simple Stokes' flow problem, I
> have run into problems with PETSc preconditioners. In particular, I would
> like to use ILU (no parallel version) along with GMRES to solve my linear
> system but the solver just hangs indefinitely
> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
> is at 100% but even for a tiny system (59x59 for minimal test case), the
> solver does not seem to manage to push through it after 30 mins.
>
> PETSc version is 3.6 and the matrix for the minimal test case is as
> follows :
> http://pastebin.com/t3fvdkaS
>

Hanging is a bug. We will check it out.


> It contains zero diagonal entries, has a condition number of around 1e3
> but is definitely non-singular. Direct solvers manage to solve the system
> as well as GMRES without preconditioner (although after many iterations for
> a 59x59 system..).
>

This will never work. Direct solvers work because they pivot away the
zeros, but ILU is defined by having no pivoting.

  Thanks,

     Matt


> Playing with the available options here
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html
> did not seem to solve the issue (even after activating diagonal_fill and/or
> nonzeros_along_diagonal) although sometimes error 71 is returned which
> stands for zero pivot detected. Are there yet other options that I have not
> considered? The default ILU factorization in MATLAB returns satisfactory
> problems without errors so surely it must be possible with PETSc?
>
> As for the choice of ILU, I agree it might be suboptimal in this setting
> but I do need it for benchmarking purposes.
>
> Best regards,
>
> Gary
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/e36e3b03/attachment-0001.html>

From knepley at gmail.com  Tue Oct 27 09:13:10 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 09:13:10 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
Message-ID: <CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>

On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch> wrote:
>
>> Dear petsc-users,
>>
>> While using the FEniCS package to Solve a simple Stokes' flow problem, I
>> have run into problems with PETSc preconditioners. In particular, I would
>> like to use ILU (no parallel version) along with GMRES to solve my linear
>> system but the solver just hangs indefinitely
>> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
>> is at 100% but even for a tiny system (59x59 for minimal test case), the
>> solver does not seem to manage to push through it after 30 mins.
>>
>> PETSc version is 3.6 and the matrix for the minimal test case is as
>> follows :
>> http://pastebin.com/t3fvdkaS
>>
>
> Hanging is a bug. We will check it out.
>

I do not have any way to read in this ASCII. Can you output a binary version

  -mat_view binary:mat.bin

  Thanks,

     Matt


> It contains zero diagonal entries, has a condition number of around 1e3
>> but is definitely non-singular. Direct solvers manage to solve the system
>> as well as GMRES without preconditioner (although after many iterations for
>> a 59x59 system..).
>>
>
> This will never work. Direct solvers work because they pivot away the
> zeros, but ILU is defined by having no pivoting.
>
>   Thanks,
>
>      Matt
>
>
>> Playing with the available options here
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html
>> did not seem to solve the issue (even after activating diagonal_fill and/or
>> nonzeros_along_diagonal) although sometimes error 71 is returned which
>> stands for zero pivot detected. Are there yet other options that I have not
>> considered? The default ILU factorization in MATLAB returns satisfactory
>> problems without errors so surely it must be possible with PETSc?
>>
>> As for the choice of ILU, I agree it might be suboptimal in this setting
>> but I do need it for benchmarking purposes.
>>
>> Best regards,
>>
>> Gary
>>
>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/6eed69af/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct 27 09:38:17 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 27 Oct 2015 09:38:17 -0500
Subject: [petsc-users] [SLEPc] any restriction on the calling order of
	EPS functions?
In-Reply-To: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>
References: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>
Message-ID: <CAGCphBvt_J_Q16Sz0GY7-5DuVkHy9WmVTH8WLOYRvyMTQ=oMuQ@mail.gmail.com>

Denis:
Your code looks fine to me. There are examples under
slepc/src/eps/examples/tutorials
using ST with SHELL, e.g., ex10.c

Hong

Dear developers,
>
> I wonder if there are any restriction (apart from obvious) on the calling
> order of EPS functions?
> Is the following logic correct:
>
> once I create EPS object (and specified it?s type)
>    ierr = EPSCreate (mpi_communicator, eps);
>    ierr = EPSSetType (eps, const_cast<char *>(EPSKRYLOVSCHUR)); // some
> solver, doesn?t matter which
>    ierr = EPSGetST(eps, st);
>    ierr = STSetType (st, const_cast<char *>(STSHIFT));
>
> I can execute the following in arbitrary order:
>     ierr = EPSSetOperators (eps, *opA, PETSC_NULL);
>     ierr = EPSSetProblemType (eps, GHEP);
>     ierr = EPSSetInitialVector (eps, *initial_vector);
>     int ierr = EPSSetTarget (eps, *target_eigenvalue );
>     ierr = EPSSetWhichEigenpairs (eps, set_which);
>     ierr = EPSSetConvergenceTest (eps, EPS_CONV_ABS);
>     ierr = EPSSetDimensions (eps, n_eigenpairs, PETSC_DECIDE,
> PETSC_DECIDE);
>     int ierr = STSetMatMode(st,SHELL );
>
> and solve:
>     ierr = EPSSetFromOptions (eps);
>     ierr = EPSSolve (eps);
>
>
> Kind regards,
> Denis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/1f46b200/attachment.html>

From jroman at dsic.upv.es  Tue Oct 27 09:39:02 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Tue, 27 Oct 2015 10:39:02 -0400
Subject: [petsc-users] [SLEPc] any restriction on the calling order of
	EPS functions?
In-Reply-To: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>
References: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>
Message-ID: <4A62F27B-8B03-448B-AF82-E8CC182F3DBB@dsic.upv.es>

Yes, in principle you can set options in any order. Let us know if anything does not work for you.

Jose


> El 27/10/2015, a las 9:15, Denis Davydov <davydden at gmail.com> escribi?:
> 
> Dear developers,
> 
> I wonder if there are any restriction (apart from obvious) on the calling order of EPS functions?
> Is the following logic correct:
> 
> once I create EPS object (and specified it?s type)
>   ierr = EPSCreate (mpi_communicator, eps);
>   ierr = EPSSetType (eps, const_cast<char *>(EPSKRYLOVSCHUR)); // some solver, doesn?t matter which
>   ierr = EPSGetST(eps, st);
>   ierr = STSetType (st, const_cast<char *>(STSHIFT));
> 
> I can execute the following in arbitrary order:
>    ierr = EPSSetOperators (eps, *opA, PETSC_NULL);
>    ierr = EPSSetProblemType (eps, GHEP);
>    ierr = EPSSetInitialVector (eps, *initial_vector);
>    int ierr = EPSSetTarget (eps, *target_eigenvalue );
>    ierr = EPSSetWhichEigenpairs (eps, set_which);
>    ierr = EPSSetConvergenceTest (eps, EPS_CONV_ABS);
>    ierr = EPSSetDimensions (eps, n_eigenpairs, PETSC_DECIDE, PETSC_DECIDE);
>    int ierr = STSetMatMode(st,SHELL );
> 
> and solve:
>    ierr = EPSSetFromOptions (eps);
>    ierr = EPSSolve (eps);
> 
> 
> Kind regards,
> Denis 
> 


From gary.rebt at gmx.ch  Tue Oct 27 09:46:17 2015
From: gary.rebt at gmx.ch (Gary Rebt)
Date: Tue, 27 Oct 2015 15:46:17 +0100
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>,
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
Message-ID: <trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>

Thanks. Here's the binary version.

Best,

Gary
?

On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com> wrote:

On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[gary.rebt at gmx.ch]> wrote:

Dear petsc-users,
?
While using the FEniCS package to Solve a simple Stokes' flow problem, I have run into problems with PETSc preconditioners. In particular, I would like to use ILU (no parallel version) along with GMRES to solve my linear system but the solver just hangs indefinitely at?MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage is at 100% but even for a tiny system (59x59 for minimal test case), the solver does not seem to manage to push through it after 30 mins.
?
PETSc version is 3.6 and the matrix for the minimal test case is as follows :
http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
?
Hanging is a bug. We will check it out.
?
I do not have any way to read in this ASCII. Can you output a binary version
?
? -mat_view binary:mat.bin
?
? Thanks,
?
? ? ?Matt
?

It contains zero diagonal entries, has a condition number of around 1e3 but is definitely non-singular. Direct solvers manage to solve the system as well as GMRES without preconditioner (although after many iterations for a 59x59 system..).
?
This will never work. Direct solvers work because they pivot away the zeros, but ILU is defined by having no pivoting.
?
? Thanks,
?
? ? ?Matt
?

Playing with the available options here?http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html] did not seem to solve the issue (even after activating diagonal_fill and/or nonzeros_along_diagonal) although sometimes error 71 is returned which stands for zero pivot detected. Are there yet other options that I have not considered? The default ILU factorization in MATLAB returns satisfactory problems without errors so surely it must be possible with PETSc?
?
As for the choice of ILU, I agree it might be suboptimal in this setting but I do need it for benchmarking purposes.
?
Best regards,
?
Gary?
?--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener?
?--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mat.bin
Type: application/octet-stream
Size: 137656 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/b09a47db/attachment-0001.bin>

From davydden at gmail.com  Tue Oct 27 10:01:15 2015
From: davydden at gmail.com (Denis Davydov)
Date: Tue, 27 Oct 2015 16:01:15 +0100
Subject: [petsc-users] [SLEPc] any restriction on the calling order of
	EPS functions?
In-Reply-To: <4A62F27B-8B03-448B-AF82-E8CC182F3DBB@dsic.upv.es>
References: <8D4CE896-1600-47B8-A292-13BE53EB23C6@gmail.com>
	<4A62F27B-8B03-448B-AF82-E8CC182F3DBB@dsic.upv.es>
Message-ID: <B3ED8F25-05D5-4359-B701-130A361EF943@gmail.com>

Thanks Jose and Hong for the prompt reply. 
I will let you know if I encounter any issues due to calling function in different order.

Kind regards,
Denis 

> On 27 Oct 2015, at 15:39, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
> Yes, in principle you can set options in any order. Let us know if anything does not work for you.
> 
> Jose

From hzhang at mcs.anl.gov  Tue Oct 27 11:13:39 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 27 Oct 2015 11:13:39 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
Message-ID: <CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>

Gary :
I tested your mat.bin using
petsc/src/ksp/ksp/examples/tutorials/ex10.c
./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
...
  Mat Object:   1 MPI processes
    type: seqaij
    rows=588, cols=588
    total: nonzeros=11274, allocated nonzeros=11274
    total number of mallocs used during MatSetValues calls =0
      using I-node routines: found 291 nodes, limit used is 5
Number of iterations =   0
Residual norm 24.2487

It does not converge, neither hangs.
As you said, matrix is non-singular, LU gives a solution
./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
  0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
  1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
Number of iterations =   1
  Residual norm < 1.e-12

Is this the same matrix as you mentioned?

Hong


>
>
> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
>
> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
> gary.rebt at gmx.ch]> wrote:
>
> Dear petsc-users,
>
> While using the FEniCS package to Solve a simple Stokes' flow problem, I
> have run into problems with PETSc preconditioners. In particular, I would
> like to use ILU (no parallel version) along with GMRES to solve my linear
> system but the solver just hangs indefinitely
> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
> is at 100% but even for a tiny system (59x59 for minimal test case), the
> solver does not seem to manage to push through it after 30 mins.
>
> PETSc version is 3.6 and the matrix for the minimal test case is as
> follows :
> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>
> Hanging is a bug. We will check it out.
>
> I do not have any way to read in this ASCII. Can you output a binary
> version
>
>   -mat_view binary:mat.bin
>
>   Thanks,
>
>      Matt
>
>
> It contains zero diagonal entries, has a condition number of around 1e3
> but is definitely non-singular. Direct solvers manage to solve the system
> as well as GMRES without preconditioner (although after many iterations for
> a 59x59 system..).
>
> This will never work. Direct solvers work because they pivot away the
> zeros, but ILU is defined by having no pivoting.
>
>   Thanks,
>
>      Matt
>
>
> Playing with the available options here
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
> did not seem to solve the issue (even after activating diagonal_fill and/or
> nonzeros_along_diagonal) although sometimes error 71 is returned which
> stands for zero pivot detected. Are there yet other options that I have not
> considered? The default ILU factorization in MATLAB returns satisfactory
> problems without errors so surely it must be possible with PETSc?
>
> As for the choice of ILU, I agree it might be suboptimal in this setting
> but I do need it for benchmarking purposes.
>
> Best regards,
>
> Gary
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/8a1f0850/attachment.html>

From davydden at gmail.com  Tue Oct 27 11:28:30 2015
From: davydden at gmail.com (Denis Davydov)
Date: Tue, 27 Oct 2015 17:28:30 +0100
Subject: [petsc-users] soname seems to be absent in OS-X
Message-ID: <6C83934B-C769-4C05-9235-96EE7C22944D@gmail.com>

Dear developers,

It seems that the compiled PETSc library does not have a soname (-Wl,-install_name=xyz.dylib in OS-X).
I compile PETSc/SLEPc using Homebrew both on OS-X and Linux and judging from ldd/otool there is indeed a difference:

Linux (soname is there):
$ ldd .linuxbrew_openblas/opt/slepc/lib/libslepc.so | grep "petsc"
    libpetsc.so.3.6 => /home/woody/iwtm/iwtm108/.linuxbrew_openblas/Cellar/petsc/3.6.2/real/lib/libpetsc.so.3.6 (0x00007fac7822f000)
OS-X (no soname):

$ otool -L /usr/local/opt/slepc/lib/libslepc.dylib | grep "petsc"
    /usr/local/opt/petsc/real/lib/libpetsc.3.6.2.dylib (compatibility version 3.6.0, current version 3.6.2)

I do not see `-Wl,-soname=xyz` in linking flags and nothing like `-Wl,-install_name=xyz` is there on OS-X either.
Any mac users can comment on this?

p.s. as Macports web is down i can?t check what folks are doing there.

Kind regards,
Denis 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/f31919c4/attachment.html>

From knepley at gmail.com  Tue Oct 27 11:44:28 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 11:44:28 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
Message-ID: <CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>

On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:

> Gary :
> I tested your mat.bin using
> petsc/src/ksp/ksp/examples/tutorials/ex10.c
> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
> ...
>   Mat Object:   1 MPI processes
>     type: seqaij
>     rows=588, cols=588
>     total: nonzeros=11274, allocated nonzeros=11274
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 291 nodes, limit used is 5
> Number of iterations =   0
> Residual norm 24.2487
>
> It does not converge, neither hangs.
> As you said, matrix is non-singular, LU gives a solution
> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
>   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
> 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
> 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
> Number of iterations =   1
>   Residual norm < 1.e-12
>
> Is this the same matrix as you mentioned?
>

Hong, could you run ILU on it as well?

  Thanks,

    Matt


> Hong
>
>
>>
>>
>> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
>> wrote:
>>
>> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
>> gary.rebt at gmx.ch]> wrote:
>>
>> Dear petsc-users,
>>
>> While using the FEniCS package to Solve a simple Stokes' flow problem, I
>> have run into problems with PETSc preconditioners. In particular, I would
>> like to use ILU (no parallel version) along with GMRES to solve my linear
>> system but the solver just hangs indefinitely
>> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
>> is at 100% but even for a tiny system (59x59 for minimal test case), the
>> solver does not seem to manage to push through it after 30 mins.
>>
>> PETSc version is 3.6 and the matrix for the minimal test case is as
>> follows :
>> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>>
>> Hanging is a bug. We will check it out.
>>
>> I do not have any way to read in this ASCII. Can you output a binary
>> version
>>
>>   -mat_view binary:mat.bin
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>> It contains zero diagonal entries, has a condition number of around 1e3
>> but is definitely non-singular. Direct solvers manage to solve the system
>> as well as GMRES without preconditioner (although after many iterations for
>> a 59x59 system..).
>>
>> This will never work. Direct solvers work because they pivot away the
>> zeros, but ILU is defined by having no pivoting.
>>
>>   Thanks,
>>
>>      Matt
>>
>>
>> Playing with the available options here
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
>> did not seem to solve the issue (even after activating diagonal_fill and/or
>> nonzeros_along_diagonal) although sometimes error 71 is returned which
>> stands for zero pivot detected. Are there yet other options that I have not
>> considered? The default ILU factorization in MATLAB returns satisfactory
>> problems without errors so surely it must be possible with PETSc?
>>
>> As for the choice of ILU, I agree it might be suboptimal in this setting
>> but I do need it for benchmarking purposes.
>>
>> Best regards,
>>
>> Gary
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>  --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/4fd750da/attachment-0001.html>

From knepley at gmail.com  Tue Oct 27 12:32:31 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 12:32:31 -0500
Subject: [petsc-users] Question - read text file
In-Reply-To: <CAKaLTH2+va34bRbgBOaMZUPQ79CMq1gtrnyK=M3xbwQtLA=bjA@mail.gmail.com>
References: <CAKaLTH2+va34bRbgBOaMZUPQ79CMq1gtrnyK=M3xbwQtLA=bjA@mail.gmail.com>
Message-ID: <CAMYG4Gm3HB_PyD16JgqDWP_YxK+d8j5O5E2C6dfAWtNUJHn4hw@mail.gmail.com>

On Fri, Oct 23, 2015 at 2:21 PM, Marcelo Xavier Guterres <
m.guterres at gmail.com> wrote:

> Dear friends,
>
> I need to read a matrix from a text file (txt).
>
> This matrix has a high dimension (2000 X 2000) .
>
> Example: date.txt - provision in the text file:
>
> 1 2 3 5
> 5 5 7 8
> 5 6 8 9
>
>
> What's the proper way to perform the action?
>

Read each row in and call MatSetValues() for that row.

  Thanks,

    Matt


> thanks for help
>
>
> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> -- -- -- -- -- -- -- -- --
> Marcelo Xavier Guterres
> -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- --
> -- -- -- -- -- -- -- -- -
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/723f71ab/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct 27 12:36:33 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 27 Oct 2015 12:36:33 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
	<CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
Message-ID: <CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>

Matt:

> On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
>
>> Gary :
>> I tested your mat.bin using
>> petsc/src/ksp/ksp/examples/tutorials/ex10.c
>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
>> ...
>>   Mat Object:   1 MPI processes
>>     type: seqaij
>>     rows=588, cols=588
>>     total: nonzeros=11274, allocated nonzeros=11274
>>     total number of mallocs used during MatSetValues calls =0
>>       using I-node routines: found 291 nodes, limit used is 5
>> Number of iterations =   0
>> Residual norm 24.2487
>>
>
>> It does not converge, neither hangs.
>>
>
This is the default GMRES/ILU.
Hong


> As you said, matrix is non-singular, LU gives a solution
>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
>>   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
>> 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
>>   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
>> 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
>> Number of iterations =   1
>>   Residual norm < 1.e-12
>>
>> Is this the same matrix as you mentioned?
>>
>
> Hong, could you run ILU on it as well?
>
>   Thanks,
>
>     Matt
>
>
>> Hong
>>
>>
>>>
>>>
>>> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
>>> wrote:
>>>
>>> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
>>> gary.rebt at gmx.ch]> wrote:
>>>
>>> Dear petsc-users,
>>>
>>> While using the FEniCS package to Solve a simple Stokes' flow problem, I
>>> have run into problems with PETSc preconditioners. In particular, I would
>>> like to use ILU (no parallel version) along with GMRES to solve my linear
>>> system but the solver just hangs indefinitely
>>> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
>>> is at 100% but even for a tiny system (59x59 for minimal test case), the
>>> solver does not seem to manage to push through it after 30 mins.
>>>
>>> PETSc version is 3.6 and the matrix for the minimal test case is as
>>> follows :
>>> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>>>
>>> Hanging is a bug. We will check it out.
>>>
>>> I do not have any way to read in this ASCII. Can you output a binary
>>> version
>>>
>>>   -mat_view binary:mat.bin
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>> It contains zero diagonal entries, has a condition number of around 1e3
>>> but is definitely non-singular. Direct solvers manage to solve the system
>>> as well as GMRES without preconditioner (although after many iterations for
>>> a 59x59 system..).
>>>
>>> This will never work. Direct solvers work because they pivot away the
>>> zeros, but ILU is defined by having no pivoting.
>>>
>>>   Thanks,
>>>
>>>      Matt
>>>
>>>
>>> Playing with the available options here
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
>>> did not seem to solve the issue (even after activating diagonal_fill and/or
>>> nonzeros_along_diagonal) although sometimes error 71 is returned which
>>> stands for zero pivot detected. Are there yet other options that I have not
>>> considered? The default ILU factorization in MATLAB returns satisfactory
>>> problems without errors so surely it must be possible with PETSc?
>>>
>>> As for the choice of ILU, I agree it might be suboptimal in this setting
>>> but I do need it for benchmarking purposes.
>>>
>>> Best regards,
>>>
>>> Gary
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>  --
>>> What most experimenters take for granted before they begin their
>>> experiments is infinitely more interesting than any results to which their
>>> experiments lead.
>>> -- Norbert Wiener
>>>
>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/094fd314/attachment.html>

From knepley at gmail.com  Tue Oct 27 12:38:04 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 12:38:04 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
	<CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
	<CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
Message-ID: <CAMYG4Gkuw047afjLquZnw2RVm=jvGOD--La+J_Q9M2BFJ-+6KA@mail.gmail.com>

On Tue, Oct 27, 2015 at 12:36 PM, Hong <hzhang at mcs.anl.gov> wrote:

> Matt:
>
>> On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Gary :
>>> I tested your mat.bin using
>>> petsc/src/ksp/ksp/examples/tutorials/ex10.c
>>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
>>> ...
>>>   Mat Object:   1 MPI processes
>>>     type: seqaij
>>>     rows=588, cols=588
>>>     total: nonzeros=11274, allocated nonzeros=11274
>>>     total number of mallocs used during MatSetValues calls =0
>>>       using I-node routines: found 291 nodes, limit used is 5
>>> Number of iterations =   0
>>> Residual norm 24.2487
>>>
>>
>>> It does not converge, neither hangs.
>>>
>>
> This is the default GMRES/ILU.
>

  Thanks,

     Matt


> Hong
>
>
>> As you said, matrix is non-singular, LU gives a solution
>>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
>>>   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
>>> 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
>>> 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
>>> Number of iterations =   1
>>>   Residual norm < 1.e-12
>>>
>>> Is this the same matrix as you mentioned?
>>>
>>
>> Hong, could you run ILU on it as well?
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Hong
>>>
>>>
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
>>>> gary.rebt at gmx.ch]> wrote:
>>>>
>>>> Dear petsc-users,
>>>>
>>>> While using the FEniCS package to Solve a simple Stokes' flow problem,
>>>> I have run into problems with PETSc preconditioners. In particular, I would
>>>> like to use ILU (no parallel version) along with GMRES to solve my linear
>>>> system but the solver just hangs indefinitely
>>>> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
>>>> is at 100% but even for a tiny system (59x59 for minimal test case), the
>>>> solver does not seem to manage to push through it after 30 mins.
>>>>
>>>> PETSc version is 3.6 and the matrix for the minimal test case is as
>>>> follows :
>>>> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>>>>
>>>> Hanging is a bug. We will check it out.
>>>>
>>>> I do not have any way to read in this ASCII. Can you output a binary
>>>> version
>>>>
>>>>   -mat_view binary:mat.bin
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>> It contains zero diagonal entries, has a condition number of around 1e3
>>>> but is definitely non-singular. Direct solvers manage to solve the system
>>>> as well as GMRES without preconditioner (although after many iterations for
>>>> a 59x59 system..).
>>>>
>>>> This will never work. Direct solvers work because they pivot away the
>>>> zeros, but ILU is defined by having no pivoting.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>> Playing with the available options here
>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
>>>> did not seem to solve the issue (even after activating diagonal_fill and/or
>>>> nonzeros_along_diagonal) although sometimes error 71 is returned which
>>>> stands for zero pivot detected. Are there yet other options that I have not
>>>> considered? The default ILU factorization in MATLAB returns satisfactory
>>>> problems without errors so surely it must be possible with PETSc?
>>>>
>>>> As for the choice of ILU, I agree it might be suboptimal in this
>>>> setting but I do need it for benchmarking purposes.
>>>>
>>>> Best regards,
>>>>
>>>> Gary
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/168d69b7/attachment-0001.html>

From hzhang at mcs.anl.gov  Tue Oct 27 12:40:28 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 27 Oct 2015 12:40:28 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
	<CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
	<CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
Message-ID: <CAGCphBtTpFr72y1umw0r39n==FNcM8x1wk0UQDY+RGtG0OAX+w@mail.gmail.com>

Here is the reason why it does not converge:
./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
-ksp_converged_reason
Linear solve did not converge due to DIVERGED_NANORINF iterations 0

KSP Object: 1 MPI processes
  type: gmres
    GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
Orthogonalization with no iterative refinement
    GMRES: happy breakdown tolerance 1e-30
  maximum iterations=10000, initial guess is zero
  tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
  left preconditioning
  using PRECONDITIONED norm type for convergence test
PC Object: 1 MPI processes
  type: ilu
    ILU: out-of-place factorization
...

Hong

On Tue, Oct 27, 2015 at 12:36 PM, Hong <hzhang at mcs.anl.gov> wrote:

> Matt:
>
>> On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
>>
>>> Gary :
>>> I tested your mat.bin using
>>> petsc/src/ksp/ksp/examples/tutorials/ex10.c
>>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
>>> ...
>>>   Mat Object:   1 MPI processes
>>>     type: seqaij
>>>     rows=588, cols=588
>>>     total: nonzeros=11274, allocated nonzeros=11274
>>>     total number of mallocs used during MatSetValues calls =0
>>>       using I-node routines: found 291 nodes, limit used is 5
>>> Number of iterations =   0
>>> Residual norm 24.2487
>>>
>>
>>> It does not converge, neither hangs.
>>>
>>
> This is the default GMRES/ILU.
> Hong
>
>
>> As you said, matrix is non-singular, LU gives a solution
>>> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
>>>   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
>>> 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
>>>   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
>>> 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
>>> Number of iterations =   1
>>>   Residual norm < 1.e-12
>>>
>>> Is this the same matrix as you mentioned?
>>>
>>
>> Hong, could you run ILU on it as well?
>>
>>   Thanks,
>>
>>     Matt
>>
>>
>>> Hong
>>>
>>>
>>>>
>>>>
>>>> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
>>>> wrote:
>>>>
>>>> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
>>>> gary.rebt at gmx.ch]> wrote:
>>>>
>>>> Dear petsc-users,
>>>>
>>>> While using the FEniCS package to Solve a simple Stokes' flow problem,
>>>> I have run into problems with PETSc preconditioners. In particular, I would
>>>> like to use ILU (no parallel version) along with GMRES to solve my linear
>>>> system but the solver just hangs indefinitely
>>>> at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage
>>>> is at 100% but even for a tiny system (59x59 for minimal test case), the
>>>> solver does not seem to manage to push through it after 30 mins.
>>>>
>>>> PETSc version is 3.6 and the matrix for the minimal test case is as
>>>> follows :
>>>> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>>>>
>>>> Hanging is a bug. We will check it out.
>>>>
>>>> I do not have any way to read in this ASCII. Can you output a binary
>>>> version
>>>>
>>>>   -mat_view binary:mat.bin
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>> It contains zero diagonal entries, has a condition number of around 1e3
>>>> but is definitely non-singular. Direct solvers manage to solve the system
>>>> as well as GMRES without preconditioner (although after many iterations for
>>>> a 59x59 system..).
>>>>
>>>> This will never work. Direct solvers work because they pivot away the
>>>> zeros, but ILU is defined by having no pivoting.
>>>>
>>>>   Thanks,
>>>>
>>>>      Matt
>>>>
>>>>
>>>> Playing with the available options here
>>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
>>>> did not seem to solve the issue (even after activating diagonal_fill and/or
>>>> nonzeros_along_diagonal) although sometimes error 71 is returned which
>>>> stands for zero pivot detected. Are there yet other options that I have not
>>>> considered? The default ILU factorization in MATLAB returns satisfactory
>>>> problems without errors so surely it must be possible with PETSc?
>>>>
>>>> As for the choice of ILU, I agree it might be suboptimal in this
>>>> setting but I do need it for benchmarking purposes.
>>>>
>>>> Best regards,
>>>>
>>>> Gary
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>  --
>>>> What most experimenters take for granted before they begin their
>>>> experiments is infinitely more interesting than any results to which their
>>>> experiments lead.
>>>> -- Norbert Wiener
>>>>
>>>
>>>
>>
>>
>> --
>> What most experimenters take for granted before they begin their
>> experiments is infinitely more interesting than any results to which their
>> experiments lead.
>> -- Norbert Wiener
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/db321b17/attachment.html>

From bsmith at mcs.anl.gov  Tue Oct 27 13:50:26 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 27 Oct 2015 13:50:26 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
	on the diagonal
In-Reply-To: <CAGCphBtTpFr72y1umw0r39n==FNcM8x1wk0UQDY+RGtG0OAX+w@mail.gmail.com>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
	<CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
	<CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
	<CAGCphBtTpFr72y1umw0r39n==FNcM8x1wk0UQDY+RGtG0OAX+w@mail.gmail.com>
Message-ID: <80DF55CB-BCAD-463D-80B6-DEEA579864F9@mcs.anl.gov>


> On Oct 27, 2015, at 12:40 PM, Hong <hzhang at mcs.anl.gov> wrote:
> 
> Here is the reason why it does not converge:
> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view -ksp_converged_reason
> Linear solve did not converge due to DIVERGED_NANORINF iterations 0

   This means it found a zero pivot either in the factorization or in the first attempt to do a triangular solve.  You can try 

-pc_factor_nonzeros_along_diagonal

and or 

-pc_factor_shift_type nonzero 

to generate a usable LU factorization but these are ad hoc fixes.



  Barry

> 
> KSP Object: 1 MPI processes
>   type: gmres
>     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement
>     GMRES: happy breakdown tolerance 1e-30
>   maximum iterations=10000, initial guess is zero
>   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
>   left preconditioning
>   using PRECONDITIONED norm type for convergence test
> PC Object: 1 MPI processes
>   type: ilu
>     ILU: out-of-place factorization
> ...
> 
> Hong
> 
> On Tue, Oct 27, 2015 at 12:36 PM, Hong <hzhang at mcs.anl.gov> wrote:
> Matt:
> On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
> Gary :
> I tested your mat.bin using
> petsc/src/ksp/ksp/examples/tutorials/ex10.c
> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
> ...
>   Mat Object:   1 MPI processes
>     type: seqaij
>     rows=588, cols=588
>     total: nonzeros=11274, allocated nonzeros=11274
>     total number of mallocs used during MatSetValues calls =0
>       using I-node routines: found 291 nodes, limit used is 5
> Number of iterations =   0
> Residual norm 24.2487
> 
> It does not converge, neither hangs.
>  
> This is the default GMRES/ILU.
> Hong 
>  
> As you said, matrix is non-singular, LU gives a solution
> ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
>   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
>   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
> Number of iterations =   1
>   Residual norm < 1.e-12
> 
> Is this the same matrix as you mentioned?
> 
> Hong, could you run ILU on it as well?
> 
>   Thanks,
> 
>     Matt
>  
> Hong
> 
> 
>  
> 
> On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[gary.rebt at gmx.ch]> wrote:
> 
> Dear petsc-users,
>  
> While using the FEniCS package to Solve a simple Stokes' flow problem, I have run into problems with PETSc preconditioners. In particular, I would like to use ILU (no parallel version) along with GMRES to solve my linear system but the solver just hangs indefinitely at MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage is at 100% but even for a tiny system (59x59 for minimal test case), the solver does not seem to manage to push through it after 30 mins.
>  
> PETSc version is 3.6 and the matrix for the minimal test case is as follows :
> http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
>  
> Hanging is a bug. We will check it out.
>  
> I do not have any way to read in this ASCII. Can you output a binary version
>  
>   -mat_view binary:mat.bin
>  
>   Thanks,
>  
>      Matt
>  
> 
> It contains zero diagonal entries, has a condition number of around 1e3 but is definitely non-singular. Direct solvers manage to solve the system as well as GMRES without preconditioner (although after many iterations for a 59x59 system..).
>  
> This will never work. Direct solvers work because they pivot away the zeros, but ILU is defined by having no pivoting.
>  
>   Thanks,
>  
>      Matt
>  
> 
> Playing with the available options here http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html] did not seem to solve the issue (even after activating diagonal_fill and/or nonzeros_along_diagonal) although sometimes error 71 is returned which stands for zero pivot detected. Are there yet other options that I have not considered? The default ILU factorization in MATLAB returns satisfactory problems without errors so surely it must be possible with PETSc?
>  
> As for the choice of ILU, I agree it might be suboptimal in this setting but I do need it for benchmarking purposes.
>  
> Best regards,
>  
> Gary 
>  --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener 
>  --
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> 


From mirzadeh at gmail.com  Tue Oct 27 14:00:09 2015
From: mirzadeh at gmail.com (Mohammad Mirzadeh)
Date: Tue, 27 Oct 2015 15:00:09 -0400
Subject: [petsc-users] soname seems to be absent in OS-X
In-Reply-To: <6C83934B-C769-4C05-9235-96EE7C22944D@gmail.com>
References: <6C83934B-C769-4C05-9235-96EE7C22944D@gmail.com>
Message-ID: <CAKsPKw=o3VcfaW3guUr+wathrZHfJW0mDhUkPYip_uV5Lh=USQ@mail.gmail.com>

Denis,

I use Homebrew to manage petsc and all related dependencies on OSX which
uses .dylib for shared libs instead of .so. However, I have not had any
issues linking against them just the usual way by passing -L/path/to/lib
-lwhatever (you need to pass -Wl,-rpath,/path/to/lib as well if lib is not
in run-time search path).

Hope this helps.


On Tue, Oct 27, 2015 at 12:28 PM, Denis Davydov <davydden at gmail.com> wrote:

> Dear developers,
>
> It seems that the compiled PETSc library does not have a soname
> (-Wl,-install_name=xyz.dylib in OS-X).
> I compile PETSc/SLEPc using Homebrew both on OS-X and Linux and judging
> from ldd/otool there is indeed a difference:
>
> Linux (soname is there):
>
> $ ldd .linuxbrew_openblas/opt/slepc/lib/libslepc.so | grep "petsc"
>     libpetsc.so.3.6 => /home/woody/iwtm/iwtm108/.linuxbrew_openblas/Cellar/petsc/3.6.2/real/lib/libpetsc.so.3.6 (0x00007fac7822f000)
>
> OS-X (no soname):
>
> $ otool -L /usr/local/opt/slepc/lib/libslepc.dylib | grep "petsc"
>     /usr/local/opt/petsc/real/lib/libpetsc.3.6.2.dylib (compatibility version 3.6.0, current version 3.6.2)
>
>
> I do not see `-Wl,-soname=xyz` in linking flags and nothing like
> `-Wl,-install_name=xyz` is there on OS-X either.
> Any mac users can comment on this?
>
> p.s. as Macports web is down i can?t check what folks are doing there.
>
> Kind regards,
> Denis
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/ac09f253/attachment.html>

From hzhang at mcs.anl.gov  Tue Oct 27 14:08:19 2015
From: hzhang at mcs.anl.gov (Hong)
Date: Tue, 27 Oct 2015 14:08:19 -0500
Subject: [petsc-users] ILU preconditioner hangs with some zero elements
 on the diagonal
In-Reply-To: <80DF55CB-BCAD-463D-80B6-DEEA579864F9@mcs.anl.gov>
References: <trinity-68de397b-dce6-4ed7-ab99-d8dcabbf6ab4-1445954776330@3capp-gmx-bs10>
	<CAMYG4GkeVP8Vt9PagKgDFt5fK3cxWfXQiCOT_gAMS21Z4gci0w@mail.gmail.com>
	<CAMYG4G=b5WcHdmCb+drz0=_zz5g1TxtwBzCyK2Da9cO9Y8bj6w@mail.gmail.com>
	<trinity-12728cfb-56bc-449d-b6d3-a7b16fe73beb-1445957177028@3capp-gmx-bs10>
	<CAGCphBvTDvFgT-La347bmP4j2GpXVVSVqAgx3wTq395g3C7a=Q@mail.gmail.com>
	<CAMYG4G=O91aqvmK+6eHkOLLxKyJjQ4hoVEA2nergC_5rHW_U1w@mail.gmail.com>
	<CAGCphBtn22X8A5YmXQrPU6i8pmYF4Tw8PQzgedZFmzOFWaPDrg@mail.gmail.com>
	<CAGCphBtTpFr72y1umw0r39n==FNcM8x1wk0UQDY+RGtG0OAX+w@mail.gmail.com>
	<80DF55CB-BCAD-463D-80B6-DEEA579864F9@mcs.anl.gov>
Message-ID: <CAGCphBv=C4wcE31=xpUpY714hUs0qmQ7yM1=W+-ztmpm9ajNVQ@mail.gmail.com>

I did a quick test:
1) ilu with any shift type does not converge
2) gmres/4 bjacobi/ilu + shift_type nonzero does not converge
3) mpiexec -n 4 ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor -sub_pc_type lu
-sub_pc_factor_shift_type nonzero
...
0 KSP Residual norm 3.284826093129e+03
  1 KSP Residual norm 2.802972716423e+03
  2 KSP Residual norm 2.039112137210e+03
...
 24 KSP Residual norm 2.666350543810e-02
Number of iterations =  24
Residual norm 0.0179698

Hong

On Tue, Oct 27, 2015 at 1:50 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 27, 2015, at 12:40 PM, Hong <hzhang at mcs.anl.gov> wrote:
> >
> > Here is the reason why it does not converge:
> > ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
> -ksp_converged_reason
> > Linear solve did not converge due to DIVERGED_NANORINF iterations 0
>
>    This means it found a zero pivot either in the factorization or in the
> first attempt to do a triangular solve.  You can try
>
> -pc_factor_nonzeros_along_diagonal
>
> and or
>
> -pc_factor_shift_type nonzero
>
> to generate a usable LU factorization but these are ad hoc fixes.
>
>
>
>   Barry
>
> >
> > KSP Object: 1 MPI processes
> >   type: gmres
> >     GMRES: restart=30, using Classical (unmodified) Gram-Schmidt
> Orthogonalization with no iterative refinement
> >     GMRES: happy breakdown tolerance 1e-30
> >   maximum iterations=10000, initial guess is zero
> >   tolerances:  relative=1e-05, absolute=1e-50, divergence=10000.
> >   left preconditioning
> >   using PRECONDITIONED norm type for convergence test
> > PC Object: 1 MPI processes
> >   type: ilu
> >     ILU: out-of-place factorization
> > ...
> >
> > Hong
> >
> > On Tue, Oct 27, 2015 at 12:36 PM, Hong <hzhang at mcs.anl.gov> wrote:
> > Matt:
> > On Tue, Oct 27, 2015 at 11:13 AM, Hong <hzhang at mcs.anl.gov> wrote:
> > Gary :
> > I tested your mat.bin using
> > petsc/src/ksp/ksp/examples/tutorials/ex10.c
> > ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -ksp_view
> > ...
> >   Mat Object:   1 MPI processes
> >     type: seqaij
> >     rows=588, cols=588
> >     total: nonzeros=11274, allocated nonzeros=11274
> >     total number of mallocs used during MatSetValues calls =0
> >       using I-node routines: found 291 nodes, limit used is 5
> > Number of iterations =   0
> > Residual norm 24.2487
> >
> > It does not converge, neither hangs.
> >
> > This is the default GMRES/ILU.
> > Hong
> >
> > As you said, matrix is non-singular, LU gives a solution
> > ./ex10 -f0 $D/mat.bin -rhs 0 -ksp_monitor_true_residual -pc_type lu
> >   0 KSP preconditioned resid norm 3.298891225772e+03 true resid norm
> 2.424871130596e+01 ||r(i)||/||b|| 1.000000000000e+00
> >   1 KSP preconditioned resid norm 1.918157196467e-12 true resid norm
> 5.039404549028e-13 ||r(i)||/||b|| 2.078215409241e-14
> > Number of iterations =   1
> >   Residual norm < 1.e-12
> >
> > Is this the same matrix as you mentioned?
> >
> > Hong, could you run ILU on it as well?
> >
> >   Thanks,
> >
> >     Matt
> >
> > Hong
> >
> >
> >
> >
> > On Tue, Oct 27, 2015 at 9:10 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> >
> > On Tue, Oct 27, 2015 at 9:06 AM, Gary Rebt <gary.rebt at gmx.ch[
> gary.rebt at gmx.ch]> wrote:
> >
> > Dear petsc-users,
> >
> > While using the FEniCS package to Solve a simple Stokes' flow problem, I
> have run into problems with PETSc preconditioners. In particular, I would
> like to use ILU (no parallel version) along with GMRES to solve my linear
> system but the solver just hangs indefinitely at
> MatLUFactorNumeric_SeqAIJ_Inode without outputting anything. CPU usage is
> at 100% but even for a tiny system (59x59 for minimal test case), the
> solver does not seem to manage to push through it after 30 mins.
> >
> > PETSc version is 3.6 and the matrix for the minimal test case is as
> follows :
> > http://pastebin.com/t3fvdkaS[http://pastebin.com/t3fvdkaS]
> >
> > Hanging is a bug. We will check it out.
> >
> > I do not have any way to read in this ASCII. Can you output a binary
> version
> >
> >   -mat_view binary:mat.bin
> >
> >   Thanks,
> >
> >      Matt
> >
> >
> > It contains zero diagonal entries, has a condition number of around 1e3
> but is definitely non-singular. Direct solvers manage to solve the system
> as well as GMRES without preconditioner (although after many iterations for
> a 59x59 system..).
> >
> > This will never work. Direct solvers work because they pivot away the
> zeros, but ILU is defined by having no pivoting.
> >
> >   Thanks,
> >
> >      Matt
> >
> >
> > Playing with the available options here
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html[http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCILU.html]
> did not seem to solve the issue (even after activating diagonal_fill and/or
> nonzeros_along_diagonal) although sometimes error 71 is returned which
> stands for zero pivot detected. Are there yet other options that I have not
> considered? The default ILU factorization in MATLAB returns satisfactory
> problems without errors so surely it must be possible with PETSc?
> >
> > As for the choice of ILU, I agree it might be suboptimal in this setting
> but I do need it for benchmarking purposes.
> >
> > Best regards,
> >
> > Gary
> >  --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >  --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/fed48840/attachment-0001.html>

From pvsang002 at gmail.com  Tue Oct 27 20:18:18 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Wed, 28 Oct 2015 08:18:18 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
Message-ID: <CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>

Hi Barry,

I made my function for preallocating the DM matrix. I am using immersed
boundary method to solve a problem of solid mechanics (3D, linear
elasticity).
At every solid nodes, I have 3 unknowns for displacement in 3 directions
(nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each
equation I preallocate 27*3  = 81 entries.
When running the code, I got many of the following error  when setting
values into the preallocated matrix:

[0]PETSC ERROR: --------------------- Error Message
------------------------------------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
[0]PETSC ERROR: See docs/changes/index.html for recent updates.
[0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
[0]PETSC ERROR: See docs/index.html for manual pages.
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed
Oct 28 07:34:37 2015
[0]PETSC ERROR: Libraries linked from
/home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
[0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
[0]PETSC ERROR: Configure options --download-mpich=1
--with-scalar-type=real --with-clanguage=cxx --download-mumps=1
--download-blacs=1 --download-parmetis=1 --download-scalapack=1
--with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1
-download-cmake=1 --download-f-blas-lapack=1
[0]PETSC ERROR:
------------------------------------------------------------------------
[0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in
/home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
[0]PETSC ERROR: MatSetValuesLocal() line 1968 in
/home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
[0]PETSC ERROR: MatSetValuesStencil() line 1339 in
/home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c

Can you please explain to me what is possible reason for this error? it
looks like there is problem with the mapping from LocaltoGobal index, but
that part I just copy from the original code.
One more question I have is that, when preallocating for a node, as you can
see in my below code (based on original PETSc code), since I have 3
equations, I preallocated 3 rows (in rows[l] array), each row has a number
of columns index, all column indices of the three rows are stored in the
cols[] array. At this point I don't understand how PETSc know in that
cols[] arrays which column indices belonging to the first(second/third)
row? Please help me to understand this.

Thank you very much.
Pham

Below are my code for preallocating the DM matrix;

PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat
pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>&
isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>&
forcing_or_ghost_cells)
{
  PetscErrorCode         ierr;
  PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
  PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows =
NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
  PetscInt
istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
  MPI_Comm               comm;
  PetscScalar            *values;
  DMDABoundaryType       bx,by,bz;
  ISLocalToGlobalMapping ltog,ltogb;
  DMDAStencilType        st;

  PetscFunctionBegin;

  ierr =
DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr);
// nc = 3
  col  = 2*s + 1;
  ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
  ierr =
DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
  ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
  ierr =
PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
  ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
  ierr =
MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
  nCells = mesh->nCells;
  sasVector<int> cell_type(mesh->nCells,0);
  for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
  for(int i=0;i<forcing_or_ghost_cells.N;i++)
 cell_type[forcing_or_ghost_cells[i]] = 2;
  for(i = 0;i<nCells;i++)
  {
    slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys) +
gnx*gny*(mesh->Cells_ijk[i].k - gzs);
    cnt = 0;
    if(cell_type[i]==0) /// fluid cells
    {
      for (l=0; l<nc; l++)
      {
        cols[cnt++] = l + nc*slot;
        rows[l] = l + nc*slot;
      }
    }

    if(cell_type[i]==1) /// solid cells:
    {
      for (l=0; l<nc; l++)
      {
       for (int ll=0; ll<nc; ll++)
        for (ii=-1; ii<2; ii++) {
          for (jj=-1; jj<2; jj++) {
            for (kk=-1; kk<2; kk++) {
                cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
              }
            }
          }
        rows[l] = l + nc*(slot);
      }
    }
    if(cell_type[i]!=2)
    ierr =
MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
  }
  MatCreate(comm,&pMat);

MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
  MatSetType(pMat,MATAIJ);
  ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
  ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
  ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
  ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
  ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);

  ierr = PetscFree2(rows,cols);CHKERRQ(ierr);

  PetscFunctionReturn(0);
}

On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Thank you, Dave,
> >
> > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
> my_context) when I need to create my matrix, and forget about
> DMDASetGetMatrix()?
>
>   You can do that if that is all you need. In some contexts when something
> else in PETSc needs to create a specific matrix (like with geometric
> multigrid) then that code calls DMCreateMatrix() which would then use your
> creation routine.
>
>    So, in summary you can start by just calling your own routine and
> convert it to use DMSetGetMatrix() later if you need to.
>
>   Barry
>
> >
> > Thank you.
> >
> > Pham
> >
> > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com>
> wrote:
> > If you need to access a user defined context from within your
> CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> >
> > If your context is not a petsc object, you can use
> PetscContainerCreate(),
> >
> >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> >
> > You would then set your user context pointer inside the container and
> then use PetscObjectCompose() to attach the container to the DM
> >
> > Thanks,
> >   Dave
> >
> >
> > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
> > Hi Barry,
> >
> > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
> Mat J), the function pointer in DMDASetGetMatrix() only accept function
> with that two arguments.
> > As you suggested, I am writing a routine (based on
> DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish,
> to do that I think It needs to have one more input argument:
> My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But
> DMDASetGetMatrix() will not accept pointer of this function. Please give a
> suggestion to overcome this.
> >
> > Thank you.
> >
> > Pham
> >
> >
> > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
> in 3d)   You need to copy this routine and add whatever additional
> preallocation information you need.   Then call DMDASetGetMatrix()  so that
> the DM will use your routine to create the matrix for you.
> >
> >    Barry
> >
> >
> >
> >
> > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > >
> > > Hi,
> > >
> > > I am using DMCreateMatrix to create matrix from a existed DM object
> and defined stencil.
> > > In my code, boundary nodes need to involve many inner nodes, thus
> matrix rows corresponding to boundary nodes are almost dense. How can I
> tell petsc that those rows need to be preallocated with more entries? I
> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
> be lost.
> > >
> > > Many thanks.
> > >
> > > Sam.
> > >
> > >
> >
> >
> >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151028/774ade5b/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: configure.log
Type: text/x-log
Size: 7267954 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151028/774ade5b/attachment-0001.bin>

From knepley at gmail.com  Tue Oct 27 20:48:14 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Tue, 27 Oct 2015 20:48:14 -0500
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
	<CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
Message-ID: <CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>

On Tue, Oct 27, 2015 at 8:18 PM, Sang pham van <pvsang002 at gmail.com> wrote:

> Hi Barry,
>
> I made my function for preallocating the DM matrix. I am using immersed
> boundary method to solve a problem of solid mechanics (3D, linear
> elasticity).
> At every solid nodes, I have 3 unknowns for displacement in 3 directions
> (nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each
> equation I preallocate 27*3  = 81 entries.
> When running the code, I got many of the following error  when setting
> values into the preallocated matrix:
>

You might have memory corruption. Run with valgrind.

  Thanks,

    Matt


> [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed
> Oct 28 07:34:37 2015
> [0]PETSC ERROR: Libraries linked from
> /home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
> [0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
> [0]PETSC ERROR: Configure options --download-mpich=1
> --with-scalar-type=real --with-clanguage=cxx --download-mumps=1
> --download-blacs=1 --download-parmetis=1 --download-scalapack=1
> --with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1
> -download-cmake=1 --download-f-blas-lapack=1
> [0]PETSC ERROR:
> ------------------------------------------------------------------------
> [0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in
> /home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
> [0]PETSC ERROR: MatSetValuesLocal() line 1968 in
> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> [0]PETSC ERROR: MatSetValuesStencil() line 1339 in
> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
>
> Can you please explain to me what is possible reason for this error? it
> looks like there is problem with the mapping from LocaltoGobal index, but
> that part I just copy from the original code.
> One more question I have is that, when preallocating for a node, as you
> can see in my below code (based on original PETSc code), since I have 3
> equations, I preallocated 3 rows (in rows[l] array), each row has a number
> of columns index, all column indices of the three rows are stored in the
> cols[] array. At this point I don't understand how PETSc know in that
> cols[] arrays which column indices belonging to the first(second/third)
> row? Please help me to understand this.
>
> Thank you very much.
> Pham
>
> Below are my code for preallocating the DM matrix;
>
> PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat
> pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>&
> isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>&
> forcing_or_ghost_cells)
> {
>   PetscErrorCode         ierr;
>   PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
>   PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows =
> NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
>   PetscInt
> istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
>   MPI_Comm               comm;
>   PetscScalar            *values;
>   DMDABoundaryType       bx,by,bz;
>   ISLocalToGlobalMapping ltog,ltogb;
>   DMDAStencilType        st;
>
>   PetscFunctionBegin;
>
>   ierr =
> DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr);
> // nc = 3
>   col  = 2*s + 1;
>   ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
>   ierr =
> DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
>   ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
>   ierr =
> PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
>   ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
>   ierr =
> MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
>   nCells = mesh->nCells;
>   sasVector<int> cell_type(mesh->nCells,0);
>   for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
>   for(int i=0;i<forcing_or_ghost_cells.N;i++)
>  cell_type[forcing_or_ghost_cells[i]] = 2;
>   for(i = 0;i<nCells;i++)
>   {
>     slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys) +
> gnx*gny*(mesh->Cells_ijk[i].k - gzs);
>     cnt = 0;
>     if(cell_type[i]==0) /// fluid cells
>     {
>       for (l=0; l<nc; l++)
>       {
>         cols[cnt++] = l + nc*slot;
>         rows[l] = l + nc*slot;
>       }
>     }
>
>     if(cell_type[i]==1) /// solid cells:
>     {
>       for (l=0; l<nc; l++)
>       {
>        for (int ll=0; ll<nc; ll++)
>         for (ii=-1; ii<2; ii++) {
>           for (jj=-1; jj<2; jj++) {
>             for (kk=-1; kk<2; kk++) {
>                 cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
>               }
>             }
>           }
>         rows[l] = l + nc*(slot);
>       }
>     }
>     if(cell_type[i]!=2)
>     ierr =
> MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
>   }
>   MatCreate(comm,&pMat);
>
> MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
>   MatSetType(pMat,MATAIJ);
>   ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
>   ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
>   ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
>   ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
>   ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);
>
>   ierr = PetscFree2(rows,cols);CHKERRQ(ierr);
>
>   PetscFunctionReturn(0);
> }
>
> On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>> > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
>> >
>> > Thank you, Dave,
>> >
>> > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
>> my_context) when I need to create my matrix, and forget about
>> DMDASetGetMatrix()?
>>
>>   You can do that if that is all you need. In some contexts when
>> something else in PETSc needs to create a specific matrix (like with
>> geometric multigrid) then that code calls DMCreateMatrix() which would then
>> use your creation routine.
>>
>>    So, in summary you can start by just calling your own routine and
>> convert it to use DMSetGetMatrix() later if you need to.
>>
>>   Barry
>>
>> >
>> > Thank you.
>> >
>> > Pham
>> >
>> > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com>
>> wrote:
>> > If you need to access a user defined context from within your
>> CreateMatrix function, you can attach it to the DM via PetscObjectCompose
>> >
>> >
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
>> >
>> > If your context is not a petsc object, you can use
>> PetscContainerCreate(),
>> >
>> >
>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
>> >
>> > You would then set your user context pointer inside the container and
>> then use PetscObjectCompose() to attach the container to the DM
>> >
>> > Thanks,
>> >   Dave
>> >
>> >
>> > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
>> > Hi Barry,
>> >
>> > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
>> Mat J), the function pointer in DMDASetGetMatrix() only accept function
>> with that two arguments.
>> > As you suggested, I am writing a routine (based on
>> DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish,
>> to do that I think It needs to have one more input argument:
>> My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But
>> DMDASetGetMatrix() will not accept pointer of this function. Please give a
>> suggestion to overcome this.
>> >
>> > Thank you.
>> >
>> > Pham
>> >
>> >
>> > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>> >
>> >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
>> in 3d)   You need to copy this routine and add whatever additional
>> preallocation information you need.   Then call DMDASetGetMatrix()  so that
>> the DM will use your routine to create the matrix for you.
>> >
>> >    Barry
>> >
>> >
>> >
>> >
>> > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
>> wrote:
>> > >
>> > > Hi,
>> > >
>> > > I am using DMCreateMatrix to create matrix from a existed DM object
>> and defined stencil.
>> > > In my code, boundary nodes need to involve many inner nodes, thus
>> matrix rows corresponding to boundary nodes are almost dense. How can I
>> tell petsc that those rows need to be preallocated with more entries? I
>> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
>> be lost.
>> > >
>> > > Many thanks.
>> > >
>> > > Sam.
>> > >
>> > >
>> >
>> >
>> >
>> >
>>
>>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151027/dc639bbc/attachment.html>

From pvsang002 at gmail.com  Tue Oct 27 20:55:21 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Wed, 28 Oct 2015 08:55:21 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
	<CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
	<CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>
Message-ID: <CAOx7dQmjj_5Z354MvbNe-Qst5458+yY3Q=Yzdiwnm6c1Kwnfrw@mail.gmail.com>

Hi Matt,

Can you please answer my second question about the way PESCs understand the
column indies? :

One more question I have is that, when preallocating for a node, as you can
see in my below code (based on original PETSc code), since I have 3
equations, I preallocated 3 rows (in rows[l] array), each row has a number
of columns index, all column indices of the three rows are stored in the
cols[] array. At this point I don't understand how PETSc know in that
cols[] arrays which column indices belonging to the first(second/third)
row? Please help me to understand this.

Thank you.

Pham


On Wed, Oct 28, 2015 at 8:48 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Tue, Oct 27, 2015 at 8:18 PM, Sang pham van <pvsang002 at gmail.com>
> wrote:
>
>> Hi Barry,
>>
>> I made my function for preallocating the DM matrix. I am using immersed
>> boundary method to solve a problem of solid mechanics (3D, linear
>> elasticity).
>> At every solid nodes, I have 3 unknowns for displacement in 3 directions
>> (nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each
>> equation I preallocate 27*3  = 81 entries.
>> When running the code, I got many of the following error  when setting
>> values into the preallocated matrix:
>>
>
> You might have memory corruption. Run with valgrind.
>
>   Thanks,
>
>     Matt
>
>
>> [0]PETSC ERROR: --------------------- Error Message
>> ------------------------------------
>> [0]PETSC ERROR: Argument out of range!
>> [0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
>> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
>> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
>> [0]PETSC ERROR: See docs/index.html for manual pages.
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed
>> Oct 28 07:34:37 2015
>> [0]PETSC ERROR: Libraries linked from
>> /home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
>> [0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
>> [0]PETSC ERROR: Configure options --download-mpich=1
>> --with-scalar-type=real --with-clanguage=cxx --download-mumps=1
>> --download-blacs=1 --download-parmetis=1 --download-scalapack=1
>> --with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1
>> -download-cmake=1 --download-f-blas-lapack=1
>> [0]PETSC ERROR:
>> ------------------------------------------------------------------------
>> [0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in
>> /home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
>> [0]PETSC ERROR: MatSetValuesLocal() line 1968 in
>> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
>> [0]PETSC ERROR: MatSetValuesStencil() line 1339 in
>> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
>>
>> Can you please explain to me what is possible reason for this error? it
>> looks like there is problem with the mapping from LocaltoGobal index, but
>> that part I just copy from the original code.
>> One more question I have is that, when preallocating for a node, as you
>> can see in my below code (based on original PETSc code), since I have 3
>> equations, I preallocated 3 rows (in rows[l] array), each row has a number
>> of columns index, all column indices of the three rows are stored in the
>> cols[] array. At this point I don't understand how PETSc know in that
>> cols[] arrays which column indices belonging to the first(second/third)
>> row? Please help me to understand this.
>>
>> Thank you very much.
>> Pham
>>
>> Below are my code for preallocating the DM matrix;
>>
>> PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat
>> pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>&
>> isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>&
>> forcing_or_ghost_cells)
>> {
>>   PetscErrorCode         ierr;
>>   PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
>>   PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows =
>> NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
>>   PetscInt
>> istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
>>   MPI_Comm               comm;
>>   PetscScalar            *values;
>>   DMDABoundaryType       bx,by,bz;
>>   ISLocalToGlobalMapping ltog,ltogb;
>>   DMDAStencilType        st;
>>
>>   PetscFunctionBegin;
>>
>>   ierr =
>> DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr);
>> // nc = 3
>>   col  = 2*s + 1;
>>   ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
>>   ierr =
>> DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
>>   ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
>>   ierr =
>> PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
>>   ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
>>   ierr =
>> MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
>>   nCells = mesh->nCells;
>>   sasVector<int> cell_type(mesh->nCells,0);
>>   for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
>>   for(int i=0;i<forcing_or_ghost_cells.N;i++)
>>  cell_type[forcing_or_ghost_cells[i]] = 2;
>>   for(i = 0;i<nCells;i++)
>>   {
>>     slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys)
>> + gnx*gny*(mesh->Cells_ijk[i].k - gzs);
>>     cnt = 0;
>>     if(cell_type[i]==0) /// fluid cells
>>     {
>>       for (l=0; l<nc; l++)
>>       {
>>         cols[cnt++] = l + nc*slot;
>>         rows[l] = l + nc*slot;
>>       }
>>     }
>>
>>     if(cell_type[i]==1) /// solid cells:
>>     {
>>       for (l=0; l<nc; l++)
>>       {
>>        for (int ll=0; ll<nc; ll++)
>>         for (ii=-1; ii<2; ii++) {
>>           for (jj=-1; jj<2; jj++) {
>>             for (kk=-1; kk<2; kk++) {
>>                 cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
>>               }
>>             }
>>           }
>>         rows[l] = l + nc*(slot);
>>       }
>>     }
>>     if(cell_type[i]!=2)
>>     ierr =
>> MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
>>   }
>>   MatCreate(comm,&pMat);
>>
>> MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
>>   MatSetType(pMat,MATAIJ);
>>   ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
>>   ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
>>   ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
>>   ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
>>   ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);
>>
>>   ierr = PetscFree2(rows,cols);CHKERRQ(ierr);
>>
>>   PetscFunctionReturn(0);
>> }
>>
>> On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>>
>>> > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com>
>>> wrote:
>>> >
>>> > Thank you, Dave,
>>> >
>>> > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
>>> my_context) when I need to create my matrix, and forget about
>>> DMDASetGetMatrix()?
>>>
>>>   You can do that if that is all you need. In some contexts when
>>> something else in PETSc needs to create a specific matrix (like with
>>> geometric multigrid) then that code calls DMCreateMatrix() which would then
>>> use your creation routine.
>>>
>>>    So, in summary you can start by just calling your own routine and
>>> convert it to use DMSetGetMatrix() later if you need to.
>>>
>>>   Barry
>>>
>>> >
>>> > Thank you.
>>> >
>>> > Pham
>>> >
>>> > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com>
>>> wrote:
>>> > If you need to access a user defined context from within your
>>> CreateMatrix function, you can attach it to the DM via PetscObjectCompose
>>> >
>>> >
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
>>> >
>>> > If your context is not a petsc object, you can use
>>> PetscContainerCreate(),
>>> >
>>> >
>>> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
>>> >
>>> > You would then set your user context pointer inside the container and
>>> then use PetscObjectCompose() to attach the container to the DM
>>> >
>>> > Thanks,
>>> >   Dave
>>> >
>>> >
>>> > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com>
>>> wrote:
>>> > Hi Barry,
>>> >
>>> > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
>>> Mat J), the function pointer in DMDASetGetMatrix() only accept function
>>> with that two arguments.
>>> > As you suggested, I am writing a routine (based on
>>> DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish,
>>> to do that I think It needs to have one more input argument:
>>> My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But
>>> DMDASetGetMatrix() will not accept pointer of this function. Please give a
>>> suggestion to overcome this.
>>> >
>>> > Thank you.
>>> >
>>> > Pham
>>> >
>>> >
>>> > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov>
>>> wrote:
>>> >
>>> >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
>>> in 3d)   You need to copy this routine and add whatever additional
>>> preallocation information you need.   Then call DMDASetGetMatrix()  so that
>>> the DM will use your routine to create the matrix for you.
>>> >
>>> >    Barry
>>> >
>>> >
>>> >
>>> >
>>> > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
>>> wrote:
>>> > >
>>> > > Hi,
>>> > >
>>> > > I am using DMCreateMatrix to create matrix from a existed DM object
>>> and defined stencil.
>>> > > In my code, boundary nodes need to involve many inner nodes, thus
>>> matrix rows corresponding to boundary nodes are almost dense. How can I
>>> tell petsc that those rows need to be preallocated with more entries? I
>>> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
>>> be lost.
>>> > >
>>> > > Many thanks.
>>> > >
>>> > > Sam.
>>> > >
>>> > >
>>> >
>>> >
>>> >
>>> >
>>>
>>>
>>
>
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151028/e5b2d38c/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Oct 27 21:05:28 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 27 Oct 2015 21:05:28 -0500
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQmjj_5Z354MvbNe-Qst5458+yY3Q=Yzdiwnm6c1Kwnfrw@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
	<CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
	<CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>
	<CAOx7dQmjj_5Z354MvbNe-Qst5458+yY3Q=Yzdiwnm6c1Kwnfrw@mail.gmail.com>
Message-ID: <071843F1-F61C-4200-A663-81C06552896E@mcs.anl.gov>


> On Oct 27, 2015, at 8:55 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> 
> Hi Matt,
> 
> Can you please answer my second question about the way PESCs understand the column indies? :
> 
> One more question I have is that, when preallocating for a node, as you can see in my below code (based on original PETSc code), since I have 3 equations, I preallocated 3 rows (in rows[l] array), each row has a number of columns index, all column indices of the three rows are stored in the cols[] array. At this point I don't understand how PETSc know in that cols[] arrays which column indices belonging to the first(second/third) row? Please help me to understand this.

  The preallocation doesn't indicate which columns will have nonzeros, just the NUMBER of columns that will have nonzeros, it is only when values are actually entered into the matrix that the nonzero columns are determined.

  Also note that MatSetValuesLocal() will NOT work for your "dense rows", it only works for values that fit into the natural stencil of the matrix as defined with the DMDA. To put values into the dense rows you will need to call  MatSetValues(), not MatSetValuesLocal().

  You are trying to do something that PETSc DMDA was not designed for so it may take a little work to accomplish what you want.

  Barry

> 
> Thank you.
>  
> Pham
>  
> 
> On Wed, Oct 28, 2015 at 8:48 AM, Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Oct 27, 2015 at 8:18 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> Hi Barry,
> 
> I made my function for preallocating the DM matrix. I am using immersed boundary method to solve a problem of solid mechanics (3D, linear elasticity). 
> At every solid nodes, I have 3 unknowns for displacement in 3 directions (nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each equation I preallocate 27*3  = 81 entries. 
> When running the code, I got many of the following error  when setting values into the preallocated matrix: 
> 
> You might have memory corruption. Run with valgrind.
> 
>   Thanks,
> 
>     Matt
>  
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014 
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed Oct 28 07:34:37 2015
> [0]PETSC ERROR: Libraries linked from /home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
> [0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
> [0]PETSC ERROR: Configure options --download-mpich=1 --with-scalar-type=real --with-clanguage=cxx --download-mumps=1 --download-blacs=1 --download-parmetis=1 --download-scalapack=1 --with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1 -download-cmake=1 --download-f-blas-lapack=1
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in /home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
> [0]PETSC ERROR: MatSetValuesLocal() line 1968 in /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> [0]PETSC ERROR: MatSetValuesStencil() line 1339 in /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> 
> Can you please explain to me what is possible reason for this error? it looks like there is problem with the mapping from LocaltoGobal index, but that part I just copy from the original code.
> One more question I have is that, when preallocating for a node, as you can see in my below code (based on original PETSc code), since I have 3 equations, I preallocated 3 rows (in rows[l] array), each row has a number of columns index, all column indices of the three rows are stored in the cols[] array. At this point I don't understand how PETSc know in that cols[] arrays which column indices belonging to the first(second/third) row? Please help me to understand this.
> 
> Thank you very much.
> Pham 
> 
> Below are my code for preallocating the DM matrix; 
> 
> PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>& isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>& forcing_or_ghost_cells)
> {
>   PetscErrorCode         ierr;
>   PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
>   PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows = NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
>   PetscInt               istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
>   MPI_Comm               comm;
>   PetscScalar            *values;
>   DMDABoundaryType       bx,by,bz;
>   ISLocalToGlobalMapping ltog,ltogb;
>   DMDAStencilType        st;
> 
>   PetscFunctionBegin;
> 
>   ierr = DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr); // nc = 3
>   col  = 2*s + 1;
>   ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
>   ierr = DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
>   ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
>   ierr = PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
>   ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
>   ierr = MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
>   nCells = mesh->nCells;
>   sasVector<int> cell_type(mesh->nCells,0);
>   for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
>   for(int i=0;i<forcing_or_ghost_cells.N;i++)  cell_type[forcing_or_ghost_cells[i]] = 2;
>   for(i = 0;i<nCells;i++)
>   {
>     slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys) + gnx*gny*(mesh->Cells_ijk[i].k - gzs);
>     cnt = 0;
>     if(cell_type[i]==0) /// fluid cells
>     {
>       for (l=0; l<nc; l++)
>       {
>         cols[cnt++] = l + nc*slot;
>         rows[l] = l + nc*slot;
>       }
>     }
> 
>     if(cell_type[i]==1) /// solid cells: 
>     {
>       for (l=0; l<nc; l++)
>       {
>        for (int ll=0; ll<nc; ll++)
>         for (ii=-1; ii<2; ii++) {
>           for (jj=-1; jj<2; jj++) {
>             for (kk=-1; kk<2; kk++) {
>                 cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
>               }
>             }
>           }
>         rows[l] = l + nc*(slot);
>       }
>     }
>     if(cell_type[i]!=2)
>     ierr = MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
>   }
>   MatCreate(comm,&pMat);
>   MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
>   MatSetType(pMat,MATAIJ);
>   ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
>   ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
>   ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
>   ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
>   ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);
> 
>   ierr = PetscFree2(rows,cols);CHKERRQ(ierr);
> 
>   PetscFunctionReturn(0);
> }
> 
> On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Thank you, Dave,
> >
> > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context) when I need to create my matrix, and forget about DMDASetGetMatrix()?
> 
>   You can do that if that is all you need. In some contexts when something else in PETSc needs to create a specific matrix (like with geometric multigrid) then that code calls DMCreateMatrix() which would then use your creation routine.
> 
>    So, in summary you can start by just calling your own routine and convert it to use DMSetGetMatrix() later if you need to.
> 
>   Barry
> 
> >
> > Thank you.
> >
> > Pham
> >
> > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com> wrote:
> > If you need to access a user defined context from within your CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> >
> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> >
> > If your context is not a petsc object, you can use PetscContainerCreate(),
> >
> > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> >
> > You would then set your user context pointer inside the container and then use PetscObjectCompose() to attach the container to the DM
> >
> > Thanks,
> >   Dave
> >
> >
> > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
> > Hi Barry,
> >
> > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da, Mat J), the function pointer in DMDASetGetMatrix() only accept function with that two arguments.
> > As you suggested, I am writing a routine (based on DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish, to do that I think It needs to have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But DMDASetGetMatrix() will not accept pointer of this function. Please give a suggestion to overcome this.
> >
> > Thank you.
> >
> > Pham
> >
> >
> > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in 3d)   You need to copy this routine and add whatever additional preallocation information you need.   Then call DMDASetGetMatrix()  so that the DM will use your routine to create the matrix for you.
> >
> >    Barry
> >
> >
> >
> >
> > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I am using DMCreateMatrix to create matrix from a existed DM object and defined stencil.
> > > In my code, boundary nodes need to involve many inner nodes, thus matrix rows corresponding to boundary nodes are almost dense. How can I  tell petsc that those rows need to be preallocated with more entries? I don't want to use MatMPIAIJSetPreallocation() since advantages of DM might be lost.
> > >
> > > Many thanks.
> > >
> > > Sam.
> > >
> > >
> >
> >
> >
> >
> 
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 


From pvsang002 at gmail.com  Tue Oct 27 21:42:34 2015
From: pvsang002 at gmail.com (Sang pham van)
Date: Wed, 28 Oct 2015 09:42:34 +0700
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <071843F1-F61C-4200-A663-81C06552896E@mcs.anl.gov>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
	<CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
	<CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>
	<CAOx7dQmjj_5Z354MvbNe-Qst5458+yY3Q=Yzdiwnm6c1Kwnfrw@mail.gmail.com>
	<071843F1-F61C-4200-A663-81C06552896E@mcs.anl.gov>
Message-ID: <CAOx7dQn-O=sYyhAQaW5DYviETwzGSb7MibY6_QWbVuBa-w6O7A@mail.gmail.com>

Hi Barry,

In the original MatMPIAIJSetPreallocation(), PETSc determines column index
when preallocating the matrix: cols[cnt++] = l + nc*(slot + ii + gnx*jj +
gnx*gny*kk); If I understand correctly, while PETSc allocates the matrix
the value of cnt is important, entries value of the cols[] array is not.

I am using MatSetValuesStencil() to put values in to the matrix.

I think  my problem is fitting well with the Box Stencil (3 dof) of PETSc,
am I right?

Many thanks,

Pham

On Wed, Oct 28, 2015 at 9:05 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Oct 27, 2015, at 8:55 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Hi Matt,
> >
> > Can you please answer my second question about the way PESCs understand
> the column indies? :
> >
> > One more question I have is that, when preallocating for a node, as you
> can see in my below code (based on original PETSc code), since I have 3
> equations, I preallocated 3 rows (in rows[l] array), each row has a number
> of columns index, all column indices of the three rows are stored in the
> cols[] array. At this point I don't understand how PETSc know in that
> cols[] arrays which column indices belonging to the first(second/third)
> row? Please help me to understand this.
>
>   The preallocation doesn't indicate which columns will have nonzeros,
> just the NUMBER of columns that will have nonzeros, it is only when values
> are actually entered into the matrix that the nonzero columns are
> determined.
>
>   Also note that MatSetValuesLocal() will NOT work for your "dense rows",
> it only works for values that fit into the natural stencil of the matrix as
> defined with the DMDA. To put values into the dense rows you will need to
> call  MatSetValues(), not MatSetValuesLocal().
>
>   You are trying to do something that PETSc DMDA was not designed for so
> it may take a little work to accomplish what you want.
>
>   Barry
>
> >
> > Thank you.
> >
> > Pham
> >
> >
> > On Wed, Oct 28, 2015 at 8:48 AM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > On Tue, Oct 27, 2015 at 8:18 PM, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > Hi Barry,
> >
> > I made my function for preallocating the DM matrix. I am using immersed
> boundary method to solve a problem of solid mechanics (3D, linear
> elasticity).
> > At every solid nodes, I have 3 unknowns for displacement in 3 directions
> (nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each
> equation I preallocate 27*3  = 81 entries.
> > When running the code, I got many of the following error  when setting
> values into the preallocated matrix:
> >
> > You might have memory corruption. Run with valgrind.
> >
> >   Thanks,
> >
> >     Matt
> >
> > [0]PETSC ERROR: --------------------- Error Message
> ------------------------------------
> > [0]PETSC ERROR: Argument out of range!
> > [0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed
> Oct 28 07:34:37 2015
> > [0]PETSC ERROR: Libraries linked from
> /home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
> > [0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
> > [0]PETSC ERROR: Configure options --download-mpich=1
> --with-scalar-type=real --with-clanguage=cxx --download-mumps=1
> --download-blacs=1 --download-parmetis=1 --download-scalapack=1
> --with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1
> -download-cmake=1 --download-f-blas-lapack=1
> > [0]PETSC ERROR:
> ------------------------------------------------------------------------
> > [0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in
> /home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
> > [0]PETSC ERROR: MatSetValuesLocal() line 1968 in
> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatSetValuesStencil() line 1339 in
> /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> >
> > Can you please explain to me what is possible reason for this error? it
> looks like there is problem with the mapping from LocaltoGobal index, but
> that part I just copy from the original code.
> > One more question I have is that, when preallocating for a node, as you
> can see in my below code (based on original PETSc code), since I have 3
> equations, I preallocated 3 rows (in rows[l] array), each row has a number
> of columns index, all column indices of the three rows are stored in the
> cols[] array. At this point I don't understand how PETSc know in that
> cols[] arrays which column indices belonging to the first(second/third)
> row? Please help me to understand this.
> >
> > Thank you very much.
> > Pham
> >
> > Below are my code for preallocating the DM matrix;
> >
> > PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat
> pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>&
> isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>&
> forcing_or_ghost_cells)
> > {
> >   PetscErrorCode         ierr;
> >   PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
> >   PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows =
> NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
> >   PetscInt
>  istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
> >   MPI_Comm               comm;
> >   PetscScalar            *values;
> >   DMDABoundaryType       bx,by,bz;
> >   ISLocalToGlobalMapping ltog,ltogb;
> >   DMDAStencilType        st;
> >
> >   PetscFunctionBegin;
> >
> >   ierr =
> DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr);
> // nc = 3
> >   col  = 2*s + 1;
> >   ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
> >   ierr =
> DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
> >   ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
> >   ierr =
> PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
> >   ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
> >   ierr =
> MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
> >   nCells = mesh->nCells;
> >   sasVector<int> cell_type(mesh->nCells,0);
> >   for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
> >   for(int i=0;i<forcing_or_ghost_cells.N;i++)
> cell_type[forcing_or_ghost_cells[i]] = 2;
> >   for(i = 0;i<nCells;i++)
> >   {
> >     slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys)
> + gnx*gny*(mesh->Cells_ijk[i].k - gzs);
> >     cnt = 0;
> >     if(cell_type[i]==0) /// fluid cells
> >     {
> >       for (l=0; l<nc; l++)
> >       {
> >         cols[cnt++] = l + nc*slot;
> >         rows[l] = l + nc*slot;
> >       }
> >     }
> >
> >     if(cell_type[i]==1) /// solid cells:
> >     {
> >       for (l=0; l<nc; l++)
> >       {
> >        for (int ll=0; ll<nc; ll++)
> >         for (ii=-1; ii<2; ii++) {
> >           for (jj=-1; jj<2; jj++) {
> >             for (kk=-1; kk<2; kk++) {
> >                 cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
> >               }
> >             }
> >           }
> >         rows[l] = l + nc*(slot);
> >       }
> >     }
> >     if(cell_type[i]!=2)
> >     ierr =
> MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
> >   }
> >   MatCreate(comm,&pMat);
> >
>  MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
> >   MatSetType(pMat,MATAIJ);
> >   ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
> >   ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
> >   ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
> >   ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
> >   ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);
> >
> >   ierr = PetscFree2(rows,cols);CHKERRQ(ierr);
> >
> >   PetscFunctionReturn(0);
> > }
> >
> > On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> >
> > > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > >
> > > Thank you, Dave,
> > >
> > > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void*
> my_context) when I need to create my matrix, and forget about
> DMDASetGetMatrix()?
> >
> >   You can do that if that is all you need. In some contexts when
> something else in PETSc needs to create a specific matrix (like with
> geometric multigrid) then that code calls DMCreateMatrix() which would then
> use your creation routine.
> >
> >    So, in summary you can start by just calling your own routine and
> convert it to use DMSetGetMatrix() later if you need to.
> >
> >   Barry
> >
> > >
> > > Thank you.
> > >
> > > Pham
> > >
> > > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com>
> wrote:
> > > If you need to access a user defined context from within your
> CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> > >
> > >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> > >
> > > If your context is not a petsc object, you can use
> PetscContainerCreate(),
> > >
> > >
> http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> > >
> > > You would then set your user context pointer inside the container and
> then use PetscObjectCompose() to attach the container to the DM
> > >
> > > Thanks,
> > >   Dave
> > >
> > >
> > > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > > Hi Barry,
> > >
> > > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da,
> Mat J), the function pointer in DMDASetGetMatrix() only accept function
> with that two arguments.
> > > As you suggested, I am writing a routine (based on
> DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish,
> to do that I think It needs to have one more input argument:
> My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But
> DMDASetGetMatrix() will not accept pointer of this function. Please give a
> suggestion to overcome this.
> > >
> > > Thank you.
> > >
> > > Pham
> > >
> > >
> > > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov>
> wrote:
> > >
> > >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working
> in 3d)   You need to copy this routine and add whatever additional
> preallocation information you need.   Then call DMDASetGetMatrix()  so that
> the DM will use your routine to create the matrix for you.
> > >
> > >    Barry
> > >
> > >
> > >
> > >
> > > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com>
> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I am using DMCreateMatrix to create matrix from a existed DM object
> and defined stencil.
> > > > In my code, boundary nodes need to involve many inner nodes, thus
> matrix rows corresponding to boundary nodes are almost dense. How can I
> tell petsc that those rows need to be preallocated with more entries? I
> don't want to use MatMPIAIJSetPreallocation() since advantages of DM might
> be lost.
> > > >
> > > > Many thanks.
> > > >
> > > > Sam.
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151028/fb954a65/attachment-0001.html>

From bsmith at mcs.anl.gov  Tue Oct 27 21:54:10 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Tue, 27 Oct 2015 21:54:10 -0500
Subject: [petsc-users] DMCreateMatrix with some dense row
In-Reply-To: <CAOx7dQn-O=sYyhAQaW5DYviETwzGSb7MibY6_QWbVuBa-w6O7A@mail.gmail.com>
References: <CAOx7dQmNxofzuzejhLc3-GmShzZ1WmrFML9G3kDxDgXjZ-KHfg@mail.gmail.com>
	<5D467103-4B37-43CF-B9A1-56C54796B902@mcs.anl.gov>
	<CAOx7dQkPGc1n9xMgr19nwj0JWJ-Ntp=LkKhOcb705jpUuk1nFA@mail.gmail.com>
	<CAJ98EDqigVUMivLt=jDYK0nrHmv-fcEza-U8zD=W2i77DcJzog@mail.gmail.com>
	<CAOx7dQk4mBt8=nofrMjNjWLiQ8sD4osNJmCjtmE19xKA9T-6mw@mail.gmail.com>
	<FAB1FBA0-C0F7-4059-9120-98B2A6F1CC67@mcs.anl.gov>
	<CAOx7dQ=4NNrL-BY3LsLrJGHf219oTXQwWLcNdTTm54yVdKru4Q@mail.gmail.com>
	<CAMYG4Gkm5va=HkFNF_Ubi8GC_20nLPak9SN9n8055SV6vc8sRA@mail.gmail.com>
	<CAOx7dQmjj_5Z354MvbNe-Qst5458+yY3Q=Yzdiwnm6c1Kwnfrw@mail.gmail.com>
	<071843F1-F61C-4200-A663-81C06552896E@mcs.anl.gov>
	<CAOx7dQn-O=sYyhAQaW5DYviETwzGSb7MibY6_QWbVuBa-w6O7A@mail.gmail.com>
Message-ID: <81F38156-0817-434E-8694-511CCB141E0D@mcs.anl.gov>


> On Oct 27, 2015, at 9:42 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> 
> Hi Barry, 
> 
> In the original MatMPIAIJSetPreallocation(), PETSc determines column index when preallocating the matrix: cols[cnt++] = l + nc*(slot + ii + gnx*jj + gnx*gny*kk); If I understand correctly, while PETSc allocates the matrix the value of cnt is important, entries value of the cols[] array is not.
> 
> I am using MatSetValuesStencil() to put values in to the matrix.
> 
> I think  my problem is fitting well with the Box Stencil (3 dof) of PETSc, am I right?

  Points within the box stencil you can set with MatSetValuesStencil() but points outside the box stencil, like in the dense rows MUST be set with MatSetValues() using global numbering (which is a pain), you cannot set those values with MatSetValuesStencil().


> 
> Many thanks,
> 
> Pham
> 
> On Wed, Oct 28, 2015 at 9:05 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Oct 27, 2015, at 8:55 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> >
> > Hi Matt,
> >
> > Can you please answer my second question about the way PESCs understand the column indies? :
> >
> > One more question I have is that, when preallocating for a node, as you can see in my below code (based on original PETSc code), since I have 3 equations, I preallocated 3 rows (in rows[l] array), each row has a number of columns index, all column indices of the three rows are stored in the cols[] array. At this point I don't understand how PETSc know in that cols[] arrays which column indices belonging to the first(second/third) row? Please help me to understand this.
> 
>   The preallocation doesn't indicate which columns will have nonzeros, just the NUMBER of columns that will have nonzeros, it is only when values are actually entered into the matrix that the nonzero columns are determined.
> 
>   Also note that MatSetValuesLocal() will NOT work for your "dense rows", it only works for values that fit into the natural stencil of the matrix as defined with the DMDA. To put values into the dense rows you will need to call  MatSetValues(), not MatSetValuesLocal().
> 
>   You are trying to do something that PETSc DMDA was not designed for so it may take a little work to accomplish what you want.
> 
>   Barry
> 
> >
> > Thank you.
> >
> > Pham
> >
> >
> > On Wed, Oct 28, 2015 at 8:48 AM, Matthew Knepley <knepley at gmail.com> wrote:
> > On Tue, Oct 27, 2015 at 8:18 PM, Sang pham van <pvsang002 at gmail.com> wrote:
> > Hi Barry,
> >
> > I made my function for preallocating the DM matrix. I am using immersed boundary method to solve a problem of solid mechanics (3D, linear elasticity).
> > At every solid nodes, I have 3 unknowns for displacement in 3 directions (nc=3), my DM has box stencil. Thus, at a node I have 3 equations, for each equation I preallocate 27*3  = 81 entries.
> > When running the code, I got many of the following error  when setting values into the preallocated matrix:
> >
> > You might have memory corruption. Run with valgrind.
> >
> >   Thanks,
> >
> >     Matt
> >
> > [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> > [0]PETSC ERROR: Argument out of range!
> > [0]PETSC ERROR: Local index 1496271904 too large 121500 (max) at 0!
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: Petsc Release Version 3.4.5, Jun, 29, 2014
> > [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> > [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> > [0]PETSC ERROR: See docs/index.html for manual pages.
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: ./sasMAIN on a arch-linux-cxx-opt named sang by sang Wed Oct 28 07:34:37 2015
> > [0]PETSC ERROR: Libraries linked from /home/sang/petsc/petsc-3.4.5/arch-linux-cxx-opt/lib
> > [0]PETSC ERROR: Configure run at Thu Sep  3 23:04:15 2015
> > [0]PETSC ERROR: Configure options --download-mpich=1 --with-scalar-type=real --with-clanguage=cxx --download-mumps=1 --download-blacs=1 --download-parmetis=1 --download-scalapack=1 --with-debugging=0 --download-hypre=1 --with-fc=gfortran --download-metis=1 -download-cmake=1 --download-f-blas-lapack=1
> > [0]PETSC ERROR: ------------------------------------------------------------------------
> > [0]PETSC ERROR: ISLocalToGlobalMappingApply() line 444 in /home/sang/petsc/petsc-3.4.5/src/vec/is/utils/isltog.c
> > [0]PETSC ERROR: MatSetValuesLocal() line 1968 in /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> > [0]PETSC ERROR: MatSetValuesStencil() line 1339 in /home/sang/petsc/petsc-3.4.5/src/mat/interface/matrix.c
> >
> > Can you please explain to me what is possible reason for this error? it looks like there is problem with the mapping from LocaltoGobal index, but that part I just copy from the original code.
> > One more question I have is that, when preallocating for a node, as you can see in my below code (based on original PETSc code), since I have 3 equations, I preallocated 3 rows (in rows[l] array), each row has a number of columns index, all column indices of the three rows are stored in the cols[] array. At this point I don't understand how PETSc know in that cols[] arrays which column indices belonging to the first(second/third) row? Please help me to understand this.
> >
> > Thank you very much.
> > Pham
> >
> > Below are my code for preallocating the DM matrix;
> >
> > PetscErrorCode sasMatVecPetsc::DMCreateMatrix_DA_3d_MPIAIJ_pvs(DM da,Mat pMat, sasSmesh* mesh,sasVector<int>& solidcells,sasVector<int>& isolidcellnearBI,sasVector<int>& solidcellnearBI,sasVector<int>& forcing_or_ghost_cells)
> > {
> >   PetscErrorCode         ierr;
> >   PetscInt               xs,ys,nx,ny,i,j,slot,gxs,gys,gnx,gny;
> >   PetscInt               m,n,dim,s,*cols = NULL,k,nc,*rows = NULL,col,cnt,l,p,*dnz = NULL,*onz = NULL;
> >   PetscInt               istart,iend,jstart,jend,kstart,kend,zs,nz,gzs,gnz,ii,jj,kk;
> >   MPI_Comm               comm;
> >   PetscScalar            *values;
> >   DMDABoundaryType       bx,by,bz;
> >   ISLocalToGlobalMapping ltog,ltogb;
> >   DMDAStencilType        st;
> >
> >   PetscFunctionBegin;
> >
> >   ierr = DMDAGetInfo(da,&dim,&m,&n,&p,0,0,0,&nc,&s,&bx,&by,&bz,&st);CHKERRQ(ierr); // nc = 3
> >   col  = 2*s + 1;
> >   ierr = DMDAGetCorners(da,&xs,&ys,&zs,&nx,&ny,&nz);CHKERRQ(ierr);
> >   ierr = DMDAGetGhostCorners(da,&gxs,&gys,&gzs,&gnx,&gny,&gnz);CHKERRQ(ierr);
> >   ierr = PetscObjectGetComm((PetscObject)da,&comm);CHKERRQ(ierr);
> >   ierr = PetscMalloc2(nc,PetscInt,&rows,col*col*col*nc*nc,PetscInt,&cols);CHKERRQ(ierr);
> >   ierr = DMGetLocalToGlobalMapping(da,&ltog);CHKERRQ(ierr);
> >   ierr = MatPreallocateInitialize(comm,nc*nx*ny*nz,nc*nx*ny*nz,dnz,onz);CHKERRQ(ierr);
> >   nCells = mesh->nCells;
> >   sasVector<int> cell_type(mesh->nCells,0);
> >   for(int i=0;i<solidcells.N;i++)  cell_type[solidcells[i]] = 1;
> >   for(int i=0;i<forcing_or_ghost_cells.N;i++)  cell_type[forcing_or_ghost_cells[i]] = 2;
> >   for(i = 0;i<nCells;i++)
> >   {
> >     slot = mesh->Cells_ijk[i].i - gxs + gnx*(mesh->Cells_ijk[i].j - gys) + gnx*gny*(mesh->Cells_ijk[i].k - gzs);
> >     cnt = 0;
> >     if(cell_type[i]==0) /// fluid cells
> >     {
> >       for (l=0; l<nc; l++)
> >       {
> >         cols[cnt++] = l + nc*slot;
> >         rows[l] = l + nc*slot;
> >       }
> >     }
> >
> >     if(cell_type[i]==1) /// solid cells:
> >     {
> >       for (l=0; l<nc; l++)
> >       {
> >        for (int ll=0; ll<nc; ll++)
> >         for (ii=-1; ii<2; ii++) {
> >           for (jj=-1; jj<2; jj++) {
> >             for (kk=-1; kk<2; kk++) {
> >                 cols[cnt++] = ll + nc*(slot + ii + gnx*jj + gnx*gny*kk);
> >               }
> >             }
> >           }
> >         rows[l] = l + nc*(slot);
> >       }
> >     }
> >     if(cell_type[i]!=2)
> >     ierr = MatPreallocateSetLocal(ltog,nc,rows,ltog,cnt,cols,dnz,onz);CHKERRQ(ierr);
> >   }
> >   MatCreate(comm,&pMat);
> >   MatSetSizes(pMat,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz,nc*gnx*gny*gnz);
> >   MatSetType(pMat,MATAIJ);
> >   ierr = MatSetBlockSize(pMat,nc);CHKERRQ(ierr);
> >   ierr = MatSeqAIJSetPreallocation(pMat,0,dnz);CHKERRQ(ierr);
> >   ierr = MatMPIAIJSetPreallocation(pMat,0,dnz,0,onz);CHKERRQ(ierr);
> >   ierr = MatPreallocateFinalize(dnz,onz);CHKERRQ(ierr);
> >   ierr = MatSetLocalToGlobalMapping(pMat,ltog,ltog);CHKERRQ(ierr);
> >
> >   ierr = PetscFree2(rows,cols);CHKERRQ(ierr);
> >
> >   PetscFunctionReturn(0);
> > }
> >
> > On Sun, Oct 25, 2015 at 12:46 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> >
> > > On Oct 24, 2015, at 1:43 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> > >
> > > Thank you, Dave,
> > >
> > > Can I just call My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context) when I need to create my matrix, and forget about DMDASetGetMatrix()?
> >
> >   You can do that if that is all you need. In some contexts when something else in PETSc needs to create a specific matrix (like with geometric multigrid) then that code calls DMCreateMatrix() which would then use your creation routine.
> >
> >    So, in summary you can start by just calling your own routine and convert it to use DMSetGetMatrix() later if you need to.
> >
> >   Barry
> >
> > >
> > > Thank you.
> > >
> > > Pham
> > >
> > > On Sat, Oct 24, 2015 at 1:33 PM, Dave May <dave.mayhem23 at gmail.com> wrote:
> > > If you need to access a user defined context from within your CreateMatrix function, you can attach it to the DM via PetscObjectCompose
> > >
> > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscObjectCompose.html
> > >
> > > If your context is not a petsc object, you can use PetscContainerCreate(),
> > >
> > > http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/PetscContainerCreate.html#PetscContainerCreate
> > >
> > > You would then set your user context pointer inside the container and then use PetscObjectCompose() to attach the container to the DM
> > >
> > > Thanks,
> > >   Dave
> > >
> > >
> > > On 24 October 2015 at 06:04, Sang pham van <pvsang002 at gmail.com> wrote:
> > > Hi Barry,
> > >
> > > The routine DMCreateMatrix_DA_3d_MPIAIJ has 2 input arguments (DM da, Mat J), the function pointer in DMDASetGetMatrix() only accept function with that two arguments.
> > > As you suggested, I am writing a routine (based on DMCreateMatrix_DA_3d_MPIAIJ()) to preallocate the matrix in the way I wish, to do that I think It needs to have one more input argument: My_DMCreateMatrix_DA_3d_MPIAIJ(DM da, Mat J, void* my_context). But DMDASetGetMatrix() will not accept pointer of this function. Please give a suggestion to overcome this.
> > >
> > > Thank you.
> > >
> > > Pham
> > >
> > >
> > > On Fri, Sep 4, 2015 at 1:24 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> > >
> > >     Look at DMCreateMatrix_DA_2d_MPIAIJ  (or the 3d version if working in 3d)   You need to copy this routine and add whatever additional preallocation information you need.   Then call DMDASetGetMatrix()  so that the DM will use your routine to create the matrix for you.
> > >
> > >    Barry
> > >
> > >
> > >
> > >
> > > > On Sep 3, 2015, at 11:28 AM, Sang pham van <pvsang002 at gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I am using DMCreateMatrix to create matrix from a existed DM object and defined stencil.
> > > > In my code, boundary nodes need to involve many inner nodes, thus matrix rows corresponding to boundary nodes are almost dense. How can I  tell petsc that those rows need to be preallocated with more entries? I don't want to use MatMPIAIJSetPreallocation() since advantages of DM might be lost.
> > > >
> > > > Many thanks.
> > > >
> > > > Sam.
> > > >
> > > >
> > >
> > >
> > >
> > >
> >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> > -- Norbert Wiener
> >
> 
> 


From timothee.nicolas at gmail.com  Thu Oct 29 02:11:05 2015
From: timothee.nicolas at gmail.com (=?UTF-8?Q?Timoth=C3=A9e_Nicolas?=)
Date: Thu, 29 Oct 2015 16:11:05 +0900
Subject: [petsc-users] Matrix-free Multigrid
Message-ID: <CAGi1ndS-q00Cqm3X59PRnPpFjptkMCw-VsShM0x1FvCOuy+pqQ@mail.gmail.com>

Hello all,

Eventually, I would like to use matrix-free methods in my multigrid solver,
because I have a large problem and matrix memory storage is an issue. Since
all my matrices represent operators, it is quite straightforward to make
them matrix-free, and according to the manual, I can use multigrid on them.
However, depending on the method chosen in KSP, different matrix operations
are used, which the user has to code by himself.

That is, in my understanding, coding the subroutine for MATOP_MULT (which
in principle is sufficient to describe the matrix) is actually not
sufficient, because the KSP may want internally to get, e.g., the diagonal
with MatGetDiagonal, in which case I need a routine to define the operation
MATOP_GET_DIAGONAL (as far as I understand). This means I have to know all
the matrix related routines used internally by the KSP (which will depend
on the ksp method as well as the preconditioner choice), and hard code the
corresponding routines. After all, one could probably do this, but this is
really tedious work and I have the feeling it is not the approach intended
by the developers of PETSc.

Would someone have an advice on how to do this efficiently ? Maybe I am
totally wrong ? But when I tried to do a KSP with multigrid preconditioning
on a matrix-free matrix, I got errors complaining about MatGetDiagonal and
this sort of things.

Best

Timothee
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/a398fc3c/attachment.html>

From dave.mayhem23 at gmail.com  Thu Oct 29 04:22:07 2015
From: dave.mayhem23 at gmail.com (Dave May)
Date: Thu, 29 Oct 2015 10:22:07 +0100
Subject: [petsc-users] Matrix-free Multigrid
In-Reply-To: <CAGi1ndS-q00Cqm3X59PRnPpFjptkMCw-VsShM0x1FvCOuy+pqQ@mail.gmail.com>
References: <CAGi1ndS-q00Cqm3X59PRnPpFjptkMCw-VsShM0x1FvCOuy+pqQ@mail.gmail.com>
Message-ID: <CAJ98EDpYMARDoHYzBnyW-jruj+nSiLkivrGE2NswoWrwJTO9QQ@mail.gmail.com>

Hi Timothee,

Your thinking is correct -- however it is not as bad as you imagine.

If you want to define a matrix free definition of your operator within
geometric MG,
the Mat is required to support several methods in order to be used in
conjunction with smoother.

For Krylov methods, you definitely need to define MatMult.
Assuming you are happy to use a Jacobi preconditioner, you will need to
implement MatGetDiagonal.

I use MF operators within geometric multigrid (on all levels) and find only
these
two operations are required to support using smoothers such as
{richardson,cheby,cg,gmres}+jacobi.

If wish to define your restriction and prolongation operators in a
matrix-free manner
you will need to additionally define the operation MatMultAdd which is
called from
MatInterpolateAdd()  inside PCMG.  If R = P^T, you will also need to define
MatMultTranspose and MatMultTransposeAdd


Cheers
  Dave


On 29 October 2015 at 08:11, Timoth?e Nicolas <timothee.nicolas at gmail.com>
wrote:

> Hello all,
>
> Eventually, I would like to use matrix-free methods in my multigrid
> solver, because I have a large problem and matrix memory storage is an
> issue. Since all my matrices represent operators, it is quite
> straightforward to make them matrix-free, and according to the manual, I
> can use multigrid on them. However, depending on the method chosen in KSP,
> different matrix operations are used, which the user has to code by
> himself.
>
> That is, in my understanding, coding the subroutine for MATOP_MULT (which
> in principle is sufficient to describe the matrix) is actually not
> sufficient, because the KSP may want internally to get, e.g., the diagonal
> with MatGetDiagonal, in which case I need a routine to define the operation
> MATOP_GET_DIAGONAL (as far as I understand). This means I have to know all
> the matrix related routines used internally by the KSP (which will depend
> on the ksp method as well as the preconditioner choice), and hard code the
> corresponding routines. After all, one could probably do this, but this is
> really tedious work and I have the feeling it is not the approach intended
> by the developers of PETSc.
>
> Would someone have an advice on how to do this efficiently ? Maybe I am
> totally wrong ? But when I tried to do a KSP with multigrid preconditioning
> on a matrix-free matrix, I got errors complaining about MatGetDiagonal and
> this sort of things.
>
> Best
>
> Timothee
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/63ec1e78/attachment.html>

From vincent.huber at cemosis.fr  Thu Oct 29 10:10:22 2015
From: vincent.huber at cemosis.fr (Vincent Huber)
Date: Thu, 29 Oct 2015 16:10:22 +0100
Subject: [petsc-users] Hypre - AMS / CG
Message-ID: <CAK+p5JYHwmCsVuvLnA52ZMyqH05t675nEMErQr9E6VokzwXXkA@mail.gmail.com>

Hello all,

I want to solve the following system [1]
? ? 1/? ? ? u + ? P = f
? ? u = 0
with ?? = cst.

This produce, in the preconditioner, two systems to solve. The first one is
related to AMS (with a non null ??), the second one is a laplacian.

I use *Hypre/Ams*
<http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCHYPRE.html>
as a preconditioner to solve the first system.

I have verified my implementation to solve the full problem in sequential,
using the default parameters, on 3D academic test cases. I obtain the
second order accuracy as expected.

On *some* parallel cases, the solver related to the first system (after few
iterations) does not converge (Indefinite matrix or Indefinite
preconditioner). If I generate [2] - from the same mesh - a new
partitionning, I can obtain convergence (but that is not always true)
I have implemented my own version of the ams preconditioner following [1]
and the system (slowly) converge.
I use {Hypre-AMS OR my own implementation}/CG to solve the corresponding
system.
If I switch from hypre-ams/CG to hypre-ams/gmres, then I obtain - at least
for that case - convergence.
If I use my own implementation/CG, then I obtain - at least for that case -
convergence (but that is very slow !)

My questions are:

   - why does the hypre-ams preconditioner loose the SDP property ?
   - Do I miss anything else ?

I have tried various cycle-type and smoothing options in the hypre-ams
preconditioner with CG without success.

Vincent H

[1]For more details, see *Parallel numerical solution of the time-harmonic
Maxwell equations in mixed form*
[2] gmsh my.msh -3 -part n -o new.msh

?
Docteur Ing?nieur de rechercheCeMoSiS <http://www.cemosis.fr> -
vincent.huber at cemosis.fr
Tel: +33 (0)3 68 85 02 06
IRMA - 7, rue Ren? Descartes
67 000 Strasbourg

?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/c41452ca/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 29 10:27:36 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 29 Oct 2015 10:27:36 -0500
Subject: [petsc-users] Hypre - AMS / CG
In-Reply-To: <CAK+p5JYHwmCsVuvLnA52ZMyqH05t675nEMErQr9E6VokzwXXkA@mail.gmail.com>
References: <CAK+p5JYHwmCsVuvLnA52ZMyqH05t675nEMErQr9E6VokzwXXkA@mail.gmail.com>
Message-ID: <5D53B552-0AE0-4A0C-BB85-6E70DF269FC7@mcs.anl.gov>


  I am including the hypre issue tracker on this discussion since they obviously know much more about their solvers than we do.

  Since you have markedly different qualitative behavior between the sequential (where hypre seems fine) and the parallel, my guess is that there is an error in your generation of the problem you pass to hypre in parallel. I recommend running your code with a very small matrix on 1 and 2 processes and using MatView() or -mat_view  and other command line options to make sure that you are providing the EXACT same matrix on 1 and 2 processes to hypre. 

You can also use MatView() with a binary file and MatLoad() to generate the matrix on one process and then load it on to two processes and run the matrix that worked on one process on two processes, does it's convergence fall apart on 2 processes (this would indicate an issue with hypre) or does it have similar convergence on 2?

   Barry

> On Oct 29, 2015, at 10:10 AM, Vincent Huber <vincent.huber at cemosis.fr> wrote:
> 
> Hello all,
> 
> I want to solve the following system [1]
> ? ? 1/? ? ? u + ? P = f
> ? ? u = 0
> with ?? = cst.
> 
> This produce, in the preconditioner, two systems to solve. The first one is related to AMS (with a non null ??), the second one is a laplacian.
> 
> I use Hypre/Ams as a preconditioner to solve the first system.
> 
> I have verified my implementation to solve the full problem in sequential, using the default parameters, on 3D academic test cases. I obtain the second order accuracy as expected.
> 
> On some parallel cases, the solver related to the first system (after few iterations) does not converge (Indefinite matrix or Indefinite preconditioner). If I generate [2] - from the same mesh - a new partitionning, I can obtain convergence (but that is not always true)
> I have implemented my own version of the ams preconditioner following [1] and the system (slowly) converge.
> I use {Hypre-AMS OR my own implementation}/CG to solve the corresponding system.
> If I switch from hypre-ams/CG to hypre-ams/gmres, then I obtain - at least for that case - convergence.
> If I use my own implementation/CG, then I obtain - at least for that case - convergence (but that is very slow !)
> 
> My questions are:
> 
> 	? why does the hypre-ams preconditioner loose the SDP property ?
> 	? Do I miss anything else ?
> I have tried various cycle-type and smoothing options in the hypre-ams preconditioner with CG without success.
> 
> Vincent H
> 
> [1]For more details, see Parallel numerical solution of the time-harmonic Maxwell equations in mixed form
> [2] gmsh my.msh -3 -part n -o new.msh
> 
> ?
> Docteur Ing?nieur de rechercheCeMoSiS - vincent.huber at cemosis.fr
> Tel: +33 (0)3 68 85 02 06
> IRMA - 7, rue Ren? Descartes
> 67 000 Strasbourg
> 
> 
> 


From davydden at gmail.com  Thu Oct 29 12:32:00 2015
From: davydden at gmail.com (Denis Davydov)
Date: Thu, 29 Oct 2015 18:32:00 +0100
Subject: [petsc-users] create PC, KSP, ST separately and use in EPS
Message-ID: <55EECDD0-25B0-4528-92F2-8E621CF67D69@gmail.com>

Dear all, 

I am currently looking at ways to extend deal.II PETSc/SLEPc wrappers classes to allow 
specification of linear solvers and preconditioners to be used in SLEPc eigensolvers 
using deal.II objects and NOT the command line arguments.

All examples and unit tests I came across in SLEPc work with EPSCreate() followed by 
EPSGetST(), STGetKSP() and KSPGetPC(). This however does not fit well into object-oriented C++ code.
Thus I plan to create each objects separately and pass them around.

I would like to ask if the way I see it (in terms of PETSc/SLEPc functions called) is conceptually correct:

? PETSc: Preconditioner class:
PCCreate()
PCSetType()
PCSetFromOptions();
?

? PETSc: Solver class (uses Preconditioner class):
KSPCreate()
KSPSetType()
KSPSetPC() // set PC created above
KSPSetConvergenceTest()
KSPSetFromOptions()
?

? SLEPc: SpectralTransformation class (uses Solver class)
STCreate()
STSetType()
STSetShift() // some specific settings to transformation
STSetKSP() // set KSP created above
?

? SLEPc: Eigensolver class (uses SpectralTransformation class)
EPSCreate()
EPSSetTolerances()
EPSSetConvergenceTest()
EPSSetWhichEigenpairs()
EPSSetProblemType()
EPSSetType()
EPSArnoldiSetDelayed() // any specific options to eigensolver
EPSSetST() // set spectral transformation created above
// solve
EPSSetDimensions()
EPSSetFromOptions()
EPSSolve()
?

Do this usage model sounds correct from SLEPc perspective?
What bothers me is that i do not call PCSetOperators()/PCSetUp() and KSPSetOperators()/KSPSetUp(), 
which one would call while setting up PETSc objects to solve SLAE.
Nonetheless I assume those functions are executed somewhere inside SLEPc depending on
SHEP/GHEP and ST to be used.

Kind regards,
Denis.

From jroman at dsic.upv.es  Thu Oct 29 12:39:47 2015
From: jroman at dsic.upv.es (Jose E. Roman)
Date: Thu, 29 Oct 2015 13:39:47 -0400
Subject: [petsc-users] create PC, KSP, ST separately and use in EPS
In-Reply-To: <55EECDD0-25B0-4528-92F2-8E621CF67D69@gmail.com>
References: <55EECDD0-25B0-4528-92F2-8E621CF67D69@gmail.com>
Message-ID: <8C2944F1-F138-40CC-9685-C53DC8413525@dsic.upv.es>


> El 29/10/2015, a las 13:32, Denis Davydov <davydden at gmail.com> escribi?:
> 
> Dear all, 
> 
> I am currently looking at ways to extend deal.II PETSc/SLEPc wrappers classes to allow 
> specification of linear solvers and preconditioners to be used in SLEPc eigensolvers 
> using deal.II objects and NOT the command line arguments.

Yes, current deal.II wrappers for SLEPc are quite limited.

> 
> All examples and unit tests I came across in SLEPc work with EPSCreate() followed by 
> EPSGetST(), STGetKSP() and KSPGetPC(). This however does not fit well into object-oriented C++ code.
> Thus I plan to create each objects separately and pass them around.
> 
> I would like to ask if the way I see it (in terms of PETSc/SLEPc functions called) is conceptually correct:
> 
> ? PETSc: Preconditioner class:
> PCCreate()
> PCSetType()
> PCSetFromOptions();
> ?
> 
> ? PETSc: Solver class (uses Preconditioner class):
> KSPCreate()
> KSPSetType()
> KSPSetPC() // set PC created above
> KSPSetConvergenceTest()
> KSPSetFromOptions()
> ?
> 
> ? SLEPc: SpectralTransformation class (uses Solver class)
> STCreate()
> STSetType()
> STSetShift() // some specific settings to transformation
> STSetKSP() // set KSP created above
> ?
> 
> ? SLEPc: Eigensolver class (uses SpectralTransformation class)
> EPSCreate()
> EPSSetTolerances()
> EPSSetConvergenceTest()
> EPSSetWhichEigenpairs()
> EPSSetProblemType()
> EPSSetType()
> EPSArnoldiSetDelayed() // any specific options to eigensolver
> EPSSetST() // set spectral transformation created above
> // solve
> EPSSetDimensions()
> EPSSetFromOptions()
> EPSSolve()
> ?
> 
> Do this usage model sounds correct from SLEPc perspective?

I see no problem with this approach.

> What bothers me is that i do not call PCSetOperators()/PCSetUp() and KSPSetOperators()/KSPSetUp(), 
> which one would call while setting up PETSc objects to solve SLAE.
> Nonetheless I assume those functions are executed somewhere inside SLEPc depending on
> SHEP/GHEP and ST to be used.

Correct, EPS/ST handle the KSP coefficient matrix so KSPSetUp() must be done at EPSSetUp() and not before.

Jose

> 
> Kind regards,
> Denis.


From davydden at gmail.com  Thu Oct 29 12:45:52 2015
From: davydden at gmail.com (Denis Davydov)
Date: Thu, 29 Oct 2015 18:45:52 +0100
Subject: [petsc-users] create PC, KSP, ST separately and use in EPS
In-Reply-To: <8C2944F1-F138-40CC-9685-C53DC8413525@dsic.upv.es>
References: <55EECDD0-25B0-4528-92F2-8E621CF67D69@gmail.com>
	<8C2944F1-F138-40CC-9685-C53DC8413525@dsic.upv.es>
Message-ID: <BAF12742-511D-4CA0-BAA1-958C2BC56FF6@gmail.com>

Thanks Jose for the prompt  (as always) reply. 
I will post further questions if I encounter any issues with the outlined approach.

Kind regards,
Denis 

> On 29 Oct 2015, at 18:39, Jose E. Roman <jroman at dsic.upv.es> wrote:
> 
>> 
>> El 29/10/2015, a las 13:32, Denis Davydov <davydden at gmail.com <mailto:davydden at gmail.com>> escribi?:
>> 
>> Dear all, 
>> 
>> I am currently looking at ways to extend deal.II PETSc/SLEPc wrappers classes to allow 
>> specification of linear solvers and preconditioners to be used in SLEPc eigensolvers 
>> using deal.II objects and NOT the command line arguments.
> 
> Yes, current deal.II wrappers for SLEPc are quite limited.
> 
>> 
>> All examples and unit tests I came across in SLEPc work with EPSCreate() followed by 
>> EPSGetST(), STGetKSP() and KSPGetPC(). This however does not fit well into object-oriented C++ code.
>> Thus I plan to create each objects separately and pass them around.
>> 
>> I would like to ask if the way I see it (in terms of PETSc/SLEPc functions called) is conceptually correct:
>> 
>> ? PETSc: Preconditioner class:
>> PCCreate()
>> PCSetType()
>> PCSetFromOptions();
>> ?
>> 
>> ? PETSc: Solver class (uses Preconditioner class):
>> KSPCreate()
>> KSPSetType()
>> KSPSetPC() // set PC created above
>> KSPSetConvergenceTest()
>> KSPSetFromOptions()
>> ?
>> 
>> ? SLEPc: SpectralTransformation class (uses Solver class)
>> STCreate()
>> STSetType()
>> STSetShift() // some specific settings to transformation
>> STSetKSP() // set KSP created above
>> ?
>> 
>> ? SLEPc: Eigensolver class (uses SpectralTransformation class)
>> EPSCreate()
>> EPSSetTolerances()
>> EPSSetConvergenceTest()
>> EPSSetWhichEigenpairs()
>> EPSSetProblemType()
>> EPSSetType()
>> EPSArnoldiSetDelayed() // any specific options to eigensolver
>> EPSSetST() // set spectral transformation created above
>> // solve
>> EPSSetDimensions()
>> EPSSetFromOptions()
>> EPSSolve()
>> ?
>> 
>> Do this usage model sounds correct from SLEPc perspective?
> 
> I see no problem with this approach.
> 
>> What bothers me is that i do not call PCSetOperators()/PCSetUp() and KSPSetOperators()/KSPSetUp(), 
>> which one would call while setting up PETSc objects to solve SLAE.
>> Nonetheless I assume those functions are executed somewhere inside SLEPc depending on
>> SHEP/GHEP and ST to be used.
> 
> Correct, EPS/ST handle the KSP coefficient matrix so KSPSetUp() must be done at EPSSetUp() and not before.
> 
> Jose
> 
>> 
>> Kind regards,
>> Denis.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/1ec5b7cd/attachment.html>

From mpovolot at purdue.edu  Thu Oct 29 15:43:12 2015
From: mpovolot at purdue.edu (Michael Povolotskyi)
Date: Thu, 29 Oct 2015 16:43:12 -0400
Subject: [petsc-users] question about setting elements to a matrix
Message-ID: <563284E0.4040602@purdue.edu>

Dear PETSc developers,
I need to set many elements into a sparse Hermitian matrix.
I'd like to do this effectively.
Your documentation says that one should use MatSetValues and set as many 
elements at one call as possible. MatSetValues expects a 2D array of values.
However, my matrix elements are computed row by row. Since the matrix is 
Hermitian I compute only a half of them and apply complex conjugation 
for the rest.
Will the following procedure be effective:
1. Compute a part of the row that correspond to the lower part of the matrix
2. Set this part of the row  to the matrix with MatSetValues
3. Apply complex conjugation and compute a part of the column that 
corresponds to upper part of the matrix
4. Set this part of the column to the matrix with MatSetValues
5. move to the next row

Thank you,
Michael.

-- 
Michael Povolotskyi, PhD
Research Assistant Professor
Network for Computational Nanotechnology
Hall for Discover and Learning Research, Room 441
West Lafayette, IN 47907
Phone (765) 4949396


From bsmith at mcs.anl.gov  Thu Oct 29 15:50:03 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 29 Oct 2015 15:50:03 -0500
Subject: [petsc-users] question about setting elements to a matrix
In-Reply-To: <563284E0.4040602@purdue.edu>
References: <563284E0.4040602@purdue.edu>
Message-ID: <D736CC60-EE6C-4224-B73A-5E696EA99052@mcs.anl.gov>


> On Oct 29, 2015, at 3:43 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
> 
> Dear PETSc developers,
> I need to set many elements into a sparse Hermitian matrix.
> I'd like to do this effectively.
> Your documentation says that one should use MatSetValues and set as many elements at one call as possible. MatSetValues expects a 2D array of values.
> However, my matrix elements are computed row by row. Since the matrix is Hermitian I compute only a half of them and apply complex conjugation for the rest.
> Will the following procedure be effective:
> 1. Compute a part of the row that correspond to the lower part of the matrix
> 2. Set this part of the row  to the matrix with MatSetValues
> 3. Apply complex conjugation and compute a part of the column that corresponds to upper part of the matrix
> 4. Set this part of the column to the matrix with MatSetValues
> 5. move to the next row

  This is ok. But you should also be able to us the SBAIJ matrix which only stores the upper half and then you would create the diagonal and right portion of the matrix row and set that a row at a time.

  Barry

> 
> Thank you,
> Michael.
> 
> -- 
> Michael Povolotskyi, PhD
> Research Assistant Professor
> Network for Computational Nanotechnology
> Hall for Discover and Learning Research, Room 441
> West Lafayette, IN 47907
> Phone (765) 4949396
> 


From mpovolot at purdue.edu  Thu Oct 29 15:53:47 2015
From: mpovolot at purdue.edu (Michael Povolotskyi)
Date: Thu, 29 Oct 2015 16:53:47 -0400
Subject: [petsc-users] question about setting elements to a matrix
In-Reply-To: <D736CC60-EE6C-4224-B73A-5E696EA99052@mcs.anl.gov>
References: <563284E0.4040602@purdue.edu>
	<D736CC60-EE6C-4224-B73A-5E696EA99052@mcs.anl.gov>
Message-ID: <5632875B.8020006@purdue.edu>



On 10/29/2015 04:50 PM, Barry Smith wrote:
>> On Oct 29, 2015, at 3:43 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
>>
>> Dear PETSc developers,
>> I need to set many elements into a sparse Hermitian matrix.
>> I'd like to do this effectively.
>> Your documentation says that one should use MatSetValues and set as many elements at one call as possible. MatSetValues expects a 2D array of values.
>> However, my matrix elements are computed row by row. Since the matrix is Hermitian I compute only a half of them and apply complex conjugation for the rest.
>> Will the following procedure be effective:
>> 1. Compute a part of the row that correspond to the lower part of the matrix
>> 2. Set this part of the row  to the matrix with MatSetValues
>> 3. Apply complex conjugation and compute a part of the column that corresponds to upper part of the matrix
>> 4. Set this part of the column to the matrix with MatSetValues
>> 5. move to the next row
>    This is ok. But you should also be able to us the SBAIJ matrix which only stores the upper half and then you would create the diagonal and right portion of the matrix row and set that a row at a time.
>
>    Barry
>
Thank you.
My matrix is not blocked. Actually there are blocks, but of different size.
Do you I understand correctly that the only format for this is the 
general sparse format?
Michael.

From bsmith at mcs.anl.gov  Thu Oct 29 15:58:24 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 29 Oct 2015 15:58:24 -0500
Subject: [petsc-users] question about setting elements to a matrix
In-Reply-To: <5632875B.8020006@purdue.edu>
References: <563284E0.4040602@purdue.edu>
	<D736CC60-EE6C-4224-B73A-5E696EA99052@mcs.anl.gov>
	<5632875B.8020006@purdue.edu>
Message-ID: <E8562E1C-B4BC-448F-8B2A-15928EF29501@mcs.anl.gov>


> On Oct 29, 2015, at 3:53 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
> 
> 
> 
> On 10/29/2015 04:50 PM, Barry Smith wrote:
>>> On Oct 29, 2015, at 3:43 PM, Michael Povolotskyi <mpovolot at purdue.edu> wrote:
>>> 
>>> Dear PETSc developers,
>>> I need to set many elements into a sparse Hermitian matrix.
>>> I'd like to do this effectively.
>>> Your documentation says that one should use MatSetValues and set as many elements at one call as possible. MatSetValues expects a 2D array of values.
>>> However, my matrix elements are computed row by row. Since the matrix is Hermitian I compute only a half of them and apply complex conjugation for the rest.
>>> Will the following procedure be effective:
>>> 1. Compute a part of the row that correspond to the lower part of the matrix
>>> 2. Set this part of the row  to the matrix with MatSetValues
>>> 3. Apply complex conjugation and compute a part of the column that corresponds to upper part of the matrix
>>> 4. Set this part of the column to the matrix with MatSetValues
>>> 5. move to the next row
>>   This is ok. But you should also be able to us the SBAIJ matrix which only stores the upper half and then you would create the diagonal and right portion of the matrix row and set that a row at a time.
>> 
>>   Barry
>> 
> Thank you.
> My matrix is not blocked. Actually there are blocks, but of different size.
> Do you I understand correctly that the only format for this is the general sparse format?

  Sorry for the confusion but you should us SBAIJ with a block size of 1. We just don't have a SAIJ format since that would be completely duplicative code with SBAIJ with a block size of 1.

  Barry

> Michael.


From bsmith at mcs.anl.gov  Thu Oct 29 16:01:59 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 29 Oct 2015 16:01:59 -0500
Subject: [petsc-users] soname seems to be absent in OS-X
In-Reply-To: <6C83934B-C769-4C05-9235-96EE7C22944D@gmail.com>
References: <6C83934B-C769-4C05-9235-96EE7C22944D@gmail.com>
Message-ID: <5D62CF7C-2798-422E-856F-663904012549@mcs.anl.gov>


  Denis,

   We don't understand what purpose a soname serves on Apple or how to add it. If you need it let us know how to install PETSc so that it is set and we will do it.

   Barry

> On Oct 27, 2015, at 11:28 AM, Denis Davydov <davydden at gmail.com> wrote:
> 
> Dear developers,
> 
> It seems that the compiled PETSc library does not have a soname (-Wl,-install_name=xyz.dylib in OS-X).
> I compile PETSc/SLEPc using Homebrew both on OS-X and Linux and judging from ldd/otool there is indeed a difference:
> 
> Linux (soname is there):
> $ ldd .linuxbrew_openblas/opt/slepc/lib/libslepc.so | grep "petsc"
>     libpetsc.so.3.6 => /home/woody/iwtm/iwtm108/.linuxbrew_openblas/Cellar/petsc/3.6.2/real/lib/libpetsc.so.3.6 (0x00007fac7822f000)
> 
> OS-X (no soname):
> 
> $ otool -L /usr/local/opt/slepc/lib/libslepc.dylib | grep "petsc"
>     /usr/local/opt/petsc/real/lib/libpetsc.3.6.2.dylib (compatibility version 3.6.0, current version 3.6.2)
> 
> 
> I do not see `-Wl,-soname=xyz` in linking flags and nothing like `-Wl,-install_name=xyz` is there on OS-X either.
> Any mac users can comment on this?
> 
> p.s. as Macports web is down i can?t check what folks are doing there.
> 
> Kind regards,
> Denis 
> 


From ecoon at lanl.gov  Thu Oct 29 17:18:39 2015
From: ecoon at lanl.gov (Coon, Ethan)
Date: Thu, 29 Oct 2015 22:18:39 +0000
Subject: [petsc-users] DMDA and local-to-global scatters
Message-ID: <C5AC614A-7206-43BC-8E04-69F8B841407E@lanl.gov>

Hi Barry, all,

I?m trying to understand some extremely old code (4 years now!) that people insist on using and therefore having me support, despite the fact that the DOE doesn?t think it is important enough to pay me to support.

Barry ? can you explain to me the history and logic in your change:

https://bitbucket.org/petsc/petsc/commits/bd1fc5ae41626b6cf1674a6070035cfd93e0c1dd

that removed DMDA?s local-to-global map, in favor of doing the reverse scatter on global-to-local?  When I wrote the aforementioned LBM code, the local-to-global scatter had different semantics than the reverse global-to-local scatter.  The latter merged owned and ghosted values from the local Vec into the owned global Vec, while the former ignored ghost values and did a direct copy from owned local values to owned global values.  Under INSERT_VALUES, this was important as the latter was a race condition while the former was a well-posed operation.

Now I see that this L2G scatter has been removed (in 2014), which introduced a race condition.  It was tweaked a bit for 1 process by this commit:

https://bitbucket.org/petsc/petsc/commits/1eb28f2e8c580cb49316c983b5b6ec6c58d77ab8

which refers to a Lisandro email that I?m having trouble finding.  Fortunately this caused some errors that I did see, as opposed to the previous race conditions which I didn?t see.

Additionally there was documentation added to not do INSERT_VALUES with DMDAs at some point.  (Maybe because it causes race conditions!)  This documentation suggests to "simply compute the values directly into a global vector instead of a local one."  This isn?t a good choice in my application, where I do many time steps in local vectors, using repeated calls to ?DMLocalToLocalBegin()?, and only go back to the global vector when I want to do i/o.  Computing into global vectors requires managing two Vecs throughout the entire code, when otherwise I only manage one (except in the i/o portion of the code).  I guess the answer is to create and use a L2G forward scatter myself?  Is there a better solution I?m not thinking of?

Thanks,

Ethan



------------------------------------------------------------------------
Ethan Coon
Research Scientist
Computational Earth Science -- EES-16
Los Alamos National Laboratory
505-665-8289

http://www.lanl.gov/expertise/profiles/view/ethan-coon
------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/7d11b1f2/attachment.html>

From bsmith at mcs.anl.gov  Thu Oct 29 17:37:30 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Thu, 29 Oct 2015 17:37:30 -0500
Subject: [petsc-users] DMDA and local-to-global scatters
In-Reply-To: <C5AC614A-7206-43BC-8E04-69F8B841407E@lanl.gov>
References: <C5AC614A-7206-43BC-8E04-69F8B841407E@lanl.gov>
Message-ID: <B926E56B-558F-4C22-B578-AB419BB31143@mcs.anl.gov>


  Ethan,

   The truth is I had to introduce the restriction because the previous code was a memory hog and difficult to maintain. I couldn't figure any way to eliminate the memory hog business except by eliminating the localtolocal feature.

   My recommendation is to not try to rewrite the local to local map which is horrible but instead try to use a global vector for the computed result. Hopefully you can do this in a way to that doesn't mess up the entire code.  In the simplest case one would have something like 

   for t=0 to many time steps
      DMGlobalToLocal (xglobal, xlocal)
      DMDAVecGetArray on xlocal and global
       update xglobal arrays based on xlocal arrays
      restore arrays
  end

  If the timesteping is controlled elsewhere so your routine can only do a single time step at a time then something like

    function whose input is a local (ghosted) vector
    DMDAGetGlobalVector( ... &xglobal);
    DMDAVecGetArray on xlocal and global
       update xglobal arrays based on xlocal arrays
   restore arrays
    DMDAGlobalToLocal(xglobal, xlocal);
    DMDARestoreGlobalVector(xglobal);

   thus you can "hide" the global vector in the routine that does only the update.

   If this doesn't help then send more specifics of exactly where your localtolocal() map is used and I may have suggestions. I convinced myself that one could 
always use a global intermediate vector in the computation without additional communication to replace the localtolocal but I could be wrong.

   Sorry about the change


  Barry


> On Oct 29, 2015, at 5:18 PM, Coon, Ethan <ecoon at lanl.gov> wrote:
> 
> Hi Barry, all,
> 
> I?m trying to understand some extremely old code (4 years now!) that people insist on using and therefore having me support, despite the fact that the DOE doesn?t think it is important enough to pay me to support.
> 
> Barry ? can you explain to me the history and logic in your change:
> 
> https://bitbucket.org/petsc/petsc/commits/bd1fc5ae41626b6cf1674a6070035cfd93e0c1dd
> 
> that removed DMDA?s local-to-global map, in favor of doing the reverse scatter on global-to-local?  When I wrote the aforementioned LBM code, the local-to-global scatter had different semantics than the reverse global-to-local scatter.  The latter merged owned and ghosted values from the local Vec into the owned global Vec, while the former ignored ghost values and did a direct copy from owned local values to owned global values.  Under INSERT_VALUES, this was important as the latter was a race condition while the former was a well-posed operation.
> 
> Now I see that this L2G scatter has been removed (in 2014), which introduced a race condition.  It was tweaked a bit for 1 process by this commit:
> 
> https://bitbucket.org/petsc/petsc/commits/1eb28f2e8c580cb49316c983b5b6ec6c58d77ab8
> 
> which refers to a Lisandro email that I?m having trouble finding.  Fortunately this caused some errors that I did see, as opposed to the previous race conditions which I didn?t see.
> 
> Additionally there was documentation added to not do INSERT_VALUES with DMDAs at some point.  (Maybe because it causes race conditions!)  This documentation suggests to "simply compute the values directly into a global vector instead of a local one."  This isn?t a good choice in my application, where I do many time steps in local vectors, using repeated calls to ?DMLocalToLocalBegin()?, and only go back to the global vector when I want to do i/o.  Computing into global vectors requires managing two Vecs throughout the entire code, when otherwise I only manage one (except in the i/o portion of the code).  I guess the answer is to create and use a L2G forward scatter myself?  Is there a better solution I?m not thinking of?
> 
> Thanks,
> 
> Ethan
> 
> 
> 
> ------------------------------------------------------------------------
> Ethan Coon
> Research Scientist
> Computational Earth Science -- EES-16
> Los Alamos National Laboratory
> 505-665-8289
> 
> http://www.lanl.gov/expertise/profiles/view/ethan-coon
> ------------------------------------------------------------------------
> 


From ling.zou at inl.gov  Fri Oct 30 12:23:02 2015
From: ling.zou at inl.gov (Zou (Non-US), Ling)
Date: Fri, 30 Oct 2015 11:23:02 -0600
Subject: [petsc-users] How do I know it is steady state?
Message-ID: <CAJWxZDTC5Vh5dt_0WcFwO_VhiwJSjT3kH8nkA7ru=rT4vm6AVg@mail.gmail.com>

Hi All,

>From physics point of view, I know my simulation converges if nothing
changes any more.

I wonder how normally you do to detect if your simulation reaches steady
state from numerical point of view.
Is it a good practice to use SNES convergence as a criterion, i.e.,
SNES converges and it takes 0 iteration(s)

Thanks,

Ling
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151030/e02aab1e/attachment.html>

From zonexo at gmail.com  Sat Oct 31 11:34:44 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 1 Nov 2015 00:34:44 +0800
Subject: [petsc-users] Scaling with number of cores
Message-ID: <5634EDA4.5030203@gmail.com>

Hi,

I understand that as mentioned in the faq, due to the limitations in 
memory, the scaling is not linear. So, I am trying to write a proposal 
to use a supercomputer.

Its specs are:

Compute nodes: 82,944 nodes (SPARC64 VIIIfx; 16GB of memory per node)

8 cores / processor

Interconnect: Tofu (6-dimensional mesh/torus) Interconnect

Each cabinet contains 96 computing nodes,

One of the requirement is to give the performance of my current code 
with my current set of data, and there is a formula to calculate the 
estimated parallel efficiency when using the new large set of data

There are 2 ways to give performance:
1. Strong scaling, which is defined as how the elapsed time varies with 
the number of processors for a fixed
problem.
2. Weak scaling, which is defined as how the elapsed time varies with 
the number of processors for a
fixed problem size per processor.

I ran my cases with 48 and 96 cores with my current cluster, giving 140 
and 90 mins respectively. This is classified as strong scaling.

Cluster specs:

CPU: AMD 6234 2.4GHz

8 cores / processor (CPU)

6 CPU / node

So 48 Cores / CPU

Not sure abt the memory / node


The parallel efficiency ?En? for a given degree of parallelism ?n? 
indicates how much the program is
efficiently accelerated by parallel processing. ?En? is given by the 
following formulae. Although their
derivation processes are different depending on strong and weak scaling, 
derived formulae are the
same.

 From the estimated time, my parallel efficiency using Amdahl's law on 
the current old cluster was 52.7%.

So is my results acceptable?

For the large data set, if using 2205 nodes (2205X8cores), my expected 
parallel efficiency is only 0.5%. The proposal recommends value of > 50%.

Is it possible for this type of scaling in PETSc (>50%), when using 
17640 (2205X8) cores?

Btw, I do not have access to the system.




Sent using CloudMagic Email 
<https://cloudmagic.com/k/d/mailapp?ct=pa&cv=7.4.10&pv=5.0.2&source=email_footer_2> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151101/b4068fc4/attachment.html>

From zonexo at gmail.com  Sat Oct 31 11:41:19 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 1 Nov 2015 00:41:19 +0800
Subject: [petsc-users] summary of the bandwidth received with different
 number of MPI processes
Message-ID: <5634EF2F.3090208@gmail.com>

Hi,

It's mentioned that for a batch sys, I have to:

1. cd src/benchmarks/steams
2. make MPIVersion
3. submit MPIVersion to the batch system a number of times with 1, 2, 3, 
etc MPI processes collecting all of the output from the runs into the 
single file scaling.log.
4. copy scaling.log into the src/benchmarks/steams directory
5. ./process.py createfile ; process.py

So for 3, how do I collect all of the output from the runs into the 
single file scaling.log.

Should scaling.log look for this:

Number of MPI processes 3 Processor names  n12-06 n12-06 n12-06
Triad:        27031.0419   Rate (MB/s)
Number of MPI processes 6 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06
Triad:        53517.8980   Rate (MB/s)

...



-- 
Thank you.

Yours sincerely,

TAY wee-beng


From knepley at gmail.com  Sat Oct 31 11:47:51 2015
From: knepley at gmail.com (Matthew Knepley)
Date: Sat, 31 Oct 2015 11:47:51 -0500
Subject: [petsc-users] Scaling with number of cores
In-Reply-To: <5634EDA4.5030203@gmail.com>
References: <5634EDA4.5030203@gmail.com>
Message-ID: <CAMYG4Gm4ew=gxEGitop6=_ufPZs7=cuid8rnwW3xpKbEt1rBbw@mail.gmail.com>

On Sat, Oct 31, 2015 at 11:34 AM, TAY wee-beng <zonexo at gmail.com> wrote:

> Hi,
>
> I understand that as mentioned in the faq, due to the limitations in
> memory, the scaling is not linear. So, I am trying to write a proposal to
> use a supercomputer.
>
> Its specs are:
>
> Compute nodes: 82,944 nodes (SPARC64 VIIIfx; 16GB of memory per node)
>
> 8 cores / processor
>
> Interconnect: Tofu (6-dimensional mesh/torus) Interconnect
>
> Each cabinet contains 96 computing nodes,
>
> One of the requirement is to give the performance of my current code with
> my current set of data, and there is a formula to calculate the estimated
> parallel efficiency when using the new large set of data
>
> There are 2 ways to give performance:
> 1. Strong scaling, which is defined as how the elapsed time varies with
> the number of processors for a fixed
> problem.
> 2. Weak scaling, which is defined as how the elapsed time varies with the
> number of processors for a
> fixed problem size per processor.
>
> I ran my cases with 48 and 96 cores with my current cluster, giving 140
> and 90 mins respectively. This is classified as strong scaling.
>
> Cluster specs:
>
> CPU: AMD 6234 2.4GHz
>
> 8 cores / processor (CPU)
>
> 6 CPU / node
>
> So 48 Cores / CPU
>
> Not sure abt the memory / node
>
>
> The parallel efficiency ?En? for a given degree of parallelism ?n?
> indicates how much the program is
> efficiently accelerated by parallel processing. ?En? is given by the
> following formulae. Although their
> derivation processes are different depending on strong and weak scaling,
> derived formulae are the
> same.
>
> From the estimated time, my parallel efficiency using  Amdahl's law on the
> current old cluster was 52.7%.
>
> So is my results acceptable?
>
> For the large data set, if using 2205 nodes (2205X8cores), my expected
> parallel efficiency is only 0.5%. The proposal recommends value of > 50%.
>
> The problem with this analysis is that the estimated serial fraction from
Amdahl's Law  changes as a function
of problem size, so you cannot take the strong scaling from one problem and
apply it to another without a
model of this dependence.

Weak scaling does model changes with problem size, so I would measure weak
scaling on your current
cluster, and extrapolate to the big machine. I realize that this does not
make sense for many scientific
applications, but neither does requiring a certain parallel efficiency.

  Thanks,

     Matt

> Is it possible for this type of scaling in PETSc (>50%), when using 17640
> (2205X8) cores?
>
> Btw, I do not have access to the system.
>
>
>
>
> Sent using CloudMagic Email
> <https://cloudmagic.com/k/d/mailapp?ct=pa&cv=7.4.10&pv=5.0.2&source=email_footer_2>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151031/e1778def/attachment.html>

From bsmith at mcs.anl.gov  Sat Oct 31 12:17:37 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 31 Oct 2015 12:17:37 -0500
Subject: [petsc-users] summary of the bandwidth received with different
	number of MPI processes
In-Reply-To: <5634EF2F.3090208@gmail.com>
References: <5634EF2F.3090208@gmail.com>
Message-ID: <C9EF975C-0766-46F6-84A8-4C809F393FA0@mcs.anl.gov>


  Yes, just put the output from running with 1 2 etc processes in order into the file

> On Oct 31, 2015, at 11:41 AM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> Hi,
> 
> It's mentioned that for a batch sys, I have to:
> 
> 1. cd src/benchmarks/steams
> 2. make MPIVersion
> 3. submit MPIVersion to the batch system a number of times with 1, 2, 3, etc MPI processes collecting all of the output from the runs into the single file scaling.log.
> 4. copy scaling.log into the src/benchmarks/steams directory
> 5. ./process.py createfile ; process.py
> 
> So for 3, how do I collect all of the output from the runs into the single file scaling.log.
> 
> Should scaling.log look for this:
> 
> Number of MPI processes 3 Processor names  n12-06 n12-06 n12-06
> Triad:        27031.0419   Rate (MB/s)
> Number of MPI processes 6 Processor names  n12-06 n12-06 n12-06 n12-06 n12-06 n12-06
> Triad:        53517.8980   Rate (MB/s)
> 
> ...
> 
> 
> 
> -- 
> Thank you.
> 
> Yours sincerely,
> 
> TAY wee-beng
> 


From jcrean01 at gmail.com  Sat Oct 31 19:52:03 2015
From: jcrean01 at gmail.com (Jared Crean)
Date: Sat, 31 Oct 2015 20:52:03 -0400
Subject: [petsc-users] PetscOptionsGetString Not Finding Option
Message-ID: <56356233.3060207@gmail.com>

Hello,
     I am trying to use PetscOptionsGetString to retrieve the value of 
an option in the options database, but the value returned in the last 
argument indicates the option was not found.   In the attached code (a 
modified version of ksp example 2), the string "-ksp_pc_side" is passed 
in as the argument name. If I run the code as

     ./jc2 -pc_type ilu -ksp_pc_side right

     I get the output:

     option -ksp_pc_side was found

     from line 71 of the file.  Petsc does not complain of unused 
options when the program finishes.  Am I using this function incorrectly?

     Jared Crean
-------------- next part --------------
A non-text attachment was scrubbed...
Name: jc2.c
Type: text/x-csrc
Size: 6214 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151031/c5ca8364/attachment.c>

From zonexo at gmail.com  Sat Oct 31 20:43:06 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 1 Nov 2015 09:43:06 +0800
Subject: [petsc-users] Scaling with number of cores
In-Reply-To: <CAMYG4Gm4ew=gxEGitop6=_ufPZs7=cuid8rnwW3xpKbEt1rBbw@mail.gmail.com>
References: <5634EDA4.5030203@gmail.com>
	<CAMYG4Gm4ew=gxEGitop6=_ufPZs7=cuid8rnwW3xpKbEt1rBbw@mail.gmail.com>
Message-ID: <56356E2A.8090000@gmail.com>


On 1/11/2015 12:47 AM, Matthew Knepley wrote:
> On Sat, Oct 31, 2015 at 11:34 AM, TAY wee-beng <zonexo at gmail.com 
> <mailto:zonexo at gmail.com>> wrote:
>
>     Hi,
>
>     I understand that as mentioned in the faq, due to the limitations
>     in memory, the scaling is not linear. So, I am trying to write a
>     proposal to use a supercomputer.
>
>     Its specs are:
>
>     Compute nodes: 82,944 nodes (SPARC64 VIIIfx; 16GB of memory per node)
>
>     8 cores / processor
>
>     Interconnect: Tofu (6-dimensional mesh/torus) Interconnect
>
>     Each cabinet contains 96 computing nodes,
>
>     One of the requirement is to give the performance of my current
>     code with my current set of data, and there is a formula to
>     calculate the estimated parallel efficiency when using the new
>     large set of data
>
>     There are 2 ways to give performance:
>     1. Strong scaling, which is defined as how the elapsed time varies
>     with the number of processors for a fixed
>     problem.
>     2. Weak scaling, which is defined as how the elapsed time varies
>     with the number of processors for a
>     fixed problem size per processor.
>
>     I ran my cases with 48 and 96 cores with my current cluster,
>     giving 140 and 90 mins respectively. This is classified as strong
>     scaling.
>
>     Cluster specs:
>
>     CPU: AMD 6234 2.4GHz
>
>     8 cores / processor (CPU)
>
>     6 CPU / node
>
>     So 48 Cores / CPU
>
>     Not sure abt the memory / node
>
>
>     The parallel efficiency ?En? for a given degree of parallelism ?n?
>     indicates how much the program is
>     efficiently accelerated by parallel processing. ?En? is given by
>     the following formulae. Although their
>     derivation processes are different depending on strong and weak
>     scaling, derived formulae are the
>     same.
>
>     From the estimated time, my parallel efficiency using  Amdahl's
>     law on the current old cluster was 52.7%.
>
>     So is my results acceptable?
>
>     For the large data set, if using 2205 nodes (2205X8cores), my
>     expected parallel efficiency is only 0.5%. The proposal recommends
>     value of > 50%.
>
> The problem with this analysis is that the estimated serial fraction 
> from Amdahl's Law  changes as a function
> of problem size, so you cannot take the strong scaling from one 
> problem and apply it to another without a
> model of this dependence.
>
> Weak scaling does model changes with problem size, so I would measure 
> weak scaling on your current
> cluster, and extrapolate to the big machine. I realize that this does 
> not make sense for many scientific
> applications, but neither does requiring a certain parallel efficiency.
Ok I check the results for my weak scaling it is even worse for the 
expected parallel efficiency. From the formula used, it's obvious it's 
doing some sort of exponential extrapolation decrease. So unless I can 
achieve a near > 90% speed up when I double the cores and problem size 
for my current 48/96 cores setup, extrapolating from about 96 nodes to 
10,000 nodes will give a much lower expected parallel efficiency for the 
new case.

However, it's mentioned in the FAQ that due to memory requirement, it's 
impossible to get >90% speed when I double the cores and problem size 
(ie linear increase in performance), which means that I can't get >90% 
speed up when I double the cores and problem size for my current 48/96 
cores setup. Is that so?

So is it fair to say that the main problem does not lie in my 
programming skills, but rather the way the linear equations are solved?

Thanks.
>
>   Thanks,
>
>      Matt
>
>     Is it possible for this type of scaling in PETSc (>50%), when
>     using 17640 (2205X8) cores?
>
>     Btw, I do not have access to the system.
>
>
>
>
>     Sent using CloudMagic Email
>     <https://cloudmagic.com/k/d/mailapp?ct=pa&cv=7.4.10&pv=5.0.2&source=email_footer_2>
>
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their 
> experiments is infinitely more interesting than any results to which 
> their experiments lead.
> -- Norbert Wiener

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151101/e10eea1c/attachment.html>

From bsmith at mcs.anl.gov  Sat Oct 31 21:00:49 2015
From: bsmith at mcs.anl.gov (Barry Smith)
Date: Sat, 31 Oct 2015 21:00:49 -0500
Subject: [petsc-users] Scaling with number of cores
In-Reply-To: <56356E2A.8090000@gmail.com>
References: <5634EDA4.5030203@gmail.com>
	<CAMYG4Gm4ew=gxEGitop6=_ufPZs7=cuid8rnwW3xpKbEt1rBbw@mail.gmail.com>
	<56356E2A.8090000@gmail.com>
Message-ID: <0BEDF334-C0C6-49D1-9FE8-D178181C5F00@mcs.anl.gov>


> On Oct 31, 2015, at 8:43 PM, TAY wee-beng <zonexo at gmail.com> wrote:
> 
> 
> On 1/11/2015 12:47 AM, Matthew Knepley wrote:
>> On Sat, Oct 31, 2015 at 11:34 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>> Hi,
>> 
>> I understand that as mentioned in the faq, due to the limitations in memory, the scaling is not linear. So, I am trying to write a proposal to use a supercomputer.
>> Its specs are:
>> Compute nodes: 82,944 nodes (SPARC64 VIIIfx; 16GB of memory per node)
>> 
>> 8 cores / processor
>> Interconnect: Tofu (6-dimensional mesh/torus) Interconnect
>> Each cabinet contains 96 computing nodes,
>> One of the requirement is to give the performance of my current code with my current set of data, and there is a formula to calculate the estimated parallel efficiency when using the new large set of data
>> There are 2 ways to give performance:
>> 1. Strong scaling, which is defined as how the elapsed time varies with the number of processors for a fixed
>> problem. 
>> 2. Weak scaling, which is defined as how the elapsed time varies with the number of processors for a
>> fixed problem size per processor.
>> I ran my cases with 48 and 96 cores with my current cluster, giving 140 and 90 mins respectively. This is classified as strong scaling.
>> Cluster specs:
>> CPU: AMD 6234 2.4GHz
>> 8 cores / processor (CPU)
>> 6 CPU / node
>> So 48 Cores / CPU
>> Not sure abt the memory / node
>> 
>> The parallel efficiency ?En? for a given degree of parallelism ?n? indicates how much the program is
>> efficiently accelerated by parallel processing. ?En? is given by the following formulae. Although their
>> derivation processes are different depending on strong and weak scaling, derived formulae are the
>> same.
>> From the estimated time, my parallel efficiency using  Amdahl's law on the current old cluster was 52.7%.
>> So is my results acceptable?
>> For the large data set, if using 2205 nodes (2205X8cores), my expected parallel efficiency is only 0.5%. The proposal recommends value of > 50%.
>> The problem with this analysis is that the estimated serial fraction from Amdahl's Law  changes as a function
>> of problem size, so you cannot take the strong scaling from one problem and apply it to another without a
>> model of this dependence.
>> 
>> Weak scaling does model changes with problem size, so I would measure weak scaling on your current
>> cluster, and extrapolate to the big machine. I realize that this does not make sense for many scientific
>> applications, but neither does requiring a certain parallel efficiency.
> Ok I check the results for my weak scaling it is even worse for the expected parallel efficiency. From the formula used, it's obvious it's doing some sort of exponential extrapolation decrease. So unless I can achieve a near > 90% speed up when I double the cores and problem size for my current 48/96 cores setup,     extrapolating from about 96 nodes to 10,000 nodes will give a much lower expected parallel efficiency for the new case.
> 
> However, it's mentioned in the FAQ that due to memory requirement, it's impossible to get >90% speed when I double the cores and problem size (ie linear increase in performance), which means that I can't get >90% speed up when I double the cores and problem size for my current 48/96 cores setup. Is that so? 

  What is the output of -ksp_view -log_summary on the problem and then on the problem doubled in size and number of processors?

  Barry

> 
> So is it fair to say that the main problem does not lie in my programming skills, but rather the way the linear equations are solved?
> 
> Thanks.
>> 
>>   Thanks,
>> 
>>      Matt
>> Is it possible for this type of scaling in PETSc (>50%), when using 17640 (2205X8) cores?
>> Btw, I do not have access to the system.
>> 
>> 
>> 
>> Sent using CloudMagic Email
>> 
>> 
>> 
>> -- 
>> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
>> -- Norbert Wiener
> 


From zonexo at gmail.com  Sat Oct 31 23:26:45 2015
From: zonexo at gmail.com (TAY wee-beng)
Date: Sun, 1 Nov 2015 12:26:45 +0800
Subject: [petsc-users] summary of the bandwidth received with different
 number of MPI processes
In-Reply-To: <C9EF975C-0766-46F6-84A8-4C809F393FA0@mcs.anl.gov>
References: <5634EF2F.3090208@gmail.com>
	<C9EF975C-0766-46F6-84A8-4C809F393FA0@mcs.anl.gov>
Message-ID: <56359485.7000004@gmail.com>


On 1/11/2015 1:17 AM, Barry Smith wrote:
>    Yes, just put the output from running with 1 2 etc processes in order into the file
Hi,

I just did but I got some errors.

The scaling.log file is:

Number of MPI processes 3 Processor names  n12-06 n12-06 n12-06
Triad:        27031.0419   Rate (MB/s)
Number of MPI processes 6 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06
Triad:        53517.8980   Rate (MB/s)
Number of MPI processes 12 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06
Triad:        53162.5346   Rate (MB/s)
Number of MPI processes 24 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06
Triad:       101455.6581   Rate (MB/s)
Number of MPI processes 48 Processor names  n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07
Triad:       115575.8960   Rate (MB/s)
Number of MPI processes 96 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07
Triad:       223742.1796   Rate (MB/s)
Number of MPI processes 192 Processor names  n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 n12-06 
n12-06 n12-06 n12-06 n12-06 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 n12-07 
n12-07 n12-07 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 
n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 
n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 
n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 
n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 n12-09 
n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 
n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 
n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 
n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 
n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10 n12-10
Triad:       436940.9859   Rate (MB/s)

When I tried to run "./process.py createfile ; process.py", I got

np  speedup
Traceback (most recent call last):
   File "./process.py", line 110, in <module>
     process(len(sys.argv)-1)
   File "./process.py", line 34, in process
     speedups[sizes] = triads[sizes]/triads[1]
KeyError: 1
Traceback (most recent call last):
   File "./process.py", line 110, in <module>
     process(len(sys.argv)-1)
   File "./process.py", line 34, in process
     speedups[sizes] = triads[sizes]/triads[1]
KeyError: 1

How can I solve it? Thanks.

>> On Oct 31, 2015, at 11:41 AM, TAY wee-beng <zonexo at gmail.com> wrote:
>>
>> Hi,
>>
>> It's mentioned that for a batch sys, I have to:
>>
>> 1. cd src/benchmarks/steams
>> 2. make MPIVersion
>> 3. submit MPIVersion to the batch system a number of times with 1, 2, 3, etc MPI processes collecting all of the output from the runs into the single file scaling.log.
>> 4. copy scaling.log into the src/benchmarks/steams directory
>> 5. ./process.py createfile ; process.py
>>
>> So for 3, how do I collect all of the output from the runs into the single file scaling.log.
>>
>> Should scaling.log look for this:
>>
>> Number of MPI processes 3 Processor names  n12-06 n12-06 n12-06
>> Triad:        27031.0419   Rate (MB/s)
>> Number of MPI processes 6 Processor names  n12-06 n12-06 n12-06 n12-06 n12-06 n12-06
>> Triad:        53517.8980   Rate (MB/s)
>>
>> ...
>>
>>
>>
>> -- 
>> Thank you.
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>