[petsc-users] Pipelined solvers PIPECG etc fail

[Sascha Schnepp]

Hi,

I cannot reproduce this on my system. All norm types work for me. 
If you use KSP_NORM_NONE you have to add something like -ksp_max_it 30
but this works too.

I have seen this error though. For me, it occurred when I messed up the memory space
by allocating too few memory in some place and overwriting other memory in consequence.

My guess is that the actual error occurs (long) before line 153 in pipecg.c

Sascha

On 20 Nov 2014, at 16:06, Håkon Strandenes <haakon at hakostra.net> wrote:

> Ok:
> 
> If I run KSP ex 2 with parameters "-ksp_type pipecg -ksp_norm_type natural", it works.
> 
> If I try *any* other norm, it fails, but with a different error message, for example if I try "preconditioned":
> 
> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
> [0]PETSC ERROR: Invalid argument
> [0]PETSC ERROR: Scalar value must be same on all processes, argument # 2
> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html for trouble shooting.
> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown
> [0]PETSC ERROR: ./ex2 on a arch-linux2-c-debug named hakostra-kontor by hakostra Thu Nov 20 16:04:37 2014
> [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5 --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging --with-debugger=gdb
> [0]PETSC ERROR: #1 VecAXPY() line 634 in /opt/petsc/git-debug/source/src/vec/vec/interface/rvector.c
> [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 153 in /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c
> [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/src/ksp/ksp/interface/itfunc.c
> [0]PETSC ERROR: #4 main() line 75 in /home/hakostra/Dropbox/KSP-ex2/ex2.c
> 
> Håkon
> 
> 
> On 20. nov. 2014 15:49, Sascha Schnepp wrote:
>> Hi,
>> 
>> this is an error that occurs when calling ksp->converged and having
>> ksp->normtype == KSP_NORM_NONE
>> 
>> In pipecg.c the error occurs here:
>> line99:  ierr       = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr); /* test for convergence */
>> 
>> It can be fixed (though there might be a nicer way) using this replacement code, wich explicitly calls KSPConvergendSkip for the respective norm:
>>   if (ksp->normtype == KSP_NORM_NONE) {
>>     ierr = KSPConvergedSkip (ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr);
>>   } else {
>>     ierr = (*ksp->converged)(ksp,0,dp,&ksp->reason,ksp->cnvP);CHKERRQ(ierr);
>>   }
>> 
>> 
>> The issue could be related to this code in
>> KSPCreate_PIPECG
>>   ierr = KSPSetSupportedNorm(ksp,KSP_NORM_UNPRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr);
>>   ierr = KSPSetSupportedNorm(ksp,KSP_NORM_PRECONDITIONED,PC_LEFT,2);CHKERRQ(ierr);
>>   ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NATURAL,PC_LEFT,2);CHKERRQ(ierr);
>>   ierr = KSPSetSupportedNorm(ksp,KSP_NORM_NONE,PC_LEFT,2);CHKERRQ(ierr);
>> where all norms have the same priority.
>> 
>> For the non-pipelined version of CG the unpreconditioned norm has a priority level of 3.
>> 
>> The modifications are independent. Choosing different default priorities will still make line 99 fail if type NONE is chosen explicitly.
>> 
>> Cheers,
>> Sascha
>> 
>> 
>> Begin forwarded message:
>> 
>>> On Thu, Nov 20, 2014 at 4:05 AM, H?kon Strandenes <haakon at hakostra.net>
>>> wrote:
>>> 
>>>> Hi,
>>>> 
>>>> First: thanks to the list and Barry for solving my previous problem on
>>>> loading vectors from HDF5 files. It works great!
>>>> 
>>>> Now a second problem: I want to try the pipelined KSP solvers, PIPECG in
>>>> particular. However, my program fails on the first KSPSolve() with the
>>>> message below. The same happens if I try PIPECR.
>>>> 
>>> 
>>> These are very new, so there could be bugs lurking. It would help our
>>> testing if this is
>>> reproducuble on a standard example. Does this happen if you try to solve
>>> KSP ex2
>>> or SNES ex5 with PIPECG?
>>> 
>>>   Thanks,
>>> 
>>>     Matt
>>> 
>>> 
>>>> I have tried this on three MPI implementations, on and two of them fully
>>>> support MPI 3 (I have read http://www.mcs.anl.gov/petsc/
>>>> documentation/faq.html#pipelined). I am using latest PETSc from Git repo.
>>>> 
>>>> I have tried to turn off the preconditioner without any effect.
>>>> 
>>>> The equation system is the result of a finite-volume discretization of a
>>>> Poisson-like equation if that helps anything. A normal CG solver works like
>>>> a charm, converging fast and effective.
>>>> 
>>>> Are there any tricks to get this working? Anything special I have to set
>>>> and/or define to get this solver running?
>>>> 
>>>> Thanks in advance.
>>>> 
>>>> Best regards,
>>>> H?kon Strandenes
>>>> 
>>>> 
>>>> Error message:
>>>> 
>>>> [0]PETSC ERROR: --------------------- Error Message
>>>> --------------------------------------------------------------
>>>> [0]PETSC ERROR: Object is in wrong state
>>>> [0]PETSC ERROR: Use KSPConvergedSkip() with KSPNormType of KSP_NORM_NONE
>>>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html
>>>> for trouble shooting.
>>>> [0]PETSC ERROR: Petsc Release Version 3.5.2, unknown
>>>> [0]PETSC ERROR: madns on a arch-linux2-c-debug named hakostra-kontor by
>>>> hakostra Thu Nov 20 11:04:28 2014
>>>> [0]PETSC ERROR: Configure options --prefix=/opt/petsc/git-debug/install
>>>> --with-shared-libraries --download-hypre --with-64-bit-indices --with-hdf5
>>>> --with-hdf5-dir=/opt/HDF5/1.8.13/install --with-debugging
>>>> --with-debugger=gdb
>>>> [0]PETSC ERROR: #1 KSPConvergedDefault() line 710 in
>>>> /opt/petsc/git-debug/source/src/ksp/ksp/interface/iterativ.c
>>>> [0]PETSC ERROR: #2 KSPSolve_PIPECG() line 99 in
>>>> /opt/petsc/git-debug/source/src/ksp/ksp/impls/cg/pipecg/pipecg.c
>>>> [0]PETSC ERROR: #3 KSPSolve() line 459 in /opt/petsc/git-debug/source/
>>>> src/ksp/ksp/interface/itfunc.c
>>>> [0]PETSC ERROR: #4 findPressure() line 53 in /opt/MaDNS/source/src/FVM2.cpp
>>>> 
>>> 
>> 
>>