<div dir="ltr">Hi folks,<div><br></div><div>I am trying to track down a bug that is sometimes triggered when solving a singular system (poisson+neumann). It only seems to happen in parallel and halfway through the run. I can provide detailed information about the actual problem, but the error message I get boils down to this:</div><div><br></div><div><p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Invalid argument</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Scalar value must be same on all processes, argument # 2</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: See <a href="http://www.mcs.anl.gov/petsc/documentation/faq.html">http://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Petsc Release Version 3.6.3, Dec, 03, 2015 </span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: ./two_fluid_2d on a linux named bazantserver1 by mohammad Wed Oct  5 21:14:47 2016</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: Configure options PETSC_ARCH=linux --prefix=/usr/local --with-clanguage=cxx --with-c-support --with-shared-libraries --download-hypre --download-metis --download-parmetis --download-ml --download-superlu_dist COPTFLAGS=" -O3 -march=native" CXXOPTFLAGS=" -O3 -march=native" FOPTFLAGS=" -O3 -march=native"</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #1 VecShift() line 1480 in /tmp/petsc-3.6.3/src/vec/vec/utils/vinv.c</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #2 MatNullSpaceRemove() line 348 in /tmp/petsc-3.6.3/src/mat/interface/matnull.c</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #3 KSP_RemoveNullSpace() line 207 in /tmp/petsc-3.6.3/include/petsc/private/kspimpl.h</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #4 KSP_PCApply() line 243 in /tmp/petsc-3.6.3/include/petsc/private/kspimpl.h</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #5 KSPInitialResidual() line 63 in /tmp/petsc-3.6.3/src/ksp/ksp/interface/itres.c</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #6 KSPSolve_BCGS() line 50 in /tmp/petsc-3.6.3/src/ksp/ksp/impls/bcgs/bcgs.c</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">[0]PETSC ERROR: #7 KSPSolve() line 604 in /tmp/petsc-3.6.3/src/ksp/ksp/interface/itfunc.c</span></p></div><div><br></div><div>I understand this is somewhat vague question, but any idea what could cause this sort of problem? This was on 2 processors. The same code runs fine on a single processor. Also the solution seems to converge fine on previous iterations, e.g. this is the convergence info from the last iteration before the code breaks:</div><div><br></div><div><p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  0 KSP preconditioned resid norm 6.814085878146e+01 true resid norm 2.885308600701e+00 ||r(i)||/||b|| 1.000000000000e+00</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  1 KSP preconditioned resid norm 3.067319980814e-01 true resid norm 8.480307326867e-02 ||r(i)||/||b|| 2.939133555699e-02</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  2 KSP preconditioned resid norm 1.526405979843e-03 true resid norm 1.125228519827e-03 ||r(i)||/||b|| 3.899855008762e-04</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  3 KSP preconditioned resid norm 2.199423175998e-05 true resid norm 4.232832916628e-05 ||r(i)||/||b|| 1.467029528695e-05</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  4 KSP preconditioned resid norm 5.382291463582e-07 true resid norm 8.438732856334e-07 ||r(i)||/||b|| 2.924724535283e-07</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  5 KSP preconditioned resid norm 9.495525177398e-09 true resid norm 1.408250768598e-08 ||r(i)||/||b|| 4.880763077669e-09</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  6 KSP preconditioned resid norm 9.249233376169e-11 true resid norm 2.795840275267e-10 ||r(i)||/||b|| 9.689917655907e-11</span></p>
<p style="margin:0px;font-size:11px;line-height:normal;font-family:monaco;color:rgb(246,239,220);background-color:rgb(2,43,54)"><span style="font-variant-ligatures:no-common-ligatures">  7 KSP preconditioned resid norm 1.138293762641e-12 true resid norm 2.559058680281e-12 ||r(i)||/||b|| 8.869272006674e-13</span></p></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">Also, if it matters, this is using hypre as PC and bcgs as KSP.</span></div><div><span style="font-variant-ligatures:no-common-ligatures"><br></span></div><div><span style="font-variant-ligatures:no-common-ligatures">Thanks</span></div></div>