<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Tue, Oct 25, 2016 at 7:57 PM, Olivier Mesnard <span dir="ltr"><<a href="mailto:olivier.mesnard8@gmail.com" target="_blank">olivier.mesnard8@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><span class=""><div style="color:rgb(0,0,0)"><span style="color:rgb(34,34,34)">On 25 October 2016 at 19:59, Matthew Knepley </span><span dir="ltr" style="color:rgb(34,34,34)"><<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>></span><span style="color:rgb(34,34,34)"> wrote:</span><br></div></span><div class="gmail_extra"><div class="gmail_quote"><span class=""><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><span class="m_3336516428159232092gmail-">On Tue, Oct 25, 2016 at 6:22 PM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="m_3336516428159232092gmail-m_6978760639828642268gmail-"><br>
> On Oct 25, 2016, at 5:39 PM, Olivier Mesnard <<a href="mailto:olivier.mesnard8@gmail.com" target="_blank">olivier.mesnard8@gmail.com</a>> wrote:<br>
><br>
> On 25 October 2016 at 17:51, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>> wrote:<br>
><br>
> Olivier,<br>
><br>
> In theory you do not need to change anything else. Are you using a different matrix object for the velocity_ksp object than the poisson_ksp object?<br>
><br>
> The matrix is different for the velocity_ksp and the poisson_ksp.<br>
><br>
> The code change in PETSc is very little but we have a report from another CFD user who also had problems with the change so there may be some subtle bug that we can't figure out causing things to not behave properly.<br>
><br>
> First run the 3.7.4 code with -poisson_ksp_view and verify that when it prints the matrix information it prints something like has attached null space if it does not print that it means that somehow the matrix is not properly getting the matrix attached.<br>
><br>
> When running with 3.7.4 and -poisson_ksp_view, the output shows that the nullspace is not attached to the KSP (as it was with 3.5.4); however the print statement is now under the Mat info (which is expected when moving from KSPSetNullSpace to MatSetNullSpace?).<br>
<br>
</span> Good, this is how it should be.<br>
<span class="m_3336516428159232092gmail-m_6978760639828642268gmail-">><br>
> Though older versions had MatSetNullSpace() they didn't necessarily associate it with the KSP so it was not expected to work as a replacement for KSPSetNullSpace() with older versions.<br>
><br>
> Because our other user had great difficulty trying to debug the issue feel free to send us at <a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a> your code with instructions on building and running and we can try to track down the problem. Better than hours and hours spent with fruitless email. We will, of course, not distribute the code and will delete in when we are finished with it.<br>
><br>
> The code is open-source and hosted on GitHub (<a href="https://github.com/barbagroup/PetIBM)" rel="noreferrer" target="_blank">https://github.com/barbagroup<wbr>/PetIBM)</a>.<br>
> I just pushed the branches `feature-compatible-petsc-3.7` and `revert-compatible-petsc-3.5` that I used to observe this problem.<br>
><br>
</span> Thanks, I'll get back to you if I discover anything</blockquote><div><br></div></span><div>Obviously GAMG is behaving quite differently (1 vs 2 levels and a much sparser coarse problem in 3.7).</div><div><br></div><div>Could you try one thing for me before we start running it? Run with</div><div><br></div><div> -poisson_mg_coarse_sub_pc_type svd</div><div><br></div><div>and see what happens on 2 procs for 3.7?</div><div><br></div></div></div></div></blockquote></span><div><div style="color:rgb(0,0,0);display:inline">Hi Matt,</div></div><div><div style="color:rgb(0,0,0);display:inline"><br></div></div><div><div style="color:rgb(0,0,0);display:inline">With </div><span style="font-size:12.8px">-poisson_mg_coarse_sub_pc_type svd<div style="color:rgb(0,0,0);display:inline">,</div></span></div><div><div style="color:rgb(0,0,0);display:inline">it ran normally on 1 proc but not on 2 procs the end of the output says:</div></div><div><div style="color:rgb(0,0,0);display:inline"><br></div></div><div><div style="color:rgb(0,0,0);display:inline">"** On entry to DGESVD parameter number 6 had an illegal value</div><span style="color:rgb(0,0,0)">"</span></div></div></div></div></blockquote><div><br></div><div>Something is wrong with your 3.7 installation. That parameter is a simple size, so I suspect</div><div>memory corruption. Run with valgrind.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><div style="color:rgb(0,0,0);display:inline">I attached the log file.</div></div><div><div class="h5"><div><div style="color:rgb(0,0,0);display:inline"></div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div> Thanks,</div><div><br></div><div> Matt</div><div><div class="m_3336516428159232092gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="m_3336516428159232092gmail-m_6978760639828642268gmail-HOEnZb"><div class="m_3336516428159232092gmail-m_6978760639828642268gmail-h5">
<br>
> PETSc (both 3.5.4 and 3.7.4) was configured as follow:<br>
> export PETSC_ARCH="linux-gnu-dbg"<br>
> ./configure --PETSC_ARCH=$PETSC_ARCH \<br>
> --with-cc=gcc \<br>
> --with-cxx=g++ \<br>
> --with-fc=gfortran \<br>
> --COPTFLAGS="-O0" \<br>
> --CXXOPTFLAGS="-O0" \<br>
> --FOPTFLAGS="-O0" \<br>
> --with-debugging=1 \<br>
> --download-fblaslapack \<br>
> --download-mpich \<br>
> --download-hypre \<br>
> --download-yaml \<br>
> --with-x=1<br>
><br>
> Our code was built using the following commands:<br>
> mkdir petibm-build<br>
> cd petibm-build<br>
> export PETSC_DIR=<directory of PETSc><br>
> export PETSC_ARCH="linux-gnu-dbg"<br>
> export PETIBM_DIR=<directory of PetIBM git repo><br>
> $PETIBM_DIR/configure --prefix=$PWD \<br>
> CXX=$PETSC_DIR/$PETSC_ARCH/bi<wbr>n/mpicxx \<br>
> CXXFLAGS="-g -O0 -std=c++11"<br>
> make all<br>
> make install<br>
><br>
> Then<br>
> cd examples<br>
> make examples<br>
><br>
> The example of the lid-driven cavity I was talking about can be found in the folder `examples/2d/convergence/lidDr<wbr>ivenCavity20/20/`<br>
><br>
> To run it:<br>
> mpiexec -n N <path-to-petibm-build>/bin/pet<wbr>ibm2d -directory <path-to-example><br>
><br>
> Let me know if you need more info. Thank you.<br>
><br>
> Barry<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
> > On Oct 25, 2016, at 4:38 PM, Olivier Mesnard <<a href="mailto:olivier.mesnard8@gmail.com" target="_blank">olivier.mesnard8@gmail.com</a>> wrote:<br>
> ><br>
> > Hi all,<br>
> ><br>
> > We develop a CFD code using the PETSc library that solves the Navier-Stokes equations using the fractional-step method from Perot (1993).<br>
> > At each time-step, we solve two systems: one for the velocity field, the other, a Poisson system, for the pressure field.<br>
> > One of our test-cases is a 2D lid-driven cavity flow (Re=100) on a 20x20 grid using 1 or 2 procs.<br>
> > For the Poisson system, we usually use CG preconditioned with GAMG.<br>
> ><br>
> > So far, we have been using PETSc-3.5.4, and we would like to update the code with the latest release: 3.7.4.<br>
> ><br>
> > As suggested in the changelog of 3.6, we replaced the routine `KSPSetNullSpace()` with `MatSetNullSpace()`.<br>
> ><br>
> > Here is the list of options we use to configure the two solvers:<br>
> > * Velocity solver: prefix `-velocity_`<br>
> > -velocity_ksp_type bcgs<br>
> > -velocity_ksp_rtol 1.0E-08<br>
> > -velocity_ksp_atol 0.0<br>
> > -velocity_ksp_max_it 10000<br>
> > -velocity_pc_type jacobi<br>
> > -velocity_ksp_view<br>
> > -velocity_ksp_monitor_true_re<wbr>sidual<br>
> > -velocity_ksp_converged_reaso<wbr>n<br>
> > * Poisson solver: prefix `-poisson_`<br>
> > -poisson_ksp_type cg<br>
> > -poisson_ksp_rtol 1.0E-08<br>
> > -poisson_ksp_atol 0.0<br>
> > -poisson_ksp_max_it 20000<br>
> > -poisson_pc_type gamg<br>
> > -poisson_pc_gamg_type agg<br>
> > -poisson_pc_gamg_agg_nsmooths 1<br>
> > -poissonksp_view<br>
> > -poisson_ksp_monitor_true_res<wbr>idual<br>
> > -poisson_ksp_converged_reason<br>
> ><br>
> > With 3.5.4, the case runs normally on 1 or 2 procs.<br>
> > With 3.7.4, the case runs normally on 1 proc but not on 2.<br>
> > Why? The Poisson solver diverges because of an indefinite preconditioner (only with 2 procs).<br>
> ><br>
> > We also saw that the routine `MatSetNullSpace()` was already available in 3.5.4.<br>
> > With 3.5.4, replacing `KSPSetNullSpace()` with `MatSetNullSpace()` led to the Poisson solver diverging because of an indefinite matrix (on 1 and 2 procs).<br>
> ><br>
> > Thus, we were wondering if we needed to update something else for the KSP, and not just modifying the name of the routine?<br>
> ><br>
> > I have attached the output files from the different cases:<br>
> > * `run-petsc-3.5.4-n1.log` (3.5.4, `KSPSetNullSpace()`, n=1)<br>
> > * `run-petsc-3.5.4-n2.log`<br>
> > * `run-petsc-3.5.4-nsp-n1.log` (3.5.4, `MatSetNullSpace()`, n=1)<br>
> > * `run-petsc-3.5.4-nsp-n2.log`<br>
> > * `run-petsc-3.7.4-n1.log` (3.7.4, `MatSetNullSpace()`, n=1)<br>
> > * `run-petsc-3.7.4-n2.log`<br>
> ><br>
> > Thank you for your help,<br>
> > Olivier<br>
> > <run-petsc-3.5.4-n1.log><run-p<wbr>etsc-3.5.4-n2.log><run-petsc-3<wbr>.5.4-nsp-n1.log><run-petsc-3.5<wbr>.4-nsp-n2.log><run-petsc-3.7.4<wbr>-n1.log><run-petsc-3.7.4-n2.lo<wbr>g><br>
><br>
><br>
<br>
</div></div></blockquote></div></div></div><span class="m_3336516428159232092gmail-HOEnZb"><font color="#888888"><br><br clear="all"><div><br></div>-- <br><div class="m_3336516428159232092gmail-m_6978760639828642268gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</font></span></div></div>
</blockquote></div></div></div><br></div></div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div>
</div></div>