<div dir="ltr"><div dir="ltr">And I just tested it with GAMG and it seems fine. And hypre ran, but it is not clear that it used GPUs....<div><br></div><div><div>14:13 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type hypre -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5 -ksp_view</div><div>lid velocity = 0.0625, prandtl # = 1., grashof # = 1.</div><div> 0 SNES Function norm 0.239155 </div><div>KSP Object: 1 MPI processes</div><div> type: fgmres</div><div> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div> happy breakdown tolerance 1e-30</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> right preconditioning</div><div> using UNPRECONDITIONED norm type for convergence test</div><div>PC Object: 1 MPI processes</div><div> type: hypre</div><div> HYPRE BoomerAMG preconditioning</div><div> Cycle type V</div><div> Maximum number of levels 25</div><div> Maximum number of iterations PER hypre call 1</div><div> Convergence tolerance PER hypre call 0.</div><div> Threshold for strong coupling 0.25</div><div> Interpolation truncation factor 0.</div><div> Interpolation: max elements per row 0</div><div> Number of levels of aggressive coarsening 0</div><div> Number of paths for aggressive coarsening 1</div><div> Maximum row sums 0.9</div><div> Sweeps down 1</div><div> Sweeps up 1</div><div> Sweeps on coarse 1</div><div> Relax down symmetric-SOR/Jacobi</div><div> Relax up symmetric-SOR/Jacobi</div><div> Relax on coarse Gaussian-elimination</div><div> Relax weight (all) 1.</div><div> Outer relax weight (all) 1.</div><div> Using CF-relaxation</div><div> Not using more complex smoothers.</div><div> Measure type local</div><div> Coarsen type Falgout</div><div> Interpolation type classical</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 1 MPI processes</div><div> type: seqaijcusparse</div><div> rows=64, cols=64, bs=4</div><div> total: nonzeros=1024, allocated nonzeros=1024</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 16 nodes, limit used is 5</div><div> 1 SNES Function norm 6.80716e-05 </div><div>KSP Object: 1 MPI processes</div><div> type: fgmres</div><div> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement</div><div> happy breakdown tolerance 1e-30</div><div> maximum iterations=10000, initial guess is zero</div><div> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.</div><div> right preconditioning</div><div> using UNPRECONDITIONED norm type for convergence test</div><div>PC Object: 1 MPI processes</div><div> type: hypre</div><div> HYPRE BoomerAMG preconditioning</div><div> Cycle type V</div><div> Maximum number of levels 25</div><div> Maximum number of iterations PER hypre call 1</div><div> Convergence tolerance PER hypre call 0.</div><div> Threshold for strong coupling 0.25</div><div> Interpolation truncation factor 0.</div><div> Interpolation: max elements per row 0</div><div> Number of levels of aggressive coarsening 0</div><div> Number of paths for aggressive coarsening 1</div><div> Maximum row sums 0.9</div><div> Sweeps down 1</div><div> Sweeps up 1</div><div> Sweeps on coarse 1</div><div> Relax down symmetric-SOR/Jacobi</div><div> Relax up symmetric-SOR/Jacobi</div><div> Relax on coarse Gaussian-elimination</div><div> Relax weight (all) 1.</div><div> Outer relax weight (all) 1.</div><div> Using CF-relaxation</div><div> Not using more complex smoothers.</div><div> Measure type local</div><div> Coarsen type Falgout</div><div> Interpolation type classical</div><div> linear system matrix = precond matrix:</div><div> Mat Object: 1 MPI processes</div><div> type: seqaijcusparse</div><div> rows=64, cols=64, bs=4</div><div> total: nonzeros=1024, allocated nonzeros=1024</div><div> total number of mallocs used during MatSetValues calls =0</div><div> using I-node routines: found 16 nodes, limit used is 5</div><div> 2 SNES Function norm 4.093e-11 </div><div>Number of SNES iterations = 2</div></div><div><br></div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Nov 2, 2018 at 2:10 PM Smith, Barry F. <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
> On Nov 2, 2018, at 1:03 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> FYI, I seem to have the new GPU machine at ORNL (summitdev) working with GPUs. That is good enough for now.<br>
> Thanks,<br>
<br>
Excellant!<br>
<br>
> <br>
> 14:00 master= ~/petsc/src/snes/examples/tutorials$ jsrun -n 1 ./ex19 -dm_vec_type cuda -dm_mat_type aijcusparse -pc_type none -ksp_type fgmres -snes_monitor_short -snes_rtol 1.e-5 -ksp_view<br>
> lid velocity = 0.0625, prandtl # = 1., grashof # = 1.<br>
> 0 SNES Function norm 0.239155 <br>
> KSP Object: 1 MPI processes<br>
> type: fgmres<br>
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> happy breakdown tolerance 1e-30<br>
> maximum iterations=10000, initial guess is zero<br>
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
> right preconditioning<br>
> using UNPRECONDITIONED norm type for convergence test<br>
> PC Object: 1 MPI processes<br>
> type: none<br>
> linear system matrix = precond matrix:<br>
> Mat Object: 1 MPI processes<br>
> type: seqaijcusparse<br>
> rows=64, cols=64, bs=4<br>
> total: nonzeros=1024, allocated nonzeros=1024<br>
> total number of mallocs used during MatSetValues calls =0<br>
> using I-node routines: found 16 nodes, limit used is 5<br>
> 1 SNES Function norm 6.82338e-05 <br>
> KSP Object: 1 MPI processes<br>
> type: fgmres<br>
> restart=30, using Classical (unmodified) Gram-Schmidt Orthogonalization with no iterative refinement<br>
> happy breakdown tolerance 1e-30<br>
> maximum iterations=10000, initial guess is zero<br>
> tolerances: relative=1e-05, absolute=1e-50, divergence=10000.<br>
> right preconditioning<br>
> using UNPRECONDITIONED norm type for convergence test<br>
> PC Object: 1 MPI processes<br>
> type: none<br>
> linear system matrix = precond matrix:<br>
> Mat Object: 1 MPI processes<br>
> type: seqaijcusparse<br>
> rows=64, cols=64, bs=4<br>
> total: nonzeros=1024, allocated nonzeros=1024<br>
> total number of mallocs used during MatSetValues calls =0<br>
> using I-node routines: found 16 nodes, limit used is 5<br>
> 2 SNES Function norm 3.346e-10 <br>
> Number of SNES iterations = 2<br>
> 14:01 master= ~/petsc/src/snes/examples/tutorials$ <br>
> <br>
> <br>
> <br>
> On Thu, Nov 1, 2018 at 9:33 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> <br>
> On Wed, Oct 31, 2018 at 12:30 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> <br>
> On Wed, Oct 31, 2018 at 6:59 AM Karl Rupp <<a href="mailto:rupp@iue.tuwien.ac.at" target="_blank">rupp@iue.tuwien.ac.at</a>> wrote:<br>
> Hi Mark,<br>
> <br>
> ah, I was confused by the Python information at the beginning of <br>
> configure.log. So it is picking up the correct compiler.<br>
> <br>
> Have you tried uncommenting the check for GNU?<br>
> <br>
> Yes, but I am getting an error that the cuda files do not find mpi.h.<br>
> <br>
> <br>
> I'm getting a make error.<br>
> <br>
> Thanks, <br>
<br>
</blockquote></div>