<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> Yikes, sorry I missed that the first run was CPU and the second GPU.<div><br></div><div> The run on the CPU is indicative of a very bad preconditioner. It doesn't really converge. When the true residual norm jumps by a factor of 10^3 at the first iteration, this means the ILU preconditioner is just not appropriate or reasonable. The "convergence" of the preconditioned residual norm is meaningless.</div><div><br></div><div><blockquote type="cite"><div dir="ltr"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;">> 0 KSP preconditioned resid norm 1.236208833927e-08 true resid norm 1.413045088306e-03 ||r(i)||/||b|| 3.745397377137e+01<br>> 1 KSP preconditioned resid norm 1.664973208594e-10 true resid norm 3.463939828700e+00 ||r(i)||/||b|| 9.181470043910e+04<br>> 2 KSP preconditioned resid norm 8.366983092820e-14 true resid norm 9.171051852915e-02 ||r(i)||/||b|| 2.430866066466e+03<br>> 3 KSP preconditioned resid norm 1.386354386207e-14 true resid norm 1.905770367881e-02 ||r(i)||/||b|| 5.051408052270e+02<br>> 4 KSP preconditioned resid norm 4.635883581096e-15 true resid norm 7.285180695640e-03 ||r(i)||/||b|| 1.930999717931e+02<br>> 5 KSP preconditioned resid norm 1.974093227402e-15 true resid norm 2.953370060898e-03 ||r(i)||/||b|| 7.828161020018e+01<br>> 6 KSP preconditioned resid norm 1.182781787023e-15 true resid norm 2.288756945462e-03 ||r(i)||/||b|| 6.066546871987e+01<br>> 7 KSP preconditioned resid norm 6.221244366707e-16 true resid norm 1.263339414861e-03 ||r(i)||/||b|| 3.348589631014e+01</blockquote></div></div></blockquote><div><br></div><div> I won't worry about the GPU behavior (it is just due to slightly different numerical computations on the GPU and not surprising.)</div><div><br></div><div> You need to use a different preconditioner, even on the CPU.</div><br><div><br><blockquote type="cite"><div>On Aug 30, 2023, at 9:51 PM, Junchao Zhang <junchao.zhang@gmail.com> wrote:</div><br class="Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr"><br><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 30, 2023 at 8:46 PM Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
What convergence do you get without the GPU matrix and vector operations?</blockquote><div>Barry, that was in the original email </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> <br>
<br>
Can you try the GPU run with -ksp_type gmres -ksp_pc_side right ?<br>
<br>
For certain problems, ILU can produce catastrophically bad preconditioners.<br>
Barry<br>
<br>
<br>
<br>
> On Aug 30, 2023, at 4:41 PM, Ramoni Z. Sedano Azevedo <<a href="mailto:ramoni.zsedano@gmail.com" target="_blank">ramoni.zsedano@gmail.com</a>> wrote:<br>
> <br>
> Hello,<br>
> <br>
> I'm executing a code in Fortran using PETSc with MPI via CPU and I would like to execute it using GPU.<br>
> PETSc is configured as follows:<br>
> ./configure \<br>
> --prefix=${PWD}/installdir \<br>
> --with-fortran \<br>
> --with-fortran-kernels=true \<br>
> --with-cuda \<br>
> --download-fblaslapack \<br>
> --with-scalar-type=complex \<br>
> --with-precision=double \<br>
> --with-debugging=0 \<br>
> --with-x=0 \<br>
> --with-gnu-compilers=1 \<br>
> --with-cc=mpicc \<br>
> --with-cxx=mpicxx \<br>
> --with-fc=mpif90 \<br>
> --with-make-exec=make<br>
> <br>
> The parameters for using MPI on CPU are:<br>
> mpirun -np $ntasks ./${executable} \<br>
> -A_mat_type mpiaij \<br>
> -P_mat_type mpiaij \<br>
> -em_ksp_monitor_true_residual \<br>
> -em_ksp_type bcgs \<br>
> -em_pc_type bjacobi \<br>
> -em_sub_pc_type ilu \<br>
> -em_sub_pc_factor_levels 3 \<br>
> -em_sub_pc_factor_fill 6 \<br>
> < ./Parameters.inp<br>
> <br>
> Code output:<br>
> Solving for Hz fields<br>
> bnorm 3.7727507818834821E-005<br>
> xnorm 2.3407405211699372E-016<br>
> Residual norms for em_ solve.<br>
> 0 KSP preconditioned resid norm 1.236208833927e-08 true resid norm 1.413045088306e-03 ||r(i)||/||b|| 3.745397377137e+01<br>
> 1 KSP preconditioned resid norm 1.664973208594e-10 true resid norm 3.463939828700e+00 ||r(i)||/||b|| 9.181470043910e+04<br>
> 2 KSP preconditioned resid norm 8.366983092820e-14 true resid norm 9.171051852915e-02 ||r(i)||/||b|| 2.430866066466e+03<br>
> 3 KSP preconditioned resid norm 1.386354386207e-14 true resid norm 1.905770367881e-02 ||r(i)||/||b|| 5.051408052270e+02<br>
> 4 KSP preconditioned resid norm 4.635883581096e-15 true resid norm 7.285180695640e-03 ||r(i)||/||b|| 1.930999717931e+02<br>
> 5 KSP preconditioned resid norm 1.974093227402e-15 true resid norm 2.953370060898e-03 ||r(i)||/||b|| 7.828161020018e+01<br>
> 6 KSP preconditioned resid norm 1.182781787023e-15 true resid norm 2.288756945462e-03 ||r(i)||/||b|| 6.066546871987e+01<br>
> 7 KSP preconditioned resid norm 6.221244366707e-16 true resid norm 1.263339414861e-03 ||r(i)||/||b|| 3.348589631014e+01<br>
> 8 KSP preconditioned resid norm 3.800488678870e-16 true resid norm 9.015738978063e-04 ||r(i)||/||b|| 2.389699054959e+01<br>
> 9 KSP preconditioned resid norm 2.498733213989e-16 true resid norm 7.194509577987e-04 ||r(i)||/||b|| 1.906966559396e+01<br>
> 10 KSP preconditioned resid norm 1.563017112250e-16 true resid norm 5.055208317846e-04 ||r(i)||/||b|| 1.339926385310e+01<br>
> 11 KSP preconditioned resid norm 8.733803057628e-17 true resid norm 3.171941303660e-04 ||r(i)||/||b|| 8.407502872682e+00<br>
> 12 KSP preconditioned resid norm 4.907010803529e-17 true resid norm 1.868311755294e-04 ||r(i)||/||b|| 4.952120782177e+00<br>
> 13 KSP preconditioned resid norm 2.214070343700e-17 true resid norm 8.760421740830e-05 ||r(i)||/||b|| 2.322025028236e+00<br>
> 14 KSP preconditioned resid norm 1.333171674446e-17 true resid norm 5.984548368534e-05 ||r(i)||/||b|| 1.586255948119e+00<br>
> 15 KSP preconditioned resid norm 7.696778066646e-18 true resid norm 3.786809196913e-05 ||r(i)||/||b|| 1.003726303656e+00<br>
> 16 KSP preconditioned resid norm 3.863008301366e-18 true resid norm 1.284864871601e-05 ||r(i)||/||b|| 3.405644702988e-01<br>
> 17 KSP preconditioned resid norm 2.061402843494e-18 true resid norm 1.054741071688e-05 ||r(i)||/||b|| 2.795681805311e-01<br>
> 18 KSP preconditioned resid norm 1.062033155108e-18 true resid norm 3.992776343462e-06 ||r(i)||/||b|| 1.058319664960e-01<br>
> converged reason 2<br>
> total number of relaxations 18<br>
> ========================================<br>
> <br>
> The parameters for GPU usage are:<br>
> mpirun -np $ntasks ./${executable} \<br>
> -A_mat_type aijcusparse \<br>
> -P_mat_type aijcusparse \<br>
> -vec_type cuda \<br>
> -use_gpu_aware_mpi 0 \<br>
> -em_ksp_monitor_true_residual \<br>
> -em_ksp_type bcgs \<br>
> -em_pc_type bjacobi \<br>
> -em_sub_pc_type ilu \<br>
> -em_sub_pc_factor_levels 3 \<br>
> -em_sub_pc_factor_fill 6 \<br>
> < ./Parameters.inp<br>
> <br>
> Code output:<br>
> Solving for Hz fields<br>
> bnorm 3.7727507818834821E-005<br>
> xnorm 2.3407405211699372E-016<br>
> Residual norms for em_ solve.<br>
> 0 KSP preconditioned resid norm 1.236220954395e-08 true resid norm 3.772750781883e-05 ||r(i)||/||b|| 1.000000000000e+00<br>
> 1 KSP preconditioned resid norm 0.000000000000e+00 true resid norm 3.772750781883e-05 ||r(i)||/||b|| 1.000000000000e+00<br>
> converged reason 3<br>
> total number of relaxations 1<br>
> ========================================<br>
> <br>
> Clearly the code running on GPU is not converging correctly.<br>
> Has anyone experienced this problem?<br>
> <br>
> Sincerely,<br>
> Ramoni Z. S. Azevedo<br>
> <br>
<br>
</blockquote></div></div>
</div></blockquote></div><br></div></body></html>