<html><head><meta http-equiv="content-type" content="text/html; charset=us-ascii"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> In general you should expect similar but not identical conference behavior. <div><br></div><div> I suggest running with all the monitoring you can. -ksp_monitor_true_residual -fieldsplit_0_monitor_true_residual -fieldsplit_1_monitor_true_residual and compare the various convergence between the CPU and GPU. Also run with -ksp_view and check that the various solver options are the same (they should be).</div><div><br></div><div> Barry</div><div><br><div><br><blockquote type="cite"><div>On Oct 30, 2022, at 11:02 AM, Carl-Johan Thore via petsc-users <petsc-users@mcs.anl.gov> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">Hi,<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">I'm solving a topology optimization problem with Stokes flow discretized by a stabilized Q1-Q0 finite element method<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">and using BiCGStab with the fieldsplit preconditioner to solve the linear systems. The implementation<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">is based on DMStag, runs on Ubuntu via WSL2, and works fine with PETSc-3.18.1 on multiple CPU cores and the following<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">options for the preconditioner:<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_ksp_type preonly \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_pc_type gamg \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_pc_gamg_reuse_interpolation 0 \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_1_ksp_type preonly \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_1_pc_type jacobi<span class="Apple-converted-space"> </span><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">However, when I enable GPU computations by adding two options -<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">...<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-dm_vec_type cuda \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-dm_mat_type aijcusparse \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_ksp_type preonly \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_pc_type gamg \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_0_pc_gamg_reuse_interpolation 0 \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_1_ksp_type preonly \<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">-fieldsplit_1_pc_type jacobi<span class="Apple-converted-space"> </span><o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">- KSP still works fine the first couple of topology optimization iterations but then<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">stops with "Linear solve did not converge due to DIVERGED_DTOL ..".<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">My question is whether I should expect the GPU versions of the linear solvers and pre-conditioners<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">to function exactly as their CPU counterparts (I got this impression from the documentation),<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">in which case I've probably made some mistake in my own code, or whether there are other/additional<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;">settings or modifications I should use to run on the GPU (an NVIDIA Quadro T2000)?<o:p></o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><o:p> </o:p></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="SV">Kind regards,<o:p></o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="SV"><o:p> </o:p></span></div><div style="margin: 0cm; font-size: 11pt; font-family: Calibri, sans-serif;"><span lang="SV">Carl-Johan</span></div></div></div></blockquote></div><br></div></body></html>