<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Aug 17, 2020, at 3:24 PM, nicola varini <<a href="mailto:nicola.varini@gmail.com" class="">nicola.varini@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class=""><div class="">Hi Mark, I do confirm that hypre with boomeramg is working fine and is pretty fast. <br class=""></div><div class="">However, none of the GAMG option works.<br class=""></div><div class="">Did anyone ever succeeded in usign hypre with petsc on gpu?<br class=""></div><div class="">I did manage to compile hypre on gpu but I do get the following error:<br class=""></div><div class="">=======<br class=""> CC gpuhypre/obj/vec/vec/impls/hypre/vhyp.o<br class="">In file included from /opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/detail/config.h:22,<br class=""> from /opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/execution_policy.h:23,<br class=""> from /users/nvarini/hypre/include/_hypre_utilities.h:1129,<br class=""> from /users/nvarini/hypre/include/_hypre_IJ_mv.h:14,<br class=""> from /scratch/snx3000/nvarini/petsc-3.13.3/include/../src/vec/vec/impls/hypre/vhyp.h:6,<br class=""> from /scratch/snx3000/nvarini/petsc-3.13.3/src/vec/vec/impls/hypre/vhyp.c:7:<br class="">/opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/version.h:83:1: error: unknown type name 'namespace'<br class=""> namespace thrust<br class=""> ^~~~~~~~~<br class="">/opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/version.h:84:1: error: expected '=', ',', ';', 'asm' or '__attribute__' before '{' token<br class=""> {<br class=""> ^<br class="">In file included from /opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/detail/config/config.h:28,<br class=""> from /opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/detail/config.h:23,<br class=""> from /opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/execution_policy.h:23,<br class=""> from /users/nvarini/hypre/include/_hypre_utilities.h:1129,<br class=""> from /users/nvarini/hypre/include/_hypre_IJ_mv.h:14,<br class=""> from /scratch/snx3000/nvarini/petsc-3.13.3/include/../src/vec/vec/impls/hypre/vhyp.h:6,<br class=""> from /scratch/snx3000/nvarini/petsc-3.13.3/src/vec/vec/impls/hypre/vhyp.c:7:<br class="">/opt/nvidia/cudatoolkit10/10.1.105_3.27-7.0.1.1_4.1__ga311ce7/include/thrust/detail/config/cpp_compatibility.h:21:10: fatal error: cstddef: No such file or directory<br class=""> #include <cstddef><br class=""> ^~~~~~~~~<br class="">compilation terminated.<br class=""></div></div></div></blockquote><div><br class=""></div><div>Nicola,</div><div><br class=""></div><div>Interfacing PETSc with HYPRE-GPU is still work in progress.</div><div><br class=""></div><div>In the meantime, if you want to play with it, you can checkout the PETSc branch <a href="https://gitlab.com/petsc/petsc/-/tree/stefanozampini/hypre-cuda-rebased-v3" class="">https://gitlab.com/petsc/petsc/-/tree/stefanozampini/hypre-cuda-rebased-v3</a> and the HYPRE branch <a href="https://github.com/hypre-space/hypre/tree/PETScFix" class="">https://github.com/hypre-space/hypre/tree/PETScFix</a> and start experimenting.</div><div>However, I do not guarantee it will work properly; for boomerAMG to work on GPU, you must use cudaUnifiedMemory (default memory allocator in the PETSc branch),and only PMIS interpolation and Jacobi smoothing are available on GPU. Ruipeng (in cc) can confirm.</div><div><br class=""></div><div>I will resume working on this thing in the next weeks and I hope it will be stable soon.</div><div><br class=""></div><div>Regards,</div><div>Stefano</div><br class=""><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class=""><br class="">=======<br class=""></div><div class="">Nicola<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno ven 14 ago 2020 alle ore 20:13 Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">You can try Hypre. If that fails then there is a problem with your system.<div class=""><br class=""><div class="">And you can run with -info and grep on GAMG and send the output and I can see if I see anything funny.</div><div class=""><br class=""></div><div class="">If this is just a Lapacian with a stable discretization and not crazy material parameters then stretched grids are about the only thing that can hurt the solver.</div></div><div class=""><br class=""></div><div class="">Do both of your solves fail in a similar way?</div><div class=""><br class=""></div><div class="">On the CPU you can try this with large subdomains, preferably (in serial ideally):</div><div class="">-ampere_mg_levels_ksp_type richardson<br class="">-ampere_mg_levels_pc_type sor<br class=""></div><div class=""><br class=""></div><div class="">And check that there are no unused options with -options_left. GAMG can fail with bad eigen estimates, but these parameters look fine.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 14, 2020 at 5:01 AM nicola varini <<a href="mailto:nicola.varini@gmail.com" target="_blank" class="">nicola.varini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">Dear Barry, yes it gives the same problems. <br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 13 ago 2020 alle ore 23:22 Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> Does the same thing work (with GAMG) if you run on the same problem on the same machine same number of MPI ranks but make a new PETSC_ARCH that does NOT use the GPUs?<div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""> Ideally one gets almost identical convergence with CPUs or GPUs (same problem, same machine) but a bug or numerically change "might" affect this.<br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 13, 2020, at 10:28 AM, nicola varini <<a href="mailto:nicola.varini@gmail.com" target="_blank" class="">nicola.varini@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class="">Dear Barry, you are right. The Cray argument checking is incorrect. It does work with download-fblaslapack.<br class=""></div>However it does fail to converge. Is there anything obviously wrong with my petscrc? <br class=""></div>Anything else am I missing?<br class=""><br class=""></div>Thanks<br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 13 ago 2020 alle ore 03:17 Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""><br class=""></div> The QR is always done on the CPU, we don't have generic calls to blas/lapack go to the GPU currently.<div class=""><br class=""></div><div class=""> The error message is:</div><div class=""><br class=""></div><div class=""> On entry to __cray_mgm_dgeqrf, parameter 7 had an illegal value (info = -7)</div><div class=""><br class=""></div><div class=""> argument 7 is &LWORK which is defined by</div><div class=""><br class=""></div><div class=""> PetscBLASInt LWORK=N*bs;</div><div class=""><br class=""></div><div class=""> and</div><div class=""><br class=""></div><div class=""> N=nSAvec is the column block size of new P.</div><div class=""><br class=""></div><div class=""> Presumably this is a huge run with many processes so using the debugger is not practical?</div><div class=""><br class=""></div><div class=""> We need to see what these variables are</div><div class=""><br class=""></div><div class=""> N, bs, nSAvec </div><div class=""><br class=""></div><div class=""> perhaps nSAvec is zero which could easily upset LAPACK. </div><div class=""><br class=""></div><div class=""> Crudest thing would be to just put a print statement in the code before the LAPACK call of if they are called many times add an error check like that</div><div class=""> generates an error if any of these three values are 0 (or negative).</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""> It is not impossible that the Cray argument checking is incorrect and the value passed in is fine. You can check this by using --download-fblaslapack and see if the same or some other error comes up.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 12, 2020, at 7:19 PM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class="">Can you reproduce this on the CPU? <div class="">The QR factorization seems to be failing. That could be from bad data or a bad GPU QR.</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Aug 12, 2020 at 4:19 AM nicola varini <<a href="mailto:nicola.varini@gmail.com" target="_blank" class="">nicola.varini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div class="">Dear all, following the suggestions I did resubmit the simulation with the petscrc below.<br class=""></div>However I do get the following error:<br class="">========<br class=""><div class=""> 7362 [592]PETSC ERROR: #1 formProl0() line 748 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/agg.c<br class=""> 7363 [339]PETSC ERROR: Petsc has generated inconsistent data<br class=""> 7364 [339]PETSC ERROR: xGEQRF error<br class=""> 7365 [339]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" target="_blank" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.<br class=""> 7366 [339]PETSC ERROR: Petsc Release Version 3.13.3, Jul 01, 2020<br class=""> 7367 [339]PETSC ERROR: /users/nvarini/gbs_test_nicola/bin/gbs_daint_gpu_gnu on a named nid05083 by nvarini Wed Aug 12 10:06:15 2020<br class=""> 7368 [339]PETSC ERROR: Configure options --with-cc=cc --with-fc=ftn --known-mpi-shared-libraries=1 --known-mpi-c-double-complex=1 --known-mpi-int64_t=1 --known-mpi-long-double=1 --with-batch=1 --known-64-bit-blas-indices=0 --LIBS=-lstdc++ --with-cxxlib-autodetect=0 --with-scalapa ck=1 --with-cxx=CC --with-debugging=0 --with-hypre-dir=/opt/cray/pe/tpsl/19.06.1/GNU/8.2/haswell --prefix=/scratch/snx3000/nvarini/petsc3.13.3-gpu --with-cuda=1 --with-cuda-c=nvcc --with-cxxlib-autodetect=0 --COPTFLAGS=-I/opt/cray/pe/mpt/7.7.10/gni/mpich-intel/16.0/include - -with-cxx=CC --CXXOPTFLAGS=-I/opt/cray/pe/mpt/7.7.10/gni/mpich-intel/16.0/include<br class=""> 7369 [592]PETSC ERROR: #2 PCGAMGProlongator_AGG() line 1063 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/agg.c<br class=""> 7370 [592]PETSC ERROR: #3 PCSetUp_GAMG() line 548 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/gamg.c<br class=""> 7371 [592]PETSC ERROR: #4 PCSetUp() line 898 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/interface/precon.c<br class=""> 7372 [592]PETSC ERROR: #5 KSPSetUp() line 376 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7373 [592]PETSC ERROR: #6 KSPSolve_Private() line 633 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7374 [316]PETSC ERROR: #3 PCSetUp_GAMG() line 548 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/gamg.c<br class=""> 7375 [339]PETSC ERROR: #1 formProl0() line 748 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/agg.c<br class=""> 7376 [339]PETSC ERROR: #2 PCGAMGProlongator_AGG() line 1063 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/agg.c<br class=""> 7377 [339]PETSC ERROR: #3 PCSetUp_GAMG() line 548 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/gamg.c<br class=""> 7378 [339]PETSC ERROR: #4 PCSetUp() line 898 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/interface/precon.c<br class=""> 7379 [339]PETSC ERROR: #5 KSPSetUp() line 376 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7380 [592]PETSC ERROR: #7 KSPSolve() line 853 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7381 [339]PETSC ERROR: #6 KSPSolve_Private() line 633 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7382 [339]PETSC ERROR: #7 KSPSolve() line 853 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/ksp/interface/itfunc.c<br class=""> 7383 On entry to __cray_mgm_dgeqrf, parameter 7 had an illegal value (info = -7)<br class=""> 7384 [160]PETSC ERROR: #3 PCSetUp_GAMG() line 548 in /scratch/snx3000/nvarini/petsc-3.13.3/src/ksp/pc/impls/gamg/gamg.c<br class="">========<br class=""></div><div class=""><br class=""></div><div class="">I did try other pc_gamg_type but they fails as well.<br class=""></div><div class=""><br class=""></div><div class=""><br class="">#PETSc Option Table entries:<br class="">-ampere_dm_mat_type aijcusparse<br class="">-ampere_dm_vec_type cuda<br class="">-ampere_ksp_atol 1e-15<br class="">-ampere_ksp_initial_guess_nonzero yes<br class="">-ampere_ksp_reuse_preconditioner yes<br class="">-ampere_ksp_rtol 1e-7<br class="">-ampere_ksp_type dgmres<br class="">-ampere_mg_levels_esteig_ksp_max_it 10<br class="">-ampere_mg_levels_esteig_ksp_type cg<br class="">-ampere_mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05<br class="">-ampere_mg_levels_ksp_type chebyshev<br class="">-ampere_mg_levels_pc_type jacobi<br class="">-ampere_pc_gamg_agg_nsmooths 1<br class="">-ampere_pc_gamg_coarse_eq_limit 10<br class="">-ampere_pc_gamg_reuse_interpolation true<br class="">-ampere_pc_gamg_square_graph 1<br class="">-ampere_pc_gamg_threshold 0.05<br class="">-ampere_pc_gamg_threshold_scale .0<br class="">-ampere_pc_gamg_type agg<br class="">-ampere_pc_type gamg<br class="">-dm_mat_type aijcusparse<br class="">-dm_vec_type cuda<br class="">-log_view<br class="">-poisson_dm_mat_type aijcusparse<br class="">-poisson_dm_vec_type cuda<br class="">-poisson_ksp_atol 1e-15<br class="">-poisson_ksp_initial_guess_nonzero yes<br class="">-poisson_ksp_reuse_preconditioner yes<br class="">-poisson_ksp_rtol 1e-7<br class="">-poisson_ksp_type dgmres<br class="">-poisson_log_view<br class="">-poisson_mg_levels_esteig_ksp_max_it 10<br class="">-poisson_mg_levels_esteig_ksp_type cg<br class="">-poisson_mg_levels_ksp_chebyshev_esteig 0,0.05,0,1.05<br class="">-poisson_mg_levels_ksp_max_it 1<br class="">-poisson_mg_levels_ksp_type chebyshev<br class="">-poisson_mg_levels_pc_type jacobi<br class="">-poisson_pc_gamg_agg_nsmooths 1<br class="">-poisson_pc_gamg_coarse_eq_limit 10<br class="">-poisson_pc_gamg_reuse_interpolation true<br class="">-poisson_pc_gamg_square_graph 1<br class="">-poisson_pc_gamg_threshold 0.05<br class="">-poisson_pc_gamg_threshold_scale .0<br class="">-poisson_pc_gamg_type agg<br class="">-poisson_pc_type gamg<br class=""><div class="">-use_mat_nearnullspace true</div><div class="">#End of PETSc Option Table entries<br class=""></div></div><div class=""><br class=""></div><div class="">Regards,<br class=""><br class=""></div><div class="">Nicola<br class=""></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno mar 4 ago 2020 alle ore 17:57 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class=""><div dir="ltr" class=""><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 4, 2020 at 6:35 AM Stefano Zampini <<a href="mailto:stefano.zampini@gmail.com" target="_blank" class="">stefano.zampini@gmail.com</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class="">Nicola,</div><div class=""><br class=""></div><div class="">You are actually not using the GPU properly, since you use HYPRE preconditioning, which is CPU only. One of your solvers is actually slower on “GPU”.</div><div class="">For a full AMG GPU, you can use PCGAMG, with cheby smoothers and with Jacobi preconditioning. Mark can help you out with the specific command line options.</div><div class="">When it works properly, everything related to PC application is offloaded to the GPU, and you should expect to get the well-known and branded 10x (maybe more) speedup one is expecting from GPUs during KSPSolve</div><div class=""><br class=""></div></div></blockquote><div class=""><br class=""></div><div class="">The speedup depends on the machine, but on SUMMIT, using enough CPUs to saturate the memory bus vs all 6 GPUs the speedup is a function of problem subdomain size. I saw 10x at about 100K equations/process.</div><div class=""> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class=""><div class=""></div><div class="">Doing what you want to do is one of the last optimization steps of an already optimized code before entering production. Yours is not even optimized for proper GPU usage yet.</div><div class="">Also, any specific reason why you are using dgmres and fgmres?</div><div class=""><br class=""></div><div class="">PETSc has not been designed with multi-threading in mind. You can achieve “overlap” of the two solves by splitting the communicator. But then you need communications to let the two solutions talk to each other.</div><div class=""><br class=""></div><div class="">Thanks</div><div class="">Stefano</div><div class=""><br class=""><div class=""><br class=""><blockquote type="cite" class=""><div class="">On Aug 4, 2020, at 12:04 PM, nicola varini <<a href="mailto:nicola.varini@gmail.com" target="_blank" class="">nicola.varini@gmail.com</a>> wrote:</div><br class=""><div class=""><div dir="ltr" class=""><div class=""><div class=""><div class=""><div class=""><div class="">Dear all, thanks for your replies. The reason why I've asked if it is possible to overlap poisson and ampere is because they roughly<br class=""></div>take the same amount of time. Please find in attachment the profiling logs for only CPU and only GPU.<br class=""></div>Of course it is possible to split the MPI communicator and run each solver on different subcommunicator, however this would involve more communication.<br class=""></div>Did anyone ever tried to run 2 solvers with hyperthreading? <br class=""></div><div class="">Thanks<br class=""></div></div><div class=""><div class=""><div class=""><div class=""><br class=""></div></div></div></div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno dom 2 ago 2020 alle ore 14:09 Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> ha scritto:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">I suspect that the Poisson and Ampere's law solve are not coupled. You might be able to duplicate the communicator and use two threads. You would want to configure PETSc with threadsafty and threads and I think it could/should work, but this mode is never used by anyone.<br class=""><div class=""><br class=""></div><div class="">That said, I would not recommend doing this unless you feel like playing in computer science, as opposed to doing application science. The best case scenario you get a speedup of 2x. That is a strict upper bound, but you will never come close to it. Your hardware has some balance of CPU to GPU processing rate. Your application has a balance of volume of work for your two solves. They have to be the same to get close to 2x speedup and that ratio(s) has to be 1:1. To be concrete, from what little I can guess about your applications let's assume that the cost of each of these two solves is about the same (eg, Laplacians on your domain and the best case scenario). But, GPU machines are configured to have roughly 1-10% of capacity in the GPUs, these days, that gives you an upper bound of about 10% speedup. That is noise. Upshot, unless you configure your hardware to match this problem, and the two solves have the same cost, you will not see close to 2x speedup. Your time is better spent elsewhere.</div><div class=""><br class=""></div><div class="">Mark</div></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Aug 1, 2020 at 3:24 PM Jed Brown <<a href="mailto:jed@jedbrown.org" target="_blank" class="">jed@jedbrown.org</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">You can use MPI and split the communicator so n-1 ranks create a DMDA for one part of your system and the other rank drives the GPU in the other part. They can all be part of the same coupled system on the full communicator, but PETSc doesn't currently support some ranks having their Vec arrays on GPU and others on host, so you'd be paying host-device transfer costs on each iteration (and that might swamp any performance benefit you would have gotten).<br class="">
<br class="">
In any case, be sure to think about the execution time of each part. Load balancing with matching time-to-solution for each part can be really hard.<br class="">
<br class="">
<br class="">
Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank" class="">bsmith@petsc.dev</a>> writes:<br class="">
<br class="">
> Nicola,<br class="">
><br class="">
> This is really viable or practical at this time with PETSc. It is not impossible but requires careful coding with threads, another possibility is to use one half of the virtual GPUs for each solve, this is also not trivial. I would recommend first seeing what kind of performance you can get on the GPU for each type of solve and revist this idea in the future.<br class="">
><br class="">
> Barry<br class="">
><br class="">
><br class="">
><br class="">
><br class="">
>> On Jul 31, 2020, at 9:23 AM, nicola varini <<a href="mailto:nicola.varini@gmail.com" target="_blank" class="">nicola.varini@gmail.com</a>> wrote:<br class="">
>> <br class="">
>> Hello, I would like to know if it is possible to overlap CPU and GPU with DMDA.<br class="">
>> I've a machine where each node has 1P100+1Haswell.<br class="">
>> I've to resolve Poisson and Ampere equation for each time step.<br class="">
>> I'm using 2D DMDA for each of them. Would be possible to compute poisson <br class="">
>> and ampere equation at the same time? One on CPU and the other on GPU?<br class="">
>> <br class="">
>> Thanks<br class="">
</blockquote></div>
</blockquote></div>
<span id="gmail-m_-287257760015557249gmail-m_7295191563877526157gmail-m_7509251495650781420gmail-m_3795355602041451874gmail-m_-2894221877128162253gmail-m_-630451931610969904gmail-m_5452938391528559121gmail-m_-3233498012180817199cid:f_kdfro0bi0" class=""><out_gpu></span><span id="gmail-m_-287257760015557249gmail-m_7295191563877526157gmail-m_7509251495650781420gmail-m_3795355602041451874gmail-m_-2894221877128162253gmail-m_-630451931610969904gmail-m_5452938391528559121gmail-m_-3233498012180817199cid:f_kdfro8hf1" class=""><out_nogpu></span></div></blockquote></div><br class=""></div></div></blockquote></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></div></blockquote></div>
<span id="gmail-m_-287257760015557249gmail-m_7295191563877526157gmail-m_7509251495650781420gmail-m_3795355602041451874cid:f_kdsygc0w0" class=""><out_miniapp_f_poisson></span></div></blockquote></div><br class=""></div></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></body></html>