<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;"><div><br></div> The examples that use DM, in particular DMDA all trivially support using the GPU with -dm_mat_type aijcusparse -dm_vec_type cuda<div><br></div><div><br><div><br><blockquote type="cite"><div>On Jul 17, 2023, at 1:45 AM, Ng, Cho-Kuen <cho@slac.stanford.edu> wrote:</div><br class="Apple-interchange-newline"><div><meta charset="UTF-8"><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Barry,</div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Thank you so much for the clarification.<span class="Apple-converted-space"> </span><br></div><div class="elementToProof" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I see that ex104.c and ex300.c use MatXAIJSetPreallocation(). Are there other tutorials available?</div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div class="elementToProof ContentPasted0" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div><div id="appendonsend" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"></div><hr tabindex="-1" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; display: inline-block; width: 934.90625px;"><span style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; float: none; display: inline !important;"></span><div id="divRplyFwdMsg" dir="ltr" style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Saturday, July 15, 2023 8:36 AM<br><b>To:</b><span class="Apple-converted-space"> </span>Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu">cho@slac.stanford.edu</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 18px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; line-break: after-white-space;"><div><br></div> <div> Cho,</div><div><br></div><div> We currently have a crappy API for turning on GPU support, and our documentation is misleading in places. </div><div><br></div><div> People constantly say "to use GPU's with PETSc you only need to use -mat_type aijcusparse (for example)" This is incorrect.</div><div><br></div><div> This does not work with code that uses the convenience Mat constructors such as MatCreateAIJ(), MatCreateAIJWithArrays etc. It only works if you use the constructor approach of MatCreate(), MatSetSizes(), MatSetFromOptions(), MatXXXSetPreallocation(). ... Similarly you need to use VecCreate(), VecSetSizes(), VecSetFromOptions() and -vec_type cuda</div><div><br></div><div> If you use DM to create the matrices and vectors then you can use <span style="font-variant-ligatures: no-common-ligatures;">-</span><span style="color: rgb(180, 36, 25); font-variant-ligatures: no-common-ligatures;"><b>dm_mat_type aijcusparse</b></span><span style="font-variant-ligatures: no-common-ligatures;"><span class="Apple-converted-space"> </span>-dm_vec_type cuda</span></div><div><br></div><div> Sorry for the confusion.</div><div><br></div><div> Barry</div><div><br></div><div><br></div><div><br><div><br><blockquote type="cite"><div>On Jul 15, 2023, at 8:03 AM, Matthew Knepley <<a href="mailto:knepley@gmail.com">knepley@gmail.com</a>> wrote:</div><br class="x_Apple-interchange-newline"><div><div dir="ltr"><div dir="ltr">On Sat, Jul 15, 2023 at 1:44 AM Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu">cho@slac.stanford.edu</a>> wrote:<br></div><div class="x_gmail_quote"><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="x_msg2927322207553750716"><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Matt,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">After inserting 2 lines in the code:</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"> ierr = MatCreate(PETSC_COMM_WORLD,&A);CHKERRQ(ierr); <span class="Apple-converted-space"> </span><br><div> ierr = MatSetFromOptions(A);CHKERRQ(ierr);</div><div> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,</div><div> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);</div><div><br></div><div>"There are no unused options." However, there is no improvement on the GPU performance.</div></div></div></div></blockquote><div><br></div><div>1. MatCreateAIJ() sets the type, and in fact it overwrites the Mat you created in steps 1 and 2. This is detailed in the manual.</div><div><br></div><div>2. You should replace MatCreateAIJ(), with MatSetSizes() before MatSetFromOptions().</div><div><br></div><div> THanks,</div><div><br></div><div> Matt</div><div> </div><blockquote class="x_gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div class="x_msg2927322207553750716"><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><div>Thanks,</div><div>Cho<br></div></div><div id="x_m_2927322207553750716appendonsend"></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><hr style="display: inline-block; width: 907.546875px;"><div id="x_m_2927322207553750716divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, July 14, 2023 5:57 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>; Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>;<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div><div dir="ltr"><div dir="ltr">On Fri, Jul 14, 2023 at 7:57 PM Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>> wrote:<br></div><div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I managed to pass the following options to PETSc using a GPU node on Perlmutter.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"> -mat_type aijcusparse -vec_type cuda -log_view -options_left</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Below is a summary of the test using 4 MPI tasks and 1 GPU per task.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">o #PETSc Option Table entries:<div><span> </span>-log_view</div><div><span> </span>-mat_type aijcusparse</div><div> -options_left</div><div> -vec_type cuda</div><div> #End of PETSc Option Table entries</div><div> WARNING! There are options you set that were not used!</div><div> WARNING! could be spelling mistake, etc!</div><div> There is one unused database option. It is:</div><div> Option left: name:-mat_type value: aijcusparse</div><div><br></div><div>The -mat_type option has not been used. In the application code, we use</div><div><br></div><div> ierr = MatCreateAIJ(PETSC_COMM_WORLD,mlocal,mlocal,m,n,<div> d_nz,PETSC_NULL,o_nz,PETSC_NULL,&A);;CHKERRQ(ierr);</div></div></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div></div></div></blockquote><div><br></div><div>If you create the Mat this way, then you need MatSetFromOptions() in order to set the type from the command line.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">o The percent flops on the GPU for KSPSolve is 17%.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">In comparison with a CPU run using 16 MPI tasks, the GPU run is an order of magnitude slower. How can I improve the GPU performance?</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div><div id="x_m_2927322207553750716x_m_-8026140834471843556appendonsend"></div><hr style="display: inline-block; width: 889.984375px;"><div id="x_m_2927322207553750716x_m_-8026140834471843556divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, June 30, 2023 7:57 AM<br><b>To:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>>; Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>;<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Barry, Mark and Matt,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Thank you all for the suggestions. I will modify the code so we can pass runtime options.</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div><div id="x_m_2927322207553750716x_m_-8026140834471843556x_appendonsend"></div><hr style="display: inline-block; width: 889.984375px;"><div id="x_m_2927322207553750716x_m_-8026140834471843556x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Barry Smith <<a href="mailto:bsmith@petsc.dev" target="_blank">bsmith@petsc.dev</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, June 30, 2023 7:01 AM<br><b>To:</b><span class="Apple-converted-space"> </span>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span>Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>>; Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>>;<span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div><div><br></div> Note that options like -mat_type aijcusparse -vec_type cuda only work if the program is set up to allow runtime swapping of matrix and vector types. If you have a call to MatCreateMPIAIJ() or other specific types then then these options do nothing but because Mark had you use -options_left the program will tell you at the end that it did not use the option so you will know.<div><br><blockquote type="cite"><div>On Jun 30, 2023, at 9:30 AM, Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:</div><br><div><div dir="ltr">PetscCall(PetscInitialize(&argc, &argv, NULL, help)); gives us the args and you run:<br><div><br></div><div>a.out -mat_type aijcusparse -vec_type cuda -log_view -options_left</div><div><br></div><div>Mark</div></div><br><div><div dir="ltr">On Fri, Jun 30, 2023 at 6:16 AM Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br></div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div dir="ltr"><div dir="ltr">On Fri, Jun 30, 2023 at 1:13 AM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Mark,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">The application code reads in parameters from an input file, where we can put the PETSc runtime options. Then we pass the options to PetscInitialize(...). Does that sounds right?</div></div></div></blockquote><div><br></div><div>PETSc will read command line argument automatically in PetscInitialize() unless you shut it off.</div><div><br></div><div> Thanks,</div><div><br></div><div> Matt</div><div> </div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div><div id="x_m_2927322207553750716x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577appendonsend"></div><hr style="display: inline-block; width: 845.0625px;"><div id="x_m_2927322207553750716x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Thursday, June 29, 2023 8:32 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Mark,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks for the information. How do I put the runtime options for the executable, say, a.out, which does not have the provision to append arguments? Do I need to change the C++ main to read in the options?</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div><div id="x_m_2927322207553750716x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_appendonsend"></div><hr style="display: inline-block; width: 845.0625px;"><div id="x_m_2927322207553750716x_m_-8026140834471843556x_x_m_-2363626647532502450m_5514417947815199577x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size: 11pt;"><b>From:</b><span class="Apple-converted-space"> </span>Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>><br><b>Sent:</b><span class="Apple-converted-space"> </span>Thursday, June 29, 2023 5:55 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Ng, Cho-Kuen <<a href="mailto:cho@slac.stanford.edu" target="_blank">cho@slac.stanford.edu</a>><br><b>Cc:</b><span class="Apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><span class="Apple-converted-space"> </span><<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [petsc-users] Using PETSc GPU backend</font><div> </div></div><div><div dir="ltr">Run with options: -mat_type aijcusparse -vec_type cuda -log_view -options_left<div><br></div><div>The last column of the performance data (from -log_view) will be the percent flops on the GPU. Check that that is > 0.</div><div><br></div><div>The end of the output will list the options that were used and options that were _not_ used (if any). Check that there are no options left.</div><div><br></div><div>Mark</div></div><br><div><div dir="ltr">On Thu, Jun 29, 2023 at 7:50 PM Ng, Cho-Kuen via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br></div><blockquote style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-style: solid; border-left-color: rgb(204, 204, 204); padding-left: 1ex;"><div><div dir="ltr"><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">I installed PETSc on Perlmutter using "spack install<span class="Apple-converted-space"> </span><span style="background: rgb(255, 255, 255);">petsc+cuda+zoltan</span><span style="background-color: rgb(255, 255, 255);">"<span class="Apple-converted-space"> </span></span>and used it by "<span style="font-family: Arial, Helvetica, sans-serif; font-size: 12pt; background: rgb(255, 255, 255);">spack load petsc/fwge6pf</span>". Then I compiled the application code (purely CPU code) linking to the petsc package, hoping that I can get performance improvement using the petsc GPU backend. However, the timing was the same using the same number of MPI tasks with and without GPU accelerators. Have I missed something in the process, for example, setting up PETSc options at runtime to use the GPU backend?</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;"><br></div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Thanks,</div><div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt;">Cho<br></div></div></div></blockquote></div></div></div></div></div></blockquote></div><br clear="all"><div><br></div><span>--<span class="Apple-converted-space"> </span></span><br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></blockquote></div></div></blockquote></div><br></div></div></div></div></blockquote></div><br clear="all"><div><br></div><span>--<span class="Apple-converted-space"> </span></span><br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a><br></div></div></div></div></div></div></div></div></div></div></div></blockquote></div><br clear="all"><div><br></div><span class="x_gmail_signature_prefix">--<span class="Apple-converted-space"> </span></span><br><div dir="ltr" class="x_gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>-- Norbert Wiener</div><div><br></div><div><a href="http://www.cse.buffalo.edu/~knepley/" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div></div></div></div></div></div></div></div></div></blockquote></div></div></div></div></blockquote></div><br></div></body></html>