<div dir="ltr"><div dir="ltr"><div>Treb, putting this on the list.</div><div><br></div><div>Treb has ECP early access to Frontier and has some problems:</div><div><br></div><div>** first he has error from hypre:</div><div><br></div><div>[0]PETSC ERROR: #1 VecGetArrayForHYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/vec/vec/impls/hypre/vhyp.c:95<br></div><div><br></div><div>We had another stack trace that I can not find that came from a Vec routine, copy to the device as I recall.</div><div><br></div><div>** The hypre folks could not do much with that so I suggested using aijhipsparse and he got this error message.</div><div><br></div><div>Looks like just a segv in MatAssemblyEnd_SeqAIJ.</div><div><br></div><div>Treb, </div><div>1) this error might be reproducible on one processor. Could you try to scale this problem down.</div><div>2) I assume this was built with debugging=1</div><div>3) if you can get it to fail on one process then you might be able to get a good stack trace with a line number with a debugger. GDB is available (on Crusher) but you need to do a few things.</div><div>4) You might see if you can get some AMD help.<br></div><div><br></div><div>Thanks,</div><div>Mark</div><div><br></div><div><br></div><div><br></div><div><br style="color:rgb(80,0,80)"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 27, 2023 at 4:46 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hey Mark-<br><div>This is a new issue that doesn't seem to be hypre. It's not using the -mat_type aijhipsparse in this run. Can you interpret these petsc errors? Seems like it's just crashing. Wasn't doing this last night. I was using less nodes however.</div><div>[39872]PETSC ERROR: ------------------------------------------------------------------------<br>[39872]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38267] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 1<br> -is_view ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br>[42800]PETSC ERROR: ------------------------------------------------------------------------<br>[38266] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[17088]PETSC ERROR: ------------------------------------------------------------------------<br>[17088]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>----------------------------------------<br>Viewer (-is_view) options:<br>[41728]PETSC ERROR: ------------------------------------------------------------------------<br>[38267] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[10256]PETSC ERROR: ------------------------------------------------------------------------<br>[10256]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br> -is_view draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br>[56]PETSC ERROR: ------------------------------------------------------------------------<br>[38270] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[41128]PETSC ERROR: ------------------------------------------------------------------------<br> -is_view binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br>[42496]PETSC ERROR: ------------------------------------------------------------------------<br>[42496]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38265] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 32768; storage space: 0 unneeded,378944 used<br>[10256]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br> -is_view ascii[:[filename][:[format][:append]]]: Prints object to stdout or ASCII file (PetscOptionsGetViewer)<br>[24]PETSC ERROR: ------------------------------------------------------------------------<br>[24]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38268] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 32768; storage space: 0 unneeded,378944 used<br>[4128]PETSC ERROR: ------------------------------------------------------------------------<br> -is_view socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br>[60]PETSC ERROR: ------------------------------------------------------------------------<br>[60]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38269] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 32768; storage space: 0 unneeded,378944 used<br>[10260]PETSC ERROR: ------------------------------------------------------------------------<br> -is_view draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br>[28]PETSC ERROR: ------------------------------------------------------------------------<br>[28]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38265] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br></div><div>[4132]PETSC ERROR: ------------------------------------------------------------------------<br> -is_view binary[:[filename][:[format][:append]]]: Saves object to a binary file (PetscOptionsGetViewer)<br>[56]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38268] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br></div><div>MPICH ERROR [Rank 10260] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier01491] - Abort(59) (rank 10260 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process\<br> 10260<br><br>aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 10260<br> -is_view draw[:[drawtype][:filename|format]] Draws object (PetscOptionsGetViewer)<br>[28]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38268] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 196<br>[4132]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br> -is_view saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>[60]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38269] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 196<br>[24]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br> -is_view socket[:port]: Pushes object to a Unix socket (PetscOptionsGetViewer)<br>[4128]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38271] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>[56]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br> -is_view saws[:communicatorname]: Publishes object to SAWs (PetscOptionsGetViewer)<br>[13736]PETSC ERROR: ------------------------------------------------------------------------<br>[13736]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38265] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[28]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[6801] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>[4132]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38268] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[60]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[6800] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>[13740]PETSC ERROR: ------------------------------------------------------------------------<br>[13740]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br>[38264] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 32768; storage space: 0 unneeded,378944 used<br>[24]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[6804] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>[4128]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[38266] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br></div><div>[56]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[6805] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>[13736]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38269] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[28]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[6802] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 0; storage space: 0 unneeded,0 used<br>[4132]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[38264] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>[60]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[6806] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 0; storage space: 0 unneeded,0 used<br>[13740]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[38267] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>[24]PETSC ERROR: to get more information on the crash.<br>[6802] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>[4128]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[38270] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>[56]PETSC ERROR: to get more information on the crash.<br>[6803] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 0; storage space: 0 unneeded,0 used<br>[13736]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[13736]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[38264] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 196<br>[28]PETSC ERROR: to get more information on the crash.<br>[6806] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>[4128]PETSC ERROR: to get more information on the crash.<br>[38264] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 0)/(num_localrows 32768) < 0.6. Do not use CompressedRow routines.<br>[60]PETSC ERROR: to get more information on the crash.<br>[6802] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0<br>[13740]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[38265] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>MPICH ERROR [Rank 24] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier00021] - Abort(59) (rank 24 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 24<br><br>[6807] <mat> MatAssemblyEnd_SeqAIJ(): Matrix size: 32768 X 0; storage space: 0 unneeded,0 used<br>[4132]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[38268] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>MPICH ERROR [Rank 56] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier00025] - Abort(59) (rank 56 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 56<br><br>[6803] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>[13736]PETSC ERROR: to get more information on the crash.<br></div><div>[38269] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>MPICH ERROR [Rank 28] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier00021] - Abort(59) (rank 28 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28<br><br>[6806] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0<br>[4132]PETSC ERROR: to get more information on the crash.<br>[38264] <mat> MatSeqAIJCheckInode(): Found 32768 nodes out of 32768 rows. Not using Inode routines<br>MPICH ERROR [Rank 60] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier00025] - Abort(59) (rank 60 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 60<br><br>aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 60<br>[6807] <mat> MatAssemblyEnd_SeqAIJ(): Number of mallocs during MatSetValues() is 0<br>[13740]PETSC ERROR: configure using --with-debugging=yes, recompile, link, and run<br>[38266] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782<br>aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 24<br>[6802] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 32768)/(num_localrows 32768) > 0.6. Use CompressedRow routines.<br>MPICH ERROR [Rank 4128] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier00609] - Abort(59) (rank 4128 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process 4\<br>128<br>[38271] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 56<br>[6803] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0<br>MPICH ERROR [Rank 13736] [job id 1277040.1] [Sun Feb 26 22:32:21 2023] [frontier01996] - Abort(59) (rank 13736 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 59) - process\<br> 13736<br><br>[38267] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374782<br>aborting job:<br>application called MPI_Abort(MPI_COMM_WORLD, 59) - process 28<br>[6806] <mat> MatCheckCompressedRow(): Found the ratio (num_zerorows 32768)/(num_localrows 32768) > 0.6. Use CompressedRow routines.<br><br>[38270] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374781<br>[13740]PETSC ERROR: to get more information on the crash.<br>[6807] <mat> MatAssemblyEnd_SeqAIJ(): Maximum nonzeros in any row is 0<br>[4160]PETSC ERROR: ------------------------------------------------------------------------<br>[4160]PETSC ERROR: Caught signal number 15 Terminate: Some process (or the batch system) has told this process to end<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 25, 2023 at 11:01 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">There is something here. It looks like an error from hypre, but you do have some sort of stack trace.<div>PETSc is catching an error here:</div><div><br></div><div>[0]PETSC ERROR: #1 VecGetArrayForHYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/vec/vec/impls/hypre/vhyp.c:95<br></div><div><br></div><div>You might send this whole output to PETSc and see if someone can help.</div><div><br></div><div>Mark</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Feb 25, 2023 at 12:37 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">from the 8192 node run:<br><div>[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[32016] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[30655] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Invalid argument<br>[32017] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[30653] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: HYPRE_MEMORY_DEVICE expects a device vector. You need to enable PETSc device support, for example, in some cases, -vec_type cuda<br>[32018] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[30649] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[0]PETSC ERROR: WARNING! There are option(s) set that were not used! Could be the program crashed before they were used or a spelling mistake, etc!<br>[32021] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[30654] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Option left: name:-diff_ksp_converged_reason (no value)<br>[32019] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[30652] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Option left: name:-diff_ksp_max_it value: 50<br>[32023] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[30650] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Option left: name:-diff_ksp_norm_type value: unpreconditioned<br>[32020] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[30648] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[0]PETSC ERROR: Option left: name:-diff_ksp_rtol value: 1.e-6<br>[32022] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[32021] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Option left: name:-diff_ksp_type value: gmres<br>[609] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[32017] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[0]PETSC ERROR: Option left: name:-diff_pc_type value: jacobi<br>[611] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[32018] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: Option left: name:-options_left (no value)<br>[615] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[32016] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[0]PETSC ERROR: Option left: name:-proj-mac_mat_type value: aijhipsparse<br>[608] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[610] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br></div><div>then further down</div><div>[10191] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[1978] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #1 VecGetArrayForHYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/vec/vec/impls/hypre/vhyp.c:95<br>[10184] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[1977] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[0]PETSC ERROR: #2 VecHYPRE_IJVectorPushVecRead() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/vec/vec/impls/hypre/vhyp.c:138<br>[10185] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[10186] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #3 PCApply_HYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/pc/impls/hypre/hypre.c:433<br>[6081] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[10188] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #4 PCApply() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/pc/interface/precon.c:441<br>[6083] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10189] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #5 PCApplyBAorAB() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/pc/interface/precon.c:711<br>[6085] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10190] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #6 KSP_PCApplyBAorAB() at /gpfs/alpine/world-shared/geo127/petsc_treb/petsc/include/petsc/private/kspimpl.h:416<br>[6086] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10191] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #7 KSPGMRESCycle() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/ksp/impls/gmres/gmres.c:147<br>[6087] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10187] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[0]PETSC ERROR: #8 KSPSolve_GMRES() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/ksp/impls/gmres/gmres.c:228<br>[6080] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[10184] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374780<br>[0]PETSC ERROR: #9 KSPSolve_Private() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/ksp/interface/itfunc.c:899<br>[6082] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10185] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374778<br>[0]PETSC ERROR: #10 KSPSolve() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/ksp/ksp/interface/itfunc.c:1071<br>[6084] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br>[10189] <sys> PetscCommDuplicate(): Using internal PETSc communicator 1140850689 -2080374783<br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 3:57 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">good idea. the global one is unused for small problems. waiting for large job to run to see if this fixes that problem.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 11:04 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I think you added the prefixes like a year ago, so the prefixes should work. <div>Try both and see which one is used with -options_left</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 1:14 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I am using 3.18.4.<br><div><br></div><div>Is aijhipsparse the global -mat_type or should this be the prefixed one for the solve where I was getting the problem, i.e., -proj_mac_mat_type aijhipsparse</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 9:28 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Oh, its 'aijhipsparse'<div>And you def want v3.18.4</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 11:29 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I rana small problem with<div>-proj_mac_mat_type hipsparse<br></div><div>and get</div><div>[10]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<br>[10]PETSC ERROR: Unknown type. Check for miss-spelling or missing package: <a href="https://petsc.org/release/install/install/#external-packages" target="_blank">https://petsc.org/release/install/install/#external-packages</a><br>[10]PETSC ERROR: Unknown Mat type given: hipsparse<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 4:09 AM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 24, 2023 at 3:35 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">More info from the stack. This is a full machine run on Frontier and I get this before I get into the first solve. It may or may not be same error as before but hopefully there's more here for you to debug.<div>[1540]PETSC ERROR: ------------------------------------------------------------------------<br>[1540]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>[1540]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>[1540]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" target="_blank">https://petsc.org/release/faq/#valgrind</a> and <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a><br>[1540]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>[1540]PETSC ERROR: The line numbers in the error traceback are not always exact.<br>[1540]PETSC ERROR: #1 hypre_ParCSRMatrixMigrate()<br>[1540]PETSC ERROR: #2 MatBindToCPU_HYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/mat/impls/hypre/mhypre.c:1255<br></div></div></blockquote><div><br></div><div>This looks like the copy to device call:</div><div><br></div><div>src/mat/impls/hypre/mhypre.c:1260: PetscCallExternal(hypre_ParCSRMatrixMigrate, parcsr, hmem);<br></div><div><br></div><div>This makes sense. You assemble it in the host and it gets sent to the device.</div><div><br></div><div>I assume you are using -mat_type hypre.</div><div>To get moving you could try -mat_type hipsparse </div><div> </div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>[1540]PETSC ERROR: #3 MatAssemblyEnd_HYPRE() at /gpfs/alpine/geo127/world-shared/petsc_treb/petsc/src/mat/impls/hypre/mhypre.c:1332<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Feb 22, 2023 at 9:26 AM Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi David,<br>
<br>
I am not sure how much information I get here for this segfault. All I can see is you wanted to migrate (copy) a matrix (on device?) to host, and it failed at somewhere in the function. The function itself looks simple and fine to me. We may need to check if everything is sane prior to the point. I am happy to help further.<br>
<br>
Thanks<br>
<br>
-Rui Peng<br>
<br>
________________________________________<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>><br>
Sent: Wednesday, February 22, 2023 9:17 AM<br>
To: Yang, Ulrike Meier<br>
Cc: Li, Rui Peng; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><br>
Subject: Re: Frontier<br>
<br>
Hi Ulrike, Rui Peng-<br>
<br>
I am running into a hypre problem on Frontier. I already passed it by Mark and here is what we get out of the stack:<br>
[1704]PETSC ERROR: ------------------------------------------------------------------------<br>
[1704]PETSC ERROR: Caught signal number 11 SEGV: Segmentation Violation, probably memory access out of range<br>
[1704]PETSC ERROR: Try option -start_in_debugger or -on_error_attach_debugger<br>
[1704]PETSC ERROR: or see <a href="https://petsc.org/release/faq/#valgrind" rel="noreferrer" target="_blank">https://petsc.org/release/faq/#valgrind</a><<a href="https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-Md5I8Mcw$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https://petsc.org/release/faq/*valgrind__;Iw!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-Md5I8Mcw$</a>> and <a href="https://petsc.org/release/faq/" rel="noreferrer" target="_blank">https://petsc.org/release/faq/</a><<a href="https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-Mm8jIngI$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https://petsc.org/release/faq/__;!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-Mm8jIngI$</a>><br>
[1704]PETSC ERROR: --------------------- Stack Frames ------------------------------------<br>
[1704]PETSC ERROR: The line numbers in the error traceback are not always exact.<br>
[1704]PETSC ERROR: #1 hypre_ParCSRMatrixMigrate()<br>
<br>
and then Mark got this:<br>
(new_py-env) 07:24 1 adams/landau-ex1-fix= ~/Codes/petsc2$ git grep hypre_ParCSRMatrixMigrate<br>
src/mat/impls/hypre/mhypre.c: PetscCallExternal(hypre_ParCSRMatrixMigrate, parcsr, hmem);<br>
src/mat/impls/hypre/mhypre.c: PetscCallExternal(hypre_ParCSRMatrixMigrate,parcsr, HYPRE_MEMORY_HOST);<br>
<br>
Any help debugging this would be appreciated. Thanks. ANd let me know if you need to be added to my Frontier project for access. I am on through this Friday.<br>
<br>
David<br>
<br>
<br>
<br>
On Sat, Feb 11, 2023 at 10:35 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
I got on Frontier yesterday. Here's how it went. Had to use PrgEnv-cray to build petsc-hypre. PrgEnv-amd was having some problems. Also their default rocm/5.3.0 was problematic so backed off to rocm/<a href="http://5.2.0." rel="noreferrer" target="_blank">5.2.0.</a><<a href="https://urldefense.us/v3/__http://5.2.0.__;!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-M2j0U4y8$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http://5.2.0.__;!!G2kpM7uM-TzIFchu!kUAHuocSRof5_aTlZtjYLNna1q86tr06UuRvUcmqBdCqWkovEx-X9Y-M2j0U4y8$</a>> They did make 5.4.0 available yesterday but I stuck with 5.2.0. I got everything built and working. Scaling is excellent thus far. Performance is a little bit better than Crusher. And I am taking the scaling test up to higher concurrencies. Here's the comparison to Crusher. Same scaling test that we have been previously discussing.<br>
[image.png]<br>
<br>
On Fri, Feb 10, 2023 at 9:17 AM Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>> wrote:<br>
I haven’t seen this before. Is this from PETSc?<br>
<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Sent: Friday, February 10, 2023 09:14 AM<br>
To: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>><br>
Subject: Re: Frontier<br>
<br>
I am on Frontier today for 10 days. Building petsc-hypre. I do get this warning. ANything I should worry about?<br>
=============================================================================================<br>
***** WARNING *****<br>
Branch "master" is specified, however remote branch "origin/master" also exists!<br>
Proceeding with using the remote branch. To use the local branch (manually checkout local<br>
branch and) - rerun configure with option --download-hypre-commit=HEAD)<br>
=============================================================================================<br>
<br>
On Tue, Feb 7, 2023 at 7:09 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
I should also say that the timestep includes other solves as well like advection and Helmholtz but the latter is not hyper, rather petsc Jacobi.<br>
<br>
On Tue, Feb 7, 2023, 5:44 PM Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>> wrote:<br>
Great. Thanks for the new figure and explanation<br>
Ulrike<br>
<br>
<br>
Get Outlook for iOS<<a href="https://urldefense.us/v3/__https:/aka.ms/o0ukef__;!!G2kpM7uM-TzIFchu!iyppRHJeX4wNsqIy0mSCIAWwwmzeqaLTfY5V1Q98MhNlzvJp0jgUuPeYYGehcqSN$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https:/aka.ms/o0ukef__;!!G2kpM7uM-TzIFchu!iyppRHJeX4wNsqIy0mSCIAWwwmzeqaLTfY5V1Q98MhNlzvJp0jgUuPeYYGehcqSN$</a>><br>
________________________________<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Sent: Tuesday, February 7, 2023 4:39:01 PM<br>
To: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>> <<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>>><br>
Subject: Re: Frontier<br>
<br>
Hi Ulrike-<br>
In this scaling problem I use hypre to solve the pressure-Poisson problem in my projection method for incompressible Navier-Stokes. The preconditioner is set up once and re-used. This particular scaling problem is not time-dependent, that is, the grid is not moving so I don't have to redefine solver stencils, etc. I run 10 timesteps of this and average the time.<br>
<br>
When I did the July runs I thought it was anomalous data because it was slower. But I have seen this before where something may have been updated the previous 6 months and caused an uptick in performance. This anomaly was one of the reasons why I ran this recent test again besides making sure the new hypre release is performing the same. So, let's just forget the July data. Here is Feb 2022 vs. Jan 2023, with either boxes or nodes on x axis:<br>
<br>
On Tue, Feb 7, 2023 at 3:18 PM Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>> wrote:<br>
<br>
Hi David,<br>
<br>
I am still trying to understand the figures and your use of hypre:<br>
<br>
<br>
<br>
When you are using hypre, do you just solve one system? Or is this a time dependent problem where you need to solve systems many times?<br>
<br>
If the latter do you set up the preconditioner once and reuse it, or do you set up every time?<br>
<br>
<br>
<br>
Now I have some questions about the figures:<br>
<br>
It seems in this plot you are using 256 boxes per node and get better performance with hypre in July 2022 than in February 2022. Is this correct?<br>
<br>
Here performance in July 2022 is worse than in February 2022 using 512 boxes per node:<br>
<br>
Performance is now back to previous better performance. I really wonder what happened in July. Do you have any idea? But the numbers of February 2022 are similar to what you have in the plot you sent below.<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Sent: Wednesday, February 1, 2023 06:00 PM<br>
To: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>><br>
Subject: Re: Frontier<br>
<br>
<br>
<br>
I'd be glad to show you the data in case you're interested.<br>
<br>
<br>
<br>
On Wed, Feb 1, 2023 at 5:55 PM Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>> wrote:<br>
<br>
Never mind. I read your new message before the one you sent before. So, the figures are correct then<br>
<br>
<br>
<br>
Get Outlook for iOS<<a href="https://urldefense.us/v3/__https:/aka.ms/o0ukef__;!!G2kpM7uM-TzIFchu!jtnof7503SCD3sNKdbf-8RTND6Q2FRuyxk2zGyChnupBjnjN-TS7Fjzp1tA-1lor$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__https:/aka.ms/o0ukef__;!!G2kpM7uM-TzIFchu!jtnof7503SCD3sNKdbf-8RTND6Q2FRuyxk2zGyChnupBjnjN-TS7Fjzp1tA-1lor$</a>><br>
<br>
________________________________<br>
<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Sent: Wednesday, February 1, 2023 5:08:03 PM<br>
To: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>> <<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>>><br>
Subject: Re: Frontier<br>
<br>
<br>
<br>
Just checked. That was a different scaling plot where the weak scaling started with N=2 nodes for the 512 box problem (not N=1). So, I can do the same for the new executable and see what we get. Should have labelled the previous figure with more detail because with log scale it is difficult to see the abscissa of the first datapoint<br>
<br>
<br>
<br>
In previous weak scaling I have put several on one plot and annotate with the starting node count for each curve:<br>
<br>
<br>
<br>
On Wed, Feb 1, 2023 at 4:56 PM Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>> wrote:<br>
<br>
Hi David,<br>
<br>
I was referring to the figure below in my previous email.<br>
<br>
The timings are different, so you were probably running something a bit different, but it shows some nice improvement.<br>
<br>
Thanks<br>
<br>
Ulrike<br>
<br>
<br>
<br>
From: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>><br>
Sent: Tuesday, July 26, 2022 4:35 PM<br>
To: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>>; <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>><br>
Cc: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Subject: Re: Frontier<br>
<br>
<br>
<br>
Hi David,<br>
<br>
<br>
<br>
Thank you for the scaling result which looks nice. The slight performance improvement was probably from recent code optimizations.<br>
<br>
<br>
<br>
About 64 integers, I assumed you were talking about hypre’s bigInt option on GPUs (? Correct me if wrong). I don’t see why you have to use it instead of mixedInt. I believe mixedInt can handle as big problems as bigInt can do (@Ulrike is it correct?). Having a 60B or 300B global size doesn’t seem to be an obstacle to me for mixedInt.<br>
<br>
<br>
<br>
Hope this makes sense.<br>
<br>
<br>
<br>
. -- .- .. .-.. / ..-. .-. --- -- / .-. ..- .. .--. . -. --. / .-.. ..<br>
<br>
Rui Peng Li<br>
<br>
Center for Applied Scientific Computing<br>
<br>
Lawrence Livermore National Laboratory<br>
<br>
P.O. Box 808, L-561 Livermore, CA 94551<br>
<br>
phone - (925) 422-6037, email - <a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>><br>
<br>
<br>
<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Date: Tuesday, July 26, 2022 at 3:40 PM<br>
To: <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>> <<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>, Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Subject: Re: Frontier<br>
<br>
Ok, looks like my build has worked. It reproduces the weak scaling numbers that I had in Feb and in May and in fact the times are slightly better.<br>
<br>
<br>
<br>
Ruipeng/Ulrike: 64 integers seems to be the sticking point for me since my runs are high d.o.f. and they're only going to get bigger so having hypre run with 64 int on GPU is probably needed. The largest problem that I run for my scaling test on Crusher is about 6B dof on 128 nodes. On Frontier we will certainly be 10x that problem size and probably 50x.<br>
<br>
<br>
<br>
Mark: I still would like to get an official build from you when you get back from vacation just to have that in a safe place and to make sure we are on the same page.<br>
<br>
<br>
<br>
Here's the configure file I used:<br>
<br>
#!/usr/bin/python3<br>
if __name__ == '__main__':<br>
import sys<br>
import os<br>
sys.path.insert(0, os.path.abspath('config'))<br>
import configure<br>
configure_options = [<br>
'--download-hypre',<br>
'--download-hypre-commit=master',<br>
'--download-hypre-configure-arguments=--enable-bigint=no --enable-mixedint=yes',<br>
'--prefix=/gpfs/alpine/world-shared/geo127/petsc_treb/arch-crusher-amd-opt-int64-master',<br>
'--with-64-bit-indices=1',<br>
'--with-cc=cc',<br>
'--with-cxx=CC',<br>
'--with-debugging=0',<br>
'--with-fc=ftn',<br>
'--with-hip',<br>
'--with-hipc=hipcc',<br>
'--with-mpiexec=srun',<br>
'LIBS=-L/opt/cray/pe/mpich/8.1.16/gtl/lib -lmpi_gtl_hsa',<br>
'PETSC_ARCH=arch-olcf-crusher-amd-opt-int64-master',<br>
'PETSC_DIR=/gpfs/alpine/world-shared/geo127/petsc_treb/petsc',<br>
]<br>
configure.petsc_configure(configure_options)<br>
<br>
<br>
<br>
And here's the module list:<br>
<br>
Currently Loaded Modules:<br>
1) craype-x86-trento<br>
2) libfabric/<a href="http://1.15.0.0" rel="noreferrer" target="_blank">1.15.0.0</a><<a href="https://urldefense.us/v3/__http:/1.15.0.0__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJeI6a5Rk$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/1.15.0.0__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJeI6a5Rk$</a>><br>
3) craype-network-ofi<br>
4) perftools-base/22.05.0<br>
5) xpmem/2.4.4-2.3_2.12__gff0e1d9.shasta<br>
6) cray-pmi/6.1.2<br>
7) rocm/5.1.0<br>
8) subversion/1.14.1<br>
9) emacs/28.1<br>
10) amd/5.1.0<br>
11) craype/2.7.15<br>
12) cray-dsmml/0.2.2<br>
13) cray-mpich/8.1.16<br>
14) cray-libsci/<a href="http://21.08.1.2" rel="noreferrer" target="_blank">21.08.1.2</a><<a href="https://urldefense.us/v3/__http:/21.08.1.2__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJSW0Wh3g$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/21.08.1.2__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJSW0Wh3g$</a>><br>
15) PrgEnv-amd/8.3.3<br>
16) xalt/1.3.0<br>
17) DefApps/default<br>
18) cray-hdf5-parallel/<a href="http://1.12.1.1" rel="noreferrer" target="_blank">1.12.1.1</a><<a href="https://urldefense.us/v3/__http:/1.12.1.1__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJMJWAYCE$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/1.12.1.1__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJMJWAYCE$</a>><br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 8:42 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
<br>
'--download-hypre-commit=master',<br>
<br>
<br>
<br>
You might want:<br>
<br>
<br>
<br>
'--download-hypre-commit=origin/master',<br>
<br>
<br>
<br>
But, You should ask questions on the mailing list petsc-maint <<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a><mailto:<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a>>> (not archived).<br>
<br>
<br>
<br>
Mark<br>
<br>
ps, I am on vacation and will be back on the1st<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 9:00 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
I am not getting anywhere with this. I'll have to wait for Mark to do the petsc build with hypre.<br>
<br>
<br>
<br>
I tried the following to get the hypre master branch but I am not sure if this is the right incantation:<br>
<br>
'--download-hypre',<br>
<br>
'--download-hypre-commit=master',<br>
<br>
<br>
<br>
I did get a build with that but still get same problem with scaling.<br>
<br>
<br>
<br>
Here's my configure script:<br>
<br>
#!/usr/bin/python3<br>
if __name__ == '__main__':<br>
import sys<br>
import os<br>
sys.path.insert(0, os.path.abspath('config'))<br>
import configure<br>
configure_options = [<br>
'--download-hypre',<br>
'--download-hypre-commit=master',<br>
'--download-hypre-configure-arguments=--enable-bigint=no --enable-mixedint=yes',<br>
'--prefix=/gpfs/alpine/world-shared/geo127/petsc_treb/arch-crusher-amd-opt-int64-master',<br>
'--with-64-bit-indices=1',<br>
'--with-cc=cc',<br>
'--with-cxx=CC',<br>
'--with-debugging=0',<br>
'--with-fc=ftn',<br>
'--with-hip',<br>
'--with-hipc=hipcc',<br>
'--with-mpiexec=srun',<br>
'LIBS=-L/opt/cray/pe/mpich/8.1.16/gtl/lib -lmpi_gtl_hsa',<br>
'PETSC_ARCH=arch-olcf-crusher-amd-opt-int64-master',<br>
]<br>
configure.petsc_configure(configure_options)<br>
<br>
<br>
<br>
Currently Loaded Modules:<br>
1) craype-x86-trento<br>
2) libfabric/<a href="http://1.15.0.0" rel="noreferrer" target="_blank">1.15.0.0</a><<a href="https://urldefense.us/v3/__http:/1.15.0.0__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJeI6a5Rk$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/1.15.0.0__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJeI6a5Rk$</a>><br>
3) craype-network-ofi<br>
4) perftools-base/22.05.0<br>
5) xpmem/2.4.4-2.3_2.12__gff0e1d9.shasta<br>
6) cray-pmi/6.1.2<br>
7) emacs/27.2<br>
8) rocm/5.1.0<br>
9) subversion/1.14.1<br>
10) amd/5.1.0<br>
11) craype/2.7.15<br>
12) cray-dsmml/0.2.2<br>
13) cray-mpich/8.1.16<br>
14) cray-libsci/<a href="http://21.08.1.2" rel="noreferrer" target="_blank">21.08.1.2</a><<a href="https://urldefense.us/v3/__http:/21.08.1.2__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJSW0Wh3g$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/21.08.1.2__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJSW0Wh3g$</a>><br>
15) PrgEnv-amd/8.3.3<br>
16) xalt/1.3.0<br>
17) DefApps/default<br>
18) cray-hdf5-parallel/<a href="http://1.12.1.1" rel="noreferrer" target="_blank">1.12.1.1</a><<a href="https://urldefense.us/v3/__http:/1.12.1.1__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJMJWAYCE$" rel="noreferrer" target="_blank">https://urldefense.us/v3/__http:/1.12.1.1__;!!G2kpM7uM-TzIFchu!n794qlYekpYcOxHOM02fRuGjxyA6-PY6Bp_NJGcse4LvoqXq878zIvHJMJWAYCE$</a>><br>
<br>
<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 11:20 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
that was wrong mpich. I got much further in the configure. How do I know if I got the master branch of hypre?<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 10:46 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
I use the following configure:<br>
<br>
#!/usr/bin/python3<br>
if __name__ == '__main__':<br>
import sys<br>
import os<br>
sys.path.insert(0, os.path.abspath('config'))<br>
import configure<br>
configure_options = [<br>
'--download-hypre',<br>
'--download-hypre-commit=master',<br>
'--download-hypre-configure-arguments=--enable-bigint=no --enable-mixedint=yes',<br>
'--prefix=/gpfs/alpine/world-shared/geo127/petsc_treb/arch-crusher-cray-opt-int64-master',<br>
'--with-64-bit-indices=1',<br>
'--with-cc=cc',<br>
'--with-cxx=CC',<br>
'--with-debugging=0',<br>
'--with-fc=ftn',<br>
'--with-hip',<br>
'--with-hipc=hipcc',<br>
'--with-mpiexec=srun',<br>
'LIBS=-L/opt/cray/pe/mpich/8.1.12/gtl/lib -lmpi_gtl_hsa',<br>
'PETSC_ARCH=arch-olcf-crusher-cray-opt-int64-master',<br>
]<br>
configure.petsc_configure(configure_options)<br>
<br>
<br>
<br>
and get:<br>
<br>
=============================================================================================<br>
Configuring PETSc to compile on your system<br>
=============================================================================================<br>
============================================================================================= ***** WARNING: Using default optimization C flags -O You might consider manually setting optimal optimization flags for your system with COPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= ***** WARNING: Using default Cxx optimization flags -O You might consider manually setting optimal optimization flags for your system with CXXOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= ***** WARNING: Using default FORTRAN optimization flags -O You might consider manually setting optimal optimization flags for your system with FOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= ============================================================================================= ***** WARNING: Using default HIP optimization flags -g -O3 You might consider manually setting optimal optimization flags for your system with HIPOPTFLAGS="optimization flags" see config/examples/arch-*-opt.py for examples ============================================================================================= TESTING: checkFortranLibraries from config.compilers(config/BuildSystem/config/compilers.py:835) *******************************************************************************<br>
OSError while running ./configure<br>
-------------------------------------------------------------------------------<br>
Cannot run executables created with FC. If this machine uses a batch system<br>
to submit jobs you will need to configure using ./configure with the additional option --with-batch.<br>
Otherwise there is problem with the compilers. Can you compile and run code with your compiler 'ftn'?<br>
*******************************************************************************<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 9:50 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
I think recent builds have been hypre v2.25<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 9:49 AM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
so instead of just<br>
<br>
'--download-hypre',<br>
<br>
add<br>
<br>
'--download-hypre',<br>
'--download-hypre-commit=master',<br>
<br>
<br>
<br>
???<br>
<br>
<br>
<br>
On Thu, Jul 21, 2022 at 9:47 AM Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>> wrote:<br>
<br>
As Ulrike said, AMD recently found bugs regarding the bigInt issue, which have been fixed in the current master. I suggest using the master branch of hypre if possible.<br>
<br>
<br>
<br>
Thanks<br>
<br>
<br>
<br>
. -- .- .. .-.. / ..-. .-. --- -- / .-. ..- .. .--. . -. --. / .-.. ..<br>
<br>
Rui Peng Li<br>
<br>
Center for Applied Scientific Computing<br>
<br>
Lawrence Livermore National Laboratory<br>
<br>
P.O. Box 808, L-561 Livermore, CA 94551<br>
<br>
phone - (925) 422-6037, email - <a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>><br>
<br>
<br>
<br>
From: Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Date: Thursday, July 21, 2022 at 9:41 AM<br>
To: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>>, Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>><br>
Cc: <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>> <<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>>><br>
Subject: RE: Frontier<br>
<br>
Actually, I think it was 2000 nodes!<br>
<br>
<br>
<br>
From: Yang, Ulrike Meier<br>
Sent: Thursday, July 21, 2022 9:40 AM<br>
To: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>>; Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>><br>
Cc: <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>><br>
Subject: RE: Frontier<br>
<br>
<br>
<br>
Which version of hypre are you using for this?<br>
<br>
We recently found one bug in the mixed-int version, however that should have been an issue also in your previous runs that apparently were working.<br>
<br>
Note that recent runs by AMD on Frontier with hypre were successful on more than 200 nodes using mixed-int, so we should be able to get this to work somehow for you guys. They also found the bug in mixed-int.<br>
<br>
Ulrike<br>
<br>
<br>
<br>
From: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Sent: Thursday, July 21, 2022 9:30 AM<br>
To: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>><br>
Cc: <a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a><mailto:<a href="mailto:MFAdams@LBL.GOV" target="_blank">MFAdams@LBL.GOV</a>>; Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Subject: Re: Frontier<br>
<br>
<br>
<br>
Hi Ruipeng and Ulrike<br>
<br>
You asked if we need 64 int for gpus and I think we definitely do need it. Currently I cannot scale past that 2B degree of freedom mark that you mentioned. I am not sure what happened between Mark's Cray build in February and his amd build in May but currently I cannot scale past 32 nodes on Crusher. This is unfortunate because given the success over the past 6 months I have told ECP that we are fully ready for Frontier. Now, we are not. Hopefully we can figure this out pretty soon and be ready to take a shot on Frontier when they let us on.<br>
<br>
David<br>
<br>
<br>
<br>
On Mon, Jul 18, 2022 at 5:03 PM Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>> wrote:<br>
<br>
Hi All,<br>
<br>
<br>
<br>
Building with unified memory will *not* change the default parameters of AMG. Are you using the master branch of hypre or some release version? I think our previous fix should be included in the latest release.<br>
<br>
<br>
<br>
Please let me know if I can further help<br>
<br>
<br>
<br>
Thanks<br>
<br>
<br>
<br>
. -- .- .. .-.. / ..-. .-. --- -- / .-. ..- .. .--. . -. --. / .-.. ..<br>
<br>
Rui Peng Li<br>
<br>
Center for Applied Scientific Computing<br>
<br>
Lawrence Livermore National Laboratory<br>
<br>
P.O. Box 808, L-561 Livermore, CA 94551<br>
<br>
phone - (925) 422-6037, email - <a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>><br>
<br>
<br>
<br>
From: Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>><br>
Date: Monday, July 18, 2022 at 1:55 PM<br>
To: David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>><br>
Cc: Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>>, Yang, Ulrike Meier <<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a><mailto:<a href="mailto:yang11@llnl.gov" target="_blank">yang11@llnl.gov</a>>><br>
Subject: Re: Frontier<br>
<br>
<br>
<br>
<br>
<br>
On Mon, Jul 18, 2022 at 4:35 PM David Trebotich <<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a><mailto:<a href="mailto:dptrebotich@lbl.gov" target="_blank">dptrebotich@lbl.gov</a>>> wrote:<br>
<br>
When I run with Mark's newest build then I get stuck in the nnz bin counts for the first solve (proj_mac). Here's the stack:<br>
<br>
[0]PETSC ERROR: #1 jac->setup() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/impls/hypre/hypre.c:420<br>
<br>
<br>
<br>
This is the same place where we got this hypre error "(12)" before.<br>
<br>
Recall this error message means that there is a zero row in the matrix.<br>
<br>
<br>
<br>
I may have been using the master branch of hypre when I built that working version.<br>
<br>
Maybe this branch was fixed to accept zero rows?<br>
<br>
<br>
<br>
Rui Peng: I am building with UVM now. Does that change the defaults in hypre?<br>
<br>
For instance, does hypre use Falgout coursening if UVM is available?<br>
<br>
<br>
<br>
<br>
<br>
[0]PETSC ERROR: #2 PCSetUp_HYPRE() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/impls/hypre/hypre.c:237<br>
[0]PETSC ERROR: #3 PCSetUp() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/pc/interface/precon.c:949<br>
[0]PETSC ERROR: #4 KSPSetUp() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:314<br>
[0]PETSC ERROR: #5 KSPSolve_Private() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:792<br>
[0]PETSC ERROR: #6 KSPSolve() at /gpfs/alpine/csc314/scratch/adams/petsc/src/ksp/ksp/interface/itfunc.c:1061<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
And here's my .petscrc:<br>
<br>
-help<br>
<br>
-proj_mac_mat_type hypre<br>
-proj_mac_pc_type hypre<br>
-proj_mac_pc_hypre_type boomeramg<br>
-proj_mac_pc_hypre_boomeramg_no_CF<br>
-proj_mac_pc_hypre_boomeramg_agg_nl 0<br>
-proj_mac_pc_hypre_boomeramg_coarsen_type PMIS<br>
-proj_mac_pc_hypre_boomeramg_interp_type ext+i<br>
-proj_mac_pc_hypre_boomeramg_print_statistics<br>
-proj_mac_pc_hypre_boomeramg_relax_type_all l1scaled-Jacobi<br>
-proj_mac_pc_hypre_SetSpGemmUseCusparse 0<br>
<br>
-proj_mac_ksp_type gmres<br>
-proj_mac_ksp_max_it 50<br>
-proj_mac_ksp_rtol 1.e-12<br>
-proj_mac_ksp_atol 1.e-30<br>
<br>
-use_gpu_aware_mpi 0<br>
<br>
-info<br>
-log_view<br>
-history PETSc.history<br>
-options_left<br>
<br>
-visc_pc_type jacobi<br>
<br>
-visc_pc_hypre_type boomeramg<br>
-visc_ksp_type gmres<br>
-visc_ksp_max_it 50<br>
-visc_ksp_rtol 1.e-12<br>
<br>
-diff_pc_type jacobi<br>
-diff_pc_hypre_type boomeramg<br>
-diff_ksp_type gmres<br>
-diff_ksp_max_it 50<br>
-diff_ksp_rtol 1.e-6<br>
<br>
-proj_mac_ksp_converged_reason<br>
-visc_ksp_converged_reason<br>
-diff_ksp_converged_reason<br>
-proj_mac_ksp_norm_type unpreconditioned<br>
-diff_ksp_norm_type unpreconditioned<br>
-visc_ksp_norm_type unpreconditioned<br>
<br>
<br>
<br>
On Mon, Jul 18, 2022 at 1:30 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a><mailto:<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>>> wrote:<br>
<br>
<br>
<br>
<br>
<br>
On Mon, Jul 18, 2022 at 4:18 PM Li, Rui Peng <<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a><mailto:<a href="mailto:li50@llnl.gov" target="_blank">li50@llnl.gov</a>>> wrote:<br>
<br>
Yes, there is no need for enable-unified-memory, unless you want to use non-GPU supported parameter of AMG (such as Falgout coarsening) which needs unified memory since it will run on CPUs.<br>
<br>
<br>
<br>
Got it, Will not use UVM.<br>
<br>
<br>
<br>
UVM is unified memory. Our expert from AMD told us to not use unified memory yet. Maybe it's working now, but I haven't tried.<br>
<br>
64-bit integer: Sorry, I did not make it clear. "mixed-int" is a more efficient approach for problems with > 2B dofs where the local integer type is kept in 32-bit while the global one is 64-bit. This is the only way we currently support on GPUs. hypre also has "--enable-big-int" which has all the integers (local and global) in 64-bit, which we don't have on GPUs. For some users, it is difficult for their code to handle two integer types (in mixed-int), so they prefer the old "big-int" approach. That's why I was asking. If "mixed-int" works for you, that's ideal. No need to bother.<br>
<br>
<br>
<br>
I see. I only care about the interface so the current parameters are fine.<br>
<br>
<br>
<br>
--enable-bigint=no --enable-mixedint=yes<br>
<br>
<br>
<br>
I think PETSc should always use this, with 64 bit ints, because we only care about the interface and I trust the local problem will be < 2B.<br>
<br>
<br>
<br>
Thanks,<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
<br>
<br>
--<br>
<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
--<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
--<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
--<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
<br>
<br>
--<br>
----------------------<br>
David Trebotich<br>
Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>
Applied Numerical Algorithms Group<br>
<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><mailto:<a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a>><br>
(510) 486-5984 office<br>
(510) 384-6868 mobile<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr">----------------------<br>David Trebotich<br>Lawrence Berkeley National Laboratory<br>
Computational Research Division<br>Applied Numerical Algorithms Group<br><a href="mailto:treb@lbl.gov" target="_blank">treb@lbl.gov</a><br>(510) 486-5984 office<br>(510) 384-6868 mobile<br></div>
</blockquote></div></div>