<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
Hi Junchao, thank you for the info. I compiled the main branch of PETSc in another machine that has the openmpi/4.1.4/gcc-11.2.1-cuda-11.7 toolchain and don't see the fortran compilation error. It might have been related to gcc-9.3.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
I tried the case again, 2 CPUs and one GPU and get this error now:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted1">
<span style="font-family: "Courier New", monospace;">terminate called after throwing an instance of 'thrust::system::system_error'</span>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">terminate called after throwing an instance of 'thrust::system::system_error'</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> what(): parallel_for failed: cudaErrorInvalidConfiguration: invalid configuration argument</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> what(): parallel_for failed: cudaErrorInvalidConfiguration: invalid configuration argument</span></div>
<div><br class="ContentPasted1">
</div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">Program received signal SIGABRT: Process abort signal.</span></div>
<div><br class="ContentPasted1">
</div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">Backtrace for this error:</span></div>
<div><br class="ContentPasted1">
</div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">Program received signal SIGABRT: Process abort signal.</span></div>
<div><br class="ContentPasted1">
</div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">Backtrace for this error:</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#0 0x2000397fcd8f in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#1 0x2000397fb657 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#0 0x2000397fcd8f in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#1 0x2000397fb657 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#2 0x2000000604d7 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#2 0x2000000604d7 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#3 0x200039cb9628 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#4 0x200039c93eb3 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#5 0x200039364a97 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#6 0x20003935f6d3 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#7 0x20003935f78f in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#8 0x20003935fc6b in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#3 0x200039cb9628 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#4 0x200039c93eb3 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#5 0x200039364a97 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#6 0x20003935f6d3 in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#7 0x20003935f78f in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#8 0x20003935fc6b in ???</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#9 0x11ec425b in _ZN6thrust8cuda_cub14throw_on_errorE9cudaErrorPKc</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/system/cuda/detail/util.h:225</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#10 0x11ec425b in _ZN6thrust8cuda_cub20uninitialized_fill_nINS0_3tagENS_10device_ptrIiEEmiEET0_RNS0_16execution_policyIT_EES5_T1_RKT2_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#9 0x11ec425b in _ZN6thrust8cuda_cub14throw_on_errorE9cudaErrorPKc</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/system/cuda/detail/util.h:225</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#10 0x11ec425b in _ZN6thrust8cuda_cub20uninitialized_fill_nINS0_3tagENS_10device_ptrIiEEmiEET0_RNS0_16execution_policyIT_EES5_T1_RKT2_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/system/cuda/detail/uninitialized_fill.h:88</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#11 0x11efa263 in _ZN6thrust20uninitialized_fill_nINS_8cuda_cub3tagENS_10device_ptrIiEEmiEET0_RKNS_6detail21execution_policy_baseIT_EES5_T1_RKT2_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/system/cuda/detail/uninitialized_fill.h:88</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#11 0x11efa263 in _ZN6thrust20uninitialized_fill_nINS_8cuda_cub3tagENS_10device_ptrIiEEmiEET0_RKNS_6detail21execution_policy_baseIT_EES5_T1_RKT2_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/uninitialized_fill.inl:55</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#12 0x11efa263 in _ZN6thrust6detail23allocator_traits_detail23default_construct_rangeINS_16device_allocatorIiEENS_10device_ptrIiEEmEENS0_10disable_ifIXsrNS1_37needs_default_construct_via_allocatorIT_NS0_15pointer_elementIT0_E4typeEEE5valueEvE4typeERS9_SB_T1_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/allocator/default_construct_range.inl:93</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#13 0x11efa263 in _ZN6thrust6detail23default_construct_rangeINS_16device_allocatorIiEENS_10device_ptrIiEEmEEvRT_T0_T1_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/allocator/default_construct_range.inl:104</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/uninitialized_fill.inl:55</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#12 0x11efa263 in _ZN6thrust6detail23allocator_traits_detail23default_construct_rangeINS_16device_allocatorIiEENS_10device_ptrIiEEmEENS0_10disable_ifIXsrNS1_37needs_default_construct_via_allocatorIT_NS0_15pointer_elementIT0_E4typeEEE5valueEvE4typeERS9_SB_T1_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/allocator/default_construct_range.inl:93</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#13 0x11efa263 in _ZN6thrust6detail23default_construct_rangeINS_16device_allocatorIiEENS_10device_ptrIiEEmEEvRT_T0_T1_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/allocator/default_construct_range.inl:104</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#14 0x11efa263 in _ZN6thrust6detail18contiguous_storageIiNS_16device_allocatorIiEEE19default_construct_nENS0_15normal_iteratorINS_10device_ptrIiEEEEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/contiguous_storage.inl:254</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#15 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE12default_initEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:220</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#14 0x11efa263 in _ZN6thrust6detail18contiguous_storageIiNS_16device_allocatorIiEEE19default_construct_nENS0_15normal_iteratorINS_10device_ptrIiEEEEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/contiguous_storage.inl:254</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#15 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE12default_initEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:220</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#16 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE12default_initEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:213</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#17 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEEC2Em</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:65</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#18 0x11ed7e47 in _ZN6thrust13device_vectorIiNS_16device_allocatorIiEEEC4Em</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/device_vector.h:88</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#19 0x11ed7e47 in MatSeqAIJCUSPARSECopyToGPU</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:2488</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#20 0x11eef623 in MatSeqAIJCUSPARSEMergeMats</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:4696</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#16 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEE12default_initEm</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:213</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#17 0x11efa263 in _ZN6thrust6detail11vector_baseIiNS_16device_allocatorIiEEEC2Em</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/detail/vector_base.inl:65</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#18 0x11ed7e47 in _ZN6thrust13device_vectorIiNS_16device_allocatorIiEEEC4Em</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /usr/local/cuda-11.7/include/thrust/device_vector.h:88</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#19 0x11ed7e47 in MatSeqAIJCUSPARSECopyToGPU</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:2488</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#20 0x11eef623 in MatSeqAIJCUSPARSEMergeMats</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/seq/seqcusparse/aijcusparse.cu:4696</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#21 0x11f0682b in MatMPIAIJGetLocalMatMerge_MPIAIJCUSPARSE</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:251</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#21 0x11f0682b in MatMPIAIJGetLocalMatMerge_MPIAIJCUSPARSE</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpicusparse/mpiaijcusparse.cu:251</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#22 0x133f141f in MatMPIAIJGetLocalMatMerge</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpiaij.c:5342</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#22 0x133f141f in MatMPIAIJGetLocalMatMerge</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpiaij.c:5342</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#23 0x133fe9cb in MatProductSymbolic_MPIAIJBACKEND</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpiaij.c:7368</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#23 0x133fe9cb in MatProductSymbolic_MPIAIJBACKEND</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpiaij.c:7368</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#24 0x1377e1df in MatProductSymbolic</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/interface/matproduct.c:795</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#24 0x1377e1df in MatProductSymbolic</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/interface/matproduct.c:795</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#25 0x11e4dd1f in MatPtAP</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/interface/matrix.c:9934</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#25 0x11e4dd1f in MatPtAP</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/interface/matrix.c:9934</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#26 0x130d792f in MatCoarsenApply_MISK_private</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:283</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#26 0x130d792f in MatCoarsenApply_MISK_private</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:283</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#27 0x130db89b in MatCoarsenApply_MISK</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:368</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#27 0x130db89b in MatCoarsenApply_MISK</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:368</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#28 0x130bf5a3 in MatCoarsenApply</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/coarsen.c:97</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#28 0x130bf5a3 in MatCoarsenApply</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/mat/coarsen/coarsen.c:97</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#29 0x141518ff in PCGAMGCoarsen_AGG</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/agg.c:524</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#29 0x141518ff in PCGAMGCoarsen_AGG</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/agg.c:524</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#30 0x13b3a43f in PCSetUp_GAMG</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/gamg.c:631</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#30 0x13b3a43f in PCSetUp_GAMG</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/gamg.c:631</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#31 0x1276845b in PCSetUp</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/interface/precon.c:1069</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#31 0x1276845b in PCSetUp</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/pc/interface/precon.c:1069</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#32 0x127d6cbb in KSPSetUp</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:415</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#32 0x127d6cbb in KSPSetUp</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:415</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#33 0x127dddbf in KSPSolve_Private</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:836</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#33 0x127dddbf in KSPSolve_Private</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:836</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#34 0x127e4987 in KSPSolve</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:1082</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#34 0x127e4987 in KSPSolve</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:1082</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#35 0x1280b18b in kspsolve_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/arch-linux-c-dbg/src/ksp/ksp/interface/ftn-auto/itfuncf.c:335</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#35 0x1280b18b in kspsolve_</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at /home/mnv/Software/petsc/arch-linux-c-dbg/src/ksp/ksp/interface/ftn-auto/itfuncf.c:335</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#36 0x1140945f in __globmat_solver_MOD_glmat_solver</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/pres.f90:3128</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#36 0x1140945f in __globmat_solver_MOD_glmat_solver</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/pres.f90:3128</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#37 0x119f8853 in pressure_iteration_scheme</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:1449</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#37 0x119f8853 in pressure_iteration_scheme</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:1449</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#38 0x11969bd3 in fds</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:688</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#38 0x11969bd3 in fds</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:688</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#39 0x11a10167 in main</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:6</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;">#39 0x11a10167 in main</span></div>
<div class="ContentPasted1"><span style="font-family: "Courier New", monospace;"> at ../../Source/main.f90:6</span></div>
<span style="font-family: "Courier New", monospace;">srun: error: enki12: tasks 0-1: Aborted (core dumped)</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
This was the slurm submission script in this case:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;">#!/bin/bash</span>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;"># ../../Utilities/Scripts/qfds.sh -p 2 -T db -d test.fds</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH -J test
</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH -e /home/mnv/Firemodels_fork/fds/Issues/PETSc/test.err</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH -o /home/mnv/Firemodels_fork/fds/Issues/PETSc/test.log</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --partition=debug</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --ntasks=2</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --nodes=1</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --cpus-per-task=1</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --ntasks-per-node=2</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --time=01:00:00</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">#SBATCH --gres=gpu:1</span></div>
<div><br class="ContentPasted2">
</div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">export OMP_NUM_THREADS=1</span></div>
<div><br class="ContentPasted2">
</div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;"># PETSc dir and arch:</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">export PETSC_DIR=/home/mnv/Software/petsc</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">export PETSC_ARCH=arch-linux-c-dbg</span></div>
<div><br class="ContentPasted2">
</div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;"># SYSTEM name:</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">export MYSYSTEM=enki</span></div>
<div><br class="ContentPasted2">
</div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;"># modules</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">module load cuda/11.7</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">module load gcc/11.2.1/toolset</span></div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">module load openmpi/4.1.4/gcc-11.2.1-cuda-11.7</span></div>
<div><br class="ContentPasted2">
</div>
<div class="ContentPasted2"><span style="font-family: "Courier New", monospace;">cd /home/mnv/Firemodels_fork/fds/Issues/PETSc</span></div>
<span style="font-family: "Courier New", monospace;">srun -N 1 -n 2 --ntasks-per-node 2 --mpi=pmi2 /home/mnv/Firemodels_fork/fds/Build/ompi_gnu_linux_db/fds_ompi_gnu_linux_db test.fds -vec_type mpicuda -mat_type mpiaijcusparse -pc_type gamg</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: Calibri, Helvetica, sans-serif;">The configure.log for the PETSc build is attached.
</span><span style="font-family: Calibri, Helvetica, sans-serif;" class="ContentPasted3">Another clue to what is happening is that even setting the matrices/vectors to be mpi (-vec_type mpi -mat_type mpiaij) and not requesting a gpu I get a GPU warning :</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;" class="ContentPasted3"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;" class="ContentPasted3 ContentPasted4">0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
<div class="ContentPasted4">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</div>
<div class="ContentPasted4">[1]PETSC ERROR: GPU error</div>
<div class="ContentPasted4">[1]PETSC ERROR: Cannot lazily initialize PetscDevice: cuda error 100 (cudaErrorNoDevice) : no CUDA-capable device is detected</div>
<div class="ContentPasted4">[1]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</div>
<div class="ContentPasted4">[0]PETSC ERROR: GPU error</div>
<div class="ContentPasted4">[0]PETSC ERROR: Cannot lazily initialize PetscDevice: cuda error 100 (cudaErrorNoDevice) : no CUDA-capable device is detected</div>
<div class="ContentPasted4">[0]PETSC ERROR: WARNING! There are unused option(s) set! Could be the program crashed before usage or a spelling mistake, etc!</div>
<div class="ContentPasted4">[0]PETSC ERROR: Option left: name:-pc_type value: gamg source: command line</div>
<div class="ContentPasted4">[0]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.</div>
<div class="ContentPasted4">[1]PETSC ERROR: Option left: name:-pc_type value: gamg source: command line</div>
<div class="ContentPasted4">[1]PETSC ERROR: See https://petsc.org/release/faq/ for trouble shooting.</div>
<div class="ContentPasted4">[1]PETSC ERROR: Petsc Development GIT revision: v3.19.4-946-g590ad0f52ad GIT Date: 2023-08-11 15:13:02 +0000</div>
<div class="ContentPasted4">[0]PETSC ERROR: Petsc Development GIT revision: v3.19.4-946-g590ad0f52ad GIT Date: 2023-08-11 15:13:02 +0000</div>
<div class="ContentPasted4">[0]PETSC ERROR: /home/mnv/Firemodels_fork/fds/Build/ompi_gnu_linux_db/fds_ompi_gnu_linux_db on a arch-linux-c-dbg named enki11.adlp by mnv Fri Aug 11 17:04:55 2023</div>
<div class="ContentPasted4">[0]PETSC ERROR: Configure options COPTFLAGS="-g -O2" CXXOPTFLAGS="-g -O2" FOPTFLAGS="-g -O2" FCOPTFLAGS="-g -O2" CUDAOPTFLAGS="-g -O2" --with-debugging=yes --with-shared-libraries=0 --download-suitesparse --download-hypre --download-fblaslapack
--with-cuda</div>
...</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;" class="ContentPasted3 ContentPasted4"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: Calibri, Helvetica, sans-serif;" class="ContentPasted3 ContentPasted4">I would have expected not to see GPU errors being printed out, given I did not request cuda matrix/vectors. The case run anyways, I assume it defaulted to the CPU
solver.</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: Calibri, Helvetica, sans-serif;" class="ContentPasted3 ContentPasted4">Let me know if you have any ideas as to what is happening. Thanks,</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: Calibri, Helvetica, sans-serif;" class="ContentPasted3 ContentPasted4">Marcos<br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;" class="ContentPasted3 ContentPasted4"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof ContentPasted0 ContentPasted2">
<span style="font-family: "Courier New", monospace;" class="ContentPasted3"><br>
</span></div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Junchao Zhang <junchao.zhang@gmail.com><br>
<b>Sent:</b> Friday, August 11, 2023 3:35 PM<br>
<b>To:</b> Vanella, Marcos (Fed) <marcos.vanella@nist.gov>; PETSc users list <petsc-users@mcs.anl.gov>; Satish Balay <balay@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Marcos,</div>
<div> We do not have good petsc/gpu documentation, but see <a href="https://petsc.org/main/faq/#doc-faq-gpuhowto" originalsrc="https://petsc.org/main/faq/#doc-faq-gpuhowto" shash="SVbm2+iuJ3exvUzhnkFVOaTxtLhJZwSbasXQ7nnXj6DwfcGclx7nsNCQjDMYxp5hcHlb/xZajG5kb2t/5rZ8OSTfTqzIhGummXrTsgZb+77rjc4xEPy8iTfcJWi9l9vitJVMSYVyp3THapAS1bCGmVxLHKeiLYSO9IAOfWQzMwM=">
https://petsc.org/main/faq/#doc-faq-gpuhowto</a>, and also search "<span style="color:rgb(0,128,0); font-family:Menlo,Monaco,"Courier New",monospace; font-size:14px; white-space:pre">requires: cuda"</span><span style="color:rgb(0,128,0); font-family:Menlo,Monaco,"Courier New",monospace; font-size:14px; white-space:pre">
</span>in petsc tests and you will find examples using GPU.</div>
<div> For the Fortran compile errors, attach your configure.log and Satish (Cc'ed) or others should know how to fix them.</div>
<div><br>
</div>
<div> Thanks.<br>
</div>
<div>
<div dir="ltr" class="x_gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div class="x_gmail_quote">
<div dir="ltr" class="x_gmail_attr">On Fri, Aug 11, 2023 at 2:22 PM Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov">marcos.vanella@nist.gov</a>> wrote:<br>
</div>
<blockquote class="x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div class="x_msg-6428481159717170951">
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Junchao, thanks for the explanation. Is there some development documentation on the GPU work? I'm interested learning about it.<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
I checked out the main branch and configured petsc. when compiling with gcc/gfortran I come across this error:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace">....</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace"> CUDAC arch-linux-c-opt/obj/src/mat/impls/aij/seq/seqcusparse/aijcusparse.o</span>
<div><span style="font-family:"Courier New",monospace"> CUDAC.dep arch-linux-c-opt/obj/src/mat/impls/aij/seq/seqcusparse/aijcusparse.o</span></div>
<div><span style="font-family:"Courier New",monospace"> FC arch-linux-c-opt/obj/src/ksp/f90-mod/petsckspdefmod.o</span></div>
<div><span style="font-family:"Courier New",monospace"> FC arch-linux-c-opt/obj/src/ksp/f90-mod/petscpcmod.o</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:37:61:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 37 | subroutine PCASMCreateSubdomains2D(a,b,c,d,e,f,g,h,i,z)</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace"><b>Error: Symbol ‘pcasmcreatesubdomains2d’ at (1) already has an explicit interface</b></span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:38:13:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 38 | import tIS</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: IMPORT statement at (1) only permitted in an INTERFACE body</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:39:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 39 | PetscInt a ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:40:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 40 | PetscInt b ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:41:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 41 | PetscInt c ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:42:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 42 | PetscInt d ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:43:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 43 | PetscInt e ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:44:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 44 | PetscInt f ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:45:80:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 45 | PetscInt g ! PetscInt</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:46:30:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 46 | IS h ! IS</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:47:30:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 47 | IS i ! IS</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:48:43:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 48 | PetscErrorCode z</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Unexpected data declaration statement in INTERFACE block at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">/home/mnv/Software/petsc/include/../src/ksp/f90-mod/ftn-auto-interfaces/petscpc.h90:49:10:</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace"> 49 | end subroutine PCASMCreateSubdomains2D</span></div>
<div><span style="font-family:"Courier New",monospace"> | 1</span></div>
<div><span style="font-family:"Courier New",monospace">Error: Expecting END INTERFACE statement at (1)</span></div>
<div><span style="font-family:"Courier New",monospace">make[3]: *** [gmakefile:225: arch-linux-c-opt/obj/src/ksp/f90-mod/petscpcmod.o] Error 1</span></div>
<div><span style="font-family:"Courier New",monospace">make[3]: *** Waiting for unfinished jobs....</span></div>
<div><span style="font-family:"Courier New",monospace"> CC arch-linux-c-opt/obj/src/tao/leastsquares/impls/pounders/pounders.o</span></div>
<div><span style="font-family:"Courier New",monospace"> CC arch-linux-c-opt/obj/src/ksp/pc/impls/bddc/bddcprivate.o</span></div>
<div><span style="font-family:"Courier New",monospace"> CUDAC arch-linux-c-opt/obj/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.o</span></div>
<div><span style="font-family:"Courier New",monospace"> CUDAC.dep arch-linux-c-opt/obj/src/vec/vec/impls/seq/cupm/cuda/vecseqcupm.o</span></div>
<div><span style="font-family:"Courier New",monospace">make[3]: Leaving directory '/home/mnv/Software/petsc'</span></div>
<div><span style="font-family:"Courier New",monospace">make[2]: *** [/home/mnv/Software/petsc/lib/petsc/conf/rules.doc:28: libs] Error 2</span></div>
<div><span style="font-family:"Courier New",monospace">make[2]: Leaving directory '/home/mnv/Software/petsc'</span></div>
<div><span style="font-family:"Courier New",monospace">**************************ERROR*************************************</span></div>
<div><span style="font-family:"Courier New",monospace"> Error during compile, check arch-linux-c-opt/lib/petsc/conf/make.log</span></div>
<div><span style="font-family:"Courier New",monospace"> Send it and arch-linux-c-opt/lib/petsc/conf/configure.log to
<a href="mailto:petsc-maint@mcs.anl.gov" target="_blank">petsc-maint@mcs.anl.gov</a></span></div>
<div><span style="font-family:"Courier New",monospace">********************************************************************</span></div>
<div><span style="font-family:"Courier New",monospace">make[1]: *** [makefile:45: all] Error 1</span></div>
<span style="font-family:"Courier New",monospace">make: *** [GNUmakefile:9: all] Error 2 </span><br>
</div>
<div id="x_m_-6428481159717170951appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-6428481159717170951divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Friday, August 11, 2023 3:04 PM<br>
<b>To:</b> Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov" target="_blank">marcos.vanella@nist.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU</font>
<div> </div>
</div>
<div>
<div dir="ltr">Hi, Macros,<br>
I saw MatSetPreallocationCOO_MPIAIJCUSPARSE_Basic() in the error stack. We recently refactored the COO code and got rid of that function. So could you try petsc/main?<br>
We map MPI processes to GPUs in a round-robin fashion. We query the number of visible CUDA devices (g), and assign the device (rank%g) to the MPI process (rank). In that sense, the work distribution is totally determined by your MPI work partition (i.e,
yourself). <br>
On clusters, this MPI process to GPU binding is usually done by the job scheduler like slurm. You need to check your cluster's users' guide to see how to bind MPI processes to GPUs. If the job scheduler has done that, the number of visible CUDA devices to
a process might just appear to be 1, making petsc's own mapping void.<br>
<br>
Thanks.<br>
--Junchao Zhang<br>
<div><br>
</div>
</div>
<br>
<div>
<div dir="ltr">On Fri, Aug 11, 2023 at 12:43 PM Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov" target="_blank">marcos.vanella@nist.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi Junchao, thank you for replying. I compiled petsc in debug mode and this is what I get for the case:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
terminate called after throwing an instance of 'thrust::system::system_error'
<div> what(): merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered</div>
<div><br>
</div>
<div>Program received signal SIGABRT: Process abort signal.</div>
<div><br>
</div>
<div>Backtrace for this error:</div>
<div>#0 0x15264731ead0 in ???</div>
<div>#1 0x15264731dc35 in ???</div>
<div>#2 0x15264711551f in ???</div>
<div>#3 0x152647169a7c in ???</div>
<div>#4 0x152647115475 in ???</div>
<div>#5 0x1526470fb7f2 in ???</div>
<div>#6 0x152647678bbd in ???</div>
<div>#7 0x15264768424b in ???</div>
<div>#8 0x1526476842b6 in ???</div>
<div>#9 0x152647684517 in ???</div>
<div>#10 0x55bb46342ebb in _ZN6thrust8cuda_cub14throw_on_errorE9cudaErrorPKc</div>
<div> at /usr/local/cuda/include/thrust/system/cuda/detail/util.h:224</div>
<div>#11 0x55bb46342ebb in _ZN6thrust8cuda_cub12__merge_sort10merge_sortINS_6detail17integral_constantIbLb1EEENS4_IbLb0EEENS0_3tagENS_12zip_iteratorINS_5tupleINS_10device_ptrIiEESB_NS_9null_typeESC_SC_SC_SC_SC_SC_SC_EEEENS3_15normal_iteratorISB_EE9IJCompareEEvRNS0_16execution_policyIT1_EET2_SM_T3_T4_</div>
<div> at /usr/local/cuda/include/thrust/system/cuda/detail/sort.h:1316</div>
<div>#12 0x55bb46342ebb in _ZN6thrust8cuda_cub12__smart_sort10smart_sortINS_6detail17integral_constantIbLb1EEENS4_IbLb0EEENS0_16execution_policyINS0_3tagEEENS_12zip_iteratorINS_5tupleINS_10device_ptrIiEESD_NS_9null_typeESE_SE_SE_SE_SE_SE_SE_EEEENS3_15normal_iteratorISD_EE9IJCompareEENS1_25enable_if_comparison_sortIT2_T4_E4typeERT1_SL_SL_T3_SM_</div>
<div> at /usr/local/cuda/include/thrust/system/cuda/detail/sort.h:1544</div>
<div>#13 0x55bb46342ebb in _ZN6thrust8cuda_cub11sort_by_keyINS0_3tagENS_12zip_iteratorINS_5tupleINS_10device_ptrIiEES6_NS_9null_typeES7_S7_S7_S7_S7_S7_S7_EEEENS_6detail15normal_iteratorIS6_EE9IJCompareEEvRNS0_16execution_policyIT_EET0_SI_T1_T2_</div>
<div> at /usr/local/cuda/include/thrust/system/cuda/detail/sort.h:1669</div>
<div>#14 0x55bb46317bc5 in _ZN6thrust11sort_by_keyINS_8cuda_cub3tagENS_12zip_iteratorINS_5tupleINS_10device_ptrIiEES6_NS_9null_typeES7_S7_S7_S7_S7_S7_S7_EEEENS_6detail15normal_iteratorIS6_EE9IJCompareEEvRKNSA_21execution_policy_baseIT_EET0_SJ_T1_T2_</div>
<div> at /usr/local/cuda/include/thrust/detail/sort.inl:115</div>
<div>#15 0x55bb46317bc5 in _ZN6thrust11sort_by_keyINS_12zip_iteratorINS_5tupleINS_10device_ptrIiEES4_NS_9null_typeES5_S5_S5_S5_S5_S5_S5_EEEENS_6detail15normal_iteratorIS4_EE9IJCompareEEvT_SC_T0_T1_</div>
<div> at /usr/local/cuda/include/thrust/detail/sort.inl:305</div>
<div>#16 0x55bb46317bc5 in MatSetPreallocationCOO_SeqAIJCUSPARSE_Basic</div>
<div> at /home/mnv/Software/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu:4452/" originalsrc="http://aijcusparse.cu:4452/" shash="kJqW69hMIzRIHC9m7UYVdJ7JUXvVP7/IsId+AWnsxZuVyxkvlaYovCx/RQIaH9zcbhUYRIRdH38fdvrWh+kVfzzWefq0vgU/qC1mZ2Bu2f2c5zdMo27CUXKD+TQmVlGyOkY73He5P2SEjg+9LH20/StRInmxGzJb3IKri/AQoCc=" target="_blank">aijcusparse.cu:4452</a></div>
<div>#17 0x55bb46c5b27c in MatSetPreallocationCOO_MPIAIJCUSPARSE_Basic</div>
<div> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpicusparse/<a href="http://mpiaijcusparse.cu:173/" originalsrc="http://mpiaijcusparse.cu:173/" shash="FEur/OrSN6zmA3eg6xsfBjoXdaXuCrlVVwNdx3xtK/JPNq/v9DlNCga+A+0guA2zUaCxF3Mf37S9YXen7TYsRSGjUK7adF819LLVmpqHnDyRPY+hWRTdJ4h3zYDpEn0V4/8ZH7NrdJ5Wj3wdgF4y/NT+6OtN8UhU34FgIZdk5bQ=" target="_blank">mpiaijcusparse.cu:173</a></div>
<div>#18 0x55bb46c5b27c in MatSetPreallocationCOO_MPIAIJCUSPARSE</div>
<div> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpicusparse/<a href="http://mpiaijcusparse.cu:222/" originalsrc="http://mpiaijcusparse.cu:222/" shash="nn0SuS0pIThyDCpweR9dg2rfQ4JyRef3Sr9rQ6AH/kVj7xvlntbdbAljXMIi2yO88BaHIEaxJWUGIk/sJGZTJ2iRzMmmykLtIyUD5c0F2/MgCY9G+ViPlwpDUMwCX5Ftb++HTtzDkKgW8DSuxnyuQ6ANAbDk+ERaO54QYgMtjnA=" target="_blank">mpiaijcusparse.cu:222</a></div>
<div>#19 0x55bb468e01cf in MatSetPreallocationCOO</div>
<div> at /home/mnv/Software/petsc/src/mat/utils/gcreate.c:606</div>
<div>#20 0x55bb46b39c9b in MatProductSymbolic_MPIAIJBACKEND</div>
<div> at /home/mnv/Software/petsc/src/mat/impls/aij/mpi/mpiaij.c:7547</div>
<div>#21 0x55bb469015e5 in MatProductSymbolic</div>
<div> at /home/mnv/Software/petsc/src/mat/interface/matproduct.c:803</div>
<div>#22 0x55bb4694ade2 in MatPtAP</div>
<div> at /home/mnv/Software/petsc/src/mat/interface/matrix.c:9897</div>
<div>#23 0x55bb4696d3ec in MatCoarsenApply_MISK_private</div>
<div> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:283</div>
<div>#24 0x55bb4696eb67 in MatCoarsenApply_MISK</div>
<div> at /home/mnv/Software/petsc/src/mat/coarsen/impls/misk/misk.c:368</div>
<div>#25 0x55bb4695bd91 in MatCoarsenApply</div>
<div> at /home/mnv/Software/petsc/src/mat/coarsen/coarsen.c:97</div>
<div>#26 0x55bb478294d8 in PCGAMGCoarsen_AGG</div>
<div> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/agg.c:524</div>
<div>#27 0x55bb471d1cb4 in PCSetUp_GAMG</div>
<div> at /home/mnv/Software/petsc/src/ksp/pc/impls/gamg/gamg.c:631</div>
<div>#28 0x55bb464022cf in PCSetUp</div>
<div> at /home/mnv/Software/petsc/src/ksp/pc/interface/precon.c:994</div>
<div>#29 0x55bb4718b8a7 in KSPSetUp</div>
<div> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:406</div>
<div>#30 0x55bb4718f22e in KSPSolve_Private</div>
<div> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:824</div>
<div>#31 0x55bb47192c0c in KSPSolve</div>
<div> at /home/mnv/Software/petsc/src/ksp/ksp/interface/itfunc.c:1070</div>
<div>#32 0x55bb463efd35 in kspsolve_</div>
<div> at /home/mnv/Software/petsc/src/ksp/ksp/interface/ftn-auto/itfuncf.c:320</div>
<div>#33 0x55bb45e94b32 in ???</div>
<div>#34 0x55bb46048044 in ???</div>
<div>#35 0x55bb46052ea1 in ???</div>
<div>#36 0x55bb45ac5f8e in ???</div>
<div>#37 0x1526470fcd8f in ???</div>
<div>#38 0x1526470fce3f in ???</div>
<div>#39 0x55bb45aef55d in ???</div>
<div>#40 0xffffffffffffffff in ???</div>
<div>--------------------------------------------------------------------------</div>
<div>Primary job terminated normally, but 1 process returned</div>
<div>a non-zero exit code. Per user-direction, the job has been aborted.</div>
<div>--------------------------------------------------------------------------</div>
<div>--------------------------------------------------------------------------</div>
<div>mpirun noticed that process rank 0 with PID 1771753 on node dgx02 exited on signal 6 (Aborted).</div>
<div>--------------------------------------------------------------------------</div>
<div><br>
</div>
<div>BTW, I'm curious. If I set n MPI processes, each of them building a part of the linear system, and g GPUs, how does PETSc distribute those n pieces of system matrix and rhs in the g GPUs? Does it do some load balancing algorithm? Where can I read about
this?</div>
<div>Thank you and best Regards, I can also point you to my code repo in GitHub if you want to take a closer look.</div>
<div><br>
</div>
<div>Best Regards,</div>
<div>Marcos<br>
</div>
<br>
</div>
<div id="x_m_-6428481159717170951x_m_1722876366411198553appendonsend"></div>
<hr style="display:inline-block; width:98%">
<div id="x_m_-6428481159717170951x_m_1722876366411198553divRplyFwdMsg" dir="ltr">
<font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" target="_blank">junchao.zhang@gmail.com</a>><br>
<b>Sent:</b> Friday, August 11, 2023 10:52 AM<br>
<b>To:</b> Vanella, Marcos (Fed) <<a href="mailto:marcos.vanella@nist.gov" target="_blank">marcos.vanella@nist.gov</a>><br>
<b>Cc:</b> <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a> <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>><br>
<b>Subject:</b> Re: [petsc-users] CUDA error trying to run a job with two mpi processes and 1 GPU</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>Hi, Marcos,</div>
<div> Could you build petsc in debug mode and then copy and paste the whole error stack message?</div>
<div><br>
</div>
Thanks<br clear="all">
<div>
<div dir="ltr">
<div dir="ltr">--Junchao Zhang</div>
</div>
</div>
<br>
</div>
<br>
<div>
<div dir="ltr">On Thu, Aug 10, 2023 at 5:51 PM Vanella, Marcos (Fed) via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a>> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex; border-left:1px solid rgb(204,204,204); padding-left:1ex">
<div>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Hi, I'm trying to run a parallel matrix vector build and linear solution with PETSc on 2 MPI processes + one V100 GPU. I tested that the matrix build and solution is successful in CPUs only. I'm using cuda 11.5 and cuda enabled openmpi and gcc 9.3. When I run
the job with GPU enabled I get the following error:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace">terminate called after throwing an instance of 'thrust::system::system_error'</span>
<div><span style="font-family:"Courier New",monospace"> <b>what(): merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered</b></span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace">Program received signal SIGABRT: Process abort signal.</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace">Backtrace for this error:</span></div>
<div><span style="font-family:"Courier New",monospace">terminate called after throwing an instance of 'thrust::system::system_error'</span></div>
<div><span style="font-family:"Courier New",monospace"> what(): merge_sort: failed to synchronize: cudaErrorIllegalAddress: an illegal memory access was encountered</span></div>
<div><br>
</div>
<span style="font-family:"Courier New",monospace">Program received signal SIGABRT: Process abort signal.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">I'm new to submitting jobs in slurm that also use GPU resources, so I might be doing something wrong in my submission script. This is it:</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">#!/bin/bash
<div>#SBATCH -J test</div>
<div>#SBATCH -e /home/Issues/PETSc/test.err</div>
<div>#SBATCH -o /home/Issues/PETSc/test.log</div>
<div>#SBATCH --partition=batch</div>
<div>#SBATCH --ntasks=2</div>
<div>#SBATCH --nodes=1</div>
<div>#SBATCH --cpus-per-task=1</div>
<div>#SBATCH --ntasks-per-node=2</div>
<div>#SBATCH --time=01:00:00</div>
<div>#SBATCH --gres=gpu:1</div>
<div><br>
</div>
<div>export OMP_NUM_THREADS=1</div>
<div>module load cuda/11.5</div>
<div>module load openmpi/4.1.1</div>
<div><br>
</div>
<div>cd /home/Issues/PETSc</div>
<div><b>mpirun -n 2 </b>/home/fds/Build/ompi_gnu_linux/fds_ompi_gnu_linux test.fds
<b>-vec_type mpicuda -mat_type mpiaijcusparse -pc_type gamg</b></div>
<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">If anyone has any suggestions on how o troubleshoot this please let me know.</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Thanks!</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">Marcos<br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace"><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="font-family:"Courier New",monospace"><br>
</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>