<div dir="ltr">Could be GPU resource competition. Note this test uses nsize=8.<br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 9, 2020 at 7:15 PM Mark Adams <<a href="mailto:mfadams@lbl.gov">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">And this is a Cuda 11 complex build: <a href="https://gitlab.com/petsc/petsc/-/jobs/901108135" target="_blank">https://gitlab.com/petsc/petsc/-/jobs/901108135</a></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 9, 2020 at 8:11 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">My MR is generating an error. Tee error message says cusparse has not been initialized, so I added a cuparse init, but I still get the error (appended, <strong style="box-sizing:border-box;color:rgb(48,48,48);font-family:-apple-system,system-ui,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";font-size:14px"><span title="" style="box-sizing:border-box;min-width:0px;font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13.3px;font-weight:normal;overflow:hidden;word-break:break-all"><a href="https://gitlab.com/petsc/petsc/-/tree/adams/landau-gpu-assembly" style="box-sizing:border-box;color:rgb(0,86,179);background-color:transparent;display:inline-block;max-width:12.5em;white-space:nowrap;text-overflow:ellipsis;line-height:14px;overflow:hidden" target="_blank">adams/landau-gpu-assembly</a></span></strong>).  Any ideas would be appreciated.<div><br></div><div>I am trying to reproduce this on Summit and it fails with a timeout limit of 60s, but it only runs for a few seconds (see timers). Any ideas?</div><div><br></div><div>19:58 adams/landau-gpu-assembly= ~/petsc$ make -f gmakefile test search='ksp_ksp_tutorials-ex71_bddc_cusparse' PETSC_ARCH=arch-summit-opt-gnu-cuda<br>Using MAKEFLAGS: PETSC_ARCH=arch-summit-opt-gnu-cuda search=ksp_ksp_tutorials-ex71_bddc_cusparse<br>        TEST arch-summit-opt-gnu-cuda/tests/counts/ksp_ksp_tutorials-ex71_bddc_cusparse.counts<br>not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Exceeded timeout limit of 60 s<br> ok ksp_ksp_tutorials-ex71_bddc_cusparse # SKIP Command failed so no diff<br><br># -------------<br>#   Summary<br># -------------<br># FAILED ksp_ksp_tutorials-ex71_bddc_cusparse<br># success 0/1 tests (0.0%)<br># failed 1/1 tests (100.0%)<br># todo 0/1 tests (0.0%)<br># skip 0/1 tests (0.0%)<br>#<br># Wall clock time for tests: 3 sec<br># Approximate CPU time (not incl. build time): 3.14 sec<br></div><div><br></div><div><br><div><br></div><div><br></div><div><br></div><div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><span style="box-sizing:border-box;white-space:pre-wrap">not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Error code: 201</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2391" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2391" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2391</a><span style="box-sizing:border-box;white-space:pre-wrap">#  [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2392" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2392" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2392</a><span style="box-sizing:border-box;white-space:pre-wrap">#  [1]PETSC ERROR: GPU error </span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2393" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2393" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2393</a><span style="box-sizing:border-box;white-space:pre-wrap">#  [1]PETSC ERROR: cuSPARSE error 1 (CUSPARSE_STATUS_NOT_INITIALIZED) : initialization error</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2394" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2394" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2394</a><span style="box-sizing:border-box;white-space:pre-wrap">#   [1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" rel="nofollow noopener noreferrer" style="box-sizing:border-box;background-color:transparent;color:inherit" target="_blank">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2395" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2395" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2395</a><span style="box-sizing:border-box;white-space:pre-wrap">#   [1]PETSC ERROR: Petsc Development GIT revision: v3.14.2-85-gd60087d  GIT Date: 2020-12-09 17:49:59 -0500</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2396" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2396" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2396</a><span style="box-sizing:border-box;white-space:pre-wrap">#    [1]PETSC ERROR: ../ex71 on a  named frog by petsc Wed Dec  9 18:41:10 2020</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2397" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2397" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2397</a><span style="box-sizing:border-box;white-space:pre-wrap">#  [1]PETSC ERROR: Configure options --package-prefix-hash=/home/petsc/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" --with-scalar-type=complex --with-precision=single --with-cuda-dir=/usr/local/cuda-11.0 PETSC_ARCH=arch-ci-linux-cuda11-complex</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2398" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2398" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2398</a><span style="box-sizing:border-box;white-space:pre-wrap">#     [1]PETSC ERROR: #1 MatConvert_SeqAIJ_SeqAIJCUSPARSE() line 2708 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu" target="_blank">aijcusparse.cu</a></span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2399" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2399" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank">2399</a><span style="box-sizing:border-box;white-space:pre-wrap">#   [1]PETSC ERROR: #2 MatCreate_SeqAIJCUSPARSE() line 2739 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu" target="_blank">aijcusparse.cu</a></span></div></div></div></div>
</blockquote></div>
</blockquote></div>