<html><head><meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> <div class=""> Yes, these messages are because of lack of resources even though the error message says not initialized.</div><div class=""><br class=""></div><div class=""> Barry</div><div class=""><br class=""><div><br class=""><blockquote type="cite" class=""><div class="">On Dec 9, 2020, at 8:01 PM, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" class="">junchao.zhang@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><div dir="ltr" class="">Could be GPU resource competition. Note this test uses nsize=8.<br clear="all" class=""><div class=""><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class="">--Junchao Zhang</div></div></div><br class=""></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 9, 2020 at 7:15 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">And this is a Cuda 11 complex build: <a href="https://gitlab.com/petsc/petsc/-/jobs/901108135" target="_blank" class="">https://gitlab.com/petsc/petsc/-/jobs/901108135</a></div><br class=""><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Dec 9, 2020 at 8:11 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank" class="">mfadams@lbl.gov</a>> wrote:<br class=""></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr" class="">My MR is generating an error. Tee error message says cusparse has not been initialized, so I added a cuparse init, but I still get the error (appended, <strong style="box-sizing:border-box;color:rgb(48,48,48);font-family:-apple-system,system-ui,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";font-size:14px" class=""><span title="" style="box-sizing:border-box;min-width:0px;font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13.3px;font-weight:normal;overflow:hidden;word-break:break-all" class=""><a href="https://gitlab.com/petsc/petsc/-/tree/adams/landau-gpu-assembly" style="box-sizing:border-box;color:rgb(0,86,179);background-color:transparent;display:inline-block;max-width:12.5em;white-space:nowrap;text-overflow:ellipsis;line-height:14px;overflow:hidden" target="_blank" class="">adams/landau-gpu-assembly</a></span></strong>). Any ideas would be appreciated.<div class=""><br class=""></div><div class="">I am trying to reproduce this on Summit and it fails with a timeout limit of 60s, but it only runs for a few seconds (see timers). Any ideas?</div><div class=""><br class=""></div><div class="">19:58 adams/landau-gpu-assembly= ~/petsc$ make -f gmakefile test search='ksp_ksp_tutorials-ex71_bddc_cusparse' PETSC_ARCH=arch-summit-opt-gnu-cuda<br class="">Using MAKEFLAGS: PETSC_ARCH=arch-summit-opt-gnu-cuda search=ksp_ksp_tutorials-ex71_bddc_cusparse<br class=""> TEST arch-summit-opt-gnu-cuda/tests/counts/ksp_ksp_tutorials-ex71_bddc_cusparse.counts<br class="">not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Exceeded timeout limit of 60 s<br class=""> ok ksp_ksp_tutorials-ex71_bddc_cusparse # SKIP Command failed so no diff<br class=""><br class=""># -------------<br class=""># Summary<br class=""># -------------<br class=""># FAILED ksp_ksp_tutorials-ex71_bddc_cusparse<br class=""># success 0/1 tests (0.0%)<br class=""># failed 1/1 tests (100.0%)<br class=""># todo 0/1 tests (0.0%)<br class=""># skip 0/1 tests (0.0%)<br class="">#<br class=""># Wall clock time for tests: 3 sec<br class=""># Approximate CPU time (not incl. build time): 3.14 sec<br class=""></div><div class=""><br class=""></div><div class=""><br class=""><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><br class=""></div><div class=""><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><span style="box-sizing:border-box;white-space:pre-wrap" class="">not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Error code: 201</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2391" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2391" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2391</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2392" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2392" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2392</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: GPU error </span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2393" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2393" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2393</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: cuSPARSE error 1 (CUSPARSE_STATUS_NOT_INITIALIZED) : initialization error</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2394" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2394" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2394</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" rel="nofollow noopener noreferrer" style="box-sizing:border-box;background-color:transparent;color:inherit" target="_blank" class="">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2395" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2395" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2395</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: Petsc Development GIT revision: v3.14.2-85-gd60087d GIT Date: 2020-12-09 17:49:59 -0500</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2396" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2396" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2396</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: ../ex71 on a named frog by petsc Wed Dec 9 18:41:10 2020</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2397" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2397" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2397</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: Configure options --package-prefix-hash=/home/petsc/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" --with-scalar-type=complex --with-precision=single --with-cuda-dir=/usr/local/cuda-11.0 PETSC_ARCH=arch-ci-linux-cuda11-complex</span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2398" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2398" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2398</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: #1 MatConvert_SeqAIJ_SeqAIJCUSPARSE() line 2708 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu/" target="_blank" class="">aijcusparse.cu</a></span></div><div style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)" class=""><a id="gmail-m_720156393060744899gmail-m_3274428905510193615gmail-L2399" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2399" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right" target="_blank" class="">2399</a><span style="box-sizing:border-box;white-space:pre-wrap" class=""># [1]PETSC ERROR: #2 MatCreate_SeqAIJCUSPARSE() line 2739 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu/" target="_blank" class="">aijcusparse.cu</a></span></div></div></div></div>
</blockquote></div>
</blockquote></div>
</div></blockquote></div><br class=""></div></body></html>