<div dir="ltr">My MR is generating an error. Tee error message says cusparse has not been initialized, so I added a cuparse init, but I still get the error (appended, <strong style="box-sizing:border-box;color:rgb(48,48,48);font-family:-apple-system,system-ui,"Segoe UI",Roboto,"Noto Sans",Ubuntu,Cantarell,"Helvetica Neue",sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol","Noto Color Emoji";font-size:14px"><span class="gmail-js-show-tooltip gmail-gl-min-w-0 gmail-label-branch gmail-label-truncate gmail-js-source-branch" title="" style="box-sizing:border-box;min-width:0px;font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13.3px;font-weight:normal;overflow:hidden;word-break:break-all"><a href="https://gitlab.com/petsc/petsc/-/tree/adams/landau-gpu-assembly" style="box-sizing:border-box;color:rgb(0,86,179);background-color:transparent;display:inline-block;max-width:12.5em;white-space:nowrap;text-overflow:ellipsis;line-height:14px;overflow:hidden">adams/landau-gpu-assembly</a></span></strong>).  Any ideas would be appreciated.<div><br></div><div>I am trying to reproduce this on Summit and it fails with a timeout limit of 60s, but it only runs for a few seconds (see timers). Any ideas?</div><div><br></div><div>19:58 adams/landau-gpu-assembly= ~/petsc$ make -f gmakefile test search='ksp_ksp_tutorials-ex71_bddc_cusparse' PETSC_ARCH=arch-summit-opt-gnu-cuda<br>Using MAKEFLAGS: PETSC_ARCH=arch-summit-opt-gnu-cuda search=ksp_ksp_tutorials-ex71_bddc_cusparse<br>        TEST arch-summit-opt-gnu-cuda/tests/counts/ksp_ksp_tutorials-ex71_bddc_cusparse.counts<br>not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Exceeded timeout limit of 60 s<br> ok ksp_ksp_tutorials-ex71_bddc_cusparse # SKIP Command failed so no diff<br><br># -------------<br>#   Summary<br># -------------<br># FAILED ksp_ksp_tutorials-ex71_bddc_cusparse<br># success 0/1 tests (0.0%)<br># failed 1/1 tests (100.0%)<br># todo 0/1 tests (0.0%)<br># skip 0/1 tests (0.0%)<br>#<br># Wall clock time for tests: 3 sec<br># Approximate CPU time (not incl. build time): 3.14 sec<br></div><div><br></div><div><br><div><br></div><div><br></div><div><br></div><div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">not ok ksp_ksp_tutorials-ex71_bddc_cusparse # Error code: 201</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2391" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2391" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2391</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#   [1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2392" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2392" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2392</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#      [1]PETSC ERROR: GPU error </span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2393" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2393" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2393</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#      [1]PETSC ERROR: cuSPARSE error 1 (CUSPARSE_STATUS_NOT_INITIALIZED) : initialization error</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2394" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2394" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2394</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#       [1]PETSC ERROR: See <a href="https://www.mcs.anl.gov/petsc/documentation/faq.html" class="gmail-gl-reset-color! gmail-gl-text-decoration-underline" rel="nofollow noopener noreferrer" style="box-sizing:border-box;background-color:transparent;color:inherit">https://www.mcs.anl.gov/petsc/documentation/faq.html</a> for trouble shooting.</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2395" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2395" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2395</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#      [1]PETSC ERROR: Petsc Development GIT revision: v3.14.2-85-gd60087d  GIT Date: 2020-12-09 17:49:59 -0500</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2396" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2396" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2396</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#        [1]PETSC ERROR: ../ex71 on a  named frog by petsc Wed Dec  9 18:41:10 2020</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2397" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2397" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2397</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap">#      [1]PETSC ERROR: Configure options --package-prefix-hash=/home/petsc/petsc-hash-pkgs --with-make-test-np=2 COPTFLAGS="-g -O" FOPTFLAGS="-g -O" CXXOPTFLAGS="-g -O" --with-scalar-type=complex --with-precision=single --with-cuda-dir=/usr/local/cuda-11.0 PETSC_ARCH=arch-ci-linux-cuda11-complex</span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2398" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2398" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2398</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap"># [1]PETSC ERROR: #1 MatConvert_SeqAIJ_SeqAIJCUSPARSE() line 2708 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu">aijcusparse.cu</a></span></div><div class="gmail-js-line gmail-log-line" style="box-sizing:border-box;padding:1px 8px 1px 55px;min-height:1.25rem;color:rgb(255,255,255);font-family:Menlo,"DejaVu Sans Mono","Liberation Mono",Consolas,"Ubuntu Mono","Courier New","andale mono","lucida console",monospace;font-size:13px;background-color:rgb(17,17,17)"><a id="gmail-L2399" href="https://gitlab.com/petsc/petsc/-/jobs/901108135#L2399" class="gmail-gl-link gmail-d-inline-block gmail-text-right gmail-line-number gmail-flex-shrink-0" style="box-sizing:border-box;color:rgb(102,102,102);background-color:transparent;font-size:0.875rem;padding:0px 1em 0px 8px;min-width:50px;display:inline-block;text-align:right">2399</a><span class="gmail-gl-white-space-pre-wrap" style="box-sizing:border-box;white-space:pre-wrap"># [1]PETSC ERROR: #2 MatCreate_SeqAIJCUSPARSE() line 2739 in /home/petsc/builds/KFnbdjNX/0/petsc/petsc/src/mat/impls/aij/seq/seqcusparse/<a href="http://aijcusparse.cu">aijcusparse.cu</a></span></div></div></div></div>