<div dir="ltr"><div>Jacob, </div><div> Could you have a look as it seems the "invalid device context" is in your newly added module?</div><div> Thanks<br></div><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">--Junchao Zhang</div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 14, 2022 at 12:49 AM Hao DONG <<a href="mailto:dong-hao@outlook.com">dong-hao@outlook.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US" style="overflow-wrap: break-word;">
<div class="gmail-m_-2563461583173086559WordSection1">
<p class="MsoNormal">Dear All, <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I have encountered a peculiar problem when fiddling with a code with PETSC 3.16.3 (which worked fine with PETSc 3.15). It is a very straight forward PDE-based optimization code which repeatedly solves a linearized PDE problem with KSP in
a subroutine (the rest of the code does not contain any PETSc related content). The main program provides the subroutine with an MPI comm. Then I set the comm as PETSC_COMM_WORLD to tell PETSC to attach to it (and detach with it when the solving is finished
each time). <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Strangely, I observe a CUDA failure whenever the petscfinalize is called for a *second* time. In other words, the first and second PDE calculations with GPU are fine (with correct solutions). The petsc code just fails after the SECOND
petscfinalize command is called. You can also see the PETSC config in the error message:
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: GPU error<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: cuda error 201 (cudaErrorDeviceUninitialized) : invalid device context<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: See <a href="https://petsc.org/release/faq/" target="_blank">https://petsc.org/release/faq/</a> for trouble shooting.<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: Petsc Release Version 3.16.3, unknown<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: maxwell.gpu on a named stratosphere by hao Fri Jan 14 10:21:05 2022<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: Configure options --prefix=/opt/petsc/complex-double-with-cuda --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 COPTFLAGS="-O3 -mavx2" CXXOPTFLAGS="-O3 -mavx2" FOPTFLAGS="-O3 -ffree-line-length-none -mavx2" CUDAOPTFLAGS=-O3
--with-cxx-dialect=cxx14 --with-cuda-dialect=cxx14 --with-scalar-type=complex --with-precision=double --with-cuda-dir=/usr/local/cuda --with-debugging=1<u></u><u></u></p>
<p class="MsoNormal">[1]PETSC ERROR: #1 PetscFinalize() at /home/hao/packages/petsc-current/src/sys/objects/pinit.c:1638<u></u><u></u></p>
<p class="MsoNormal">You might have forgotten to call PetscInitialize().<u></u><u></u></p>
<p class="MsoNormal">The EXACT line numbers in the error traceback are not available.<u></u><u></u></p>
<p class="MsoNormal">Instead the line number of the start of the function is given.<u></u><u></u></p>
<p class="MsoNormal">[1] #1 PetscAbortFindSourceFile_Private() at /home/hao/packages/petsc-current/src/sys/error/err.c:35<u></u><u></u></p>
<p class="MsoNormal">[1] #2 PetscLogGetStageLog() at /home/hao/packages/petsc-current/src/sys/logging/utils/stagelog.c:29<u></u><u></u></p>
<p class="MsoNormal">[1] #3 PetscClassIdRegister() at /home/hao/packages/petsc-current/src/sys/logging/plog.c:2376<u></u><u></u></p>
<p class="MsoNormal">[1] #4 MatMFFDInitializePackage() at /home/hao/packages/petsc-current/src/mat/impls/mffd/mffd.c:45<u></u><u></u></p>
<p class="MsoNormal">[1] #5 MatInitializePackage() at /home/hao/packages/petsc-current/src/mat/interface/dlregismat.c:163<u></u><u></u></p>
<p class="MsoNormal">[1] #6 MatCreate() at /home/hao/packages/petsc-current/src/mat/utils/gcreate.c:77<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">However, it doesn’t seem to affect the other part of my code, so the code can continue running until it gets to the petsc part again (the *<b>third</b>* time). Unfortunately, it doesn’t give me any further information even if I set the
debugging to yes in the configure file. It also worth noting that PETSC without CUDA (i.e. with simple MATMPIAIJ) works perfectly fine. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I am able to re-produce the problem with a toy code modified from ex11f. Please see the attached file (ex11fc.F90) for details. Essentially the code does the same thing as ex11f, but three times with a do loop. To do that I added an extra
MPI_INIT/MPI_FINALIZE to ensure that the MPI communicator is not destroyed when PETSC_FINALIZE is called. I used the PetscOptionsHasName utility to check if you have “-usecuda” in the options. So running the code with and without that option can give you
a comparison w/o CUDA. I can see that the code also fails after the second loop of the KSP operation. Could you kindly shed some lights on this problem?
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I should say that I am not even sure if the problem is from PETSc, as I also accidentally updated the NVIDIA driver (for now it is 510.06 with cuda 11.6). And it is well known that NVIDIA can give you some surprise in the updates (yes,
I know I shouldn’t have touched that if it’s not broken). But my CUDA code without PETSC (which basically does the same PDE thing, but with cusparse/cublas directly) seems to work just fine after the update. It is also possible that my petsc code related to
CUDA was not quite “legitimate” – I just use: <u></u><u></u></p>
<p class="MsoNormal"> MatSetType(A, MATMPIAIJCUSPARSE, ierr)<u></u><u></u></p>
<p class="MsoNormal">and <u></u><u></u></p>
<p class="MsoNormal"> MatCreateVecs(A, u, PETSC_NULL_VEC, ierr)<u></u><u></u></p>
<p class="MsoNormal">to make the data onto GPU. I would very much appreciate it if you could show me the “right” way to do that.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks a lot in advance, and all the best,<u></u><u></u></p>
</div>
<p class="MsoNormal">Hao<u></u><u></u></p>
</div>
</div>
</blockquote></div>