<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">Hi Hao,<div class=""><br class=""></div><div class="">Strange… these must be internal errors to CUDA, as we don’t call any any of these directly. FYI, CUDA has 2 API’s:</div><div class=""><br class=""></div><div class="">1. The simpler “runtime” API (which we use) which handles many things like CUDA context management and the like behind the scenes.</div><div class="">2. The more advanced “driver” API, where one can explicitly control the above.<br class=""><div class=""><br class="webkit-block-placeholder"></div><div class="">The errors you are seeing are from the driver API. This would indicate to me that the bug may be on NVIDIA’s side; if they were caused on our end I would expect to get CUDA runtime errors.</div><div class=""><br class=""></div><div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div>Best regards,</div><div><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<br class=""></div></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On Jan 25, 2022, at 21:52, Hao DONG <<a href="mailto:dong-hao@outlook.com" class="">dong-hao@outlook.com</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta charset="UTF-8" class=""><div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks Jacob,<span class="Apple-converted-space"> </span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Right, I forgot to change the clone -b release to main - my mistake. The c++ dialect option now works without problem. </div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">My laptop system is indeed configured with WSL2, with Linux kernel of “5.10.60.1-microsoft-standard-WSL2”. And I have a windows 11 Version 21H2 (OS Build 22000.438) as the host system. The Nvidia driver version is “510.06” with cuda 11.4.<o:p class=""></o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Interestingly, my ex11fc can now get pass the second petscfinalize. The code can now get to the third loop of kspsolve and reach the mpifinalize without a problem. So changing to main branch solves the petscfinalize error problem. However, it still complains with an error like:<span class="Apple-converted-space"> </span><o:p class=""></o:p></div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------------------------------------------------------------</div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The call to cuEventDestory failed. This is a unrecoverable error and will</div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">cause the program to abort.</div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> cuEventDestory return value: 400</div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Check the cuda.h file for what the return value means.</div><div style="margin: 0in 0in 0in 0.5in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--------------------------------------------------------------------------</div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The full log files are also attached. I also noticed there are other event-management related errors like cuEventCreate and cuIpcGetEventHandle in the log. Does it give any insights on why we have the problem?</div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Cheers,<span class="Apple-converted-space"> </span></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hao</div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Sent from<span class="Apple-converted-space"> </span><a href="https://go.microsoft.com/fwlink/?LinkId=550986" style="color: blue; text-decoration: underline;" class="">Mail</a><span class="Apple-converted-space"> </span>for Windows</div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif; border: none; padding: 0in;" class=""><b class="">From:<span class="Apple-converted-space"> </span></b><a href="mailto:jacob.fai@gmail.com" style="color: blue; text-decoration: underline;" class="">Jacob Faibussowitsch</a><br class=""><b class="">Sent:<span class="Apple-converted-space"> </span></b>Tuesday, January 25, 2022 11:19 PM<br class=""><b class="">To:<span class="Apple-converted-space"> </span></b><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">Hao DONG</a><br class=""><b class="">Cc:<span class="Apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users</a>;<span class="Apple-converted-space"> </span><a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;" class="">Junchao Zhang</a><br class=""><b class="">Subject:<span class="Apple-converted-space"> </span></b>Re: [petsc-users] Strange CUDA failure with a second petscfinalize with PETSc 3.16</div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi Hao,<o:p class=""></o:p></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I have tried to git pull on the laptop from main and re-config as you suggested.<o:p class=""></o:p></div></div></blockquote><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">It looks like you’re still on the release branch. Do <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">```<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">$ git checkout main<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">$ git pull<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">```<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Then reconfigure. This is also why the cxx dialect flag did not work, I forgot that this change had not made it to release yet.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">my laptop setup is based on WSL<o:p class=""></o:p></div></div></blockquote><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">What version of windows do you have? And what version of WSL? And what version is the linux kernel? You will need at least WSL2 and both your NVIDIA driver, windows version, and linux kernel version are required to be fairly new AFAIK to be able to run CUDA on them. See here<span class="Apple-converted-space"> </span><a href="https://docs.nvidia.com/cuda/wsl-user-guide/index.html" style="color: blue; text-decoration: underline;" class="">https://docs.nvidia.com/cuda/wsl-user-guide/index.html</a>. <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">To get your windows version:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1. Press Windows key+R<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2. Type winver in the box, and press enter<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">3. You should see a line with Version and a build number. For example on my windows machine I see “Version 21H2 (OS Build 19044.1496)”<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">To get WSL version:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1. Open WSL<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2. Type uname -r, for example I get "5.10.60.1-microsoft-standard-wsl2"<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">To get NVIDIA driver version:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1. Open up the NVIDIA control panel<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2. Click on “System Information” in the bottom left corner<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">3. You should see a dual list, “Items” and “Details”. In the details column. You should see “Driver verion”. For example on my machine I see “511.23”<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="" class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<o:p class=""></o:p></span></div></div></div></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><o:p class=""></o:p></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Jan 25, 2022, at 03:42, Hao DONG <<a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">dong-hao@outlook.com</a>> wrote:<o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi Jacob,<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks for the comments – silly that I have overlooked the debugging flag so far. Unfortunately, I am out of office for a couple of days so I cannot confirm the result on my workstation, for now.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">However, I have a laptop with nvidia graphic card (old gtx1050, which is actually slower than the cpu in terms of double precision calculation), I have tried to git pull on the laptop from main and re-config as you suggested.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">However, using “--with-cxx-dialect=14” throws up an error like:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Unknown C++ dialect: with-cxx-dialect=14<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">And omitting the “--with-cuda-dialect=cxx14” also gives me a similar complaint with:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">CUDA Error: Using CUDA with PetscComplex requires a C++ dialect at least cxx11. Use --with-cxx-dialect=xxx and --with-cuda-dialect=xxx to specify a suitable compiler.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Eventually I was able to configure and compile with the following config setup:<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">./configure --prefix=/opt/petsc/debug --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 --with-scalar-type=complex --with-precision=double --with-cuda-dir=/usr/local/cuda --with-fortran-kernels=1 --with-cxx-dialect=cxx14 --with-cuda-dialect=cxx14 --with-debugging=1<o:p class=""></o:p></div></div><div style="margin-left: 0.5in;" class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">But still, I got the same error regarding cuda (still the error code 97 thing). I attached the configure and output log of my ex11 on my laptop – is there anything that can help pinpoint the problem? I can also confirm that PETSc 3.15.2 works well with my ex11fc code with cuda, on my laptop. Sadly, my laptop setup is based on WSL, which is far from an ideal environment to test CUDA. I will let you know once I get my hands on my workstations.<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Cheers,<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hao<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Sent from<span class="apple-converted-space"> </span><a href="https://go.microsoft.com/fwlink/?LinkId=550986" style="color: blue; text-decoration: underline;" class="">Mail</a><span class="apple-converted-space"> </span>for Windows<o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:<span class="apple-converted-space"> </span></b><a href="mailto:jacob.fai@gmail.com" style="color: blue; text-decoration: underline;" class="">Jacob Faibussowitsch</a><br class=""><b class="">Sent:<span class="apple-converted-space"> </span></b>Tuesday, January 25, 2022 1:22 AM<br class=""><b class="">To:<span class="apple-converted-space"> </span></b><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">Hao DONG</a><br class=""><b class="">Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users</a>;<span class="apple-converted-space"> </span><a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;" class="">Junchao Zhang</a><br class=""><b class="">Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Strange CUDA failure with a second petscfinalize with PETSc 3.16<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hi Hao,<o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-family: Menlo-Regular, serif;" class="">Any luck reproducing the CUDA problem?</span><o:p class=""></o:p></div></div></div></div></div></div></blockquote><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Sorry for the long radio silence, I still have not been able to reproduce the problem unfortunately. I have tried on a local machine, and a few larger clusters and all return without errors both with and without cuda…<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Can you try pulling the latest version of main, reconfiguring and trying again?<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">BTW, your configure arguments are a little wonky:<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">1. --with-clanguage=c - this isn’t needed, PETSc will default to C<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">2. --with-cxx-dialect=cxx14 --with-cuda-dialect=cxx14 - use --with-cxx-dialect=14 instead, PETSc will detect that you have gnu compilers and enable gnu extensions<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">3. -with-debugging=1 - this is missing an extra dash, but you also have optimization flags set so maybe just leave this one out<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<o:p class=""></o:p></div></div></div></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><br class=""><o:p class=""></o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Jan 23, 2022, at 21:29, Hao DONG <<a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">dong-hao@outlook.com</a>> wrote:<o:p class=""></o:p></div></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">Dear Jacob, </span><o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class=""> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">Any luck reproducing the CUDA problem? - just write to check in, in case somehow the response did not reach me (this happens to my colleagues abroad sometimes, probably due to the Wall). </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class=""> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">All the best,</span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">Hao</span><o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class=""><br class=""><br class=""><br class=""></span><o:p class=""></o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">On Jan 19, 2022, at 3:01 PM, Hao DONG <</span><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">dong-hao@outlook.com</span></a><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">> wrote:</span></p></blockquote></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt;" class=""></span><o:p class=""></o:p></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks Jacob for looking into this –<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">You can see the updated source code of ex11fc in the attachment – although there is not much that I modified (except for the jabbers I outputted). I also attached the full output (ex11fc.log) along with the configure.log file. It’s an old dual Xeon workstation (one of my “production” machines) with Linux kernel 5.4.0 and gcc 9.3.<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I simply ran the code with<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">mpiexec -np 2 ex11fc -usecuda<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">for GPU test. And as stated before, calling without the “-usecuda” option shows no errors.<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Please let me know if you find anything wrong with the configure/code. <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Cheers,<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hao<o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:<span class="apple-converted-space"> </span></b><a href="mailto:jacob.fai@gmail.com" style="color: blue; text-decoration: underline;" class="">Jacob Faibussowitsch</a><br class=""><b class="">Sent:<span class="apple-converted-space"> </span></b>Wednesday, January 19, 2022 3:38 AM<br class=""><b class="">To:<span class="apple-converted-space"> </span></b><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">Hao DONG</a><br class=""><b class="">Cc:<span class="apple-converted-space"> </span></b><a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;" class="">Junchao Zhang</a>;<span class="apple-converted-space"> </span><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users</a><br class=""><b class="">Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Strange CUDA failure with a second petscfinalize with PETSc 3.16<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Apologies, forgot to mention in my previous email but can you also include a copy of the full printout of the error message that you get? It will include all the command-line flags that you ran with (if any) so I can exactly mirror your environment.<o:p class=""></o:p></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<o:p class=""></o:p></div></div></div></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><br class=""><br class=""><o:p class=""></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Jan 18, 2022, at 14:06, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" style="color: blue; text-decoration: underline;" class="">jacob.fai@gmail.com</a>> wrote:<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">Can you send your updated source file as well as your configure.log (should be $PETSC_DIR/configure.log). I will see if I can reproduce the error on my end.</span></p><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)</span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class=""><br class=""><br class=""><br class=""><br class=""></span><o:p class=""></o:p></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 11pt; font-family: Calibri, sans-serif;"><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">On Jan 17, 2022, at 23:06, Hao DONG <</span><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class=""><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">dong-hao@outlook.com</span></a><span style="font-size: 9pt; font-family: Menlo-Regular, serif;" class="">> wrote:</span></p></blockquote></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 9pt;" class=""></span><o:p class=""></o:p></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Dear Junchao and Jacob,<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks a lot for the response – I also don’t understand why this is related to the device, especially on why the procedure can be successfully finished for *<b class="">once</b>* – As instructed, I tried to add a CHKERRA() macro after (almost) every petsc line – such as the initialization, mat assemble, ksp create, solve, mat destroy, etc. However, all other petsc commands returns with error code 0. It only gives me a similar (still not very informative) error after I call the petscfinalize (again for the second time), with error code 97: <span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: GPU error<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: cuda error 709 (cudaErrorContextIsDestroyed) : context is destroyed<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: See<span class="apple-converted-space"> </span><a href="https://petsc.org/release/faq/" style="color: blue; text-decoration: underline;" class="">https://petsc.org/release/faq/</a><span class="apple-converted-space"> </span>for trouble shooting.<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Petsc Release Version 3.16.3, unknown<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: ./ex11f on a named stratosphere by donghao Tue Jan 18 11:39:43 2022<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: Configure options --prefix=/opt/petsc/complex-double-with-cuda --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 COPTFLAGS="-O3 -mavx2" CXXOPTFLAGS="-O3 -mavx2" FOPTFLAGS="-O3 -ffree-line-length-none -mavx2" CUDAOPTFLAGS=-O3 --with-cxx-dialect=cxx14 --with-cuda-dialect=cxx14 --with-scalar-type=complex --with-precision=double --with-cuda-dir=/usr/local/cuda --with-debugging=1<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #1 PetscFinalize() at /home/donghao/packages/petsc-current/src/sys/objects/pinit.c:1638<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[0]PETSC ERROR: #2 User provided function() at User file:0<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I can also confirm that rolling back to petsc 3.15 will *<b class="">not</b>* see the problem, even with the new nvidia driver. And petsc 3.16.3 with an old nvidia driver (470.42) also get this same error. So it’s probably not connected to the nvidia driver.<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Any idea on where I should look at next?<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks a lot in advance, and all the best,<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hao<o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in;" class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class="">From:<span class="apple-converted-space"> </span></b><a href="mailto:jacob.fai@gmail.com" style="color: blue; text-decoration: underline;" class="">Jacob Faibussowitsch</a><br class=""><b class="">Sent:<span class="apple-converted-space"> </span></b>Sunday, January 16, 2022 12:12 AM<br class=""><b class="">To:<span class="apple-converted-space"> </span></b><a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;" class="">Junchao Zhang</a><br class=""><b class="">Cc:<span class="apple-converted-space"> </span></b><a href="mailto:petsc-users@mcs.anl.gov" style="color: blue; text-decoration: underline;" class="">petsc-users</a>;<span class="apple-converted-space"> </span><a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">Hao DONG</a><br class=""><b class="">Subject:<span class="apple-converted-space"> </span></b>Re: [petsc-users] Strange CUDA failure with a second petscfinalize with PETSc 3.16<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I don’t quite understand how it is getting to the CUDA error to be honest. None of the code in the stack trace is anywhere near the device code. Reading the error message carefully, it first chokes on PetscLogGetStageLog() from a call to PetscClassIdRegister():<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">PetscErrorCode PetscLogGetStageLog(PetscStageLog *stageLog)<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">{<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PetscFunctionBegin;<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PetscValidPointer(stageLog,1);<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> if (!petsc_stageLog) {<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> fprintf(stderr, "PETSC ERROR: Logging has not been enabled.\nYou might have forgotten to call PetscInitialize().\n");<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> PETSCABORT(MPI_COMM_WORLD, PETSC_ERR_SUP); // Here<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> }<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> ...<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">But then jumps to PetscFinalize(). You can also see the "You might have forgotten to call PetscInitialize().” message in the error message, just under the topmost level of the stack trace.<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Can you check the value of ierr of each function call (use the CHKERRA() macro to do so)? I suspect the problem here that errors occurring previously in the program are being ignored, leading to the garbled stack trace.<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Best regards,<br class=""><br class="">Jacob Faibussowitsch<br class="">(Jacob Fai - booss - oh - vitch)<o:p class=""></o:p></div></div></div></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><br class=""><br class=""><br class=""><br class=""><br class=""><o:p class=""></o:p></div></div></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Jan 14, 2022, at 20:58, Junchao Zhang <<a href="mailto:junchao.zhang@gmail.com" style="color: blue; text-decoration: underline;" class="">junchao.zhang@gmail.com</a>> wrote:<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Jacob, <o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Could you have a look as it seems the "invalid device context" is in your newly added module?<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks<o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">--Junchao Zhang<o:p class=""></o:p></div></div></div></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div></div><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div><div class=""><div class=""><div class=""><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class="">On Fri, Jan 14, 2022 at 12:49 AM Hao DONG <<a href="mailto:dong-hao@outlook.com" style="color: blue; text-decoration: underline;" class="">dong-hao@outlook.com</a>> wrote:<o:p class=""></o:p></div></div></div></div></div><blockquote style="border-style: none none none solid; border-left-width: 1pt; border-left-color: rgb(204, 204, 204); padding: 0in 0in 0in 6pt; margin: 5pt 0in 5pt 4.8pt;" class=""><div class=""><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Dear All, <o:p class=""></o:p></div></div></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p></div><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I have encountered a peculiar problem when fiddling with a code with PETSC 3.16.3 (which worked fine with PETSc 3.15). It is a very straight forward PDE-based optimization code which repeatedly solves a linearized PDE problem with KSP in a subroutine (the rest of the code does not contain any PETSc related content). The main program provides the subroutine with an MPI comm. Then I set the comm as PETSC_COMM_WORLD to tell PETSC to attach to it (and detach with it when the solving is finished each time).<o:p class=""></o:p></div></div></div></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p></div><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Strangely, I observe a CUDA failure whenever the petscfinalize is called for a *second* time. In other words, the first and second PDE calculations with GPU are fine (with correct solutions). The petsc code just fails after the SECOND petscfinalize command is called. You can also see the PETSC config in the error message:<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: GPU error<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: cuda error 201 (cudaErrorDeviceUninitialized) : invalid device context<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: See<span class="apple-converted-space"> </span><a href="https://petsc.org/release/faq/" target="_blank" style="color: blue; text-decoration: underline;" class="">https://petsc.org/release/faq/</a><span class="apple-converted-space"> </span>for trouble shooting.<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: Petsc Release Version 3.16.3, unknown<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: maxwell.gpu on a named stratosphere by hao Fri Jan 14 10:21:05 2022<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: Configure options --prefix=/opt/petsc/complex-double-with-cuda --with-cc=mpicc --with-cxx=mpicxx --with-fc=mpif90 COPTFLAGS="-O3 -mavx2" CXXOPTFLAGS="-O3 -mavx2" FOPTFLAGS="-O3 -ffree-line-length-none -mavx2" CUDAOPTFLAGS=-O3 --with-cxx-dialect=cxx14 --with-cuda-dialect=cxx14 --with-scalar-type=complex --with-precision=double --with-cuda-dir=/usr/local/cuda --with-debugging=1<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1]PETSC ERROR: #1 PetscFinalize() at /home/hao/packages/petsc-current/src/sys/objects/pinit.c:1638<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">You might have forgotten to call PetscInitialize().<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The EXACT line numbers in the error traceback are not available.<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Instead the line number of the start of the function is given.<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #1 PetscAbortFindSourceFile_Private() at /home/hao/packages/petsc-current/src/sys/error/err.c:35<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #2 PetscLogGetStageLog() at /home/hao/packages/petsc-current/src/sys/logging/utils/stagelog.c:29<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #3 PetscClassIdRegister() at /home/hao/packages/petsc-current/src/sys/logging/plog.c:2376<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #4 MatMFFDInitializePackage() at /home/hao/packages/petsc-current/src/mat/impls/mffd/mffd.c:45<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #5 MatInitializePackage() at /home/hao/packages/petsc-current/src/mat/interface/dlregismat.c:163<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">[1] #6 MatCreate() at /home/hao/packages/petsc-current/src/mat/utils/gcreate.c:77<o:p class=""></o:p></div></div></div></div><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">However, it doesn’t seem to affect the other part of my code, so the code can continue running until it gets to the petsc part again (the *<b class="">third</b>* time). Unfortunately, it doesn’t give me any further information even if I set the debugging to yes in the configure file. It also worth noting that PETSC without CUDA (i.e. with simple MATMPIAIJ) works perfectly fine. <o:p class=""></o:p></div></div></div></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p></div><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I am able to re-produce the problem with a toy code modified from ex11f. Please see the attached file (ex11fc.F90) for details. Essentially the code does the same thing as ex11f, but three times with a do loop. To do that I added an extra MPI_INIT/MPI_FINALIZE to ensure that the MPI communicator is not destroyed when PETSC_FINALIZE is called. I used the PetscOptionsHasName utility to check if you have “-usecuda” in the options. So running the code with and without that option can give you a comparison w/o CUDA. I can see that the code also fails after the second loop of the KSP operation. Could you kindly shed some lights on this problem?<o:p class=""></o:p></div></div></div></div></div><div class=""><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p></div><div class=""><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">I should say that I am not even sure if the problem is from PETSc, as I also accidentally updated the NVIDIA driver (for now it is 510.06 with cuda 11.6). And it is well known that NVIDIA can give you some surprise in the updates (yes, I know I shouldn’t have touched that if it’s not broken). But my CUDA code without PETSC (which basically does the same PDE thing, but with cusparse/cublas directly) seems to work just fine after the update. It is also possible that my petsc code related to CUDA was not quite “legitimate” – I just use:<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> MatSetType(A, MATMPIAIJCUSPARSE, ierr)<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">and<span class="apple-converted-space"> </span><o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> MatCreateVecs(A, u, PETSC_NULL_VEC, ierr)<o:p class=""></o:p></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">to make the data onto GPU. I would very much appreciate it if you could show me the “right” way to do that. <o:p class=""></o:p></div></div></div></div><p class="MsoNormal" style="margin: 0in 0in 0in 667.2pt; font-size: 11pt; font-family: Calibri, sans-serif;"> </p><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Thanks a lot in advance, and all the best,<o:p class=""></o:p></div></div></div></div></div><div style="margin-left: 40.8pt;" class=""><div class=""><div style="margin-left: 225.6pt;" class=""><div style="margin: 0in 0in 0in 974.4pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Hao<o:p class=""></o:p></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div></div></blockquote></div></div></blockquote></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div><div class=""><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div></div></div></blockquote></div></div></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><configure.log><ex11fc.F90><ex11fc.log><o:p class=""></o:p></div></div></div></blockquote></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div class=""><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><configure-xps.log><ex11fc-xps.log><o:p class=""></o:p></div></div></blockquote></div></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div style="margin: 0in; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div></div><span id="cid:56473585-10EF-4DCC-B744-E942A5E1B8F2"><configure-main.log></span><span id="cid:BE6492F1-92F7-48FA-8445-10E51B8645DB"><ex11fc-main.log></span></div></blockquote></div><br class=""></div></body></html>