<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<div class="">1. Commenting out ierr = __initialize(dctx->device->deviceId,dci);CHKERRQ(ierr); in device/impls/cupm/cupmcontext.hpp:L199</div>
<div class=""><br class="">
</div>
<div class="">CUDA memory: 1.575GB</div>
<div class="">CUDA memory without importing torch: 0.370GB</div>
<div class=""><br class="">
</div>
<div class="">This has the same effect as commenting out L437-L440 in interface/device.cxx </div>
<div class=""><br class="">
</div>
<div class="">2. Comment out these two: </div>
<div class="">. <span class="">src/sys/objects/device/impls/cupm/cupmdevice.cxx:</span>327 [<span class="">ierr = _devices[_defaultDevice]->configure();CHKERRQ(ierr);]</span></div>
<div class="">. <span class="">src/sys/objects/device/impls/cupm/cupmdevice.cxx:326 [</span>ierr = _devices[_defaultDevice]->initialize();CHKERRQ(ierr);]</div>
<div class="">
<div class=""><br class="">
</div>
<div class="">
<div class="">CUDA memory: 1.936GB</div>
<div class="">CUDA memory without importing torch: 0.730GB</div>
</div>
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:21 AM, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<blockquote type="cite" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
They had no influence to the memory usage. </div>
</blockquote>
???????????????????????????????????????????????????????????????????????
<div class=""><br class="">
</div>
<div class="">Comment out the ierr = _devices[id]->initialize();CHKERRQ(ierr); on line 360 in cupmdevice.cxx as well.</div>
<div class=""><br class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 12:18, Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
I have tried all of these. They had no influence to the memory usage.
<div class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:15 AM, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<blockquote type="cite" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<div class="">Initializing cutlass and cusolver does not affect the memory usage. I did the following to turn them off:</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
Ok next things to try out in order:
<div class=""><br class="">
</div>
<div class="">1. src/sys/objects/device/impls/cupm/cupmcontext.hpp:178 [PetscFunctionBegin;] </div>
<div class="">Put a PetscFunctionReturn(0); right after this</div>
<div class=""><br class="">
</div>
<div class="">2. <span style="caret-color: rgb(0, 0, 0);" class="">src/sys/objects/device/impls/cupm/cupmdevice.cxx:</span>327 [<span style="caret-color: rgb(0, 0, 0);" class="">ierr = _devices[_defaultDevice]->configure();CHKERRQ(ierr);]</span></div>
<div class=""><font class=""><span style="caret-color: rgb(0, 0, 0);" class="">Comment this out</span></font></div>
<div class="">
<div class=""><br class="">
</div>
<div class="">3. <span style="caret-color: rgb(0, 0, 0);" class="">src/sys/objects/device/impls/cupm/cupmdevice.cxx:326 [</span>ierr = _devices[_defaultDevice]->initialize();CHKERRQ(ierr);]<br class="">
Comment this out</div>
<div class=""><br class="">
</div>
</div>
<div class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 12:02, Zhang, Hong <<a href="mailto:hongzhang@anl.gov" class="">hongzhang@anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Initializing cutlass and cusolver does not affect the memory usage. I did the following to turn them off:</div>
<div class=""><br class="">
</div>
<div class="">diff --git a/src/sys/objects/device/impls/cupm/cupmcontext.hpp b/src/sys/objects/device/impls/cupm/cupmcontext.hpp</div>
<div class="">index 51fed809e4d..9a5f068323a 100644</div>
<div class="">--- a/src/sys/objects/device/impls/cupm/cupmcontext.hpp</div>
<div class="">+++ b/src/sys/objects/device/impls/cupm/cupmcontext.hpp</div>
<div class="">@@ -199,7 +199,7 @@ inline PetscErrorCode CUPMContext<T>::setUp(PetscDeviceContext dctx) noexcept</div>
<div class=""> #if PetscDefined(USE_DEBUG)</div>
<div class=""> dci->timerInUse = PETSC_FALSE;</div>
<div class=""> #endif</div>
<div class="">- ierr = __initialize(dctx->device->deviceId,dci);CHKERRQ(ierr);</div>
<div class="">+ //ierr = __initialize(dctx->device->deviceId,dci);CHKERRQ(ierr);</div>
<div class=""> PetscFunctionReturn(0);</div>
<div class=""> }</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 10:53 AM, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
I don't think this is right. We want the device initialized by PETSc , we just don't want the cublas and cusolve stuff initialized. In order to see how much memory initializing the blas and solvers takes.
<div class=""><br class="">
</div>
<div class=""> So I think you need to comment things in cupminterface.hpp like cublasCreate and cusolverDnCreate.</div>
<div class=""><br class="">
</div>
<div class=""> Urgh, I hate C++ where huge chunks of real code are in header files.</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:34 AM, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Hit send too early…
<div class=""><br class="">
</div>
<div class="">If you don’t want to comment out, you can also run with "-device_enable lazy" option. Normally this is the default behavior but if -log_view or -log_summary is provided this defaults to “-device_enable eager”. See src/sys/objects/device/interface/device.cxx:398</div>
<div class=""><br class="">
</div>
<div class="">
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:29, Jacob Faibussowitsch <<a href="mailto:jacob.fai@gmail.com" class="">jacob.fai@gmail.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<blockquote type="cite" class="">
<div class="" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">
<span class="" style="color: rgb(29, 28, 29); font-family: Slack-Lato, appleLogo, sans-serif; font-size: 15px; font-variant-ligatures: common-ligatures; orphans: 2; widows: 2; background-color: rgb(248, 248, 248); text-decoration-thickness: initial;">You need
to go into the PetscInitialize() routine find where it loads the cublas and cusolve and comment out those lines then run with -log_view</span></div>
</blockquote>
<div class=""><br class="">
</div>
Comment out
<div class="">
<div class=""><br class="">
</div>
<div class="">#if (PetscDefined(HAVE_CUDA) || PetscDefined(HAVE_HIP) || PetscDefined(HAVE_SYCL))</div>
<div class=""> ierr = PetscDeviceInitializeFromOptions_Internal(PETSC_COMM_WORLD);CHKERRQ(ierr);</div>
<div class="">#endif</div>
<div class=""><br class="">
</div>
<div class="">At <span style="caret-color: rgb(0, 0, 0);" class="">src/sys/objects/pinit.c:956</span></div>
<div class=""><br class="">
</div>
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Best regards,<br class="">
<br class="">
Jacob Faibussowitsch<br class="">
(Jacob Fai - booss - oh - vitch)<br class="">
</div>
</div>
</div>
</div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:24, Barry Smith <<a href="mailto:bsmith@petsc.dev" class="">bsmith@petsc.dev</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><br class="">
</div>
<span style="color: rgb(29, 28, 29); font-family: Slack-Lato, appleLogo, sans-serif; font-size: 15px; font-variant-ligatures: common-ligatures; orphans: 2; widows: 2; background-color: rgb(248, 248, 248); text-decoration-thickness: initial;" class="">Without
log_view it does not load any cuBLAS/cuSolve immediately with -log_view it loads all that stuff at startup. You need to go into the PetscInitialize() routine find where it loads the cublas and cusolve and comment out those lines then run with -log_view</span>
<div class="">
<div style="orphans: 2; widows: 2;" class=""><font color="#1d1c1d" face="Slack-Lato, appleLogo, sans-serif" class=""><span style="caret-color: rgb(29, 28, 29); font-size: 15px; background-color: rgb(248, 248, 248);" class=""><br class="">
</span></font></div>
<div class=""><br class="">
<blockquote type="cite" class="">
<div class="">On Jan 7, 2022, at 11:14 AM, Zhang, Hong via petsc-dev <<a href="mailto:petsc-dev@mcs.anl.gov" class="">petsc-dev@mcs.anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span class="">When PETSc is initialized, it takes about 2GB CUDA memory. This is way too much for doing nothing. A test script is attached to reproduce the issue. If I remove the first line "import torch", PETSc consumes about 0.73GB, which is still significant.
Does anyone have any idea about this behavior?</span>
<div class=""><br class="">
</div>
<div class="">Thanks,</div>
<div class="">Hong<br class="">
<div class=""><br class="">
</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); font-size: 12px; line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">hongzhang@gpu02:/gpfs/jlse-fs0/users/hongzhang/Projects/pnode/examples (caidao22/update-examples)$ python3 test.py
CUDA memory before PETSc 0.000GB
CUDA memory after PETSc 0.004GB
hongzhang@gpu02:/gpfs/jlse-fs0/users/hongzhang/Projects/pnode/examples (caidao22/update-examples)$ python3 test.py -log_view :0.txt
CUDA memory before PETSc 0.000GB
CUDA memory after PETSc 1.936GB</pre>
<div class=""><br class="">
</div>
</div>
<div class="">
<pre class="c-mrkdwn__pre" data-stringify-type="pre" style="box-sizing: inherit; margin-top: 4px; margin-bottom: 4px; padding: 8px; --saf-0: rgba(var(--sk_foreground_low,29,28,29),0.13); font-size: 12px; line-height: 1.50001; font-variant-ligatures: none; white-space: pre-wrap; overflow-wrap: break-word; word-break: normal; tab-size: 4; border: 1px solid var(--saf-0); border-top-left-radius: 4px; border-top-right-radius: 4px; border-bottom-right-radius: 4px; border-bottom-left-radius: 4px; background: rgba(var(--sk_foreground_min,29,28,29),0.04); counter-reset: list-0 0 list-1 0 list-2 0 list-3 0 list-4 0 list-5 0 list-6 0 list-7 0 list-8 0 list-9 0; color: rgb(29, 28, 29); orphans: 2; widows: 2; text-decoration-thickness: initial; font-family: Monaco, Menlo, Consolas, "Courier New", monospace !important;">import torch
import sys
import os
import nvidia_smi
nvidia_smi.nvmlInit()
handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
print('CUDA memory before PETSc %.3fGB' % (info.used/1e9))
petsc4py_path = os.path.join(os.environ['PETSC_DIR'],os.environ['PETSC_ARCH'],'lib')
sys.path.append(petsc4py_path)
import petsc4py
petsc4py.init(sys.argv)
handle = nvidia_smi.nvmlDeviceGetHandleByIndex(0)
info = nvidia_smi.nvmlDeviceGetMemoryInfo(handle)
print('CUDA memory after PETSc %.3fGB' % (info.used/1e9))</pre>
<div class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>