<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Jed,</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
That seems to have worked (ridiculously well). It's now 55MB, and it's happening in the call to MatSetPreallocationCOO.</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Thank you,<br>
</div>
<div class="elementToProof">
<div style="font-family: Consolas, Courier, monospace; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="Signature">
<div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="font-size: 11pt;"><b>Philip Fackler<br>
</b></span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Research Software Engineer, Application Engineering Group</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Advanced Computing Systems Research Section</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);">Computer Science and Mathematics Division<br>
</span></div>
<div><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>Oak Ridge National Laboratory</b></span><span style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);"><br>
</span></div>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Jed Brown <jed@jedbrown.org><br>
<b>Sent:</b> Thursday, December 14, 2023 16:27<br>
<b>To:</b> Fackler, Philip <facklerpw@ornl.gov>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov>; xolotl-psi-development@lists.sourceforge.net <xolotl-psi-development@lists.sourceforge.net><br>
<b>Subject:</b> [EXTERNAL] Re: [petsc-users] Call to DMSetMatrixPreallocateSkip not changing allocation behavior</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">I had a one-character typo in the diff above. This MR to release should work now.<br>
<br>
<a href="https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e=">https://urldefense.us/v2/url?u=https-3A__gitlab.com_petsc_petsc_-2D_merge-5Frequests_7120&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=h_jIP-6WcIjR6LssfGrV6Z2DojlN_w7Me4-a4rBE074&e=</a>
<br>
<br>
Jed Brown <jed@jedbrown.org> writes:<br>
<br>
> 17 GB for a 1D DMDA, wow. :-)<br>
><br>
> Could you try applying this diff to make it work for DMDA (it's currently handled by DMPlex)?<br>
><br>
> diff --git i/src/dm/impls/da/fdda.c w/src/dm/impls/da/fdda.c<br>
> index cad4d926504..bd2a3bda635 100644<br>
> --- i/src/dm/impls/da/fdda.c<br>
> +++ w/src/dm/impls/da/fdda.c<br>
> @@ -675,19 +675,21 @@ PetscErrorCode DMCreateMatrix_DA(DM da, Mat *J)<br>
> specialized setting routines depend only on the particular preallocation<br>
> details of the matrix, not the type itself.<br>
> */<br>
> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij));<br>
> - if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij));<br>
> - if (!aij) {<br>
> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij));<br>
> - if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij));<br>
> - if (!baij) {<br>
> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij));<br>
> - if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij));<br>
> - if (!sbaij) {<br>
> - PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell));<br>
> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell));<br>
> + if (!dm->prealloc_skip) { // Flag is likely set when user intends to use MatSetPreallocationCOO()<br>
> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIAIJSetPreallocation_C", &aij));<br>
> + if (!aij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqAIJSetPreallocation_C", &aij));<br>
> + if (!aij) {<br>
> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPIBAIJSetPreallocation_C", &baij));<br>
> + if (!baij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqBAIJSetPreallocation_C", &baij));<br>
> + if (!baij) {<br>
> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISBAIJSetPreallocation_C", &sbaij));<br>
> + if (!sbaij) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSBAIJSetPreallocation_C", &sbaij));<br>
> + if (!sbaij) {<br>
> + PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatMPISELLSetPreallocation_C", &sell));<br>
> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatSeqSELLSetPreallocation_C", &sell));<br>
> + }<br>
> + if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is));<br>
> }<br>
> - if (!sell) PetscCall(PetscObjectQueryFunction((PetscObject)A, "MatISSetPreallocation_C", &is));<br>
> }<br>
> }<br>
> if (aij) {<br>
><br>
><br>
> "Fackler, Philip via petsc-users" <petsc-users@mcs.anl.gov> writes:<br>
><br>
>> I'm using the following sequence of functions related to the Jacobian matrix:<br>
>><br>
>> DMDACreate1d(..., &da);<br>
>> DMSetFromOptions(da);<br>
>> DMSetUp(da);<br>
>> DMSetMatType(da, MATAIJKOKKOS);<br>
>> DMSetMatrixPreallocateSkip(da, PETSC_TRUE);<br>
>> Mat J;<br>
>> DMCreateMatrix(da, &J);<br>
>> MatSetPreallocationCOO(J, ...);<br>
>><br>
>> I recently added the call to DMSetMatrixPreallocateSkip, hoping the allocation would be delayed to MatSetPreallocationCOO, and that it would require less memory. The documentation<<a href=""></a>https://urldefense.us/v2/url?u=https-3A__petsc.org_release_manualpages_DM_DMSetMatrixPreallocateSkip_&d=DwIBAg&c=v4IIwRuZAmwupIjowmMWUmLasxPEgYsgNI-O7C4ViYc&r=DAkLCjn8leYU-uJ-kfNEQMhPZWx9lzc4d5KgIR-RZWQ&m=v9sHqomCGBRWotign4NcwYwOpszOJehUGs_EO3eGn4SSZqxnfK7Iv15-X8nO1lii&s=IMLBs0ydxDPvuXeD6jmsq1BN_8oieHVSyG6VA9c0DyM&e=
> says that the data structures will not be preallocated. The following data from heaptrack shows that the allocation is still happening in the call to DMCreateMatrix.<br>
>><br>
>> [cid:bda9ef12-a46f-47b2-9b9b-a4b2808b6b13]<br>
>><br>
>> Can someone help me understand this?<br>
>><br>
>> Thanks,<br>
>><br>
>> Philip Fackler<br>
>> Research Software Engineer, Application Engineering Group<br>
>> Advanced Computing Systems Research Section<br>
>> Computer Science and Mathematics Division<br>
>> Oak Ridge National Laboratory<br>
</div>
</span></font></div>
</body>
</html>