<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Hi Barry,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Sorry for the slow response - yes, that makes sense, and it is a sensible default to have. I appreciate the writeup.</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
<br>
</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
Best,</div>
<div style="font-family: Aptos, Aptos_EmbeddedFont, Aptos_MSFontService, Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);" class="elementToProof">
David</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Barry Smith <bsmith@petsc.dev><br>
<b>Sent:</b> Tuesday, February 10, 2026 9:32 PM<br>
<b>To:</b> Wells, David <drwells@email.unc.edu><br>
<b>Cc:</b> Matthew Knepley <knepley@gmail.com>; petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b> Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc.</font>
<div> </div>
</div>
<div style="line-break:after-white-space">
<table border="0" cellspacing="0" cellpadding="0" width="100%" align="left" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; border:0!important; display:table!important; width:100%!important; table-layout:fixed!important; border-collapse:seperate!important; float:none!important; border-spacing:0px 0px!important">
<tbody style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; display:block!important">
<tr style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
<td valign="middle" width="1px" bgcolor="#A6A6A6" cellpadding="7px 2px 7px 2px" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; padding:7px 2px 7px 2px!important; background-color:#A6A6A6!important; width:0px!important">
</td>
<td valign="middle" width="100%" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 15px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:100%!important; background-color:#EAEAEA!important; padding:7px 5px 7px 15px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
<div style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important">
You don't often get email from bsmith@petsc.dev. <a href="https://urldefense.us/v3/__https://aka.ms/LearnAboutSenderIdentification__;!!G_uCfscf7eWS!fGPdLXU1J2z90EOLzcOjKmEhpgVOTMjLIZFkcTqlZ1InQNY5B79RxHFXzmqiSb7O09i1rWuXObNqZRj-hZmQ43XwjnQ$" style="background:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; opacity:revert!important; visibility:revert!important">
Learn why this is important</a> </div>
</td>
<td valign="middle" align="left" width="75px" bgcolor="#EAEAEA" cellpadding="7px 5px 7px 5px" color="#212121" style="background:revert!important; border:revert!important; bottom:revert!important; color:revert!important; direction:revert!important; display:revert!important; font-size:revert!important; height:revert!important; letter-spacing:revert!important; line-height:revert!important; margin:revert!important; opacity:revert!important; order:revert!important; outline:revert!important; overflow:revert!important; padding:revert!important; position:revert!important; tab-size:revert!important; table-layout:revert!important; text-align:revert!important; text-indent:revert!important; text-orientation:revert!important; text-overflow:revert!important; text-transform:revert!important; top:revert!important; vertical-align:revert!important; visibility:revert!important; white-space:revert!important; width:revert!important; word-break:revert!important; word-spacing:revert!important; writing-mode:revert!important; zoom:revert!important; width:75px!important; background-color:#EAEAEA!important; padding:7px 5px 7px 5px!important; font-family:wf_segoe-ui_normal,Segoe UI,Segoe WP,Tahoma,Arial,sans-serif!important; font-size:12px!important; font-weight:normal!important; color:#212121!important; text-align:left!important; word-wrap:break-word!important">
</td>
</tr>
</tbody>
</table>
<div>
<div><br>
</div>
1) For a fixed restart (say of 30) FGMRES needs 60 vectors, while GMRES only needs 30. This is a big disadvantage of FGMRES over GMRES.
<div><br>
</div>
<div> 2) By default PETSc GMRES uses a restart of 30 meaning it keeps 30 previous Krylov vectors (and FGMRES needs 60 vectors). You can use a smaller restart with KSPGMRESSetRestart or -ksp_gmres_restart to need less memory (of course the convergence may get
far worse or not depending on the problem.</div>
<div><br>
</div>
<div> 3) When GMRES (or FGMRES) starts up it does not immediately allocate all 30 (or whatever) restart vectors because it may be that GMRES only takes 15 steps so why allocate all of them? Instead it allocates a chunk at a time GMRES_DELTA_DIRECTIONS which
is 10 when it uses up the 10 it allocates another 10 (if needed) etc until it gets to the restart You can force GMRES to allocate all 30 (or whatever) initially instead of the chunk of a time approach by using “KSPGMRESSetPreAllocateVectors() or -ksp_gmres_preallocate
This will prevent confusion about why more vectors are allocated later and why they are not all allocated when the solve starts.</div>
<div><br>
</div>
<div>4) PETSc’s GMRES tries to use BLAS 2 operations for MDot() and MAXPY (the orthogonalization in GMRES). It can only use the BLAS 2 on vector chunks that are allocated together. By preallocating all the vectors at the beginning one gets a single chunk and
hence more efficient orthogonalization; this is more important on GPUs. For CPUs whether you have 10 or 30 vectors together doesn’t matter much at all.</div>
<div><br>
</div>
<div> I hope this clarifies why you are seeing the memory allocations. Note that these are NOT “reallocations” in the sense of KSPGMRES allocating more memory and then copying something into the new memory and freeing the old. They are just allocations of
new memory which will then be used.</div>
<div><br>
</div>
<div> Barry</div>
<div><br>
</div>
<div><br>
</div>
<div> <br id="x_lineBreakAtBeginningOfMessage">
<div><br>
<blockquote type="cite">
<div>On Feb 10, 2026, at 9:04 PM, Wells, David via petsc-users <petsc-users@mcs.anl.gov> wrote:</div>
<br class="x_Apple-interchange-newline">
<div><span style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">Hi
Matt,</span><span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Thanks for the quick response!</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
> I cannot understand precisely what is happening here. You specify a restart</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
> size when you setup the KSP. It allocates that many vecs (roughly). Why are</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
> there reallocations? Do you increase the restart size during the iteration?</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
I don't believe there are any reallocations (I didn't write this solver, but I</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
don't see any calls which set the restart size or any other relevant parameter [1]):</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
as far as I can tell, the solver just allocates a lot of vectors. I'm working</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
off of traces computed by heaptrack, which is my only insight into how this</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
works. The allocations come from KSPCreateVecs(), which is called by</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
1. KSPFGMRESGetNewVectors() (for about 1.7 GB [2] of memory)</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
2. KSPSetUp_GMRES() (for about 300 MB of memory)</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
3. KSPSetUp_FGMRES() (for about 264 MB of memory)</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
4. KSPSetWorkVecs() (for about 236 MB of memory)</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Is there some relevant set of monitoring flags I can set which will show me how</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
many vectors I allocate or use? That would also help.</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Best,</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
David</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
[1] This is IBAMR's PETScKrylovLinearSolver.</div>
<div class="x_elementToProof" style="font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
[2] This is half the total memory we use for side-centered data vectors.</div>
<div id="x_appendonsend" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid">
</div>
<hr tabindex="-1" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid; display:inline-block; width:662.46875px">
<span style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration:none; float:none; display:inline!important"></span>
<div id="x_divRplyFwdMsg" dir="ltr" style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid">
<font face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b><span class="x_Apple-converted-space"> </span>Matthew Knepley <knepley@gmail.com><br>
<b>Sent:</b><span class="x_Apple-converted-space"> </span>Tuesday, February 10, 2026 6:28 PM<br>
<b>To:</b><span class="x_Apple-converted-space"> </span>Wells, David <drwells@email.unc.edu><br>
<b>Cc:</b><span class="x_Apple-converted-space"> </span>petsc-users@mcs.anl.gov <petsc-users@mcs.anl.gov><br>
<b>Subject:</b><span class="x_Apple-converted-space"> </span>Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc.</font>
<div> </div>
</div>
<div style="font-family:Helvetica; font-size:12px; font-style:normal; font-variant-caps:normal; font-weight:400; letter-spacing:normal; orphans:2; text-align:start; text-indent:0px; text-transform:none; white-space:normal; widows:2; word-spacing:0px; text-decoration-line:none; text-decoration-style:solid">
<div dir="ltr">
<div dir="ltr">On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov">petsc-users@mcs.anl.gov</a>> wrote:</div>
<div class="x_x_gmail_quote x_x_gmail_quote_container">
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_msg-4134260419650533412">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Hello,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
<br>
</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS)
in a couple of places inside that KSP. Is there a way to change this value?</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that.</div>
<div> </div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_msg-4134260419650533412">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time.</div>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?</div>
<div><br>
</div>
<div> Thanks,</div>
<div><br>
</div>
<div> Matt</div>
<div> </div>
<blockquote class="x_x_gmail_quote" style="margin:0px 0px 0px 0.8ex; border-left-width:1px; border-left-style:solid; border-left-color:rgb(204,204,204); padding-left:1ex">
<div class="x_x_msg-4134260419650533412">
<div dir="ltr">
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
Best,</div>
<div style="font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Calibri,Helvetica,sans-serif; font-size:12pt">
David Wells</div>
</div>
</div>
</blockquote>
</div>
<div><br clear="all">
</div>
<div><br>
</div>
<span class="x_x_gmail_signature_prefix">--<span class="x_Apple-converted-space"> </span></span><br>
<div dir="ltr" class="x_x_gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener</div>
<div><br>
</div>
<div><a href="https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aBTZ4oaySeEsyg3U98-DZfMmoj0Wg8RUNEzlPdQoEnZ6pfCuP3cZSF7ib3bqpT7GBGVct81F2vjmak_Zva9875mpIc0$" target="_blank">https://www.cse.buffalo.edu/~knepley/</a></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>