[petsc-users] Limiting the number of vectors allocated at a time by fgmres etc.

[Wells, David]

Hi Matt,

Thanks for the quick response!

> I cannot understand precisely what is happening here. You specify a restart
> size when you setup the KSP. It allocates that many vecs (roughly). Why are
> there reallocations? Do you increase the restart size during the iteration?

I don't believe there are any reallocations (I didn't write this solver, but I
don't see any calls which set the restart size or any other relevant parameter [1]):
as far as I can tell, the solver just allocates a lot of vectors. I'm working
off of traces computed by heaptrack, which is my only insight into how this
works. The allocations come from KSPCreateVecs(), which is called by
1. KSPFGMRESGetNewVectors() (for about 1.7 GB [2] of memory)
2. KSPSetUp_GMRES() (for about 300 MB of memory)
3. KSPSetUp_FGMRES() (for about 264 MB of memory)
4. KSPSetWorkVecs() (for about 236 MB of memory)

Is there some relevant set of monitoring flags I can set which will show me how
many vectors I allocate or use? That would also help.

Best,
David

[1] This is IBAMR's PETScKrylovLinearSolver.
[2] This is half the total memory we use for side-centered data vectors.
________________________________
From: Matthew Knepley <knepley at gmail.com>
Sent: Tuesday, February 10, 2026 6:28 PM
To: Wells, David <drwells at email.unc.edu>
Cc: petsc-users at mcs.anl.gov <petsc-users at mcs.anl.gov>
Subject: Re: [petsc-users] Limiting the number of vectors allocated at a time by fgmres etc.

On Tue, Feb 10, 2026 at 5:32 PM Wells, David via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:
Hello,

I've been profiling the memory usage of my solver and it looks like a huge number (roughly half) of allocations are from KSPFGMRESGetNewVectors(). I read through the source code and it looks like these vectors are allocated ten at a time (FGMRES_DELTA_DIRECTIONS) in a couple of places inside that KSP. Is there a way to change this value?

We could add an option to change this delta. Actually theory suggests that a constant is not optimal, but rather we should double the number each time. I would also be willing to code that.

If not - how hard would it be to add an API to set a different initial value for that? These vectors take up a lot of memory and I would rather just one at a time.

I cannot understand precisely what is happening here. You specify a restart size when you setup the KSP. It allocates that many vecs (roughly). Why are there reallocations? Do you increase the restart size during the iteration?

  Thanks,

     Matt

Best,
David Wells


--
What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aBTZ4oaySeEsyg3U98-DZfMmoj0Wg8RUNEzlPdQoEnZ6pfCuP3cZSF7ib3bqpT7GBGVct81F2vjmak_Zva98zbh7D1M$ <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!aBTZ4oaySeEsyg3U98-DZfMmoj0Wg8RUNEzlPdQoEnZ6pfCuP3cZSF7ib3bqpT7GBGVct81F2vjmak_Zva9875mpIc0$ >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20260211/0e4c850f/attachment-0001.html>