[petsc-users] Problem creating an array of KSP objects

[Matthew Knepley]

On Thu, May 17, 2012 at 12:37 PM, Hautzenroeder, Brenna M <
bmhautz at sandia.gov> wrote:

>  I’m running into time and memory problems creating an array to store
> multiple KSP objects.
>
> We are using a Fourier method for the temporal portion of our operator;
> currently, this is anywhere between 1024-2048 time samples equating to
> 1024-2048 frequencies to advance our solution. Each frequency has its own
> matrix, meaning there are 1024-2048 linear equations of the form Ax = b.
> So, while the matrices are different for each frequency, when we propagate
> the solution one step, the 1024-2048 matrices don’t change since the
> frequencies haven’t changed between steps.
>
> We would like to store each KSP solution (all 1024-2048 of them) in an
> array so that we can reuse them for the next step since this saves us time
> factoring the matrix. The problem I am seeing is that storing these KSP
> objects in any kind of array increases the runtime and storage space
> exponentially – I am seeing times on the order of 30 mins to an hour and
> memory consumed hover around 20-30GB! Is there something inherent
>

I don't think you really mean "exponential", but just big. Did you
calculate how much space you expect? Storing a factor can take
N^3 space, and then you have 1000 of them, so it will take 1000x the space
of 1 KSP. Did you compare the storage for both cases?
There is really not enough information in your question, however

   Yes, storing KSPs is the right thing here

   No, they do not take more space than necessary

       Matt


> about the KSP object that I am missing and/or a better way to store these
> objects? I don’t have any problems if I use one KSP object and set the
> operators each time to the different matrices, except that it takes longer
> than it normally would after the first step since it has to recalculate the
> preconditioner matrix.
>
> Note: We are using distributed matrices and vectors for our linear system.
>
> I have included some pseudocode to illustrate our linear system, which
> works:
>
> KSP k;
> for steps = 1 to 100 do:
> for time = 1 to 1024 do:
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)
>                 create distributed vector b and fill with local values for
> given time
>                 create distributed vector x for result
>                 create KSP k and set matrices
>                 KSPSolve(k, b, x) => takes a little extra time each step
>
> However, the following creates time/storage problems:
>
> KSP* kArray;
> create kArray => takes inordinate amout of time and storage
> for steps = 1 to 100 do:
>         for time=1 to 1024 do:
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)
>                 create distributed vector b and fill with local values for
> given time
>                 create distributed vector x for result
>                 if time == 1 do:
> set matrices on kArray[time] => also takes an inordinate amount of time
> and storage
>                 KSPSolve(kArray[time], b, x)
>
> Any insight would be greatly appreciated!
> Brenna
>
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120517/1a5cb57c/attachment.htm>