[petsc-users] [EXTERNAL] Re: Problem creating an array of KSP objects

[Matthew Knepley]

On Fri, May 18, 2012 at 11:41 AM, Hautzenroeder, Brenna M <
bmhautz at sandia.gov> wrote:

>  Matt,****
>
> ** **
>
> Thanks for replying back to me. Actually, we are only seeing this problem
> on debug, non-fortran compilations of PETSc. When we turn on fortran and
> turn off debugging in the configuration, the behavior is normal, the timing
> is fast (as to be expected), and the storage space is more in lines with
> our calculations. Maybe this is expected behavior for all debug/non-fortran
> builds? Or maybe this is a test case for finding an edge-case?
>

No. KSP is most used part of PETSc, and has been there for > 15 years. We
run nightly valgrind tests to check for leaks.
I am confident that this is misconfiguration. Send a sample code to
petsc-maint at mcs.anl.gov if -malloc_dump does
not find the problem.

    Matt


>
>
> Anyways, thanks for your help!****
>
> ** **
>
> -Brenna****
>
> ** **
>
> *From:* petsc-users-bounces at mcs.anl.gov [mailto:
> petsc-users-bounces at mcs.anl.gov] *On Behalf Of *Matthew Knepley
> *Sent:* Thursday, May 17, 2012 10:59 AM
> *To:* PETSc users list
> *Subject:* [EXTERNAL] Re: [petsc-users] Problem creating an array of KSP
> objects****
>
> ** **
>
> On Thu, May 17, 2012 at 12:37 PM, Hautzenroeder, Brenna M <
> bmhautz at sandia.gov> wrote:****
>
>  I’m running into time and memory problems creating an array to store
> multiple KSP objects.****
>
>  ****
>
> We are using a Fourier method for the temporal portion of our operator;
> currently, this is anywhere between 1024-2048 time samples equating to
> 1024-2048 frequencies to advance our solution. Each frequency has its own
> matrix, meaning there are 1024-2048 linear equations of the form Ax = b.
> So, while the matrices are different for each frequency, when we propagate
> the solution one step, the 1024-2048 matrices don’t change since the
> frequencies haven’t changed between steps.****
>
>  ****
>
> We would like to store each KSP solution (all 1024-2048 of them) in an
> array so that we can reuse them for the next step since this saves us time
> factoring the matrix. The problem I am seeing is that storing these KSP
> objects in any kind of array increases the runtime and storage space
> exponentially – I am seeing times on the order of 30 mins to an hour and
> memory consumed hover around 20-30GB! Is there something inherent ****
>
>  ** **
>
> I don't think you really mean "exponential", but just big. Did you
> calculate how much space you expect? Storing a factor can take****
>
> N^3 space, and then you have 1000 of them, so it will take 1000x the space
> of 1 KSP. Did you compare the storage for both cases?****
>
> There is really not enough information in your question, however****
>
> ** **
>
>    Yes, storing KSPs is the right thing here****
>
> ** **
>
>    No, they do not take more space than necessary****
>
> ** **
>
>        Matt****
>
>  ****
>
>  about the KSP object that I am missing and/or a better way to store
> these objects? I don’t have any problems if I use one KSP object and set
> the operators each time to the different matrices, except that it takes
> longer than it normally would after the first step since it has to
> recalculate the preconditioner matrix.****
>
>  ****
>
> Note: We are using distributed matrices and vectors for our linear system.
> ****
>
>  ****
>
> I have included some pseudocode to illustrate our linear system, which
> works:****
>
>  ****
>
> KSP k;****
>
> for steps = 1 to 100 do:****
>
> for time = 1 to 1024 do:****
>
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)****
>
>                 create distributed vector b and fill with local values for
> given time****
>
>                 create distributed vector x for result****
>
>                 create KSP k and set matrices****
>
>                 KSPSolve(k, b, x) => takes a little extra time each step**
> **
>
>  ****
>
> However, the following creates time/storage problems:****
>
>  ****
>
> KSP* kArray;****
>
> create kArray => takes inordinate amout of time and storage****
>
> for steps = 1 to 100 do:****
>
>         for time=1 to 1024 do:****
>
>                 create distributed matrix A and fill with local values
> (A[step=1,time=1] = A[step=2,time=1], etc)****
>
>                 create distributed vector b and fill with local values for
> given time****
>
>                 create distributed vector x for result****
>
>                 if time == 1 do:****
>
> set matrices on kArray[time] => also takes an inordinate amount of time
> and storage****
>
>                 KSPSolve(kArray[time], b, x)****
>
>  ****
>
> Any insight would be greatly appreciated!****
>
> Brenna****
>
>  ****
>
>
>
> ****
>
> ** **
>
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener****
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20120518/fa375425/attachment.htm>