[petsc-users] Matrix reuse with changing structure
Matthew Knepley
knepley at gmail.com
Thu Mar 17 10:00:24 CDT 2016
On Thu, Mar 17, 2016 at 9:29 AM, Smith, Kevin R. <Kevin.R.Smith at jhuapl.edu>
wrote:
> Hi Matt,
>
>
>
> Yeah, if I need to put zeros in all potential columns at the start, I
> think that matrix will become essentially dense. The size of my problem is
> such that I need to keep the matrix sparse.
>
>
>
> Yes, what you described is exactly what I’m after.
>
Okay, I think it may be enough to 0 out the row lengths and then carry out
your MatSetValues() again. Here is how to try it
#include <../src/mat/impls/aij/seq/aij.h>
#undef __FUNCT__
#define __FUNCT__ "ClearStructure"
PetscErrorCode ClearStructure(Mat A)
{
Mat_SeqAIJ *a = (Mat_SeqAIJ *) A->data;
PetscFunctionBeginUser;
ierr = PetscMemzero(a->ailen, A->rmap->n *
sizeof(PetscInt));CHKERRQ(ierr);
PetscFunctionReturn(0);
}
Make sure you remember to put in zero for any missing spaces before
assembling the matrix. Let us know how it goes.
Thanks,
Matt
> Thanks,
>
> Kevin
>
>
>
> *From:* Matthew Knepley [mailto:knepley at gmail.com]
> *Sent:* Thursday, March 17, 2016 10:07 AM
>
> *To:* Smith, Kevin R.
> *Cc:* Barry Smith; petsc-users at mcs.anl.gov
> *Subject:* Re: [petsc-users] Matrix reuse with changing structure
>
>
>
> On Thu, Mar 17, 2016 at 8:54 AM, Smith, Kevin R. <Kevin.R.Smith at jhuapl.edu>
> wrote:
>
> Hi Barry,
>
> Thanks! So this approach will allow the non-zero structure to change
> while avoiding reallocations? From what I understood from the documentation
> and discussion so far is that the non-zero structure is locked in once you
> do SetValue calls. I would think that while this approach does prevent
> PETSc from throwing away the oversubscribed pre-allocation, it would still
> not allow the matrix structure to change. I can only predict the
> preallocation beforehand, not the structure of the matrix during its entire
> lifetime.
>
>
>
> The strategy Barry suggested means that you have to 0 out all potential
> columns at the start.
>
>
>
> So suppose you translate your small box over the entire coarse mesh. You
> might say, at any
>
> one time I never fill up more than C columns, however the strategy above
> would mean you
>
> would need to 0 out every column in all those rows.
>
>
>
> It sounds like what you want is something like this. At each iteration,
>
>
>
> 1) Wipe out nonzero structure, but keep allocation. This returns us to
>
> the situation right after MatSetPreallocation*()
>
>
>
> 2) Fill up matrix, including enough 0s to fill up the allocation
>
>
>
> If that is what you want, we can show you how to do it.
>
>
>
> Thanks,
>
>
>
> Matt
>
>
>
> Thank you very much for taking the time to discuss this with me.
>
> Kind regards,
> Kevin
>
> -----Original Message-----
> From: Barry Smith [mailto:bsmith at mcs.anl.gov]
>
> Sent: Wednesday, March 16, 2016 2:10 PM
> To: Smith, Kevin R.
> Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> Subject: Re: [petsc-users] Matrix reuse with changing structure
>
>
> You need to do two things
>
> 1) over preallocate
> 2) fill up all possible nonzero entries in the matrix at the beginning
> before calling the first MatAssemblyBegin/End()
>
> then it should run fine and efficiently (except it will always compute
> with those extra zeros but so long as they are relatively small amounts
> that is ok).
>
> The reason you need to do 2 is that Mat "throws away" any "extra"
> preallocation that you provided but did not set values into in the first
> MatAssemblyBegin/End. If you do not do 2) then it will need to do all the
> mallocs later when you set the new nonzero locations and so the over
> preallocation doesn't help at all.
>
> Barry
>
> > On Mar 16, 2016, at 11:19 AM, Smith, Kevin R. <Kevin.R.Smith at jhuapl.edu>
> wrote:
> >
> > The use case is dynamic overset mesh coupling. I can guarantee that all
> matrices will fit since I have all the information I need to do so at
> initialization. The component meshes do not themselves change, only the
> overset assembly changes. I have to over-allocate memory to ensure the
> matrix always fits to cover the range of motion.
> >
> > I did figure out a way to avoid deleting the matrix every time I solve
> in my system, so that provides some level of an optimization if PETSc
> doesn’t support restructuring sparse matrices out of the box. With this
> approach I can also avoid over-allocating memory. However, my use case is
> unlikely to hit the memory limits, even if I have to over-subscribe the
> preallocation to cover the entire range of motion. So I think there will
> still be a clear benefit to doing the allocation once, and avoiding the
> reallocation each time structure changes.
> >
> > Not sure if this figure will get filtered out by the mailing list, but
> this shows a basic overset matrix structure. The green portions are the
> overset coupling regions, which may move around the matrix as the body
> moves. The number of coupling coefficients has a maximum N which is known,
> so I can preallocate to cover the entire range of motion. Hope that helps
> describe the use case.
> >
> > <image001.png>
> >
> > From: Matthew Knepley [mailto:knepley at gmail.com]
> > Sent: Wednesday, March 16, 2016 10:37 AM
> > To: Smith, Kevin R.
> > Cc: Barry Smith; petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Matrix reuse with changing structure
> >
> > On Wed, Mar 16, 2016 at 8:23 AM, Smith, Kevin R. <
> Kevin.R.Smith at jhuapl.edu> wrote:
> > Barry - I need to avoid dynamic allocation because it results in too
> much of a slowdown.
> > Matt - The MatSetType thing did not work for some reason. It seems like
> PETSc wants me to set the preallocation after calling this method
> (suggested by the error below). I'm trying to reuse the original block of
> preallocated memory for the sparse matrix and blow away the structure each
> time. This would let me avoid repeated deallocate/allocate calls throughout
> my simulation.
> >
> > We need to understand the use case better. How can you guarantee that
> > your new matrix will fit in the old matrix memroy? If you cannot, then
> we have to reallocate anyway.
> >
> > Thanks,
> >
> > Matt
> >
> > PETSc Function: MatGetOwnershipRange
> > PETSc File:
> > /share/maul-data/smithkr1/src/petsc-3.6.0/src/mat/interface/matrix.c
> > PETSc Line: 6289
> > PETSc Error Code: 73
> > PETSc Error reference:
> > http://www.mcs.anl.gov/petsc/petsc-dev/include/petscerror.h.html
> > PETSc Message: Must call MatXXXSetPreallocation() or MatSetUp() on
> > argument 1 "mat" before MatGetOwnershipRange()
> >
> > Kind regards,
> > Kevin
> >
> > -----Original Message-----
> > From: Barry Smith [mailto:bsmith at mcs.anl.gov]
> > Sent: Tuesday, March 15, 2016 2:02 PM
> > To: Smith, Kevin R.
> > Cc: Matthew Knepley; petsc-users at mcs.anl.gov
> > Subject: Re: [petsc-users] Matrix reuse with changing structure
> >
> >
> > > On Mar 15, 2016, at 12:56 PM, Smith, Kevin R. <
> Kevin.R.Smith at jhuapl.edu> wrote:
> > >
> > >
> > > Barry - Yeah, I suspected it was doing this. In my original
> implementation I would get allocation errors.
> >
> > You need to call a MatSetOption() to tell the matrix that it is
> allowed to allocate new nonzeros.
> >
> >
> > >
> > > Matt - Thanks, I will try this solution out.
> > >
> > > Thanks for your help,
> > >
> > > Kevin
> > >
> > > -----Original Message-----
> > > From: Barry Smith [mailto:bsmith at mcs.anl.gov]
> > > Sent: Tuesday, March 15, 2016 1:48 PM
> > > To: Matthew Knepley
> > > Cc: Smith, Kevin R.; petsc-users at mcs.anl.gov
> > > Subject: Re: [petsc-users] Matrix reuse with changing structure
> > >
> > >
> > >> On Mar 15, 2016, at 12:28 PM, Matthew Knepley <knepley at gmail.com>
> wrote:
> > >>
> > >> On Tue, Mar 15, 2016 at 12:24 PM, Smith, Kevin R. <
> Kevin.R.Smith at jhuapl.edu> wrote:
> > >> Hello,
> > >>
> > >>
> > >>
> > >> Is it possible to reuse a sparse matrix and not reuse its non-zero
> structure? I have a matrix whose sparse structure changes each time. I’m
> hoping to avoid destroying and allocating new matrices each time the
> structure changes.
> > >
> > > Sure you can just add new nonzero locations at a later time. But it
> won't take out any current entries even if they are zero. So effectively
> the nonzero structure grows over time.
> > >
> > > Barry
> > >
> > >>
> > >>
> > >> Hmm, I can't find a toplevel API that does this (it would be
> > >> something like MatReset()). You can get this effect using
> > >>
> > >> MatSetType(A, MATSHELL)
> > >> MatSetType(A, <type you had before>)
> > >>
> > >> A little messy but it will work.
> > >>
> > >> Thanks,
> > >>
> > >> Matt
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Kevin
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > >> -- Norbert Wiener
> > >
> >
> >
> >
> >
> > --
> > What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> > -- Norbert Wiener
>
>
>
>
>
> --
>
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20160317/023f9919/attachment-0001.html>
More information about the petsc-users
mailing list