<div dir="ltr">I don't think this will work w/o a lot of pain. You would have to figure out how much of the preallocation you did not use in each row and add random (new) zero columns to each row as needed.<div><br></div><div>And is this really buying you much? PETSc will have to redo all the maps anyway. This is just saving one big malloc and maybe a few reductions. Right?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Mar 17, 2016 at 10:29 AM, Smith, Kevin R. <span dir="ltr"><<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Matt,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yeah, if I need to put zeros in all potential columns at the start, I think that matrix will become essentially dense. The size of my problem is such that I
need to keep the matrix sparse. <u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Yes, what you described is exactly what I’m after.
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Kevin<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Matthew Knepley [mailto:<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]
<br>
<b>Sent:</b> Thursday, March 17, 2016 10:07 AM</span></p><div><div class="h5"><br>
<b>To:</b> Smith, Kevin R.<br>
<b>Cc:</b> Barry Smith; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
<b>Subject:</b> Re: [petsc-users] Matrix reuse with changing structure<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Thu, Mar 17, 2016 at 8:54 AM, Smith, Kevin R. <<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">Hi Barry,<br>
<br>
Thanks! So this approach will allow the non-zero structure to change while avoiding reallocations? From what I understood from the documentation and discussion so far is that the non-zero structure is locked in once you do SetValue calls. I would think that
while this approach does prevent PETSc from throwing away the oversubscribed pre-allocation, it would still not allow the matrix structure to change. I can only predict the preallocation beforehand, not the structure of the matrix during its entire lifetime.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">The strategy Barry suggested means that you have to 0 out all potential columns at the start.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So suppose you translate your small box over the entire coarse mesh. You might say, at any<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">one time I never fill up more than C columns, however the strategy above would mean you<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">would need to 0 out every column in all those rows.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It sounds like what you want is something like this. At each iteration,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> 1) Wipe out nonzero structure, but keep allocation. This returns us to<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> the situation right after MatSetPreallocation*()<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> 2) Fill up matrix, including enough 0s to fill up the allocation<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If that is what you want, we can show you how to do it.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> Matt<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Thank you very much for taking the time to discuss this with me.<br>
<br>
<span>Kind regards,</span><br>
<span>Kevin</span><br>
<br>
<span>-----Original Message-----</span><br>
<span>From: Barry Smith [mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>]</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Sent: Wednesday, March 16, 2016 2:10 PM<br>
To: Smith, Kevin R.<br>
Cc: Matthew Knepley; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
Subject: Re: [petsc-users] Matrix reuse with changing structure<br>
<br>
<br>
You need to do two things<br>
<br>
1) over preallocate<br>
2) fill up all possible nonzero entries in the matrix at the beginning before calling the first MatAssemblyBegin/End()<br>
<br>
then it should run fine and efficiently (except it will always compute with those extra zeros but so long as they are relatively small amounts that is ok).<br>
<br>
The reason you need to do 2 is that Mat "throws away" any "extra" preallocation that you provided but did not set values into in the first MatAssemblyBegin/End. If you do not do 2) then it will need to do all the mallocs later when you set the new nonzero locations
and so the over preallocation doesn't help at all.<br>
<br>
Barry<br>
<br>
> On Mar 16, 2016, at 11:19 AM, Smith, Kevin R. <<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>> wrote:<br>
><br>
> The use case is dynamic overset mesh coupling. I can guarantee that all matrices will fit since I have all the information I need to do so at initialization. The component meshes do not themselves change, only the overset assembly changes. I have to over-allocate
memory to ensure the matrix always fits to cover the range of motion.<br>
><br>
> I did figure out a way to avoid deleting the matrix every time I solve in my system, so that provides some level of an optimization if PETSc doesn’t support restructuring sparse matrices out of the box. With this approach I can also avoid over-allocating
memory. However, my use case is unlikely to hit the memory limits, even if I have to over-subscribe the preallocation to cover the entire range of motion. So I think there will still be a clear benefit to doing the allocation once, and avoiding the reallocation
each time structure changes.<br>
><br>
> Not sure if this figure will get filtered out by the mailing list, but this shows a basic overset matrix structure. The green portions are the overset coupling regions, which may move around the matrix as the body moves. The number of coupling coefficients
has a maximum N which is known, so I can preallocate to cover the entire range of motion. Hope that helps describe the use case.<br>
><br>
> <image001.png><br>
><br>
> From: Matthew Knepley [mailto:<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>]<br>
> Sent: Wednesday, March 16, 2016 10:37 AM<br>
> To: Smith, Kevin R.<br>
> Cc: Barry Smith; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
> Subject: Re: [petsc-users] Matrix reuse with changing structure<br>
><br>
> On Wed, Mar 16, 2016 at 8:23 AM, Smith, Kevin R. <<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>> wrote:<br>
> Barry - I need to avoid dynamic allocation because it results in too much of a slowdown.<br>
> Matt - The MatSetType thing did not work for some reason. It seems like PETSc wants me to set the preallocation after calling this method (suggested by the error below). I'm trying to reuse the original block of preallocated memory for the sparse matrix and
blow away the structure each time. This would let me avoid repeated deallocate/allocate calls throughout my simulation.<br>
><br>
> We need to understand the use case better. How can you guarantee that<br>
> your new matrix will fit in the old matrix memroy? If you cannot, then we have to reallocate anyway.<br>
><br>
> Thanks,<br>
><br>
> Matt<br>
><br>
> PETSc Function: MatGetOwnershipRange<br>
> PETSc File:<br>
> /share/maul-data/smithkr1/src/petsc-3.6.0/src/mat/interface/matrix.c<br>
> PETSc Line: 6289<br>
> PETSc Error Code: 73<br>
> PETSc Error reference:<br>
> <a href="http://www.mcs.anl.gov/petsc/petsc-dev/include/petscerror.h.html" target="_blank">
http://www.mcs.anl.gov/petsc/petsc-dev/include/petscerror.h.html</a><br>
> PETSc Message: Must call MatXXXSetPreallocation() or MatSetUp() on<br>
> argument 1 "mat" before MatGetOwnershipRange()<br>
><br>
> Kind regards,<br>
> Kevin<br>
><br>
> -----Original Message-----<br>
> From: Barry Smith [mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>]<br>
> Sent: Tuesday, March 15, 2016 2:02 PM<br>
> To: Smith, Kevin R.<br>
> Cc: Matthew Knepley; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
> Subject: Re: [petsc-users] Matrix reuse with changing structure<br>
><br>
><br>
> > On Mar 15, 2016, at 12:56 PM, Smith, Kevin R. <<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>> wrote:<br>
> ><br>
> ><br>
> > Barry - Yeah, I suspected it was doing this. In my original implementation I would get allocation errors.<br>
><br>
> You need to call a MatSetOption() to tell the matrix that it is allowed to allocate new nonzeros.<br>
><br>
><br>
> ><br>
> > Matt - Thanks, I will try this solution out.<br>
> ><br>
> > Thanks for your help,<br>
> ><br>
> > Kevin<br>
> ><br>
> > -----Original Message-----<br>
> > From: Barry Smith [mailto:<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>]<br>
> > Sent: Tuesday, March 15, 2016 1:48 PM<br>
> > To: Matthew Knepley<br>
> > Cc: Smith, Kevin R.; <a href="mailto:petsc-users@mcs.anl.gov" target="_blank">petsc-users@mcs.anl.gov</a><br>
> > Subject: Re: [petsc-users] Matrix reuse with changing structure<br>
> ><br>
> ><br>
> >> On Mar 15, 2016, at 12:28 PM, Matthew Knepley <<a href="mailto:knepley@gmail.com" target="_blank">knepley@gmail.com</a>> wrote:<br>
> >><br>
> >> On Tue, Mar 15, 2016 at 12:24 PM, Smith, Kevin R. <<a href="mailto:Kevin.R.Smith@jhuapl.edu" target="_blank">Kevin.R.Smith@jhuapl.edu</a>> wrote:<br>
> >> Hello,<br>
> >><br>
> >><br>
> >><br>
> >> Is it possible to reuse a sparse matrix and not reuse its non-zero structure? I have a matrix whose sparse structure changes each time. I’m hoping to avoid destroying and allocating new matrices each time the structure changes.<br>
> ><br>
> > Sure you can just add new nonzero locations at a later time. But it won't take out any current entries even if they are zero. So effectively the nonzero structure grows over time.<br>
> ><br>
> > Barry<br>
> ><br>
> >><br>
> >><br>
> >> Hmm, I can't find a toplevel API that does this (it would be<br>
> >> something like MatReset()). You can get this effect using<br>
> >><br>
> >> MatSetType(A, MATSHELL)<br>
> >> MatSetType(A, <type you had before>)<br>
> >><br>
> >> A little messy but it will work.<br>
> >><br>
> >> Thanks,<br>
> >><br>
> >> Matt<br>
> >><br>
> >><br>
> >><br>
> >> Thanks,<br>
> >><br>
> >> Kevin<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> --<br>
> >> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> >> -- Norbert Wiener<br>
> ><br>
><br>
><br>
><br>
><br>
> --<br>
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
> -- Norbert Wiener<u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener<u></u><u></u></p>
</div>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>