<div dir="ltr">On Mon, Oct 21, 2013 at 11:32 AM, Barry Smith <span dir="ltr"><<a href="mailto:bsmith@mcs.anl.gov" target="_blank">bsmith@mcs.anl.gov</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><br>
The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.<br>
<br>
BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3<br>
<br>
In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory. There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.</blockquote>
<div><br></div><div>I am adding a DMDA example to look at this is detail. Here is what I have up front. Suppose that there are G grid vertices, e,g, 10^6 in</div><div>your example, so that a vector takes up dof*8G bytes. Then the 2D DMDA allocates</div>
<div><br></div><div><div> Create ltog scatter dof*8G</div><div> Create gtol scatter dof*8G</div><div> Raw indices dof*4G</div><div> Create ltogmap dof*4G</div><div> Create ltogmapb 4G</div>
</div><div>--------------------------------------------</div><div> dof*24G + 4G < 4 vectors</div><div><br></div><div>It also allocates 2 temporary vectors which are freed but your test may pick up since the OS might not have garbage collected them. I will</div>
<div>get the precise numbers for 3D, but they should be similar.</div><div><br></div><div>I don't really see the point of using a DMDA without the scatters. You could save 1 vector of storage by making the creation of the l2g maps</div>
<div>for the global vector lazy (and possibly those indices we use to remap arrays).</div><div><br></div><div> Matt</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<span class=""><font color="#888888"><br>
Barry<br>
</font></span><div class=""><div class="h5"><br>
<br>
<br>
On Oct 21, 2013, at 7:02 AM, Juha Jäykkä <<a href="mailto:juhaj@iki.fi">juhaj@iki.fi</a>> wrote:<br>
<br>
> Dear list members,<br>
><br>
> I have noticed strange memory consumption after upgrading to 3.4 series. I<br>
> never had time to properly investigate, but here is what happens [yes, this<br>
> might be a petsc4py issue, but I doubt it] is<br>
><br>
> # helpers contains _ProcessMemoryInfoProc routine which just digs the memory<br>
> # usage data from /proc<br>
> import helpers<br>
> procdata=helpers._ProcessMemoryInfoProc()<br>
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]<br>
> from petsc4py import PETSc<br>
> procdata=helpers._ProcessMemoryInfoProc()<br>
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]<br>
> da = PETSc.DA().create(sizes=[100,100,100],<br>
> proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],<br>
> boundary_type=[3,0,0],<br>
> stencil_type=PETSc.DA.StencilType.BOX,<br>
> dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)<br>
> procdata=helpers._ProcessMemoryInfoProc()<br>
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]<br>
> vec=da.createGlobalVec()<br>
> procdata=helpers._ProcessMemoryInfoProc()<br>
> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]<br>
><br>
> outputs<br>
><br>
> 48 MiB / 49348 kB<br>
> 48 MiB / 49360 kB<br>
> 381 MiB / 446228 kB<br>
> 435 MiB / 446228 kB<br>
><br>
> Which is odd: size of the actual data to be stored in the da is just about 56<br>
> megabytes, so why does creating the da consume 7 times that? And why does the<br>
> DA reserve the memory in the first place? I thought memory only gets allocated<br>
> once an associated vector is created and it indeed looks like the<br>
> createGlobalVec call does indeed allocate the right amount of data. But what<br>
> is that 330 MiB that DA().create() consumes? [It's actually the .setUp()<br>
> method that does the consuming, but that's not of much use as it needs to be<br>
> called before a vector can be created.]<br>
><br>
> Cheers,<br>
> Juha<br>
><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.<br>
-- Norbert Wiener
</div></div>