[petsc-users] memory use of a DMDA

[Barry Smith]

   Matt,

     I think you are running on 1 process where the DMDA doesn't have an optimized path, when I run on 2 processes the numbers indicate nothing proportional to dof* number of local points

dof = 12
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 182480 VecScatterCreate_PtoS()

dof = 8
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 176080 VecScatterCreate_PtoS()

dof = 4

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 169680 VecScatterCreate_PtoS()

dof = 2
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 21344 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 166480 VecScatterCreate_PtoS()

dof =2 grid is 50 by 50 instead of 100 by 100

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep VecScatter
[0] 7 6352 VecScatterCreate()
[0] 2 32 VecScatterCreateCommon_PtoS()
[0] 39 43952 VecScatterCreate_PtoS()

The IS creation in the DMDA is far more troubling

/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS

dof = 2

[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 81600 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 42016 ISLocalToGlobalMappingCreate()

dof = 4

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 163200 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 82816 ISLocalToGlobalMappingCreate()

dof = 8

~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 326400 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 164416 ISLocalToGlobalMappingCreate()

dof = 12
~/Src/petsc/test  master $ petscmpiexec -n 2 ./ex1 -malloc_log | grep IS
[0] 1 20400 ISBlockSetIndices_Block()
[0] 15 3760 ISCreate()
[0] 4 128 ISCreate_Block()
[0] 1 16 ISCreate_Stride()
[0] 2 489600 ISGetIndices_Block()
[0] 1 20400 ISLocalToGlobalMappingBlock()
[0] 7 246016 ISLocalToGlobalMappingCreate()

Here the accessing of indices is at the point level (as well as block) and hence memory usage is proportional to dof* local number of grid points. Of course it is still only proportional to the vector size. There is some improvement we could make it; with a lot of refactoring we can remove the dof* completely, with a little refactoring we can bring it down to a single dof*local number of grid points.

   I cannot understand why you are seeing memory usage 7 times more than a vector. That seems like a lot.

   Barry



On Oct 21, 2013, at 11:32 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

> 
>   The PETSc DMDA object greedily allocates several arrays of data used to set up the communication and other things like local to global mappings even before you create any vectors. This is why you see this big bump in memory usage.
> 
>   BUT I don't think it should be any worse in 3.4 than in 3.3 or earlier; at least we did not intend to make it worse. Are you sure it is using more memory than in 3.3
> 
>   In order for use to decrease the memory usage of the DMDA setup it would be helpful if we knew which objects created within it used the most memory.  There is some sloppiness in that routine of not reusing memory as well as could be, not sure how much difference that would make.
> 
> 
>   Barry
> 
> 
> 
> On Oct 21, 2013, at 7:02 AM, Juha Jäykkä <juhaj at iki.fi> wrote:
> 
>> Dear list members,
>> 
>> I have noticed strange memory consumption after upgrading to 3.4 series. I 
>> never had time to properly investigate, but here is what happens [yes, this 
>> might be a petsc4py issue, but I doubt it] is
>> 
>> # helpers contains _ProcessMemoryInfoProc routine which just digs the memory
>> # usage data from /proc
>> import helpers
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> from petsc4py import PETSc
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> da = PETSc.DA().create(sizes=[100,100,100],
>>                      proc_sizes=[PETSc.DECIDE,PETSc.DECIDE,PETSc.DECIDE],
>>                      boundary_type=[3,0,0],
>>                      stencil_type=PETSc.DA.StencilType.BOX,
>>                      dof=7, stencil_width=1, comm=PETSc.COMM_WORLD)
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> vec=da.createGlobalVec()
>> procdata=helpers._ProcessMemoryInfoProc()
>> print procdata.rss/2**20, "MiB /", procdata.os_specific[3][1]
>> 
>> outputs
>> 
>> 48 MiB / 49348 kB
>> 48 MiB / 49360 kB
>> 381 MiB / 446228 kB
>> 435 MiB / 446228 kB
>> 
>> Which is odd: size of the actual data to be stored in the da is just about 56 
>> megabytes, so why does creating the da consume 7 times that? And why does the 
>> DA reserve the memory in the first place? I thought memory only gets allocated 
>> once an associated vector is created and it indeed looks like the 
>> createGlobalVec call does indeed allocate the right amount of data. But what 
>> is that 330 MiB that DA().create() consumes? [It's actually the .setUp() 
>> method that does the consuming, but that's not of much use as it needs to be 
>> called before a vector can be created.]
>> 
>> Cheers,
>> Juha
>> 
>