DA memory consumption

[Barry Smith]

    Sometimes computing can be an experimental science. Run the same  
size DA on 1, 2, 4, 8, 16, 32 processes and gather the information  
about memory usage and make a little table.

    Here is what you should find. The amount of memory depends on the  
local size of the array, which for your example below is  
1000*1000*1000 on one process. Thus you will see that as you increase  
the number of processes you'll see the space needed per process for  
the DA decreases. It increases from 1 process to 2 because it needs  
all the ghost point data and VecScatter that are not needed on 1.

   Note also on one process a SINGLE vector for this size mesh is 8  
gigabytes so the DA is really not much of pig since it is less than  
one vector.

    Barry

On Nov 22, 2009, at 9:34 PM, Denis Teplyashin wrote:

> So this sort of memory consumption is expected? Is it possible to  
> reduce is somehow? I'm not sure about underlying petsc object but it  
> looks like these additional objects require more memory than the  
> actual vector itself.
>
> Cheers,
> Denis
>
> Matthew Knepley wrote:
>>
>> It is not simple, but it is scalable, meaning in the limit of large  
>> N, the memory will be constant on
>> each processor. When it is created, the VecScatter objects mapping  
>> global to local vecs are created.
>>
>>   Matt
>>
>> On Sun, Nov 22, 2009 at 8:29 PM, Denis Teplyashin  
>> <denist at al.com.au> wrote:
>> Hi guys,
>>
>> I'm a bit confused with distributed array memory consumption. I did  
>> a simple test like this one:
>>  ierr = DACreate3d(PETSC_COMM_WORLD, DA_NONPERIODIC,  
>> DA_STENCIL_BOX, 1000, 1000, 1000, PETSC_DECIDE, PETSC_DECIDE,  
>> PETSC_DECIDE, 1, 1, PETSC_NULL, PETSC_NULL, PETSC_NULL , &da);
>> and then checked memory with PetscMemoryGetCurrentUsage and  
>> PetscMemoryGetMaximumUsage. Running this test using mpi on one core  
>> gives me this result: current usage 3818Mb and maximum usage  
>> 7633Mb. And this is the result after creating just a DA without  
>> actual vectors. Running the same test on two cores gives me even  
>> more interesting result: rank 0 - 9552/11463Mb and rank 1 -  
>> 5735/5732Mb.
>> Is it what i should expect in general or am i doing something  
>> wrong? Is there a simple formula which could show how much memory i  
>> would need to allocate and array with given resolution?
>>
>> Thanks in advance,
>> Denis
>>
>>
>>
>> -- 
>> What most experimenters take for granted before they begin their  
>> experiments is infinitely more interesting than any results to  
>> which their experiments lead.
>> -- Norbert Wiener
>