DA memory consumption

Barry Smith bsmith at mcs.anl.gov
Sun Nov 22 21:47:53 CST 2009

    Sometimes computing can be an experimental science. Run the same  
size DA on 1, 2, 4, 8, 16, 32 processes and gather the information  
about memory usage and make a little table.

    Here is what you should find. The amount of memory depends on the  
local size of the array, which for your example below is  
1000*1000*1000 on one process. Thus you will see that as you increase  
the number of processes you'll see the space needed per process for  
the DA decreases. It increases from 1 process to 2 because it needs  
all the ghost point data and VecScatter that are not needed on 1.

   Note also on one process a SINGLE vector for this size mesh is 8  
gigabytes so the DA is really not much of pig since it is less than  
one vector.


On Nov 22, 2009, at 9:34 PM, Denis Teplyashin wrote:

> So this sort of memory consumption is expected? Is it possible to  
> reduce is somehow? I'm not sure about underlying petsc object but it  
> looks like these additional objects require more memory than the  
> actual vector itself.
> Cheers,
> Denis
> Matthew Knepley wrote:
>> It is not simple, but it is scalable, meaning in the limit of large  
>> N, the memory will be constant on
>> each processor. When it is created, the VecScatter objects mapping  
>> global to local vecs are created.
>>   Matt
>> On Sun, Nov 22, 2009 at 8:29 PM, Denis Teplyashin  
>> <denist at al.com.au> wrote:
>> Hi guys,
>> I'm a bit confused with distributed array memory consumption. I did  
>> a simple test like this one:
>> and then checked memory with PetscMemoryGetCurrentUsage and  
>> PetscMemoryGetMaximumUsage. Running this test using mpi on one core  
>> gives me this result: current usage 3818Mb and maximum usage  
>> 7633Mb. And this is the result after creating just a DA without  
>> actual vectors. Running the same test on two cores gives me even  
>> more interesting result: rank 0 - 9552/11463Mb and rank 1 -  
>> 5735/5732Mb.
>> Is it what i should expect in general or am i doing something  
>> wrong? Is there a simple formula which could show how much memory i  
>> would need to allocate and array with given resolution?
>> Thanks in advance,
>> Denis
>> -- 
>> What most experimenters take for granted before they begin their  
>> experiments is infinitely more interesting than any results to  
>> which their experiments lead.
>> -- Norbert Wiener

More information about the petsc-users mailing list