[petsc-users] Out of memory and parallel issues

[Matthew Knepley]

On Thu, Jan 1, 2015 at 2:15 AM, TAY wee-beng <zonexo at gmail.com> wrote:

>
>  On 1/1/2015 2:06 PM, Matthew Knepley wrote:
>
>  On Wed, Dec 31, 2014 at 11:20 PM, TAY wee-beng <zonexo at gmail.com> wrote:
>
>> Hi,
>>
>> I used to run my CFD code with 96 procs, with a grid size of 231 x 461 x
>> 368.
>>
>> I used MPI and partition my grid in the z direction. Hence with 96 procs
>> (8 nodes, each 12 procs), each procs has a size of 231 x 461 x 3 or 231 x
>> 461 x 4.
>>
>> It worked fine.
>>
>> Now I modified the code and added some more routines which increases the
>> fixed memory requirement per procs. However, the grid size is still the
>> same. But the code aborts while solving the Poisson eqn, saying:
>>
>> Out of memory trying to allocate XXX bytes
>>
>> I'm using PETSc with HYPRE boomeramg to solve the linear Poisson eqn. I
>> am guessing that now the amt of memory per procs is less because I added
>> some routines which uses some memory. The result is less memory available
>> for the solving of the Poisson eqn.
>>
>
>  I would try GAMG instead of Hypre. It tends to be memory light compared
> to it.
>
>    -pc_type gamg
>
>    Thanks,
>
>       Matt
>
> Hi Matt,
>
> To use gamg, must I use use DMDA to partition the grid?
>

No.


> Also, does MPI partitioning in only the z direction affect parallel
> performance? Since the MPI partition grid is almost like 2D plane with 3-4
> cell thickness.
>

It could affect load balance.


> Lastly, using 10x10 = 100 procs seems to work for now, although there's a
> wastage of 20 procs since each node has 12 procs.
>

PETSc attempts to make square domains, but you can prescribe your own
domains if that helps.

  Thanks,

     Matt


> Thanks!
>
>
>
>> I'm now changing to KSPBCGS but it seems to take forever. When I abort
>> it, the error msg is:
>>
>> Out of memory. This could be due to allocating
>> [10]PETSC ERROR: too large an object or bleeding by not properly
>> [10]PETSC ERROR: destroying unneeded objects.
>> [10]PETSC ERROR: Memory allocated 0 Memory used by process 4028370944
>> [10]PETSC ERROR: Try running with -malloc_dump or -malloc_log for info.
>>
>> I can't use more procs because some procs will have a size of 231 x 461 x
>> 2 (or even 1). This will give error since I need to reference the nearby
>> values along the z direction.
>>
>> So what options do I have? I'm thinking of these at the moment:
>>
>> 1. Remove as much fixed overhead memory per procs as possible so that
>> there's enough memory for each procs.
>>
>> 2. Re-partition my grid in both x,y direction or x,y,z direction so I
>> will not encounter extremely skew grid dimensions per procs. Btw, does
>> having extremely skew grid dimensions affect the performance in solving the
>> linear eqn?
>>
>> Are there other feasible options
>>
>> --
>> Thank you.
>>
>> Yours sincerely,
>>
>> TAY wee-beng
>>
>>
>
>
>  --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which their
> experiments lead.
> -- Norbert Wiener
>
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20150101/a796ed8b/attachment-0001.html>