[petsc-users] Domain partitioning using DAs

[Barry Smith]

On Oct 11, 2011, at 11:30 AM, Ethan Coon wrote:

> On Tue, 2011-10-11 at 11:22 -0500, Barry Smith wrote:
>> On Oct 11, 2011, at 11:19 AM, Ethan Coon wrote:
>> 
>>> 
>>> 
>>> On Sat, 2011-10-08 at 08:32 -0500, Jed Brown wrote:
>>> 
>>>> 
>>>> I'm a little surprised it doesn't find the "best" split in this simple
>>>> case.
>>> 
>>> 
>>> I swore at this for a while as well back when I implemented the
>>> "degenerate" 3D case of P=1.  It actually does quite poorly for a few
>>> common sizes (8, for example) when there is one "small" dimension that
>>> is not the Y-direction.  (If the Y-direction is the small one, it does
>>> fine as n is calculated first, see line 273 in da3.c.)  For instance, if
>>> M=N=256, P=1, and size <= 16, it ends up with (1,size,1) for the
>>> decomposition.  
>>> 
>>> I think it's worth reorganizing that calculation to pick p first, since
>>> it would do so much better for when m=n=p=PETSC_DECIDE and P << (M,N),
>>> which seems to be a much more common case.  It wouldn't help in
>>> Mohamad's case, but if he chose to use, say, 4 processors, he'd find
>>> that his Y-direction would get split in 4 instead of splitting both X
>>> and Y in 2.  
>>> 
>> 
>>   Should the logic be changed to first select the direction to be decided based relative sizes and then decide the second direction instead of having the order of directions figured out hardwired?
>> 
> 
> That seems reasonable.  In the case where M ~ N ~ P, order doesn't
> matter.
> 
> In the case where M << N ~ P, m should be done first (and permutations),
> which would suggest just doing them from smallest to largest.
> 
> I haven't considered the case where M ~ N << P.  Will do so and submit a
> patch for this (after I fix the communication bug...)

   Thanks, remember patches should be done via 3.2 repository, not petsc-dev ssh://petsc@petsc.cs.iit.edu//hg/petsc/releases/petsc-3.2

   Barrt

> 
> Ethan
> 
>>   What would that logic look like?
>> 
>>    Barry
>> 
>>> Ethan
>>> 
>>> 
>>> 
>>> 
>>> -- 
>>> ------------------------------------
>>> Ethan Coon
>>> Post-Doctoral Researcher
>>> Applied Mathematics - T-5
>>> Los Alamos National Laboratory
>>> 505-665-8289
>>> 
>>> http://www.ldeo.columbia.edu/~ecoon/
>>> ------------------------------------
>>> 
>> 
> 
> -- 
> ------------------------------------
> Ethan Coon
> Post-Doctoral Researcher
> Applied Mathematics - T-5
> Los Alamos National Laboratory
> 505-665-8289
> 
> http://www.ldeo.columbia.edu/~ecoon/
> ------------------------------------
>