[petsc-dev] DARefine produces unusable partitions

[Barry Smith]

0a1,29
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: Sum of lx across processors not equal to M 39 19!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision: cd00cd8c8d87f4b93d5195df073a02522fc5c1a8  HG Date: Fri Sep 17 15:02:40 2010 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./ex22 on a arch-gnu named bsmith-laptop.mcs.anl.gov by barrysmith Wed Sep 22 13:41:50 2010
> [0]PETSC ERROR: Libraries linked from /Users/barrysmith/Src/petsc-dev/arch-gnu/lib
> [0]PETSC ERROR: Configure run at Fri Sep 17 15:06:36 2010
> [0]PETSC ERROR: Configure options --download-mpich PETSC_ARCH=arch-gnu --with-server --with-ams-dir=/Users/barrysmith/Src/ams-dev --with-java --with-fc="gfortran -m64"
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: DACreate_1D() line 188 in src/dm/da/src/da1.c
> [0]PETSC ERROR: DASetType() line 48 in src/dm/da/src/dareg.c
> [0]PETSC ERROR: DASetTypeFromOptions_Private() line 67 in src/dm/da/src/dacreate.c
> [0]PETSC ERROR: DASetFromOptions() line 128 in src/dm/da/src/dacreate.c
> [0]PETSC ERROR: DACreate1d() line 393 in src/dm/da/src/da1.c
> [0]PETSC ERROR: DARefine() line 690 in src/dm/da/src/da.c
> [0]PETSC ERROR: DMRefine() line 259 in src/dm/da/utils/dm.c
> [0]PETSC ERROR: DMCompositeRefine() line 1387 in src/dm/da/utils/pack.c
> [0]PETSC ERROR: DMRefine() line 259 in src/dm/da/utils/dm.c
> [0]PETSC ERROR: DMRefineHierarchy() line 444 in src/dm/da/utils/dm.c
> [0]PETSC ERROR: DMMGSetDM() line 222 in src/snes/utils/damg.c
> [0]PETSC ERROR: main() line 118 in src/snes/examples/tutorials/ex22.c
> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
> [cli_0]: aborting job:
> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
Possible problem with ex22_1, diffs above
3c3
<     2 SNES Function norm 2.29297e-06 
---
>     2 SNES Function norm 2.29302e-06 
6c6
<   1 SNES Function norm 5.60828e-05 
---
>   1 SNES Function norm 5.6082e-05 
Possible problem with ex23_1, diffs above


On Sep 22, 2010, at 11:41 AM, Jed Brown wrote:

> $ cd src/dm/da/examples/tests
> $ make ex3
> $ mpiexec -n 5 ./ex3 -M 8
> 
> The issue is that the partition [0,2,4,6,7,8] is refined to
> [0,3,6,9,12,15].  Now on rank 3, the start indices (6 and 9) don't
> line up (because 9/2=4 < 6-1).  I'm about to push a patch for this
> that creates and always uses lx,ly,lz for refinement so that the
> subdomains are always nested.  The downside of this is that when the
> coarse level is imbalanced (not a serious problem), all refinements
> will preserve the coarse fractional imbalance, which may be a big
> problem on the fine meshes.
> 
> It is probably not hard to redistribute lx,ly,lz within the
> constraints that DAGetInterpolation has to work, so you can't move a
> subdomain interface by more than one coarse stencil width.  But I'm
> not sure that any redistribution is always desirable (especially if
> the user used DASetVertexDivision).
> 
> Jed