[petsc-dev] DACreate3d should work for 1 node in Z
Ethan Coon
ecoon at lanl.gov
Tue Apr 19 15:19:58 CDT 2011
Ah, the 2D case provides an error message for this case, but not the 3d.
[0]PETSC ERROR: --------------------- Error Message ------------------------------------
[0]PETSC ERROR: Argument out of range!
[0]PETSC ERROR: Local y-width of domain y 1 is smaller than stencil width s 2!
[0]PETSC ERROR:
The problem is less about the 1-node-wide domain with 1 proc in that
dimension, and more about the case where there are more than one procs
in that dimension but less than the stencil size local unknowns. To
handle this, the scatter would have to look two proc neighbors away, not
just one. This has never been handled by PETSc.
Consider the z-case with local size zl and stencil size s... I supposed
one could check for the special case where P, the number of procs, in
the z-dimension is 1 and s > zl. In that case, DMDA_BOUNDARY_GHOSTED
and DMDA_BOUNDARY_NONE both work (as evidenced by the 3D code). Then,
it need only error if
(s > zl) and ((P > 1) or (zwrap == DMDA_BOUNDARY_PERIODIC))
Alternatively, DMDA_BOUNDARY_PERIODIC could be made to work with more if
statements to set up the indices for the scatter, doing the wrap around
of the domain multiple times. Not sure how much work this would be.
At any rate, I'll fix the error message in 2D and add it to 3D to only
error in the cases that actually currently break. Doing
DMDA_BOUNDARY_PERIODIC with P=1 and s > zl (the multiple-wraps around
the domain case) will take more thought...
Ethan
On Tue, 2011-04-19 at 21:25 +0200, Jed Brown wrote:
> On Tue, 19 Apr 2011 09:18:44 -0600, Ethan Coon <ecoon at lanl.gov> wrote:
> > Can you shoot me your example? I run 1-node z direction both with and
> > without periodic BCs, and with and without ghost nodes, without a
> > valgrind complaint in DMDACreate3D.
>
> See attached very simple test case. It only crashes for stencil width
> greater than 1 (the original support request needs stencil width of 2
> for TVD FV methods),
>
> mpiexec -n 1 ./a.out # stencil width 2 in code
>
> [0]PETSC ERROR: --------------------- Error Message ------------------------------------
> [0]PETSC ERROR: Argument out of range!
> [0]PETSC ERROR: Negative index -100 at 0 location!
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: Petsc Development HG revision: 335e836c7798747e6242df554ae58fe4db9f803f HG Date: Fri Apr 15 16:04:07 2011 -0500
> [0]PETSC ERROR: See docs/changes/index.html for recent updates.
> [0]PETSC ERROR: See docs/faq.html for hints about trouble shooting.
> [0]PETSC ERROR: See docs/index.html for manual pages.
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: ./a.out on a ompi named kunyang by jed Tue Apr 19 21:22:07 2011
> [0]PETSC ERROR: Libraries linked from /home/jed/petsc/ompi/lib
> [0]PETSC ERROR: Configure run at Sat Apr 16 13:36:39 2011
> [0]PETSC ERROR: Configure options --download-blacs --download-chaco --download-hypre=externalpackages/hypre-2.7.0b.tar.gz --download-ml --download-mum
> ps --download-parms --download-scalapack --download-scotch --download-spai --download-spooles --download-sundials --download-superlu --download-superl
> u_dist --with-blas-lapack-dir=/usr --with-c2html --with-cholmod-dir=/usr --with-clib-autodetect=0 --with-cxxlib-autodetect=0 --with-fortranlib-autodet
> ect=0 --with-hdf5-dir=/usr --with-java-dir=/usr --with-lgrind --with-mpi-dir=/usr --with-parmetis-dir=/usr --with-shared-libraries --with-single-libra
> ry=0 --with-sowing --with-umfpack-dir=/usr -LIBS="-L/usr/lib/openmpi -lmpi_f90 -lmpi_f77 -lmpi_cxx -lstdc++ -lgfortran" PETSC_ARCH=ompi
> [0]PETSC ERROR: ------------------------------------------------------------------------
> [0]PETSC ERROR: VecScatterCheckIndices_Private() line 26 in /home/jed/petsc/src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: VecScatterCreate() line 1106 in /home/jed/petsc/src/vec/vec/utils/vscat.c
> [0]PETSC ERROR: DMSetUp_DA_3D() line 953 in /home/jed/petsc/src/dm/impls/da/da3.c
> [0]PETSC ERROR: DMSetUp_DA() line 22 in /home/jed/petsc/src/dm/impls/da/dareg.c
> [0]PETSC ERROR: DMSetUp() line 189 in /home/jed/petsc/src/dm/interface/dm.c
> [0]PETSC ERROR: DMDACreate3d() line 1366 in /home/jed/petsc/src/dm/impls/da/da3.c
> [0]PETSC ERROR: main() line 14 in "unknowndirectory/"da3d.c
> --------------------------------------------------------------------------
> MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
> with errorcode 63.
>
>
> I still see Valgrind noise for stencil width of 1:
>
> mpiexec -n 2 valgind ./a.out # stencil width 1 in code
>
> ==26044== Invalid read of size 2
> ==26044== at 0x4C26C2C: memcpy (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==26044== by 0x71D8D01: opal_convertor_pack (in /usr/lib/openmpi/libmpi.so.1.0.1)
> ==26044== by 0x117600C8: ??? (in /usr/lib/openmpi/openmpi/mca_btl_sm.so)
> ==26044== by 0x10F421B7: ??? (in /usr/lib/openmpi/openmpi/mca_pml_ob1.so)
> ==26044== by 0x10F30C0C: ??? (in /usr/lib/openmpi/openmpi/mca_pml_ob1.so)
> ==26044== by 0x7127650: PMPI_Isend (in /usr/lib/openmpi/libmpi.so.1.0.1)
> ==26044== by 0x6C5A103: PetscGatherMessageLengths (mpimesg.c:127)
> ==26044== by 0x697A90A: VecScatterCreate_PtoS (vpscat.c:1623)
> ==26044== by 0x69564FD: VecScatterCreate (vscat.c:1334)
> ==26044== by 0x7CB339A: DMSetUp_DA_3D (da3.c:953)
> ==26044== by 0x7D044BF: DMSetUp_DA (dareg.c:22)
> ==26044== by 0x7C76A6C: DMSetUp (dm.c:189)
> ==26044== Address 0xdfeda02 is 2 bytes inside a block of size 8 alloc'd
> ==26044== at 0x4C23918: memalign (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
> ==26044== by 0x6CC9DEF: PetscMallocAlign (mal.c:30)
> ==26044== by 0x697A5EA: VecScatterCreate_PtoS (vpscat.c:1603)
> ==26044== by 0x69564FD: VecScatterCreate (vscat.c:1334)
> ==26044== by 0x7CB339A: DMSetUp_DA_3D (da3.c:953)
> ==26044== by 0x7D044BF: DMSetUp_DA (dareg.c:22)
> ==26044== by 0x7C76A6C: DMSetUp (dm.c:189)
> ==26044== by 0x7CB6D8D: DMDACreate3d (da3.c:1366)
> ==26044== by 0x400938: main (in /home/jed/lang/c/petsc/a.out)
>
> C source code attachment (da3d.c)
> #include <petscdmda.h>
>
> int main(int argc, char *argv[])
> {
> DM da;
> PetscErrorCode ierr;
>
> PetscInitialize(&argc,&argv,0,0);
> ierr = DMDACreate3d(PETSC_COMM_WORLD,
> DMDA_BOUNDARY_PERIODIC, DMDA_BOUNDARY_PERIODIC, DMDA_BOUNDARY_PERIODIC,
> DMDA_STENCIL_STAR,
> 10, 10, 1,
> PETSC_DECIDE, PETSC_DECIDE, PETSC_DECIDE, 1, 1, /* Stencil width of 2 in original request */
> PETSC_NULL, PETSC_NULL, PETSC_NULL, &da);CHKERRQ(ierr);
> ierr = DMDestroy(da);CHKERRQ(ierr);
> PetscFinalize();
> return 0;
> }
--
------------------------------------
Ethan Coon
Post-Doctoral Researcher
Applied Mathematics - T-5
Los Alamos National Laboratory
505-665-8289
http://www.ldeo.columbia.edu/~ecoon/
------------------------------------
More information about the petsc-dev
mailing list