[petsc-users] DMDA with 0 in lx, ly, lz ... or how to create a DMDA for a subregion

[Randall Mackie]

Take a look at these posts from last year and see if they will help you at least get a slice:

https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043037.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043037.html>

https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043043.html <https://lists.mcs.anl.gov/pipermail/petsc-users/2021-January/043043.html>


Good luck,

Randy M.


> On Jan 25, 2022, at 2:52 PM, Bastian Löhrer <bastian.loehrer at tu-dresden.de> wrote:
> 
> Dear PETSc community,
> 
> in our CFD code we use a 3D DMDA to organize our data.
> 
> Now I need to compute a derived quantity in a subregion of the global domain and to write these data to disk for further post-processing.
> 
> The subregion is actually a planar slice for now, but it could also be a boxed-shaped region in future applications.
> 
> Hence, I figured I would create a new DMDA for this subregion by writing something along the lines of 
> 
> call DMDACreate3d(                 & ! https://www.mcs.anl.gov/petsc/petsc-3.8.4/docs/manualpages/DMDA/DMDACreate3d.html <https://www.mcs.anl.gov/petsc/petsc-3.8.4/docs/manualpages/DMDA/DMDACreate3d.html>
>    PETSC_COMM_WORLD,               &
>    DM_BOUNDARY_NONE,  DM_BOUNDARY_NONE,  DM_BOUNDARY_NONE, &
>    DMDA_STENCIL_BOX,               & ! <= stencil type
>    sum(lx), sum(ly), sum(lz),      & ! <= global dimension (of data) in each direction
>    px, py, pz,                     & ! <= number of processors in each dimension
>    1, 0,                           & ! <= dof per node, stencil width (no ghost cell)
>    lx, ly, lz,                     & ! <= numbers of nodes held by processors
>    dmobject, ierr )
> 
> where lx, ly and lz could look like
> 
> lx = (/  0,  1,  0 /)
> ly = (/ 16, 16 /)
> lz = (/ 16, 16 /)
> 
> Unfortunately, this does not work:
> 
> 
>> [0]PETSC ERROR: --------------------- Error Message --------------------------------------------------------------
>> [0]PETSC ERROR: Argument out of range
>> [0]PETSC ERROR: Partition in x direction is too fine! 1 2
>> [0]PETSC ERROR: See http://www.mcs.anl.gov/petsc/documentation/faq.html <http://www.mcs.anl.gov/petsc/documentation/faq.html> for trouble shooting.
>> [0]PETSC ERROR: Petsc Release Version 3.8.4, Mar, 24, 2018 
>> [0]PETSC ERROR: ./prime on a foss_debug named laptwo by bastian Tue Jan 25 23:32:30 2022
>> [0]PETSC ERROR: Configure options PETSC_ARCH=foss_debug --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --with-large-file-io=1 --with-shared-libraries=1 COPTFLAGS="   " CXXOPTFLAGS=" " FOPTFLAGS="   " --march=native --mtune=native --download-hypre=/soft/petsc-3.8.4/foss_debug/hypre-v2.12.0.tar.gz --with-debugging=yes
>> [0]PETSC ERROR: #1 DMSetUp_DA_3D() line 318 in /soft/petsc-3.8.4/src/dm/impls/da/da3.c
>> [0]PETSC ERROR: #2 DMSetUp_DA() line 25 in /soft/petsc-3.8.4/src/dm/impls/da/dareg.c
>> [0]PETSC ERROR: #3 DMSetUp() line 720 in /soft/petsc-3.8.4/src/dm/interface/dm.c
>> [0]PETSC ERROR: #4 User provided function() line 0 in User file
>> application called MPI_Abort(MPI_COMM_WORLD, 63) - process 0
> Apparently, lx, ly and lz cannot contain zeros.
> (Which would be a useful information in the documentation, too.)
> 
> Is there any workaround?
> 
> My current understanding is that I need to go the extra mile of creating an additional Communicator involving only those ranks that will share at least one cell in the subregion DMDA.
> 
> If this is the way to go? If so, how can I control which rank receives which subdomain in the subregion?  DMDACreate3d does not enable me to do so, but I need to make sure that a rank holds only those cells of the subregion which are also present in its share of the global domain.
> 
> Many thanks in advance,
> Bastian Löhrer
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20220125/9a27ea7e/attachment.html>