[petsc-users] Using DMDA for a block-structured grid approach

[Barry Smith]

> On Jun 26, 2023, at 11:44 AM, Srikanth Sathyanarayana <srcs at mpcdf.mpg.de> wrote:
> 
> Dear PETSc developers,
> 
> 
> I am currently working on a Gyrokinetic code where I essentially have to implement a block structured grid approach in one of the subdomains of the phase space coordinates. I have attached one such example in the x - v_parallel subdomains where I go from a full grid to a grid based on 4 blocks (divided along x direction) which is still Cartesian but misaligned across blocks (the grid is a very coarse representation). So the idea is to basically create a library for the existing solver and try to implement the block structured grid approach which mainly involves some sort of interpolation between the blocks to align the points.
> 
> 
> I came up with an idea to implement this using DMDA. I looked into the old threads where you have suggested using DMComposite in order to tackle such problems although a clear path for the interpolation between the DM's was not clarified. Nonetheless, my main questions were:
> 
> 1. Do you still suggest using DMComposite to approach this problem.

  Unfortunately, that is all we have for combining DM's. You can use unstructured, or structured or unstructed with quad-tree-type refinement but we don't have a "
canned" approach for combining a bunch of structured grids together efficiently and cleanly (lots of issues come up in trying to design such a thing in a distributed memory environment since some blocks may need to live on different number of MPI ranks)
> 
> 2. Is there a way to use DMDA where the user provides the allocation? My main problem is that I am not allowed to change the solvers data structure

   The allocation for what?
> 
> 3. I looked into VecCreateMPIWithArray for the user provided allocation, however I am not very sure if this Vector can be used with the DMDA operations.

   Yes, you can use these variants to create vectors that you use with DMDA; so long as they have the correct dimensions. 
> 
> 
> Overall, I request you to please let me know what you think of this approach (using DMDA) and I would be grateful if you could suggest me any alternatives.
> 
> 
> Thanks and regards,
> 
> Srikanth
> <Screenshot from 2023-06-26 17-24-32.png>