[petsc-users] TS scheme with different DAs

[Smith, Barry F.]

  Don't be too quick to dismiss switching to the DMStag you may find that it actually takes little time to convert and then you have a much less cumbersome process to manage the staggered grid. Take a look at src/dm/impls/stag/examples/tutorials/ex2.c where

const PetscInt dof0 = 0, dof1 = 1,dof2 = 1; /* 1 dof on each edge and element center */
    const PetscInt stencilWidth = 1;
    ierr = DMStagCreate2d(PETSC_COMM_WORLD,DM_BOUNDARY_NONE,DM_BOUNDARY_NONE,7,9,PETSC_DECIDE,PETSC_DECIDE,dof0,dof1,dof2,DMSTAG_STENCIL_BOX,stencilWidth,NULL,NULL,&dmSol);CHKERRQ(ierr);

BOOM, it has set up a staggered grid with 1 cell centered variable and 1 on each edge. Adding more the cell centers, vertices, or edges is trivial.

  If you want to stick to DMDA you 

"cheat". Depending on exactly what staggering you have you make the DMDA for the "smaller problem" as large as the other ones and just track zeros in those locations. For example if velocities are "edges" and T, S are on cells, make your "cells" DMDA one extra grid width wide in all three dimensions. You may need to be careful on the boundaries deepening on the types of boundary conditions. 



> On Sep 17, 2019, at 7:04 PM, Manuel Valera via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thanks Matthew, but my code is too complicated to be redone on DMStag now after spending a long time using DMDAs,
> 
> Is there a way to ensure PETSc distributes several DAs in the same way? besides manually distributing the points,
> 
> Thanks, 
> 
> On Tue, Sep 17, 2019 at 3:28 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Sep 17, 2019 at 6:15 PM Manuel Valera via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hello, petsc users,
> 
> I have integrated the TS routines in my code, but i just noticed i didn't do it optimally. I was using 3 different TS objects to integrate velocities, temperature and salinity, and it works but only for small DTs. I suspect the intermediate Runge-Kutta states are unphased and this creates the discrepancy for broader time steps, so I need to integrate the 3 quantities in the same routine. 
> 
> I tried to do this by using a 5 DOF distributed array for the RHS, where I store the velocities in the first 3 and then Temperature and Salinity in the rest. The problem is that I use a staggered grid and T,S are located in a different DA layout than the velocities. This is creating problems for me since I can't find a way to communicate the information from the result of the TS integration back to the respective DAs of each variable.
> 
> Is there a way to communicate across DAs? or can you suggest an alternative solution to this problem?
> 
> If you have a staggered discretization on a structured grid, I would recommend checking out DMStag.
> 
>   Thanks,
> 
>      MAtt
>  
> Thanks,
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/