[petsc-users] extract arbitrary subset of a DMDA

[Matthew Knepley]

On Mon, Oct 13, 2025 at 11:17 AM Semplice Matteo <
matteo.semplice at uninsubria.it> wrote:

>
>
> Inviato da Outlook per Android <https://urldefense.us/v3/__https://aka.ms/AAb9ysg__;!!G_uCfscf7eWS!bufWgyONsvb66zmOv6LgtJsh0Yr4dFXIovj8jn6nt7Ugtd0-U5Tthdyj_F0vRS7qG9mtw4BzudiTClCJsK8z$ >
>
> ------------------------------
> *Da:* Matthew Knepley <knepley at gmail.com>
> *Inviato:* Sabato, Ottobre 11, 2025 11:19:23 PM
> *A:* Semplice Matteo <matteo.semplice at uninsubria.it>
> *Cc:* PETSc <petsc-users at mcs.anl.gov>
> *Oggetto:* Re: [petsc-users] extract arbitrary subset of a DMDA
>
> On Fri, Oct 10, 2025 at 11:15 AM Matteo Semplice <
> matteo.semplice at uninsubria.it> wrote:
>
> On 10/10/2025 16:39, Matthew Knepley wrote:
>
> On Fri, Oct 10, 2025 at 9:48 AM Matteo Semplice via petsc-users <
> petsc-users at mcs.anl.gov> wrote:
>
> Dear all,
>
>      I am wondering if there is a way to extract a subset of a DMDA and
> use it as a mesh. The use case is to program a finite-difference method
> in which the domain is defined by a levelset function: if I could
> completely ignore the parts of the background DMDA that are "far away"
> from the object, I guess I would avoid some cores having almost no
> workload. I figure that I could setup a DMDA, load/compute the levelset
> on the entire box, then mark the nodes to be retained, extract the
> submesh and repartition it. I would also need a mean to transfer some
> Vec data from the DMDA to the new mesh.
>
> I guess that the extracted mesh would then become a DMPlex and it would
> not retain any DMDA flavour (like notions of which are the grid nodes
> sitting on top/bottom, left/right of a given node), right?
>
>
> If you are planning on extracting a Plex anyway, I think it would be
> easier to just
> start with a Cartesian Plex, instead of a DA, and use
> DMPlexCreateSubmesh().
>
> Hmmm... doable, but I have a couple of questions.
>
> By Cartesian Plex you mean a Plex created by DMPlexCreateBoxMesh with
> simplex=false, right?
>
> Yes, exactly
>
> And, could you point me to the routines that can perform data tranfer from
> Vecs associated to the DM to the ones asscoiated to the subDM? Is
> DMPlexGetSubpointIS the way to go?
>
> Yes. You get the IS and then you can use
>
>   https://urldefense.us/v3/__https://petsc.org/main/manualpages/Vec/VecISCopy/__;!!G_uCfscf7eWS!bufWgyONsvb66zmOv6LgtJsh0Yr4dFXIovj8jn6nt7Ugtd0-U5Tthdyj_F0vRS7qG9mtw4BzudiTCi5Y9fxK$ 
>
> Next, I will load the levelsets from the output of another code that is
> DA-based and that I'd really like to reuse some code in the setup phase
> which relies on the DA indexing. So maybe I'd rather, create the DMDA and
> the associated Vecs, do the setup phase, then DMConvert the DMDA to a
> "large" DMPlex that covers the entire box, transfer the DA Vecs to the
> "large" Plex vectors and then extract the submesh. Would this be feasible?
> If so, can you point me to the routines to transfer the vecs from the dmda
> to the large plex?
>
> Yes, this sounds doable, and once we write it, we should just put it in
> the library.
>
> In serial this is completely trivial. The DA has a known ordering and the
> Plex has a known ordering. We can use an AO
> or just a VecScatter to permute the Vec. In parallel, the DA is
> partitioned geometrically, whereas the Plex, by default, is partitioned
> using a graph partitioner, like ParMetis. To me, if this project is
> important, it seems worth it to build a simple partitioner for the Plex
> that mimics the DA. Then we are back to the trivial remapping. Finally, we
> probably want the Plex dual to the DA due to the way that DAs partition
> things.
>
> ____
> Hi Matt,
>    doing some research on parallel Plex I found this thread on the list
> from 2020
>
> https://lists.mcs.anl.gov/pipermail/petsc-users/2020-August/041934.html 
>
> I will explore this way to create a basic cartesian Plex from my
> distributed da. I think that there are enough info there, but I will come
> back if I run into troubles.
>

Sure this is doable. Creating the SF is morally the same thing as creating
the partition I was asking for with CreateBoxMesh.

As I mentioned at the end of the last post, DA partitions "vertices" and
shares "edges", which is not how we normally think in finite element land.
If you are using the Plex for finite difference, then I would just match
the DA. If you are using it for finite elements, I would create the dual
mesh instead.

  Thanks,

     Matt


> Or do you suggest that I should instead study the private methods used in
> DMPlexCreateBoxMesh?
>
> Thanks
>   Matteo
>
>

-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://urldefense.us/v3/__https://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bufWgyONsvb66zmOv6LgtJsh0Yr4dFXIovj8jn6nt7Ugtd0-U5Tthdyj_F0vRS7qG9mtw4BzudiTCoQl4j2n$  <https://urldefense.us/v3/__http://www.cse.buffalo.edu/*knepley/__;fg!!G_uCfscf7eWS!bufWgyONsvb66zmOv6LgtJsh0Yr4dFXIovj8jn6nt7Ugtd0-U5Tthdyj_F0vRS7qG9mtw4BzudiTChMNFZIy$ >
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20251013/2a39789c/attachment-0001.html>