[petsc-users] Vecs and Mats with non-contiguous parallel layout

[Smith, Barry F.]

> On Jan 30, 2019, at 7:36 PM, Dave Lee via petsc-users <petsc-users at mcs.anl.gov> wrote:
> 
> Thanks Matt,
> 
> I'm currently working through the details of the nested field split. I've decided to re-organise my application data layout so that the local portions are contiguous in the global space for consistency with the default PETSc layout.
> 
> Does the IndexSet need to be the same size on each processor in order for this IndexSet to be supplied to a FieldSplit PC? 

   No, since it is the local size each process can have a different one.
> 
> Moreover can an IndexSet have 0 entries on a particular processor and still be part of a global FieldSplit PC?

   Yes (at least in theory, hopefully there are not bugs for this corner case).
> 
> Cheers, Dave. 
> 
> 
> On Wed, Jan 30, 2019 at 11:10 PM Matthew Knepley <knepley at gmail.com> wrote:
> On Tue, Jan 29, 2019 at 7:58 PM Dave Lee via petsc-users <petsc-users at mcs.anl.gov> wrote:
> Hi,
> 
> just wondering if its possible to manually specify the parallel decomposition of data within PETSc Vecs and Mats? And if so, will the FieldSplit preconditioning handle this also?
> 
> I have an application (non-PETSc) data layout as:
> DATA[16][4][num_procs][3472]
> 
> and if possible I would like to use FieldSplit preconditioning without re-arranging this layout.
> 
> Yes. You need to call
> 
>   https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/PC/PCFieldSplitSetIS.html
> 
> in order to define the splits. Note that if you have nested splits, you will have to also call it on the
> nested PC, which stinks, but we have no other way of getting the information.
> 
> Nested splits can also be handled by implementing the DM interface, but that is a lot of work. It is,
> however, the way we handle it internally because it makes things much easier and more flexible.
> 
>   Thanks,
> 
>      Matt
>  
> Currently I have a nested FieldSplit preconditioning with 16 outer field splits, and a schur complement field split within each. however this throws up an indexing error as FieldSplit assumes that the parallel decomposition is contiguous (I think).
> 
> [0]PETSC ERROR: #1 ISComplement() line 750 in /home/dlee0035/soft/petsc-3.9.3/src/vec/is/is/utils/iscoloring.c
> [0]PETSC ERROR: #2 PCSetUp_FieldSplit() line 703 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/impls/fieldsplit/fieldsplit.c
> [0]PETSC ERROR: #3 PCSetUp() line 923 in /home/dlee0035/soft/petsc-3.9.3/src/ksp/pc/interface/precon.c
> 
> Cheers, Dave.
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/