[petsc-users] Regarding global indexing

[Barry Smith]

On Dec 29, 2011, at 7:08 AM, Jed Brown wrote:

> On Thu, Dec 29, 2011 at 01:27, Rahul Praghanmor <praghanmor at gmail.com> wrote:
> I successfully implement PETSc to solve on a single zone unstructured mesh i.e. serial implementation has done and tested.
> Now I want to implement a PETSc on a multi-zone mesh.The mesh partitioning in multi-zone is done using a separate tool, Metis. The multi-zone mesh is read in solver.I don't understand how to do the global indexing for multi-zone mesh to form a global matrix required for PETSc.
> 
> Start with a non-overlapping partition (usually what is provided by Metis). Distribute the mesh according to this partition, then compute the local size and use MPI_Scan() to compute the offset of your piece in the global ordering. You can learn the global index of ghosted points by sending it from owner to ghoster through the overlap.

  You can optionally use the PetscLayout object (I've included the manual page for PetscLayoutCreate() below since I just improved it) to avoid having to use MPI_Scan() directly. You just set the local sizes based on your new number of grid points per MPI process and use PetscLayoutSetUp() and then you can query the global starting point (rstart) etc of each process using the other routines mentioned in the manual page.

     PetscLayoutCreate - Allocates PetscLayout space and sets the map contents to the default.

    Collective on MPI_Comm

   Input Parameters:
+    comm - the MPI communicator
-    map - pointer to the map

   Level: developer

    Notes: Typical calling sequence
       PetscLayoutCreate(MPI_Comm,PetscLayout *);
       PetscLayoutSetBlockSize(PetscLayout,1);
       PetscLayoutSetSize(PetscLayout,n) or PetscLayoutSetLocalSize(PetscLayout,N);
       PetscLayoutSetUp(PetscLayout);
       Optionally use any of the following:
          PetscLayoutGetSize(PetscLayout,PetscInt *); or PetscLayoutGetLocalSize(PetscLayout,PetscInt *;)
          PetscLayoutGetRange(PetscLayout,PetscInt *rstart,PetscInt *rend); or PetscLayoutGetRanges(PetscLayout,const PetscInt *range[])
       PetscLayoutDestroy(PetscLayout);

      The PetscLayout object and methods are intended to be used in the PETSc Vec and Mat implementions; it is often not needed in  
      user codes unless you really gain something in their use.

    Fortran Notes: 
      Not available from Fortran

.seealso: PetscLayoutSetLocalSize(), PetscLayoutSetSize(), PetscLayoutGetSize(), PetscLayoutGetLocalSize(), PetscLayout, PetscLayoutDestroy(),
          PetscLayoutGetRange(), PetscLayoutGetRanges(), PetscLayoutSetBlockSize(), PetscLayoutGetBlockSize(), PetscLayoutSetUp()