[petsc-users] Some tricky problem in my multilevel feti dp code

[Mark F. Adams]

You might also be able to put everything in a global comm.  Then you have a bunch of block diagonal matrix ops.  There is no performance penalty other then in reductions (e.g., when PETSc figures out its scatter stuff) and you have to keep track of where the local problem "starts" in the global matrix, but it might be simpler.  Note, your local LU solves will now be a global looking block Jacobi with a sub LU solver, but its the same thing.


On Dec 13, 2012, at 11:43 AM, Matthew Knepley <knepley at gmail.com> wrote:

> On Thu, Dec 13, 2012 at 7:41 AM, Thomas Witkowski
> <thomas.witkowski at tu-dresden.de> wrote:
>> I have some problem in the implementation of my multilevel FETI DP code,
>> where two block structured matrices must be multiplied. I'll give my best to
>> explain the problem, may be one of you have an idea how to implement it. I
>> think, the best is to make a small example: lets assume we have 16
>> subdomains, uniformly subdividing a unit square. Each of the subdomain
>> matrices is purely local, thus they have the communicator PETSC_COMM_SELF.
>> Each of them is of size n x n. There is a coarse grid matrix, with
>> communicator PETSC_COMM_WORLD and of size m x m. The coupling matrices
>> between the global coarse grid and the local matrices are also global, so
>> they are of size 16n x m and m x 16n, respectively. So far, everything is
>> fine and works perfectly. Now I introduce four "local coarse grids", each of
>> them couples four local subdomains, and is defined on a subset communicator
>> of PETSC_COMM_WORLD. Say, each "local coarse grid" matrix is of size p x p,
>> and there are also coupling matrices of size 4n x p and p x 4n. Now I have
>> to perform a MatMatMult of the local coarse coupling matrices p x 4n with
>> the global coupling matrix 16n x m. So the final matrix is of size 4p x m.
>> But I cannot perform the MatMatMult, as the matrix sizes do not fit and the
>> communicators are not compatible.
>> 
>> Is it possible to understand, what I want to do? :) Any idea, how to
>> implement it?
> 
> It sounds like you need to redistribute the matrix before the
> MatMatMult. I think you
> can do this with MatGetSubmatrix(), if I understand your problem
> correctly. You probably
> need to move the matrix from the subcomm to the global comm first,
> with empty entries
> on some procs. I would just do it the naive way first, then profile to
> see how it does.
> 
>   Matt
> 
>> Thomas
> 
> 
> 
> --
> What most experimenters take for granted before they begin their
> experiments is infinitely more interesting than any results to which
> their experiments lead.
> -- Norbert Wiener
>