[petsc-users] Some tricky problem in my multilevel feti dp code

[Matthew Knepley]

On Thu, Dec 13, 2012 at 7:41 AM, Thomas Witkowski
<thomas.witkowski at tu-dresden.de> wrote:
> I have some problem in the implementation of my multilevel FETI DP code,
> where two block structured matrices must be multiplied. I'll give my best to
> explain the problem, may be one of you have an idea how to implement it. I
> think, the best is to make a small example: lets assume we have 16
> subdomains, uniformly subdividing a unit square. Each of the subdomain
> matrices is purely local, thus they have the communicator PETSC_COMM_SELF.
> Each of them is of size n x n. There is a coarse grid matrix, with
> communicator PETSC_COMM_WORLD and of size m x m. The coupling matrices
> between the global coarse grid and the local matrices are also global, so
> they are of size 16n x m and m x 16n, respectively. So far, everything is
> fine and works perfectly. Now I introduce four "local coarse grids", each of
> them couples four local subdomains, and is defined on a subset communicator
> of PETSC_COMM_WORLD. Say, each "local coarse grid" matrix is of size p x p,
> and there are also coupling matrices of size 4n x p and p x 4n. Now I have
> to perform a MatMatMult of the local coarse coupling matrices p x 4n with
> the global coupling matrix 16n x m. So the final matrix is of size 4p x m.
> But I cannot perform the MatMatMult, as the matrix sizes do not fit and the
> communicators are not compatible.
>
> Is it possible to understand, what I want to do? :) Any idea, how to
> implement it?

It sounds like you need to redistribute the matrix before the
MatMatMult. I think you
can do this with MatGetSubmatrix(), if I understand your problem
correctly. You probably
need to move the matrix from the subcomm to the global comm first,
with empty entries
on some procs. I would just do it the naive way first, then profile to
see how it does.

   Matt

> Thomas



--
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which
their experiments lead.
-- Norbert Wiener