[petsc-users] Parallelize the Schur complement

[Sun, Hui]

Hi Jed, thank you for your answers. However, I still don't think I get the answer. Maybe I should ask in a clearer way. 

My A00 is of size n^3 times n^3, while my A11 is of size m times m. A00 has DM, but A11 does not. 

Let's suppose I have 16 cores, and I use all 16 cores to create the DM for A00, and my A01 and A10' has 16 cores parallelization in rows, but they are sequential in columns, and A11 is also sequential. 

I want to make A11 parallelized. So maybe I can try the following, I use 8 cores to create the DM for A00, and when I do A10*A00^(-1)*A01*v, I need the same 8 cores for the rows of A01 and A10' so that the matrix multiplications can carry over. But I want to parallelize A11 as well, so maybe I want to use 2 cores for the rows of A11 and 2 cores for the columns of A11, and hence I should also have 2 cores for the columns of A01 and A10'. Then, for matrix A00 I use 8 cores, and for A10 and A01 I use 8 times 2 which is 16 cores, and for A11 I use 4 cores. However, this doesn't seem right, because since there are 16 cores for A01, these 16 cores should all have access to part of the matrix A00 because we have the operation A00^(-1)*A01. But I use 8 cores for DM. That means I should have two copies of A00, which doesn't seem quite reasonable. 

But what is a reasonable way to do this task: that is, to parallelize all four components A00, A01, A10, A11 in a consistent, efficient, and natural way. 

Best,
Hui


________________________________________
From: Jed Brown [jed at jedbrown.org]
Sent: Friday, March 20, 2015 10:32 PM
To: Sun, Hui; petsc-users at mcs.anl.gov
Subject: Re: [petsc-users] Parallelize the Schur complement

"Sun, Hui" <hus003 at ucsd.edu> writes:
> How do I parallelize it? I mean, I use DMCreateMatrix to create A00,
> and I can parallelize the rows of A01 and A10^t by the same DM
> object. But I don't know how to parallelize the columns of A01 and
> A10^t.

You create the matrix.  We don't have DM support for non-square
matrices, but you can get their sizes.  You might consider using
DMComposite or a multi-field DM when creating the matrix.  It's also
pretty likely that you don't want to actually form the Schur complement
(usually dense), and instead would be better off approximating it and
solving with it iteratively.