redistribution of vectors

[Stephan Kramer]

Matthew Knepley wrote:
> On Tue, Feb 10, 2009 at 2:11 PM, Stephan Kramer
> <stephan.kramer at imperial.ac.uk> wrote:
>   
>> Hi,
>>
>> I'm trying to pick up a linear system (matrix and rhs) that I've written out
>> in a previous parallel run using MatView and VecView in binary mode. Now
>> when I've read the matrix and rhs vector the rows are evenly distributed by
>> PETSc over the processes. The partioning previously used in the parallel run
>> however doesn't in general correspond to this even distribution (the global
>> node numbering is the same, but number of rows are not exactly equal per
>> process due to other constraints). So what I'm trying to do is redistribute
>> the read in matrix and vector to match the previously used partioning. I've
>> found that for the matrix this is easy to do using MatGetSubMatrix(), but
>> I'm a little stuck on how to do this for the vector. Is there a similar way
>> of doing this for the vector? There seems to be no way of extracting vector
>> values not stored on this processor.
>>
>> I've tried to look at src/ksp/ksp/examples/tutorials/example10.c where
>> something similar is done for a repartioning created by MatPartioning. It
>> does the redistribution of the matrix with MatGetSubMatrix() as I expected,
>> but then seems to do nothing for the vector (there is only a comment:  /*
>> need to move the vector also */ ).
>>
>> Does this mean I just have to extract the vector values locally and do some
>> redistributing with MPI myself?
>>     
>
> You can certainly use a VecScatter to redistribute a vector. You
> already have the
> indices from MatGetSubMatrix() so it should be easy.
>
> However, you can also use VecLoadIntoVector() and provide an initial
> partitioning.
>
>   Matt
>   
Ah brilliant, I must have overlooked these options (there are so many in 
PETSc!). In that case is there also something like VecLoadIntoVector for 
MatLoad, i.e. where I specify the distribution beforehand, otherwise 
I'll just go for the MatGetSubMatrix() + VecScatter().

Just a small related question, for MatGetSubMatrix where I ask for the 
rows owned by each process but all columns, should I really assemble an 
index set ranging over all global indices? This seems a bit wasteful and 
non-scalable. Or should I put in the effort and, using information I 
have over my halo regions, work out which columns might have nonzeros 
for these rows. Or am I missing something and is there an easier short-cut?

Thanks a lot
Stephan

>   
>> Cheers
>> Stephan
>>