[petsc-users] Efficiency of different choice of local rows for MPIAIJ * MPIDENSE

[Ian C. Lin]

Hi,

I am recently trying to do matrix multiplication for C = A*B, where A is a sparse matrix MPIAIJ, C and B are created as dense matrices MPIDENSE.

In matrix A, the nonzeros are not distributed evenly across the processor, meaning that if using the default setting to let each processor own similar number of rows, the number of nonzeros owned by each processor will be significantly different. So I want to use different number of local rows for each processor. In this case, does the MPIDense matrices B and C need to be in the same row-layout as A?

I mean, is something like the following is doable (A owns 3 rows and B, C own 2 rows)

           
           A                    B          C
P0  o o o o | o.         o o.       o o
      o o o o | o          o o        o o
      o o o o | o.    *.   -----  =   ----
     ---------------         o o        o o
P1  o o o o | o          o o        o o

In this case, the entries can be evenly distributed for B and C thus more memory efficient.

But I am not sure would this make communication more complicated thus slow down the overall wall time. How would you recommend to do? 
a) Let rows of A and B be both evenly distributed
b) Let A have different rows layout, and B, C evenly distributed 
c) Let A have different rows layout, and B, C follow A

Or maybe other better way that I did not think about.

Thanks a lot for your help, 
Ian