[petsc-users] Preallocation: Allocate more than the local column

[Barry Smith]

   Florian,

     I don't really understand your question so I will answer a question that I do understand that may or not be related to your question.

     To do a proper preallocation each process needs to know the rows of the matrix that it will own plus which columns are in the "diagonal portion" of the matrix. Usually one figures this information out based on the decomposition of the "mesh" that they have decided on. 

     One cannot have the matrix decide on the decomposition of the rows/columns while also trying to provide the preallocation information since the preallocation information depends exactly on the decomposition of the rows/columns. Thus you need to determine how you want to decompose the matrix YOURSELF before you create the matrix and provide the preallocation information. 

   Barry

> On Apr 18, 2016, at 6:02 AM, Florian Lindner <mailinglists at xgm.de> wrote:
> 
> Hello,
> 
> I create a MATAIJ on 4 processors using local sizes
> 
> (0) Set matrix A to local size 2 x 5
> (1) Set matrix A to local size 2 x 2
> (2) Set matrix A to local size 2 x 2
> (3) Set matrix A to local size 2 x 2
> 
> which results in a global size of 8 x 11.
> 
> Afaik the local size given is not relevant for the parallel layout, so each processors holds 2 rows x 11 cols. Accordingly I can set these using MatSetValue and alike
> 
> But it seems to be important when I do preallocation, i.e. work on the local submatrices:
> 
> (0) Local Submatrix Rows = 0 / 2, Local Submatrix Cols = 0 / 5
> (1) Local Submatrix Rows = 2 / 4, Local Submatrix Cols = 5 / 7
> (2) Local Submatrix Rows = 4 / 6, Local Submatrix Cols = 7 / 9
> (3) Local Submatrix Rows = 6 / 8, Local Submatrix Cols = 9 / 11
> 
> (retrieved with MatGetOwnershipRange / MatGetOwnershipRangeColumn)
> 
> therefore, if I do a preallocation using MatMPIAIJSetPreallocation I get errors like
> 
> [1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home/florian/software/petsc/src/mat/impls/aij/seq/aij.c
>    nnz cannot be greater than row length: local row 0 value 4 rowlength 2
> 
> Easiest way to work around that would be to set the number of local cols (here always 5/2/2/2) to 11. Setting the number of global columns to 11 would probably have the same effect.
> 
> Problem is that each processor only knows the number of local columns, so I would need some communication for that.
> 
> Another possibility that comes to my mind is to declare the Mat only after the preallocation has been determined and during preallocation the longest line has been computed.
> 
> Are my assumptions correct so far?
> How would you cope with that?
> 
> Best Regards,
> Florian
> 
>