[petsc-users] Preallocation: Allocate more than the local column

[Florian Lindner]

Hello,

I create a MATAIJ on 4 processors using local sizes

(0) Set matrix A to local size 2 x 5
(1) Set matrix A to local size 2 x 2
(2) Set matrix A to local size 2 x 2
(3) Set matrix A to local size 2 x 2

which results in a global size of 8 x 11.

Afaik the local size given is not relevant for the parallel layout, so each processors holds 2 rows x 11 cols. Accordingly I can set these using MatSetValue and alike

But it seems to be important when I do preallocation, i.e. work on the local submatrices:

(0) Local Submatrix Rows = 0 / 2, Local Submatrix Cols = 0 / 5
(1) Local Submatrix Rows = 2 / 4, Local Submatrix Cols = 5 / 7
(2) Local Submatrix Rows = 4 / 6, Local Submatrix Cols = 7 / 9
(3) Local Submatrix Rows = 6 / 8, Local Submatrix Cols = 9 / 11

(retrieved with MatGetOwnershipRange / MatGetOwnershipRangeColumn)

therefore, if I do a preallocation using MatMPIAIJSetPreallocation I get errors like

[1]PETSC ERROR: MatSeqAIJSetPreallocation_SeqAIJ() line 3567 in /home/florian/software/petsc/src/mat/impls/aij/seq/aij.c
    nnz cannot be greater than row length: local row 0 value 4 rowlength 2

Easiest way to work around that would be to set the number of local cols (here always 5/2/2/2) to 11. Setting the number of global columns to 11 would probably have the same effect.

Problem is that each processor only knows the number of local columns, so I would need some communication for that.

Another possibility that comes to my mind is to declare the Mat only after the preallocation has been determined and during preallocation the longest line has been computed.

Are my assumptions correct so far?
How would you cope with that?

Best Regards,
Florian