[petsc-users] Newbie question : sequential and parallel storage, and memory estimation

[Barry Smith]

> On Feb 15, 2017, at 9:57 PM, lixin chu <lixin_chu at yahoo.com> wrote:
> 
> Hello,
> Newbie question again - data storage and distribution related this time 
> 
> I have briefly looked at the user manual, and searched forum, trying to get some basic understanding of the data storage and distribution. Some questions I have:
> 
> 1. Sequential vs. Parallel
>     I assume that by 'sequential', we mean the matrix/vector is created in one process only ?

>     Then if this correct, I can assume that 'parallel' means each process will create a sub-matrix/vector for itself ?

   In PETSc language parallel means that parts of the matrix/vector are stored on each process. 
> 
>     Will each process eventually have the complete matrix and the RHS vector after MatAssemblyBegin() and MatAssemblyEnd(), before solve starts ?

  No, each process contains part of the matrix and part of the vector. If you stored the entire matrix on each process this would limit how large a problem you could run based on the memory had on one process.

> 
> 2. How the matrix/vector is split among the processes ?
>     I think matrix will be split by blocks of rows ?

  Yes

> Does PETSc equally divide the matrix rows (with the last taking the remainder ) ?

 More or less. You can set how the rows are divided by using MatSetSizes()  and you can use MatGetOwnershipRange() to determine how it decided to split the rows.

> 
>     If I want to create the matrix in a parallel way, do I have to divide the rows equally, or different process can have different number of rows, and some may not have any rows ?

   You can using MatSetSizes() do it anyway you want. BUT if you give some processes many rows and others zero rows you will not be using the parallel machine efficiently since some processes will have lots of work (and memory demands) and others will have no work. 
> 
>     Does RHS vector also need to be split in the exactly same blocks of rows as the matrix A (same start and end row number for the same process) ?

  Yes

> 
> 
> 3. MAT_STRUCTURALLY_SYMMETRIC
>    Do I have to input the upper triangle portion of non zero values, or lower triangle is also fine ?

  Structurally symmetric only means the nonzero pattern below the diagonal is the same as above, the numerical values below the diagonal may be different than above so you always need to provide all the values.

> 
> 
> 4. Given the matrix A and RHS b, data type and sparsity, is there a way to estimate the total RAM needed for the solve phase, let's say for GMRES algorithm ?

  Not really. It is very complicated based on the preconditioned used. Just for the GMRES solver, not counting memory for the preconditioner it is roughly restart + seven vectors worth of memory.

  Barry

> 
> many thanks !
>  
> rgds
> LX