[petsc-users] Matrix and vector type selection & memory allocation for efficient matrix import?

Matthew Knepley knepley at gmail.com
Fri Apr 20 12:52:08 CDT 2018 

On Fri, Apr 20, 2018 at 1:30 PM, Klaus Burkart <k_burkart at yahoo.com> wrote:

> In my case N=M but n for process 0, 1, 2, 3,... no_processes-1 can be
> different from the nth process like in the example where the nth
> process=Proc2 and has only two rows while all other processes have three
> rows:
>

Yes.


> Example from the PETSc webpage mentioned before:
>
>             1  2  0  |  0  3  0  |  0  4
>     Proc0   0  5  6  |  7  0  0  |  8  0
>             9  0 10  | 11  0  0  | 12  0
>     -------------------------------------
>            13  0 14  | 15 16 17  |  0  0
>     Proc1   0 18  0  | 19 20 21  |  0  0
>             0  0  0  | 22 23  0  | 24  0
>     -------------------------------------
>     Proc2  25 26 27  |  0  0 28  | 29  0
>            30  0  0  | 31 32 33  |  0 34
>
> and I need to enter different values for d_nnz and o_nnz for each row
> somewhere too
>
>      proc0: d_nnz = [2,2,2] and o_nnz = [2,2,2]
>      proc1: d_nnz = [3,3,2] and o_nnz = [2,1,1]
>      proc2: d_nnz = [1,1]   and o_nnz = [4,4]
>
> Each process only sets d_nnz and o_nnz for its LOCAL rows. Thus, it is
exactly as shown above. Proc1 sets values
only for rows 3-5.

  Thanks,

    Matt


> I simply can't identify the function(s) used to set the values for n,
> d_nnz and o_nnz for the individual local matrices allocated to all the
> processes if n isn't the same for all processes and d_nnz and o_nnz are
> different for each local matrix?
>
> Approach described on the PETSc webpage:
>
>     MatCreate <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatCreate.html#MatCreate>(...,&A);
>
>
>     MatSetType <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetType.html#MatSetType>(A,MATMPIAIJ <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MATMPIAIJ.html#MATMPIAIJ>);
>
>
>     MatSetSizes <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatSetSizes.html#MatSetSizes>(A, m,n,M,N); // for the example above using this function would set the no. of rows for Proc2 to 3 but it's 2
>
>
>
>     MatMPIAIJSetPreallocation <http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Mat/MatMPIAIJSetPreallocation.html#MatMPIAIJSetPreallocation>(A,...); // this function can be used to set values for ONE local matrix only
>
>
> In addition to that I don't know which functions to use to preallocate memory for ALL local matrices when each of them has different values for d_nnz and o_nnz.
>
> I other words, what's the code for the 3 process example above? (entering the matrix structure and allocating memory)
>
> Klaus
>
> Am Freitag, 20. April 2018, 17:13:26 MESZ hat Smith, Barry F. <
> bsmith at mcs.anl.gov> Folgendes geschrieben:
>
>
>
>   For square matrices almost always n is the same as m. On different
> processes m can be different. You get to decide what makes sense for each
> processes what its m should be.
>
>   Barry
>
>
> > On Apr 20, 2018, at 10:05 AM, Klaus Burkart <k_burkart at yahoo.com> wrote:
> >
> > I think I understood the matrix structure for parallel computation with
> the rows, diagonal (d) and off-diagonal (o) structure, where I have
> problems is how to do the setup including memory allocation in PETSc:
> >
> > Lets assume, I use a 16 core workstation (=16 processes) and the number
> of nonzeros varies in each row for both d and o and the number of rows
> assigned to each process differs too - at least for the nth process.
> >
> > Looking at the manual and http://www.mcs.anl.gov/petsc/
> petsc-current/docs/manualpages/Mat/MatCreateAIJ.html#MatCreateAIJ, I
> don't understand how to enter a global matrix when n is NOT the same for
> each process as e.g. in MatSetSizes(A, m,n,M,N); n and m are integers, not
> arrays?
> >
> >    MatCreate(...,&A);
> >
> >    MatSetType(A,MATMPIAIJ);
> >
> >    MatSetSizes(A, m,n,M,N); // seems to assume n and m are the same for
> each process which isn't even the case in the example on the page mentioned
> above?!
> >
> >    MatMPIAIJSetPreallocation(A,...);
> >
> >
> > How can I enter the parallel global-local matrix structure?
> >
> > How can the memory preallocation be done?
> >
> > Klaus
> >
> > Am Donnerstag, 19. April 2018, 01:47:59 MESZ hat Smith, Barry F. <
> bsmith at mcs.anl.gov> Folgendes geschrieben:
> >
> >
> >
> >
> > > On Apr 18, 2018, at 4:42 PM, k_burkart at yahoo.com wrote:
> > >
> > > So, practically speaking, l should invent routines to decompose the
> matrix e.g. into a block matrix structure to be able to make real use of
> PETSc ie. be able to solve a linear system using more than one process/core?
> >
> >  To really use PETSc efficiently/effectively you need to generate your
> matrix in parallel.
> >
> >  Barry
> >
> > >
> > > Klaus
> > >
> > > Von meinem Huawei-Mobiltelefon gesendet
> > >
> > >
> > > -------- Originalnachricht --------
> > > Betreff: Re: [petsc-users] Matrix and vector type selection & memory
> allocation for efficient matrix import?
> > > Von: "Smith, Barry F."
> > > An: Klaus Burkart
> > > Cc: PETSc Users List
> > >
> > >
> > >
> > > If you can only generate the nonzero allocation sequentially you can
> only solve sequentially which means your matrix is MATSEQAIJ and your
> vector is VECSEQ and your communicator is PETSC_COMM_SELF.
> > >
> > > If you pass and array for nnz, what you pass for nz is irrelevant, you
> might as well pass 0.
> > >
> > > Barry
> > >
> > >
> > > > On Apr 18, 2018, at 10:48 AM, Klaus Burkart wrote:
> > > >
> > > > More questions about matrix and vector type selection for my
> application:
> > > >
> > > > My starting point is a huge sparse matrix which can be symmetric or
> asymmetric and a rhs vector. There's no defined local or block structure at
> all, just row and column indices and the values and an array style rhs
> vector together describing the entire linear system to be solved. With
> quite some effort, I should be able to create an array nnz[N] containing
> the number of nonzeros per row in the global matrix for memory allocation
> which would leave me with MatSeqAIJSetPreallocation(M, 0, nnz); as the only
> option for efficient memory allocation ie. a MATSEQAIJ matrix and VECSEQ. I
> assume here, that 0 indicates different numbers of nonzero values in each
> row, the exact number being stored in the nnz array. Regarding this detail
> but one example assume a constant number of nz per row so I am not sure
> whether I should write 0 or NULL for nz?
> > > >
> > > > I started with:
> > > >
> > > > MatCreate(PETSC_COMM_WORLD, &M);
> > > > MatSetSizes(M, PETSC_DECIDE, PETSC_DECIDE, N, N);
> > > > MatSetFromOptions(M);
> > > >
> > > > taken from a paper and assume, the latter would set the matrix type
> to MATSEQAIJ which might conflict with PETSC_COMM_WORLD. Maybe
> decompositioning took place at an earlier stage and the authors of the
> paper were able to retrieve the local data and structure.
> > > >
> > > > What type of matrix and vector should I use for my application e.g.
> MATSEQAIJ and VECSEQ to be able to use MatSeqAIJSetPreallocation(M, 0,
> nnz); for efficient memory allocation?
> > > >
> > > > In this case, where would the decompositioning / MPI process
> allocation take place?
> > >
>



-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener

https://www.cse.buffalo.edu/~knepley/ <http://www.caam.rice.edu/~mk51/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20180420/0be0ace2/attachment-0001.html>

More information about the petsc-users mailing list