MPIAIJ and MatSetValuesBlocked

[Barry Smith]

On May 4, 2009, at 2:59 PM, Matthew Knepley wrote:

> On Mon, May 4, 2009 at 2:56 PM, Randall Mackie  
> <rlmackie862 at gmail.com> wrote:
> Jed,
>
> Can you explain how to tell if the matrix is truly blocked? What's the
> difference between a blocked matrix and one with several degrees of  
> freedom
> at each node, or are they the same thing? I'm solving Maxwell's  
> equations
> in 3D, so I have three vector field components at each node, is that  
> what
> you mean by blocked?
>
> Yes, that is blocked.

    The 3 components at each node create a little tiny blocks  
representing the coupling
between the various components at the node. If each component is  
coupled to all other
components at that node then the little blocks are dense. If there is  
only partial coupling then
the little blocks are sparse.  BAIJ treats the little blocks as dense,  
so if they are truly dense
you get the best performance. If they are "mostly filled in" then you  
get better performance
with BAIJ (over AIJ) but with the cost of doing some extra  
computations on the zeros
in the blocks.

    Barry

>
>
>   Matt
>
>
> Thanks, Randy
>
>
> Jed Brown wrote:
> Ryan Yan wrote:
> Could you make a little bit more clarification
> on why the MatSetValuesBlocked() have some advantage on blocked  
> structure?
>
> In addition to the assembly advantages that Satish pointed out, BAIJ
> requires less storage for the column indices, effectively improving  
> the
> arithmetic intensity of many kernels, and speeding up matrix
> factorization (e.g. symbolic factorization only needs to compute  
> fill in
> terms of blocks instead of individual elements).  The use of inodes  
> with
> AIJ (default when applicable) reduces the memory bandwidth  
> requirements
> of the column indices, turns point relaxation smoothers (SOR) into
> stronger block relaxation, and allows a certain amount of unrolling.
> BAIJ requires even less metadata, provides more regular memory access,
> and does more unrolling.
>
> If your matrix is truly blocked, BAIJ should provide better  
> performance
> with all preconditioners that support it.  Many third-party
> preconditioners will not work with BAIJ, so it is useful to give your
> matrix a prefix (or check the options database if you are getting your
> matrix from a DA or similar) so that you can set it's type with
> -foo_mat_type when using a preconditioner that requires it.
>
> Jed
>
>
>
>
> -- 
> What most experimenters take for granted before they begin their  
> experiments is infinitely more interesting than any results to which  
> their experiments lead.
> -- Norbert Wiener