[petsc-users] Block matrix data layout and access

[Jed Brown]

On Mon, Nov 5, 2012 at 7:10 AM, <Gregor.Matura at dlr.de> wrote:

> Maybe I have not explained my situation well enough: I do already have a
> matrix on the disk.
>

First a warning: 99% of the time, storing large matrices on disk represents
a flawed workflow. Putting a matrix on disk is an epic scalability
bottleneck that almost no amount of computation can escape. It takes most
of today's supercomputers more than half an hour to read or write their
contents to disk, assuming that they sustain maximum theoretical bandwidth
for the entire operation.


> This makes generating the matrix on-the-fly either a burden for the
> filesystem (because of to many accesses in a short time) or for the memory
> (because data has to be duplicated).
>

1. File IO APIs read in chunks, not entire files. Just read one chunk at a
time, loop through rows in the chunk, and call
MatSetValues[Blocked][Local]().

2. Be aware that independent file IO will scale terribly.

3. Recall that it's _terrible_ for your workflow to partition the matrix
entirely up-front for a specific number of processors. This means that you
can't store the matrix data in a way that could be used directly anyway.


> I want to avoid both, thus read in my matrix in one go and pass this data
> conveniently to PETSc. As far as my insights go, there are two
> possibilities with certain drawbacks, maybe you could uncover more details
> on both (like convenience of data layout, assembly time, solution speed, …).
> ****
>
> ** **
>
> 1.)MatCreateBAIJ (includes MatSetBlockSize and MatMPIBAIJSetPreallocation)
> > MatSetLocalToGlobalMapping > MatSetValuesBlockedLocal****
>
> drawbacks:****
>
> - data is copied (thus doubles memory needs)
>

Disk doesn't work like this, see above.


> ****
>
> - ALL local rows has to be read in to get information for preallocation
>

Row lengths are stored explicitly anyway, in a dedicated array if you use a
CSR-type format.


> ****
>
> - but: SetValuesBlockedLocal requires "square" layout of data (not suited
> for sparse matrices). In my case this leads to setting values (block) row
> by (block) row.
>

That is intentional. It inserts a _logically dense_ block. In typical PDE
problems, that is a row (for FD schemes) or an element stiffness matrix for
FE schemes.


> ****
>
> - and: value array does not match block structure (cf. documentation
> example of MatSetValuesBlocked): my values are sorted sequentially by
> blocks, i.e. v[] = [1,2,5,6,3,4,7,8,...]
>

Just unpack row-by-row before calling MatSetValuesBlocked(). It's
vanishingly little time.


> ****
>
> ** **
>
> 2.)MatCreateMPIAIJWithSplitArrays****
>
> drawbacks:****
>
> ? doesn't match PETSc data layout -> slower?****
>
> - not blocked (conversion to MPIBAIJ possible?)
>

This routine exists for rare and peculiar workflows. It's only used by a
handful of people and most of them shouldn't be using it. You shouldn't use
it either.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20121105/83c6d005/attachment-0001.html>