since developing object oriented software is so cumbersome in C and we are all resistent to doing it in C++

Matthew Knepley knepley at
Sat Dec 5 18:11:58 CST 2009

On Sat, Dec 5, 2009 at 6:01 PM, Jed Brown <jed at> wrote:

> On Sat, 5 Dec 2009 16:50:38 -0600, Matthew Knepley <knepley at>
> wrote:
> > You assign a few threads per element to calculate the FEM
> > integral. You could maintain this unassembled if you only need
> > actions.
> You can also store it with much less memory as just values at quadrature
> points.

Depends on the quadrature, but yes there are sometimes better storage
(especially if you have other properties like decay).

> > However, if you want an actual sparse matrix, there are a couple of
> > options
> >
> >
> >   1) Store the unassembled matrix, and run assembly after integration
> > is complete. This needs more memory, but should perform well.
> Fine, but how is this assembly done?  If it's serial then it would be a
> bottleneck, so you still need the concurrent thing below.

Vec assembly can be done on the CPU since its so small I think.

> >   2) Use atmoic operations to update. I have not seen this yet, so I am
> > unsure how is will perform.
> Atomic operations could be used per-entry but this costs on the order of
> 100 cycles on CPUs.  I think newer GPUs have atomics, but I don't know
> the cost.  Presumably it's at least as much as the latency of a read
> from memory.

Not sure. Needs to be explored. Felipe is working on it.

> When inserting in decent sized chunks, it would probably be worth taking
> per-row or larger locks to amortize the cost of the atomics.
> Additionally, you could statically partition the workload and only use
> atomics for rows/entries that were shared.

You can use partitioning/coloring techniques to increase the lockless


> Jed
What most experimenters take for granted before they begin their experiments
is infinitely more interesting than any results to which their experiments
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <>

More information about the petsc-dev mailing list