[petsc-dev] What's the point of D(A/M)GetGlobalVector?

[Jed Brown]

On Sun, 29 Aug 2010 09:16:18 -0400, Kai Germaschewski <kai.germaschewski at unh.edu> wrote:
> Some malloc()'s, like GNU libc, allow you to hook into it. Even if not, if
> you define your own malloc(), it'll get used instead of the C library one
> (depends on the order of libs on the command line). However, in that case
> you may have trouble to do your thing and then continue on to the original
> version. Actually, with dynamic libs you can do it:
> 
> http://stackoverflow.com/questions/262439/create-a-wrapper-function-for-malloc-and-free-in-c
> 
> I don't know how to handle C++, I imagine there's a way to do it on common
> systems.

GNU new calls malloc, though not necessarily every time.  Unfortunately
this is all non-portable, changes the linking procedure, and is not
appropriate for a library to do.

Note also that object creation and destruction are collective
operations, but malloc() may be called from user code on one process.
This may require GC'ing an unused cache, which involves destroying
parallel objects.

> I take your point, it'll be major hassle, and it's quite likely you can't
> make it work on every system. However, you could check at configure time,
> and just not use the cache but free() things on destroy() if the system
> doesn't support it. I'm also not sure I like the random timing distortion
> caused by having something GC-like.
> 
> Before considering the complex solutions, I think it's important to first
> answer
> (1) Is the cache something that really make a difference in performance? (I
> suppose whenever someone DAGet{Global,Local}Vector(), they will do something
> to fill that vector with values, if that near always takes >> than even a
> VecCreate(), there may be no point in caching those things in the first
> place.

Depends on the size of the vector, coarse level residual evaluation
works with very small vectors, you'd rather not add a few collective
operations to that operation.

> (2) Is there an easier way to fix the problem? One way may be to look at why
> a DACreate{Local,Global}Vector() isn't performing as well as we'd need, and
> there may ways to streamline/optimize it for the common case. A second way
> could be a "do_cache" flag on Vec's (and maybe Mat's, too). That is, do your
> VecCreate(), or DACreate{...}Vector(), and then VecSetCached(vec). That flag
> would be checked at Destroy() time and the Vector would go into the pool
> instead of being free'd. In that case, you don't eat up all mem for the
> unsuspecting user, but an advanced user would easily be able to use the
> benefit of caching where it actually matters (That's not as nice as having
> everything fully automatic, but...)

If this cache is specific to the DA, then I don't see this being
preferable to DAGet*.  If the cache is not specific to the DA, then you
need to somehow match "compatible" Vecs lying around in the cache, and I
still don't think the interface is as nice, and it seems less precise
when you want to reclaim that memory.


As an aside, talloc (http://talloc.samba.org) is an interesting
hierarchical, reference counting allocator.  You allocate hierarchically
(new memory references a "parent"), and then automatically collect all
children when the parent is freed.  There is support for various sharing
schemes, the interesting thing is that it makes it much easier to clean
up gracefully in error conditions.  If you're interesting in C memory
management systems, it's worth a look.

Jed