[petsc-dev] Adding support memkind allocators in PETSc

[Jed Brown]

Richard Mills <rtm at utk.edu> writes:

> It's been a while, but I'd like to pick up this discussion of adding a
> context to memory allocations again.

Have you heard anything back about whether move_pages() will work?

> hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size);

How much would you prefer it?  If we stupidly ask for HBM in VecCreate_*
and MatCreate_*, then our users will see catastrophic performance drops
at magic sizes and will have support questions like "I swapped these two
independent lines and my code ran 5x faster".  Then they'll hack the
source by writing

  if (moon_is_waxing() && operator_holding_tongue_in_right_cheek()) {
    policy = MEMKIND_HBW_PREFERRED;
  }

eventually making all decisions based on nonlocal information, ignoring
the advice parameter.

Then they'll get smart and register their own malloc so they don't have
to hack the library.  Then they'll try to couple their application with
another that does the same thing and now they have to write a new malloc
that makes a new set of decisions in light of the fact that multiple
libraries are being coupled.

I think we can agree that this is madness.  Where do you draw the line
and say that crappy performance is just reality?

It's hard for me not to feel like the proposed system will be such a
nightmarish maintenance burden with such little benefit over a simple
size-based allocation that it would be better for everyone if it doesn't
exist.

For example, we've already established that small allocations should
generally go in DRAM because they're either cached or not prefetched and
thus limited by latency instead of bandwidth.  Large allocations that
get used a lot should go in HBM so long as they fit.  Since we can't
determine "used a lot" or "fit" from any information possibly available
in the calling scope, there's literally no useful advice we can provide
at that point.  So don't try, just set a dumb threshold (crude tuning
parameter) or implement a profile-guided allocation policy (brittle).

Or ignore all this nonsense, implement move_pages(), and we'll have PETSc
track accesses so we can balance the pages once the app gets going.

> Of course, we'll need some way to ensure that the "advanced malloc" 

I thought AdvMalloc was short for AdvisedMalloc.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 818 bytes
Desc: not available
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150603/8b77ebb4/attachment.sig>