[petsc-dev] Adding support memkind allocators in PETSc

[Richard Mills]

On Tue, Apr 28, 2015 at 10:47 PM, Jed Brown <jed at jedbrown.org> wrote:

> Richard Mills <rtm at utk.edu> writes:
>
> [...]
> > I think many users are going to want more control than what something
> like
> > AutoHBW provides, but, as you say, a lot of the time one will only care
> > about the the substantial allocations for things like matrices and
> vectors,
> > and these also tend to be long lived--plenty of codes will do something
> > like allocate a matrix for Jacobians once and keep it around for the
> > lifetime of the run.  Maybe we should consider not using a heap manager
> for
> > these allocations, then.  For allocations above some specified threshold,
> > perhaps we (PETSc) should simply do the appropriate mmap() and mbind()
> > calls to allocate the pages we need in the desired type of memory, and
> then
> > we could use things like use move_pages() if/when appropriate (yes, I
> know
> > we don't yet have a good way to make such decisions).  This would mean
> > PETSc getting more into the lower level details of memory management, but
> > maybe this is appropriate (an unavoidable) as more kinds of
> > user-addressable memory get introduced.  I think is actually less
> horrible
> > than it sounds, because, really, we would just want to do this for the
> > largest allocations.  (And this is somewhat analogous to how many
> malloc()
> > implementations work, anyway: Use sbrk() for the small stuff, and mmap()
> > for the big stuff.)
>
> I say just use malloc (or posix_memalign) for everything.  PETSc can't
> do a better job of the fancy stuff and these normal functions are
> perfectly sufficient.
>
> >> That is a regression relative to move_pages.  Just make move_pages work.
> >> That's the granularity I've been asking for all along.
> >
> > Cannot practically be done using a heap manager system like memkind.  But
> > we can do this if we do our own mmap() calls, as discussed above.
>
> In practice, we would still use malloc(), but set mallopt
> M_MMAP_THRESHOLD if needed and call move_pages.  The reality is that
> with 4 KiB pages, it doesn't even matter if your "large" allocation is
> not page aligned.  The first and last page don't matter--they're small
> enough to be inexpensive to re-fetch from DRAM and don't use up that
> much extra space if you map them into MCDRAM.
>

Hmm.  That may be a pretty good solution for DRAM vs. MCDRAM.  What about
when we further complicate things by adding some large pool of NVRAM?  One
might want some sufficiently large arrays to go into MCDRAM, but other
large arrays to go to NVRAM or DRAM.  I guess we can still do the
appropriate move_pages() to get things into the right places, but I can
also see wanting to do things like use a much large page size for giant
data sets going into NVRAM (which you won't be able to do without a copy to
a different mapped region).  And if there are these and other
complications... then maybe we should be using a heap manager like
memkind.  It would simplify quite a few things EXCEPT we'd have to deal
with the virtual address changing when we want to change "kind" of memory.
But maybe this would not be so bad, using an approach like Karli outlined.

--Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150429/6bae035a/attachment.html>