[petsc-dev] Adding support memkind allocators in PETSc

[Matthew Knepley]

On Tue, Apr 28, 2015 at 5:26 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Apr 27, 2015, at 1:51 PM, Richard Mills <rtm at utk.edu> wrote:
> >
> > All,
> >
> > I'd like to add support for the allocators provided by the 'memkind'
> library (https://github.com/memkind/memkind).  I've discussed memkind a
> little bit with some of you off-list.  Briefly, memkind provides a "user
> extensible heap manager built on top of jemalloc which enables control of
> memory characteristics and a partitioning of the heap between kinds of
> memory".  The immediate motivation is to support placement of critical data
> structures into the high bandwidth on-package memory that will be available
> with Intel's "Knights Landing" generation of Xeon Phi processor (like on
> the upcoming NERSC Cori machine), but what the library provides is more
> general, and it can also be used for placing data in memory such as
> nonvolatile RAM (NVRAM), which will be appearing in more systems.
> >
> > I'm with Jed in thinking that, ideally, PETSc (or its users) shouldn't
> have to make decisions about the optimal way to solve the "packing problem"
> of what should go into high-bandwidth memory.  (In fact, I think this is a
> really interesting research problem that relates to some work on
> memory-adaptation in scientific applications that I did back when I was
> doing my Ph.D. research, e.g.,
> http://www.climatemodeling.org/~rmills/pubs/JGC_mmlib_2007.pdf.)
> However, right now I'd like to take the baby step of coming up with a
> mechanism to simply tag PETSc objects with a kind of memory that is
> preferred, and then having associated allocations reflect that preference
> (or requirement, if the user wants allocations to fail if such memory is
> not available).  Later we can worry about how to move data structures in
> and out of a kind of memory.
> >
> > It might make sense to add an option for certain PETSc classes--Mat and
> Vec are the most obvious here--to prefer allocations in a certain kind of
> memory.  Or, would it make more sense to add such an option at the
> PetscObject level?
> >
> > I think it is possible to add the memkind support without breaking all
> of the interfaces used throughout PETSc for PetscMalloc(), etc.
>
>   I don't think having this as a goal is useful at all! Just break the
> current interface; add an abstract "memkind" argument to all PetscMalloc()
> and Free() calls that indicates any additional information about the memory
> requested. By making it "abstract" it will always just be there and on
> systems without any special memory options it just doesn't do anything.
>

Since Malloc() is so pervasive, I think it would be useful to have a 2
level interface here. The standard Malloc() would call
you advanced PlacedMalloc(), and anyone could call that function, but I
think its just cruel to make everyone allocating
memory give arguments they do not understand or need.

  Matt


>    Barry
>
>    Note: it is not clear to me how this could be helpful on its own
> because I agree with Jed how is the user when creating the object supposed
> to know the optimal place to put the object?  For more complex objects it
> may be that different parts of the object would be stored in different
> types of memory etc etc.
>
>
> >  I recently sat with Chris Cantalupo, the main memkind developer, and
> walked him through PETSc's allocation routines, and we came up with the
> following: The imalloc() function pointer could have an implementation
> something like
> >
> > PetcErrorCode PetscMemkindMalloc(size_t size, const char *func, const
> char *file, void **result)
> >
> > {
> >
> >     struct memkind *kind;
> >
> >     int err;
> >
> >
> >     if (*result == NULL) {
> >
> >         kind = MEMKIND_DEFAULT;
> >
> >     }
> >
> >     else {
> >
> >         kind = (struct memkind *)(*result);
> >
> >     }
> >
> >
> >     err = memkind_posix_memalign(kind, result, 16, size);
> >
> >     return PosixErrToPetscErr(err);
> >
> > }
> >
> >
> > and ifree will look something like:
> >
> > PetcErrorCode PetscMemkindFree(void *ptr, int a, const char *func, const
> char *file)
> >
> > {
> >
> >     memkind_free(0, ptr);
> >
> >     return 0;
> >
> > }
> >
> >
> > This gives us (1) a method of passing the kind of memory without
> modifying the petsc allocation routine calling sequence, and (2) support a
> fall back code path legacy applications which will not set the pointer to
> NULL.  Or am I missing something?
> >
> > Thoughts?  I'd like to hash out something soon and start writing some
> code.
> >
> > --Richard
>
>


-- 
What most experimenters take for granted before they begin their
experiments is infinitely more interesting than any results to which their
experiments lead.
-- Norbert Wiener
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150428/411f224e/attachment.html>