[petsc-dev] Adding support memkind allocators in PETSc

[Richard Mills]

All,

I'd like to add support for the allocators provided by the 'memkind'
library (https://github.com/memkind/memkind).  I've discussed memkind a
little bit with some of you off-list.  Briefly, memkind provides a "user
extensible heap manager built on top of jemalloc which enables control of
memory characteristics and a partitioning of the heap between kinds of
memory".  The immediate motivation is to support placement of critical data
structures into the high bandwidth on-package memory that will be available
with Intel's "Knights Landing" generation of Xeon Phi processor (like on
the upcoming NERSC Cori machine), but what the library provides is more
general, and it can also be used for placing data in memory such as
nonvolatile RAM (NVRAM), which will be appearing in more systems.

I'm with Jed in thinking that, ideally, PETSc (or its users) shouldn't have
to make decisions about the optimal way to solve the "packing problem" of
what should go into high-bandwidth memory.  (In fact, I think this is a
really interesting research problem that relates to some work on
memory-adaptation in scientific applications that I did back when I was
doing my Ph.D. research, e.g.,
http://www.climatemodeling.org/~rmills/pubs/JGC_mmlib_2007.pdf.)  However,
right now I'd like to take the baby step of coming up with a mechanism to
simply tag PETSc objects with a kind of memory that is preferred, and then
having associated allocations reflect that preference (or requirement, if
the user wants allocations to fail if such memory is not available).  Later
we can worry about how to move data structures in and out of a kind of
memory.

It might make sense to add an option for certain PETSc classes--Mat and Vec
are the most obvious here--to prefer allocations in a certain kind of
memory.  Or, would it make more sense to add such an option at the
PetscObject level?

I think it is possible to add the memkind support without breaking all of
the interfaces used throughout PETSc for PetscMalloc(), etc.  I recently
sat with Chris Cantalupo, the main memkind developer, and walked him
through PETSc's allocation routines, and we came up with the following: The
imalloc() function pointer could have an implementation something like

PetcErrorCode PetscMemkindMalloc(size_t size, const char *func, const char
*file, void **result)

{

    struct memkind *kind;

    int err;



    if (*result == NULL) {

        kind = MEMKIND_DEFAULT;

    }

    else {

        kind = (struct memkind *)(*result);

    }



    err = memkind_posix_memalign(kind, result, 16, size);

    return PosixErrToPetscErr(err);

}


and ifree will look something like:



PetcErrorCode PetscMemkindFree(void *ptr, int a, const char *func, const
char *file)

{

    memkind_free(0, ptr);

    return 0;

}


This gives us (1) a method of passing the kind of memory without modifying
the petsc allocation routine calling sequence, and (2) support a fall back
code path legacy applications which will not set the pointer to NULL.  Or
am I missing something?

Thoughts?  I'd like to hash out something soon and start writing some code.

--Richard
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150427/5c351ed0/attachment.html>