[petsc-dev] Adding support memkind allocators in PETSc

[Barry Smith]

> On Apr 27, 2015, at 2:30 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Tue, Apr 28, 2015 at 5:26 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Apr 27, 2015, at 1:51 PM, Richard Mills <rtm at utk.edu> wrote:
> >
> > All,
> >
> > I'd like to add support for the allocators provided by the 'memkind' library (https://github.com/memkind/memkind).  I've discussed memkind a little bit with some of you off-list.  Briefly, memkind provides a "user extensible heap manager built on top of jemalloc which enables control of memory characteristics and a partitioning of the heap between kinds of memory".  The immediate motivation is to support placement of critical data structures into the high bandwidth on-package memory that will be available with Intel's "Knights Landing" generation of Xeon Phi processor (like on the upcoming NERSC Cori machine), but what the library provides is more general, and it can also be used for placing data in memory such as nonvolatile RAM (NVRAM), which will be appearing in more systems.
> >
> > I'm with Jed in thinking that, ideally, PETSc (or its users) shouldn't have to make decisions about the optimal way to solve the "packing problem" of what should go into high-bandwidth memory.  (In fact, I think this is a really interesting research problem that relates to some work on memory-adaptation in scientific applications that I did back when I was doing my Ph.D. research, e.g., http://www.climatemodeling.org/~rmills/pubs/JGC_mmlib_2007.pdf.)  However, right now I'd like to take the baby step of coming up with a mechanism to simply tag PETSc objects with a kind of memory that is preferred, and then having associated allocations reflect that preference (or requirement, if the user wants allocations to fail if such memory is not available).  Later we can worry about how to move data structures in and out of a kind of memory.
> >
> > It might make sense to add an option for certain PETSc classes--Mat and Vec are the most obvious here--to prefer allocations in a certain kind of memory.  Or, would it make more sense to add such an option at the PetscObject level?
> >
> > I think it is possible to add the memkind support without breaking all of the interfaces used throughout PETSc for PetscMalloc(), etc.
> 
>   I don't think having this as a goal is useful at all! Just break the current interface; add an abstract "memkind" argument to all PetscMalloc() and Free() calls that indicates any additional information about the memory requested. By making it "abstract" it will always just be there and on systems without any special memory options it just doesn't do anything.
> 
> Since Malloc() is so pervasive, I think it would be useful to have a 2 level interface here. The standard Malloc() would call
> you advanced PlacedMalloc(), and anyone could call that function, but I think its just cruel to make everyone allocating
> memory give arguments they do not understand or need.

    MPI_Comm argument?  PETSc users rarely need to call PetscMalloc() themselves and if they do call it then they should know the properties of the memory they are allocating. Most users won't even notice the change. 

   Note that I'd like to add this argument independent of memkind. 

   Barry

> 
>   Matt
>  
>    Barry
> 
>    Note: it is not clear to me how this could be helpful on its own because I agree with Jed how is the user when creating the object supposed to know the optimal place to put the object?  For more complex objects it may be that different parts of the object would be stored in different types of memory etc etc.
> 
> 
> >  I recently sat with Chris Cantalupo, the main memkind developer, and walked him through PETSc's allocation routines, and we came up with the following: The imalloc() function pointer could have an implementation something like
> >
> > PetcErrorCode PetscMemkindMalloc(size_t size, const char *func, const char *file, void **result)
> >
> > {
> >
> >     struct memkind *kind;
> >
> >     int err;
> >
> >
> >     if (*result == NULL) {
> >
> >         kind = MEMKIND_DEFAULT;
> >
> >     }
> >
> >     else {
> >
> >         kind = (struct memkind *)(*result);
> >
> >     }
> >
> >
> >     err = memkind_posix_memalign(kind, result, 16, size);
> >
> >     return PosixErrToPetscErr(err);
> >
> > }
> >
> >
> > and ifree will look something like:
> >
> > PetcErrorCode PetscMemkindFree(void *ptr, int a, const char *func, const char *file)
> >
> > {
> >
> >     memkind_free(0, ptr);
> >
> >     return 0;
> >
> > }
> >
> >
> > This gives us (1) a method of passing the kind of memory without modifying the petsc allocation routine calling sequence, and (2) support a fall back code path legacy applications which will not set the pointer to NULL.  Or am I missing something?
> >
> > Thoughts?  I'd like to hash out something soon and start writing some code.
> >
> > --Richard
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener