[petsc-dev] Adding support memkind allocators in PETSc

[Richard Mills]

Hi Folks,

It's been a while, but I'd like to pick up this discussion of adding a
context to memory allocations again.

The immediate motivation I have is that I'd like to support use of the
memkind library (https://github.com/memkind/memkind), though adding a
context to PetscMallocN() (or making some other interface, say
PetscAdvMalloc() or whatever) could have much broader utility than simply
memkind support (which Jed doesn't like anyway, and I share some of his
concerns).  For the sake of having a concrete example, I'll discuss memkind
here.

Memkind's memkind_malloc() works like malloc() but takes a memkind_t
argument to specify some desired property of the memory being allocated.
For example,

 hugetlb_str = (char *)memkind_malloc(MEMKIND_HUGETLB, size);

returns a pointer to memory allocated using huge pages, and

hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size);

allocates memory from a high-bandwidth region if it's available and
elsewhere if not (specifying MEMKIND_HBW will insist on the allocation
coming from high-bandwidth memory, failing if it's not available).

It should be straightforward to add a variant of PetscMalloc() that accepts
a context: I'll call this PetscAdvMalloc(), for now, though we can come up
with a better name later.  This will allow passing on the memkind_t via
this context to the underlying memkind allocator, and we can have some
mechanism to set a default context (in the case of Memkind, this is likely
MEMKIND_DEFAULT) that gets used when plain PetscMalloc() gets called.

Of course, we'll need some way to ensure that the "advanced malloc" gets
used to allocated the critical data structures.  As a low-level way to
start, it may make sense to simply add a way to stash a context in Vec and
Mat objects.  Maybe have VecSetAdvMallocCtx(), and if that context gets
set, then PetscAdvMalloc() is used for the allocations associated with the
contents of that object.  It would probably be better to eventually have a
higher-level way to do this, e.g., support standard settings in the options
database that PETSc uses to construct the appropriate arguments to
underlying allocators that are supported, but I think just adding a way to
set this context directly is an appropriate first step.

Does this sound like a reasonable thing for me to prototype, or are others
thinking something very different?  Please let me know.  I'm getting more
access to early systems I can experiment on, and I'd really like to move
forward on trying things with high bandwidth memory (imperfect as our APIs
for using it are).

Best regards,
Richard


On Wed, Apr 29, 2015 at 11:10 PM, Richard Mills <rtm at utk.edu> wrote:

> On Wed, Apr 29, 2015 at 1:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>
>>
>>   Forget about the issue of "changing" PetscMallocN() or adding a new
>> interface instead, that is a minor syntax and annoyance issue:
>>
>>   The question is "is it worth exploring adding a context for certain
>> memory allocations that would allow us to "do" various things to the memory
>> and "indicate" properties of the memory"? I think, though I agree with Jed
>> that it could be fraught with difficulties, that is is worthwhile playing
>> around with this.
>>
>>   Barry
>>
>>
> I vote "yes".  One might want to, say
>
> * Give hints via something like madvise() on how/when the memory might be
> accessed.
> * Specify a preferred "kind" of memory (and behavior if the preferred kind
> is not available, or perhaps even specify a priority on how hard to try to
> get the preferred memory kind)
> * Specify something like a preference to interleave allocation blocks
> between different kinds of memory
>
> I'm sure we can come up with plenty of other possibilities, some of which
> might actually be useful, many of which will be useful only for very
> contrived cases, and some that are not useful today but may become useful
> as memory systems evolve.
>
> --Richard
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20150603/333cfa3b/attachment.html>