[petsc-dev] Adding support memkind allocators in PETSc

[Barry Smith]

  The beauty of git/bitbucket is one can make branches to try out anything they want even if some cranky old conservative PETSc developer thinks it is worse then consorting with the devil.

   As I said before I think that "additional argument" to advised_malloc should be a living object which one can change over time as opposed to just a "flag" type argument that only effects the malloc at malloc time. Of course the "living part" can be implemented later.

   Barry

Yes, Jed has already transformed himself into a cranky old conservative PETSc developer


> On Jun 3, 2015, at 7:33 PM, Richard Mills <rtm at utk.edu> wrote:
> 
> Hi Folks,
> 
> It's been a while, but I'd like to pick up this discussion of adding a context to memory allocations again.
> 
> The immediate motivation I have is that I'd like to support use of the memkind library (https://github.com/memkind/memkind), though adding a context to PetscMallocN() (or making some other interface, say PetscAdvMalloc() or whatever) could have much broader utility than simply memkind support (which Jed doesn't like anyway, and I share some of his concerns).  For the sake of having a concrete example, I'll discuss memkind here.
> 
> Memkind's memkind_malloc() works like malloc() but takes a memkind_t argument to specify some desired property of the memory being allocated.  For example, 
> 
>  hugetlb_str = (char *)memkind_malloc(MEMKIND_HUGETLB, size);
> 
> returns a pointer to memory allocated using huge pages, and 
> 
>  hbw_preferred_str = (char *)memkind_malloc(MEMKIND_HBW_PREFERRED, size);
> 
> allocates memory from a high-bandwidth region if it's available and elsewhere if not (specifying MEMKIND_HBW will insist on the allocation coming from high-bandwidth memory, failing if it's not available).
> 
> It should be straightforward to add a variant of PetscMalloc() that accepts a context: I'll call this PetscAdvMalloc(), for now, though we can come up with a better name later.  This will allow passing on the memkind_t via this context to the underlying memkind allocator, and we can have some mechanism to set a default context (in the case of Memkind, this is likely MEMKIND_DEFAULT) that gets used when plain PetscMalloc() gets called.
> 
> Of course, we'll need some way to ensure that the "advanced malloc" gets used to allocated the critical data structures.  As a low-level way to start, it may make sense to simply add a way to stash a context in Vec and Mat objects.  Maybe have VecSetAdvMallocCtx(), and if that context gets set, then PetscAdvMalloc() is used for the allocations associated with the contents of that object.  It would probably be better to eventually have a higher-level way to do this, e.g., support standard settings in the options database that PETSc uses to construct the appropriate arguments to underlying allocators that are supported, but I think just adding a way to set this context directly is an appropriate first step.
>   
> Does this sound like a reasonable thing for me to prototype, or are others thinking something very different?  Please let me know.  I'm getting more access to early systems I can experiment on, and I'd really like to move forward on trying things with high bandwidth memory (imperfect as our APIs for using it are).
> 
> Best regards,
> Richard
> 
> 
> On Wed, Apr 29, 2015 at 11:10 PM, Richard Mills <rtm at utk.edu> wrote:
> On Wed, Apr 29, 2015 at 1:28 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
>   Forget about the issue of "changing" PetscMallocN() or adding a new interface instead, that is a minor syntax and annoyance issue:
> 
>   The question is "is it worth exploring adding a context for certain memory allocations that would allow us to "do" various things to the memory and "indicate" properties of the memory"? I think, though I agree with Jed that it could be fraught with difficulties, that is is worthwhile playing around with this.
> 
>   Barry
> 
> 
> I vote "yes".  One might want to, say
> 
> * Give hints via something like madvise() on how/when the memory might be accessed.
> * Specify a preferred "kind" of memory (and behavior if the preferred kind is not available, or perhaps even specify a priority on how hard to try to get the preferred memory kind)
> * Specify something like a preference to interleave allocation blocks between different kinds of memory
> 
> I'm sure we can come up with plenty of other possibilities, some of which might actually be useful, many of which will be useful only for very contrived cases, and some that are not useful today but may become useful as memory systems evolve.
> 
> --Richard
>