[petsc-dev] Adding support memkind allocators in PETSc

[Barry Smith]

  Forget about the issue of "changing" PetscMallocN() or adding a new interface instead, that is a minor syntax and annoyance issue:  

  The question is "is it worth exploring adding a context for certain memory allocations that would allow us to "do" various things to the memory and "indicate" properties of the memory"? I think, though I agree with Jed that it could be fraught with difficulties, that is is worthwhile playing around with this.

  Barry


> On Apr 29, 2015, at 3:17 PM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Thu, Apr 30, 2015 at 6:13 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:
> 
> > On Apr 29, 2015, at 12:29 AM, Richard Mills <rtm at utk.edu> wrote:
> >
> > I think this is maybe getting away from the heart of the discussion, but anyway: Barry, you are talking about being able to mark certain allocated regions as "optional" and then have those allocations disappear, right?  I could see, say, having such an "optional" allocation that is backed by a file on some sort of "fast" storage (some super-fancy SSD or NVRAM) and whose pages can be evicted if the memory is needed.  I don't know that I like a pointer to that just being nullified if the eviction occurs, though.  For a case like this, I'd like to do something like have a Vec and only have this happen to the array of values if no "get" is outstanding.  This puts us back with an array to the numerical values that could get swapped around.
> 
>    There are many possibilities from "just release this memory" to "if you really need to you can move this to slower memory".
> 
>    For example if after a DMRestoreLocalVector() the array memory could be marked as "release if you need the space" then on the next DMGetLocalVector() it would check if the memory had been released and allocated it again. If it had not been released then it would just use it.
> 
>    Clearly you can't just mark memory that is being passed around randomly in user code as "release if need the space" but you can for carefully controlled memory.
> 
> I still see no convincing rationale for changing the PetscMalloc interface. If, as you say, few people use it, then there
> is no reason to change it. We can just change our internal interface, and leave that top level interface alone. Moreover,
> since none of the interface changes are very specific I think it needs time to be shaken out. If at the end, things get
> faster and more understandable, we can replace PetscMalloc.
> 
>    Matt
>  
> 
>   Barry
> 
> >
> > On Tue, Apr 28, 2015 at 8:44 PM, Jed Brown <jed at jedbrown.org> wrote:
> > Barry Smith <bsmith at mcs.anl.gov> writes:
> > >   The special malloc would need to save the locations at which it set
> > >   the addresses and then switch the address to NULL. Then the code
> > >   that used those locations would have to know that they that they may
> > >   be set to NULL and hence check them before use.
> >
> > And then PetscMalloc(...,&tmp); foo->data = tmp; creates SEGV at some
> > unpredictable time.  Awesome!
> >
> > >   I am not saying this particular thing would be practical or not,
> > >   just that if we had a concept of a malloc context for each malloc
> > >   there are many games we could try that we couldn't try otherwise and
> > >   this is just one of them.
> >
> > I'm not convinced, except in the case of mixing in madvise hints.
> >
> 
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener