[petsc-dev] Adding support memkind allocators in PETSc

[Karl Rupp]

Hi,

 > (...)
>
> If we want to move data from one memory kind to another, I believe that
> we need to be able to deal with the virtual address changing.  Yes, this
> is a pain because extra bookkeeping is involved.  Maybe we don't want to
> bother with supporting something like this in PETSc.  But I don't know
> of any good way around this.  I have discussed with Chris the idea of
> adding support for asynchronously copying pages between different kinds
> of memory (maybe have a memdup() analog to strdup()) and he had some
> ideas about how this might be done efficiently.  But, again, I don't
> know of a good way to move data to a different memory kind while keeping
> the same virtual address.  If I'm misunderstanding something about what
> is possible with Linux (or other *nix), please let me know--I'd really
> like to be wrong on this.

let me bring up another spin of this thought: Currently we have related 
issues with managing memory on GPUs. The way we address this topic there 
is that we have a plain host-buffer, and a buffer allocated on the GPU. 
A separate flag keeps track of which buffer holds the most recent data 
(host, GPU, or both). What if we extend this system slightly such that 
we can also deal with HBM?

Benefits:
  - Changes to code base mainly in *GetArrayReadWrite(), returning the 
'correct' buffer.
  - Command line options as well as APIs for enabling/disabling HBM can 
be easily provided.
  - DRAM fallback always available, even if HBM exhausted.
  - Similar code and logic for dealing with HBM and GPUs.

Disadvantages:
  - Depending on the actual implementation, we may need extra memory 
(data duplication in HBM and DRAM). Since DRAM >> HBM, this may not be a 
big issue.
  - Some parts of PETSc allocate memory directly rather than using 
standard types. These will not use HBM then. May not be 
performance-critical, though...
  - Asynchronous copies between DRAM and HBM remain tricky.
  - 'Politics': The approach is not as fancy as writing heap managers 
and other low-level stuff, so it's harder to sell to e.g. program managers.

Best regards,
Karli