[petsc-dev] Using multiple mallocs with PETSc

[Richard Mills]

On Fri, Mar 10, 2017 at 11:52 AM, Barry Smith <bsmith at mcs.anl.gov> wrote:

>
> > On Mar 10, 2017, at 12:04 AM, Richard Mills <richardtmills at gmail.com>
> wrote:
> [...]
> > Personally, I think that memkind is something we should support it in
> PETSc.
>
>    Yes, the question is how do we "support it"? Do we have a PETSc malloc
> API and then it can direct the calls down to "classic malloc(), memkind,
> .....? At this time I think saying the PETSc malloc API is "just" the
> memkind API would be a mistake.  If we have a PETSc malloc API* what is
> crucial to have in the PETSc malloc API? And what should the API look like.
> Three things that could/should possibly be exposed are: "kind of memory",
> alignment (if doing SIMD this might/is important), page size (these three
> things are exposed by memkind/hbm_xxx().
>
>
> * PETSc currently does have a Petsc malloc API but it has always been kept
> exceedingly limited, so the question is more if we extend it and how we
> extend it.
>

I think there are two questions here: What is the long-term solution (hard
to answer) and what is the short-term fix we can implement right now so
that Mr. Hong can make his adjoint code work in a slightly less kludgy
way.  Shortest term, I think Hong can do basically what he has been doing,
but he should have two "malloc" commands (and "high-bandwidth" and a
"normal DRAM" version) that he sets, both of which use the memkind
interface.  He should just use one "free" command, which is uses the
memkind interface but does not specify the "kind" of memory -- this will
work to free anything that memkind has allocated.  This approach requires
no changes to the existing PETSc API.

Less short term, I like Barry's idea of allocating a bit of extra memory to
track things like what free() should be used.  If we want to avoid tracking
as much global state but also detect memory corruption, we could store a
pointer to the free() routine and then write some extra stuff in there (we
will have a whole cache line) to help detect corruption (if error checking
is turned on).  If we don't care about global state, then whatever.

I also like the idea of a stack for the malloc()s to use.  I actually like
this both for the "advise" version and just as an alternative to having to
do a PetscMallocClear() followed by a new PetscMallocSet() every time one
wants to switch from something like a "high-bandwidth" allocator back to
some default.

--Richard


>
>
> >  That doesn't mean it's the ideal solution, but it has utility now.  We
> can certainly support something else when something better arrives.
>
>   Disappointing on my Mac.
>
> WARNING: Could not locate OpenMP support. This warning can also occur if
> compiler already supports OpenMP
> checking omp.h usability... no
> checking omp.h presence... no
> checking for omp.h... no
> omp.h is required for this library
>
>    Now I have read through all the old email from Jed to understand why he
> hates memkind so much.
>
>   Barry
>
> >
> > --Richard
> >
> >
> > Jeff
> >
> > --Richard
> >
> >
> > Jeff
> >
> > Perhaps I should try memkind calls since they may become much better.
> >
> > Hong (Mr.)
> >
> > --
> > Jeff Hammond
> > jeff.science at gmail.com
> > http://jeffhammond.github.io/
> >
> >
> >
> >
> > --
> > Jeff Hammond
> > jeff.science at gmail.com
> > http://jeffhammond.github.io/
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20170313/7bda4256/attachment.html>