[petsc-dev] Soliciting suggestions for linear solver work under SciDAC 4 Institutes

[Patrick Sanan]

Maybe a general class of opportunities for PETSc could be wrapped up
under "good abstractions for memory space awareness". That is, all
these topics have been discussed recently:

1.  Re Jeff's excellent suggestion about OpenMP, it would be nice if
more were done to promote the alternative, MPI-based shared memory
capability. It's quite clear that this is a good way to go, but this
doesn't mean that people (who often equate shared memory parallelism
with threads) will use it, so the more that can be done to make the
internals of the library use this approach and the more that the
classes (DM in particular) can make it easy to do things "the right
way", the better.

2. As we saw from Richard's talk on KNL, that device will feature
(when used a certain way) one NUMA domain which can provide much
greater memory bandwidth. As in the discussions here before on the
topic, it's a real challenge to figure out how to to make something
like this usable via PETSc, and an even greater one to pick defaults
in a way that won't sometimes produce bewildering performance results.
Nevertheless, there's a good chance this is the kind of hardware that
people will be running PETSc on, and given that this is something
which attacks memory bandwidth bottlenecks, something that could speed
up a lot of PETSc code.

3. People will likely also be running on machines with
coprocessors/accelerators/etc for a while, and there needs to be
plumbing to deal with the fact that each device may provide an extra
memory space which needs to be managed properly in the flat MPI
environment. This is of course related to point 1, as good use of
MPI-3 shared memory seems like a reasonable way forward.

4. Related topics re MPI-4/endpoints and how PETSc really could be
used properly from an existing multi-threaded environment. Karl has
talked about the challenges of doing this the right way a lot
recently.

With all of these, introducing some clever abstractions to allow these
sorts of things to be as transparent/automatic/encapsulated as
possible might be very valuable and well-used additions to the
library.

On Sat, Jul 2, 2016 at 1:13 AM, Jeff Hammond <jeff.science at gmail.com> wrote:
> Obviously, holistic support for OpenMP is critical to the future of PETSc
> :-D
>
> On a more serious note, Matt and I have discussed the use of PETSc for
> sparse multidimensional array computations for dimensions greater than 2,
> also known as tensor computations. The associated paper describing previous
> work with dense arrays is
> http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html.  There
> was even an unsuccessful SciDAC application proposal that described how
> PETSc could be used for that domain when sparsity is important.  To start,
> all we'd need is sparse matrix x sparse matrix multiplication, which I hear
> the multigrid folks also need.  Sparse times dense is also important.
> Sparse tensor factorization would also help, but I get that there are enough
> open math questions there that it might be impractical to try to implement
> something in PETSc in the near future.
>
> Maybe I am just biased because I spend all of my time reading
> www.nextplatform.com, but I hear machine learning is becoming an important
> HPC workload.  While the most hyped efforts related to running inaccurate -
> the technical term is half-precision - dense matrix multiplication as fast
> as possible, I suspect that more elegant approaches will prevail.
> Presumably there is something that PETSc can do to enable machine learning
> algorithms.  As most of the existing approaches use silly programming models
> based on MapReduce, it can't be too hard for PETSc to do better.
>
> Jeff
>
> On Fri, Jul 1, 2016 at 2:32 PM, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>
>>
>>    The DOE SciDAC institutes have supported PETSc linear solver
>> research/code development for the past fifteen years.
>>
>>     This email is to solicit ideas for linear solver research/code
>> development work for the next round of SciDAC institutes (which will be a 4
>> year period) in PETSc. Please send me any ideas, no matter how crazy, on
>> things you feel are missing, broken, or incomplete in PETSc with regard to
>> linear solvers that we should propose to work on. In particular, issues
>> coming from particular classes of applications would be good. Generic "multi
>> physics" coupling types of things are too general (and old :-)) while  work
>> for extreme large scale is also out since that is covered under another call
>> (ECP). But particular types of optimizations etc for existing or new codes
>> could be in, just not for the very large scale.
>>
>>     Rough ideas and pointers to publications are all useful. There is an
>> extremely short fuse so the sooner the better,
>>
>>     Thanks
>>
>>       Barry
>>
>>
>>
>
>
>
> --
> Jeff Hammond
> jeff.science at gmail.com
> http://jeffhammond.github.io/