<div dir="ltr">Efficient and scalable MatGetSubmatrix() and assemble submatrices into a matrix -- used for multiphysic simulation, e.g., wash project we are doing now.<div><br></div><div>Hong</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Jul 2, 2016 at 2:46 AM, Patrick Sanan <span dir="ltr"><<a href="mailto:patrick.sanan@gmail.com" target="_blank">patrick.sanan@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Maybe a general class of opportunities for PETSc could be wrapped up<br>
under "good abstractions for memory space awareness". That is, all<br>
these topics have been discussed recently:<br>
<br>
1. Re Jeff's excellent suggestion about OpenMP, it would be nice if<br>
more were done to promote the alternative, MPI-based shared memory<br>
capability. It's quite clear that this is a good way to go, but this<br>
doesn't mean that people (who often equate shared memory parallelism<br>
with threads) will use it, so the more that can be done to make the<br>
internals of the library use this approach and the more that the<br>
classes (DM in particular) can make it easy to do things "the right<br>
way", the better.<br>
<br>
2. As we saw from Richard's talk on KNL, that device will feature<br>
(when used a certain way) one NUMA domain which can provide much<br>
greater memory bandwidth. As in the discussions here before on the<br>
topic, it's a real challenge to figure out how to to make something<br>
like this usable via PETSc, and an even greater one to pick defaults<br>
in a way that won't sometimes produce bewildering performance results.<br>
Nevertheless, there's a good chance this is the kind of hardware that<br>
people will be running PETSc on, and given that this is something<br>
which attacks memory bandwidth bottlenecks, something that could speed<br>
up a lot of PETSc code.<br>
<br>
3. People will likely also be running on machines with<br>
coprocessors/accelerators/etc for a while, and there needs to be<br>
plumbing to deal with the fact that each device may provide an extra<br>
memory space which needs to be managed properly in the flat MPI<br>
environment. This is of course related to point 1, as good use of<br>
MPI-3 shared memory seems like a reasonable way forward.<br>
<br>
4. Related topics re MPI-4/endpoints and how PETSc really could be<br>
used properly from an existing multi-threaded environment. Karl has<br>
talked about the challenges of doing this the right way a lot<br>
recently.<br>
<br>
With all of these, introducing some clever abstractions to allow these<br>
sorts of things to be as transparent/automatic/encapsulated as<br>
possible might be very valuable and well-used additions to the<br>
library.<br>
<div class="HOEnZb"><div class="h5"><br>
On Sat, Jul 2, 2016 at 1:13 AM, Jeff Hammond <<a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a>> wrote:<br>
> Obviously, holistic support for OpenMP is critical to the future of PETSc<br>
> :-D<br>
><br>
> On a more serious note, Matt and I have discussed the use of PETSc for<br>
> sparse multidimensional array computations for dimensions greater than 2,<br>
> also known as tensor computations. The associated paper describing previous<br>
> work with dense arrays is<br>
> <a href="http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html" rel="noreferrer" target="_blank">http://www.eecs.berkeley.edu/Pubs/TechRpts/2012/EECS-2012-210.html</a>. There<br>
> was even an unsuccessful SciDAC application proposal that described how<br>
> PETSc could be used for that domain when sparsity is important. To start,<br>
> all we'd need is sparse matrix x sparse matrix multiplication, which I hear<br>
> the multigrid folks also need. Sparse times dense is also important.<br>
> Sparse tensor factorization would also help, but I get that there are enough<br>
> open math questions there that it might be impractical to try to implement<br>
> something in PETSc in the near future.<br>
><br>
> Maybe I am just biased because I spend all of my time reading<br>
> <a href="http://www.nextplatform.com" rel="noreferrer" target="_blank">www.nextplatform.com</a>, but I hear machine learning is becoming an important<br>
> HPC workload. While the most hyped efforts related to running inaccurate -<br>
> the technical term is half-precision - dense matrix multiplication as fast<br>
> as possible, I suspect that more elegant approaches will prevail.<br>
> Presumably there is something that PETSc can do to enable machine learning<br>
> algorithms. As most of the existing approaches use silly programming models<br>
> based on MapReduce, it can't be too hard for PETSc to do better.<br>
><br>
> Jeff<br>
><br>
> On Fri, Jul 1, 2016 at 2:32 PM, Barry Smith <<a href="mailto:bsmith@mcs.anl.gov">bsmith@mcs.anl.gov</a>> wrote:<br>
>><br>
>><br>
>> The DOE SciDAC institutes have supported PETSc linear solver<br>
>> research/code development for the past fifteen years.<br>
>><br>
>> This email is to solicit ideas for linear solver research/code<br>
>> development work for the next round of SciDAC institutes (which will be a 4<br>
>> year period) in PETSc. Please send me any ideas, no matter how crazy, on<br>
>> things you feel are missing, broken, or incomplete in PETSc with regard to<br>
>> linear solvers that we should propose to work on. In particular, issues<br>
>> coming from particular classes of applications would be good. Generic "multi<br>
>> physics" coupling types of things are too general (and old :-)) while work<br>
>> for extreme large scale is also out since that is covered under another call<br>
>> (ECP). But particular types of optimizations etc for existing or new codes<br>
>> could be in, just not for the very large scale.<br>
>><br>
>> Rough ideas and pointers to publications are all useful. There is an<br>
>> extremely short fuse so the sooner the better,<br>
>><br>
>> Thanks<br>
>><br>
>> Barry<br>
>><br>
>><br>
>><br>
><br>
><br>
><br>
> --<br>
> Jeff Hammond<br>
> <a href="mailto:jeff.science@gmail.com">jeff.science@gmail.com</a><br>
> <a href="http://jeffhammond.github.io/" rel="noreferrer" target="_blank">http://jeffhammond.github.io/</a><br>
</div></div></blockquote></div><br></div>