[petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

[Jed Brown]

"Smith, Barry F. via petsc-dev" <petsc-dev at mcs.anl.gov> writes:

>> On May 16, 2019, at 6:16 PM, Mills, Richard Tran <rtmills at anl.gov> wrote:
>> 
>> OK, so this thread is two months old but I saw some things recently that reminded me of it.
>> 
>> To answer Barry's first question: I think that "AI" was used more than "HPC" during the presentation because the HPC community's ridiculous focus on rankings in the TOP500 list has resulted in machines that aren't truly good for much other than xGEMM operations. And if you are looking around for something to justify your xGEMM machine, well, deep neural nets fit the bill pretty well.
>
>    Is it actually a subset of deep neural networks? For example I can see deep neural networks that use convolutions as being good on GPUs because the amount of data that defines the network is very small and can stay near the compute unit (and has a good reuse since it is applied at each stencil point). On the other hand deep neural networks based on dense matrices are only BLAS level 2 (dense matrix vector products) and are probably totally impractical anyways so won't be super good on GPUs, right? Meanwhile anything based on sparse matrices would give low performance. 

They are usually batched so there are several to many "columns"; enough
to boost arithmetic intensity even for "fully connected" layers.