[petsc-dev] https://www.dursi.ca/post/hpc-is-dying-and-mpi-is-killing-it.html

[Mills, Richard Tran]

OK, so this thread is two months old but I saw some things recently that reminded me of it.

To answer Barry's first question: I think that "AI" was used more than "HPC" during the presentation because the HPC community's ridiculous focus on rankings in the TOP500 list has resulted in machines that aren't truly good for much other than xGEMM operations. And if you are looking around for something to justify your xGEMM machine, well, deep neural nets fit the bill pretty well. (Yes, the fact that GPUs are really good for, well, *graphics* and this is a huge market -- way, way bigger than HPC -- is a contributing factor.)

On the health of HPC sales, I, like Bill, was thinking of what I see in earnings reports from companies like Intel and NVIDIA. Yes, a lot of this is driven by AI applications in data centers, but the same hardware gets used for what I think of as more "traditional" HPC.

As for the increasing use of MPI in machine learning, two major examples are

* Uber's Horovod framework: https://eng.uber.com/horovod/
* Microsoft's Cognitive Toolkit (CNTK) uses MPI for parallel training: https://docs.microsoft.com/en-us/cognitive-toolkit/multiple-gpus-and-machines

There are other examples, too, but Uber and Microsoft are pretty big players. I'm seeing a lot of examples of people using Horovod, in particular.

--Richard

On 3/19/19 4:11 PM, Gropp, William D wrote:
There is a sort of citation for the increasing use of MPI in distributed ML/DL - Torsten has a recent paper on demystifying ML with a graph based on published papers. Not the same, but interesting.

On the health of HPC, sales figures are available (along with attendance at SC) and these show HPC is healthy if not growing at unsustainable rates :)

Bill


On Mar 19, 2019 3:45 PM, "Smith, Barry F. via petsc-dev" <petsc-dev at mcs.anl.gov><mailto:petsc-dev at mcs.anl.gov> wrote:


> On Mar 19, 2019, at 12:27 AM, Mills, Richard Tran via petsc-dev <petsc-dev at mcs.anl.gov><mailto:petsc-dev at mcs.anl.gov> wrote:
>
> I've seen this quite some time ago. Others in this thread have already articulated many of the same criticisms I have with the material in this blog post, as well as some of the problems that I have with MPI, so I'll content myself by asking the following:
>
> If HPC is as dying as this guy says it is, then
>
> * Why did DOE just announce today that they are spending $500 million on the first (there are *more* coming?) US-based exascale computing system?

   Why was the acronym AI used more often than HPC during the presentation?

>
> * Why are companies like Intel, NVIDIA, Mellanox, etc., managing to sell so much HPC hardware?

   Citation
>
> and if it is all the fault of MPI, then
>
> * Why have a bunch of the big machine-learning shops actually been moving towards more use of MPI?

   Citation

>
> Yeah, MPI has plenty of warts. So does Fortran -- yet that hasn't killed scientific computing.
>
> --Richard
>
> On 3/17/19 1:12 PM, Smith, Barry F. via petsc-dev wrote:
>>   I stubbled on this today; I should have seen it years ago.
>>
>>   Barry
>>
>>
>



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-dev/attachments/20190516/406453d5/attachment-0001.html>