[petsc-users] Petsc on AMD EPYC

[Jed Brown]

Should all work from the logs I sent. I'll mention a couple points here:

1. Set NPS4 in the BIOS -- it improves memory bandwidth for jobs with good memory locality, such as MPI.

2. Make sure to get a chassis that supports the fastest memory (DDR4-3200 for Zen 2).

3. Use BLIS instead of MKL or OpenBLAS. You can use a package manager or configure PETSc with --download-blis --download-f2cblaslapack.

  https://github.com/flame/blis/blob/master/docs/Performance.md#zen2-results

Blaise A Bourdin <bourdin at lsu.edu> writes:

> That would be great.
> What would be even greater would be if I could run the same test on LSU machines (Xeon 8260).
> Blaise
>
>
> On Mar 19, 2021, at 10:27 AM, Jed Brown <jed at jedbrown.org<mailto:jed at jedbrown.org>> wrote:
>
> Blaise A Bourdin <bourdin at lsu.edu<mailto:bourdin at lsu.edu>> writes:
>
> Hi,
>
> I am pricing out a new small cluster for my group and have been out of the HW loop for a while.
> Does anybody on the list have experience running petsc on recent generation AMD EPYC?
> I assume that the intel will not generate optimized code for non-intel COU. How about gcc / clang?
> My metric is not pure performance but rather performance over price.
>
> I have a 2-socket 7452, which is a great price point for our sort of work. If you're purely memory bandwidth-limited, you can get fewer cores, but that doesn't save a lot of money. You'd probably want to compare pricing with the new Zen 3 chips too. I'll forward you a log file from a multigrid solver with some analysis. Suffice it to say, it matches to significantly outperforms (depending on the operation) a 2-socket Xeon 8280.
>
> I do mostly implicit finite element codes and 90% of my walltime is KSP / SNES solves
>
> Regards,
> Blaise
>
> --
> A.K. & Shirley Barton Professor of  Mathematics
> Adjunct Professor of Mechanical Engineering
> Adjunct of the Center for Computation & Technology
> Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin
>
> --
> A.K. & Shirley Barton Professor of  Mathematics
> Adjunct Professor of Mechanical Engineering
> Adjunct of the Center for Computation & Technology
> Louisiana State University, Lockett Hall Room 344, Baton Rouge, LA 70803, USA
> Tel. +1 (225) 578 1612, Fax  +1 (225) 578 4276 Web http://www.math.lsu.edu/~bourdin