[petsc-users] Direct PETSc to use MCDRAM on KNL and other optimizations for KNL

[Zhang, Hong]

For LCRC, it is better to follow the example on their website (http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/)

#!/bin/bash

#SBATCH -J mympijobname
#SBATCH -A myaccount
#SBATCH -p knlall
#SBATCH -C knl,quad,cache
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH -t 00:15:00

export I_MPI_FABRICS=shm:tmi
srun ./ex_modify

I would not recommend you to use flat mode unless you really want to seriously optimize your code for best performance on KNL.

When configuring petsc on KNL, adding options --with-avx512-kernels --with-memalign=64 may speed up your code a bit.

Hong (Mr.)

On Feb 27, 2019, at 5:02 PM, Sajid Ali via petsc-users <petsc-users at mcs.anl.gov<mailto:petsc-users at mcs.anl.gov>> wrote:


Hi Junchao,

I’m confused with the syntax. If I submit the following as my job script, I get an error :

#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00

module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify -ts_type cn -prop_steps 25 -pc_type gamg -ts_monitor -log_view


The error is :
srun: cluster configuration lacks support for cpu binding
srun: error: Unable to create step for job 916208: More processors requested than permitted

I’m following the advice as given at slide 33 of https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf

For further info, I’m using LCRC at ANL.

Thank You,
Sajid Ali
Applied Physics
Northwestern University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20190228/3f3f62fb/attachment-0001.html>