<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
For LCRC, it is better to follow the example on their website (<a href="http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/" class="">http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/</a>)
<div class=""><span style="color: rgb(55, 61, 65); font-family: "Courier New", "DejaVu Sans Mono", monospace, sans-serif; font-size: 15.4px; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(242, 242, 242);" class=""><br class="">
</span></div>
<div class=""><span style="color: rgb(55, 61, 65); font-family: "Courier New", "DejaVu Sans Mono", monospace, sans-serif; font-size: 15.4px; orphans: 2; white-space: pre-wrap; widows: 2; background-color: rgb(242, 242, 242);" class="">#!/bin/bash</span>
<pre style="margin-top: 0px; margin-bottom: 0.64286em; font-size: 15.4px; line-height: 1.42857em; font-family: "Courier New", "DejaVu Sans Mono", monospace, sans-serif; white-space: pre-wrap; color: rgb(55, 61, 65); font-variant-ligatures: normal; orphans: 2; widows: 2; background-color: rgb(242, 242, 242);" class="">#SBATCH -J mympijobname
#SBATCH -A myaccount
#SBATCH -p knlall
#SBATCH -C knl,quad,cache
#SBATCH -N 2
#SBATCH --ntasks-per-node=64
#SBATCH -t 00:15:00
export I_MPI_FABRICS=shm:tmi
srun ./ex_modify</pre>
<div class="">I would not recommend you to use flat mode unless you really want to seriously optimize your code for best performance on KNL.</div>
<div class=""><br class="">
</div>
<div class="">When configuring petsc on KNL, adding options --with-avx512-kernels --with-memalign=64 may speed up your code a bit.</div>
<div class=""><br class="">
</div>
<div class="">Hong (Mr.)</div>
<div class=""><br class="">
</div>
<div>
<blockquote type="cite" class="">
<div class="">On Feb 27, 2019, at 5:02 PM, Sajid Ali via petsc-users <<a href="mailto:petsc-users@mcs.anl.gov" class="">petsc-users@mcs.anl.gov</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div dir="ltr" class="">
<div class="markdown-here-wrapper" style="">
<p style="margin:0px 0px 1.2em!important" class="">Hi Junchao, </p>
<p style="margin:0px 0px 1.2em!important" class="">I’m confused with the syntax. If I submit the following as my job script, I get an error :</p>
<pre style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;font-size:1em;line-height:1.2em;margin:1.2em 0px" class=""><code style="font-size:0.85em;font-family:Consolas,Inconsolata,Courier,monospace;margin:0px 0.15em;padding:0px 0.3em;white-space:pre-wrap;border:1px solid rgb(234,234,234);background-color:rgb(248,248,248);border-radius:3px;display:inline;white-space:pre;overflow:auto;border-radius:3px;border:1px solid rgb(204,204,204);padding:0.5em 0.7em;display:block!important" class="">#!/bin/bash
#SBATCH --job-name=petsc_test
#SBATCH -N 1
#SBATCH -C knl,quad,flat
#SBATCH -p apsxrmd
#SBATCH --time=1:00:00
module load intel/18.0.3-d6gtsxs
module load intel-parallel-studio/cluster.2018.3-xvnfrfz
module load numactl-2.0.12-intel-18.0.3-wh44iog
srun -n 64 -c 64 --cpu_bind=cores numactl -m 1 aps ./ex_modify -ts_type cn -prop_steps 25 -pc_type gamg -ts_monitor -log_view
</code></pre>
<p style="margin:0px 0px 1.2em!important" class="">The error is :<br class="">
srun: cluster configuration lacks support for cpu binding<br class="">
srun: error: Unable to create step for job 916208: More processors requested than permitted</p>
<p style="margin:0px 0px 1.2em!important" class="">I’m following the advice as given at slide 33 of
<a href="https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf" class="">
https://www.nersc.gov/assets/Uploads/02-using-cori-knl-nodes-20170609.pdf</a></p>
<p style="margin:0px 0px 1.2em!important" class="">For further info, I’m using LCRC at ANL.
</p>
<p style="margin:0px 0px 1.2em!important" class="">Thank You,<br class="">
Sajid Ali<br class="">
Applied Physics<br class="">
Northwestern University</p>
<div title="MDH:PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0ciI+PGRpdiBkaXI9Imx0
ciI+PGRpdj5IaSBKdW5jaGFvLCA8YnI+PGJyPjwvZGl2PkknbSBjb25mdXNlZCB3aXRoIHRoZSBz
eW50YXguIElmIEkgc3VibWl0IHRoZSBmb2xsb3dpbmcgYXMgbXkgam9iIHNjcmlwdCwgSSBnZXQg
YW4gZXJyb3IgOjxicj48L2Rpdj48ZGl2IGRpcj0ibHRyIj48YnI+PC9kaXY+PGRpdiBkaXI9Imx0
ciI+YGBgPGJyPiMhL2Jpbi9iYXNoPGJyPiNTQkFUQ0ggLS1qb2ItbmFtZT1wZXRzY190ZXN0PGJy
PiNTQkFUQ0ggLU4gMSA8YnI+I1NCQVRDSCAtQyBrbmwscXVhZCxmbGF0PGJyPiNTQkFUQ0ggLXAg
YXBzeHJtZDxicj4jU0JBVENIIC0tdGltZT0xOjAwOjAwPGJyPjxicj5tb2R1bGUgbG9hZCBpbnRl
bC8xOC4wLjMtZDZndHN4czxicj5tb2R1bGUgbG9hZCBpbnRlbC1wYXJhbGxlbC1zdHVkaW8vY2x1
c3Rlci4yMDE4LjMteHZuZnJmejxicj5tb2R1bGUgbG9hZCBudW1hY3RsLTIuMC4xMi1pbnRlbC0x
OC4wLjMtd2g0NGlvZzwvZGl2PjxkaXYgZGlyPSJsdHIiPnNydW4gLW4gNjQgLWMgNjQgLS1jcHVf
YmluZD1jb3JlcyBudW1hY3RsIC1tIDEgYXBzIC4vZXhfbW9kaWZ5IC10c190eXBlIGNuIC1wcm9w
X3N0ZXBzIDI1IC1wY190eXBlIGdhbWcgLXRzX21vbml0b3IgLWxvZ192aWV3IDxicj5gYGA8YnI+
PGJyPjwvZGl2PjxkaXY+VGhlIGVycm9yIGlzIDogPGJyPnNydW46IGNsdXN0ZXIgY29uZmlndXJh
dGlvbiBsYWNrcyBzdXBwb3J0IGZvciBjcHUgYmluZGluZzxicj5zcnVuOiBlcnJvcjogVW5hYmxl
IHRvIGNyZWF0ZSBzdGVwIGZvciBqb2IgOTE2MjA4OiBNb3JlIHByb2Nlc3NvcnMgcmVxdWVzdGVk
IHRoYW4gcGVybWl0dGVkPGJyPjwvZGl2PjxkaXYgZGlyPSJsdHIiPjxicj48L2Rpdj48ZGl2Pkkn
bSBmb2xsb3dpbmcgdGhlIGFkdmljZSBhcyBnaXZlbiBhdCBzbGlkZSAzMyBvZiBodHRwczovL3d3
dy5uZXJzYy5nb3YvYXNzZXRzL1VwbG9hZHMvMDItdXNpbmctY29yaS1rbmwtbm9kZXMtMjAxNzA2
MDkucGRmPGJyPjxicj48L2Rpdj48ZGl2PkZvciBmdXJ0aGVyIGluZm8sIEknbSB1c2luZyBMQ1JD
IGF0IEFOTC4gPGJyPjwvZGl2PjxkaXYgZGlyPSJsdHIiPjxkaXY+PGRpdj48ZGl2PjxkaXY+PGJy
PjwvZGl2PjxkaXY+VGhhbmsgWW91LCA8YnI+PC9kaXY+PGRpdj48ZGl2IGRpcj0ibHRyIiBjbGFz
cz0iZ21haWxfc2lnbmF0dXJlIj48ZGl2IGRpcj0ibHRyIj48ZGl2IHN0eWxlPSJmb250LXNpemU6
IDEyLjhweDsiPlNhamlkIEFsaTxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LXNpemU6IDEyLjhw
eDsiPkFwcGxpZWQgUGh5c2ljczxicj48L2Rpdj48ZGl2IHN0eWxlPSJmb250LXNpemU6IDEyLjhw
eDsiPk5vcnRod2VzdGVybiBVbml2ZXJzaXR5PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+
PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+PC9kaXY+" style="height:0;width:0;max-height:0;max-width:0;overflow:hidden;font-size:0em;padding:0;margin:0" class="">
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</body>
</html>