<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
On 9/25/19 11:38 AM, Mark Adams via petsc-dev wrote:<br>
<blockquote type="cite" cite="mid:CADOhEh5iXtQduYO5O5=yhCkUnT2qN3P=S7dHC6MGC10hVsr-eg@mail.gmail.com">
<div dir="ltr">[...]<br>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
> jsrun does take -n. It just has other args. I am trying to check if it<br>
> requires other args. I thought it did but let me check.<br>
<br>
<a href="https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/" rel="noreferrer" target="_blank" moz-do-not-send="true">https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/</a><br>
<br>
-n --nrs Number of resource sets<br>
<br>
</blockquote>
<div><br>
</div>
<div>-n is still supported. There are two versions of everything. One letter ones and more explanatory ones.</div>
</div>
</div>
</blockquote>
Yes, it's supported, but it's a little different than what "-n" usually does in mpiexec, where it means the number of processes. For 'jsrun', it means the number of resource sets, which is multiplied by the "tasks per resource set" specified by "-a" to get
the MPI process count. I think if we can specify that "-a 1" is part of our "mpiexec", then we should be OK with using -n as PETSc normally does.<br>
<br>
--Richard<br>
<blockquote type="cite" cite="mid:CADOhEh5iXtQduYO5O5=yhCkUnT2qN3P=S7dHC6MGC10hVsr-eg@mail.gmail.com">
<div dir="ltr">
<div class="gmail_quote">
<div><br>
</div>
<div>In fact they have a nice little tool to viz layouts and they give you the command line with this short form, eg,</div>
<div><br>
</div>
<div><a href="https://jsrunvisualizer.olcf.ornl.gov/?s1f0o01n6c4g1r14d1b21l0=" moz-do-not-send="true">https://jsrunvisualizer.olcf.ornl.gov/?s1f0o01n6c4g1r14d1b21l0=</a><br>
</div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
Beta2 Change (October 17):<br>
-n was be replaced by -nnodes<br>
<br>
So its not the same functionality as 'mpiexec -n'<br>
</blockquote>
<div><br>
</div>
<div>I am still waiting for an interactive shell to test just -n. That really should run</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
Either way - please try the above branch </blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<br>
Satish<br>
<br>
> <br>
> <br>
> ><br>
> > And then configure needs to run some binaries for some checks - here<br>
> > perhaps '-n 1' doesn't matter. [MPICH defaults to 1, OpenMPI defaults<br>
> > to ncore]. So perhaps mpiexec is required for this purpose on summit?<br>
> ><br>
> > And then there is this code to escape spaces in path - for<br>
> > windows. [but we have to make sure this is not in code-path for user<br>
> > specified --with-mpiexec="jsrun -g 1"<br>
> ><br>
> > Satish<br>
> ><br>
> > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
> ><br>
> > > No luck,<br>
> > ><br>
> > > On Wed, Sep 25, 2019 at 10:01 AM Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank" moz-do-not-send="true">balay@mcs.anl.gov</a>><br>
> > wrote:<br>
> > ><br>
> > > > Mark,<br>
> > > ><br>
> > > > Can you try the fix in branch balay/fix-mpiexec-shell-escape and see<br>
> > if it<br>
> > > > works?<br>
> > > ><br>
> > > > Satish<br>
> > > ><br>
> > > > On Wed, 25 Sep 2019, Balay, Satish via petsc-dev wrote:<br>
> > > ><br>
> > > > > Mark,<br>
> > > > ><br>
> > > > > Can you send configure.log from mark/fix-cuda-with-gamg-pintocpu<br>
> > branch?<br>
> > > > ><br>
> > > > > Satish<br>
> > > > ><br>
> > > > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
> > > > ><br>
> > > > > > I double checked that a clean build of your (master) branch has<br>
> > this<br>
> > > > error<br>
> > > > > > by my branch (mark/fix-cuda-with-gamg-pintocpu), which may include<br>
> > > > stuff<br>
> > > > > > from Barry that is not yet in master, works.<br>
> > > > > ><br>
> > > > > > On Wed, Sep 25, 2019 at 5:26 AM Karl Rupp via petsc-dev <<br>
> > > > > > <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank" moz-do-not-send="true">
petsc-dev@mcs.anl.gov</a>> wrote:<br>
> > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > On 9/25/19 11:12 AM, Mark Adams via petsc-dev wrote:<br>
> > > > > > > > I am using karlrupp/fix-cuda-streams, merged with master, and I<br>
> > > > get this<br>
> > > > > > > > error:<br>
> > > > > > > ><br>
> > > > > > > > Could not execute "['jsrun -g\\ 1 -c\\ 1 -a\\ 1<br>
> > --oversubscribe -n<br>
> > > > 1<br>
> > > > > > > > printenv']":<br>
> > > > > > > > Error, invalid argument: 1<br>
> > > > > > > ><br>
> > > > > > > > My branch mark/fix-cuda-with-gamg-pintocpu seems to work but I<br>
> > did<br>
> > > > edit<br>
> > > > > > > > the jsrun command but Karl's branch still fails. (SUMMIT was<br>
> > down<br>
> > > > today<br>
> > > > > > > > so there could have been updates).<br>
> > > > > > > ><br>
> > > > > > > > Any suggestions?<br>
> > > > > > ><br>
> > > > > > > Looks very much like a systems issue to me.<br>
> > > > > > ><br>
> > > > > > > Best regards,<br>
> > > > > > > Karli<br>
> > > > > > ><br>
> > > > > ><br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > ><br>
> ><br>
> ><br>
> <br>
<br>
</blockquote>
</div>
</div>
</blockquote>
<br>
</body>
</html>