<div dir="ltr">I did test this and sent the log (error).</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 25, 2019 at 2:58 PM Balay, Satish <<a href="mailto:balay@mcs.anl.gov">balay@mcs.anl.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">I made changes and asked to retest with the latest changes.<br>
<br>
Satish<br>
<br>
On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
<br>
> Oh, and I tested the branch and it didn't work. file was attached.<br>
> <br>
> On Wed, Sep 25, 2019 at 2:38 PM Mark Adams <<a href="mailto:mfadams@lbl.gov" target="_blank">mfadams@lbl.gov</a>> wrote:<br>
> <br>
> ><br>
> ><br>
> > On Wed, Sep 25, 2019 at 2:23 PM Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>> wrote:<br>
> ><br>
> >> On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
> >><br>
> >> > On Wed, Sep 25, 2019 at 12:44 PM Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
> >> wrote:<br>
> >> ><br>
> >> > > Can you retry with updated balay/fix-mpiexec-shell-escape branch?<br>
> >> > ><br>
> >> > ><br>
> >> > > current mpiexec interface/code in petsc is messy.<br>
> >> > ><br>
> >> > > Its primarily needed for the test suite. But then - you can't easily<br>
> >> > > run the test suite on machines like summit.<br>
> >> > ><br>
> >> > > Also - it assumes mpiexec provided supports '-n 1'. However if one<br>
> >> > > provides non-standard mpiexec such as --with-mpiexec="jsrun -g 1" -<br>
> >> > > what is the appropriate thing here?<br>
> >> > ><br>
> >> ><br>
> >> > jsrun does take -n. It just has other args. I am trying to check if it<br>
> >> > requires other args. I thought it did but let me check.<br>
> >><br>
> >><br>
> >> <a href="https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/" rel="noreferrer" target="_blank">https://www.olcf.ornl.gov/for-users/system-user-guides/summitdev-quickstart-guide/</a><br>
> >><br>
> >> -n --nrs Number of resource sets<br>
> >><br>
> >><br>
> > -n is still supported. There are two versions of everything. One letter<br>
> > ones and more explanatory ones.<br>
> ><br>
> > In fact they have a nice little tool to viz layouts and they give you the<br>
> > command line with this short form, eg,<br>
> ><br>
> > <a href="https://jsrunvisualizer.olcf.ornl.gov/?s1f0o01n6c4g1r14d1b21l0=" rel="noreferrer" target="_blank">https://jsrunvisualizer.olcf.ornl.gov/?s1f0o01n6c4g1r14d1b21l0=</a><br>
> ><br>
> ><br>
> ><br>
> >> Beta2 Change (October 17):<br>
> >> -n was be replaced by -nnodes<br>
> >><br>
> >> So its not the same functionality as 'mpiexec -n'<br>
> >><br>
> ><br>
> > I am still waiting for an interactive shell to test just -n. That really<br>
> > should run<br>
> ><br>
> ><br>
> >><br>
> >> Either way - please try the above branch<br>
> ><br>
> ><br>
> >> Satish<br>
> >><br>
> >> ><br>
> >> ><br>
> >> > ><br>
> >> > > And then configure needs to run some binaries for some checks - here<br>
> >> > > perhaps '-n 1' doesn't matter. [MPICH defaults to 1, OpenMPI defaults<br>
> >> > > to ncore]. So perhaps mpiexec is required for this purpose on summit?<br>
> >> > ><br>
> >> > > And then there is this code to escape spaces in path - for<br>
> >> > > windows. [but we have to make sure this is not in code-path for user<br>
> >> > > specified --with-mpiexec="jsrun -g 1"<br>
> >> > ><br>
> >> > > Satish<br>
> >> > ><br>
> >> > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
> >> > ><br>
> >> > > > No luck,<br>
> >> > > ><br>
> >> > > > On Wed, Sep 25, 2019 at 10:01 AM Balay, Satish <<a href="mailto:balay@mcs.anl.gov" target="_blank">balay@mcs.anl.gov</a>><br>
> >> > > wrote:<br>
> >> > > ><br>
> >> > > > > Mark,<br>
> >> > > > ><br>
> >> > > > > Can you try the fix in branch balay/fix-mpiexec-shell-escape and<br>
> >> see<br>
> >> > > if it<br>
> >> > > > > works?<br>
> >> > > > ><br>
> >> > > > > Satish<br>
> >> > > > ><br>
> >> > > > > On Wed, 25 Sep 2019, Balay, Satish via petsc-dev wrote:<br>
> >> > > > ><br>
> >> > > > > > Mark,<br>
> >> > > > > ><br>
> >> > > > > > Can you send configure.log from mark/fix-cuda-with-gamg-pintocpu<br>
> >> > > branch?<br>
> >> > > > > ><br>
> >> > > > > > Satish<br>
> >> > > > > ><br>
> >> > > > > > On Wed, 25 Sep 2019, Mark Adams via petsc-dev wrote:<br>
> >> > > > > ><br>
> >> > > > > > > I double checked that a clean build of your (master) branch<br>
> >> has<br>
> >> > > this<br>
> >> > > > > error<br>
> >> > > > > > > by my branch (mark/fix-cuda-with-gamg-pintocpu), which may<br>
> >> include<br>
> >> > > > > stuff<br>
> >> > > > > > > from Barry that is not yet in master, works.<br>
> >> > > > > > ><br>
> >> > > > > > > On Wed, Sep 25, 2019 at 5:26 AM Karl Rupp via petsc-dev <<br>
> >> > > > > > > <a href="mailto:petsc-dev@mcs.anl.gov" target="_blank">petsc-dev@mcs.anl.gov</a>> wrote:<br>
> >> > > > > > ><br>
> >> > > > > > > ><br>
> >> > > > > > > ><br>
> >> > > > > > > > On 9/25/19 11:12 AM, Mark Adams via petsc-dev wrote:<br>
> >> > > > > > > > > I am using karlrupp/fix-cuda-streams, merged with master,<br>
> >> and I<br>
> >> > > > > get this<br>
> >> > > > > > > > > error:<br>
> >> > > > > > > > ><br>
> >> > > > > > > > > Could not execute "['jsrun -g\\ 1 -c\\ 1 -a\\ 1<br>
> >> > > --oversubscribe -n<br>
> >> > > > > 1<br>
> >> > > > > > > > > printenv']":<br>
> >> > > > > > > > > Error, invalid argument: 1<br>
> >> > > > > > > > ><br>
> >> > > > > > > > > My branch mark/fix-cuda-with-gamg-pintocpu seems to work<br>
> >> but I<br>
> >> > > did<br>
> >> > > > > edit<br>
> >> > > > > > > > > the jsrun command but Karl's branch still fails. (SUMMIT<br>
> >> was<br>
> >> > > down<br>
> >> > > > > today<br>
> >> > > > > > > > > so there could have been updates).<br>
> >> > > > > > > > ><br>
> >> > > > > > > > > Any suggestions?<br>
> >> > > > > > > ><br>
> >> > > > > > > > Looks very much like a systems issue to me.<br>
> >> > > > > > > ><br>
> >> > > > > > > > Best regards,<br>
> >> > > > > > > > Karli<br>
> >> > > > > > > ><br>
> >> > > > > > ><br>
> >> > > > > ><br>
> >> > > > ><br>
> >> > > > ><br>
> >> > > ><br>
> >> > ><br>
> >> > ><br>
> >> ><br>
> >><br>
> >><br>
> <br>
<br>
</blockquote></div>