[Swift-devel] command line ssh provider...

Michael Wilde wilde at mcs.anl.gov
Fri Jan 13 18:00:01 CST 2012 

Latest update on this:

I was trying to get coasters to work with jobmanager ssh-cl:local, communicado to bridled.

Its close now. You need to:

- make sure you set the right hostname in sites.xml :)
- create valid x509 proxies on both sides
  -- I sourced /opt/osg/setup.sh and then ran grid-proxy-init manually
  -- also I *think* need to source this in your .bashrc or equiv so that
     the remote side gets the right CADIR in its env
- set GLOBUS_TCP_PORT_RANGE=50000,51000
  -- Mihael says this should get exported from client
  -- I added it to .bashrc to be sure it was set on both sides

Once I had done that, coasters booted OK.  Then I hit a suspected problem in the local provider: it was not accepting jobs, but seemed set up OK. Mihael is investigating.

Its very likely that this *will* work end to end e.g. from communicado to PADS using ssh-cl:pbs.

Another good test is to access eg surveyor, and intrepid using an OTP via ssh-cl.

- Mike



----- Original Message -----
> From: "Ben Clifford" <benc at hawaga.org.uk>
> To: "Michael Wilde" <wilde at mcs.anl.gov>
> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Jonathan Monette" <jonmon at mcs.anl.gov>
> Sent: Friday, January 13, 2012 8:26:59 AM
> Subject: Re: [Swift-devel] command line ssh provider...
> SSH_AUTH_SOCK is the variable I intended to refer to. But if that's
> working for you, then my suggestion probably isn't the problem...
> 
> On Jan 13, 2012, at 12:47 PM, Michael Wilde wrote:
> 
> > I ssh to communicado from my mac using the following command:
> >
> >    ssh -A -t login.ci.uchicago.edu ssh -A -t
> >    communicado.ci.uchicago.edu
> >
> > then I get the following ssh env vars, and the basic ssh-cl provider
> > seems to work:
> >
> > com$ env | grep -i ssh
> > SSH_CLIENT=128.135.125.155 47429 22
> > SSH_TTY=/dev/pts/0
> > SSH_AUTH_SOCK=/tmp/ssh-iGZFq22173/agent.22173
> > SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
> > CVS_RSH=cvs-ssh
> > SSH_CONNECTION=128.135.125.155 47429 128.135.125.17 22
> > com$ export | grep -i ssh
> > declare -x CVS_RSH="cvs-ssh"
> > declare -x SSH_ASKPASS="/usr/libexec/openssh/gnome-ssh-askpass"
> > declare -x SSH_AUTH_SOCK="/tmp/ssh-iGZFq22173/agent.22173"
> > declare -x SSH_CLIENT="128.135.125.155 47429 22"
> > declare -x SSH_CONNECTION="128.135.125.155 47429 128.135.125.17 22"
> > declare -x SSH_TTY="/dev/pts/0"
> > com$
> >
> > (I still have problems, unrelated I think, with getting coasters to
> > work with ssh-cl).
> >
> > - Mike
> >
> > ----- Original Message -----
> >> From: "Ben Clifford" <benc at hawaga.org.uk>
> >> To: "Jonathan Monette" <jonmon at mcs.anl.gov>
> >> Cc: "Michael Wilde" <wilde at mcs.anl.gov>, "Swift Devel"
> >> <swift-devel at ci.uchicago.edu>
> >> Sent: Friday, January 13, 2012 3:00:07 AM
> >> Subject: Re: [Swift-devel] command line ssh provider...
> >> one guess, based only on reading this thread, is that the SSH_AGENT
> >> environment variable from your login session (which tells the 'ssh'
> >> commandline program how to get back to the agent that it should
> >> use)
> >> is not getting passed all the way through swift and ssh-ci to the
> >> ssh
> >> command executed in there. I didn't look at the code, though, or
> >> try
> >> to determine the truth of this in any way.
> >>
> >> On Jan 13, 2012, at 3:24 AM, Jonathan Monette wrote:
> >>
> >>> I am getting a different problem. The provider does not seem to be
> >>> using an agent.
> >>>
> >>> Starting from my macbook I can ssh -A jonmon at login.ci.uchicago.edu
> >>> and then do ssh -A jonmon at communicado.ci.uchicago.edu and then ssh
> >>> -A jonmon at bridled.ci.uchicago.edu in the terminal and none of them
> >>> require a password.
> >>>
> >>> However if I ssh -A jonmon at login.ci.uchicago.edu and then ssh -A
> >>> jonmon at communicado.ci.uchicago.edu, then start a Swift run that
> >>> does
> >>> a simple hostname call on bridled.ci.uchicago.edu I am prompted
> >>> for
> >>> my ci password every time.
> >>>
> >>> I am more than certain that this is a configuration issue so I ask
> >>> for suggestions. My next step is to completely undo all my ssh
> >>> keys
> >>> in the authorized key files and start fresh with new keys and
> >>> passphrases that are not in my macbook keychain. I do not really
> >>> want to basically revert back to nothing regarding ssh
> >>> configuration
> >>> but this seems to be my only alternative. Any suggestions?
> >>>
> >>> On Jan 12, 2012, at 9:19 PM, Michael Wilde wrote:
> >>>
> >>>> The boostrap log shows this:
> >>>>
> >>>> com$ cat ~/coaster-bootstrap-1460623968.log
> >>>> using plain mode
> >>>> BS: http://communicado.ci.uchicago.edu:45621
> >>>> Failed to download bootstrap jar from
> >>>> http://communicado.ci.uchicago.edu:45621
> >>>> com$
> >>>>
> >>>> - Mike
> >>>>
> >>>> ----- Original Message -----
> >>>>> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> >>>>> To: "Michael Wilde" <wilde at mcs.anl.gov>
> >>>>> Cc: "Jonathan Monette" <jonmon at mcs.anl.gov>, "Swift Devel"
> >>>>> <swift-devel at ci.uchicago.edu>
> >>>>> Sent: Thursday, January 12, 2012 8:34:36 PM
> >>>>> Subject: Re: [Swift-devel] command line ssh provider...
> >>>>> Can't test it right now because UCDavis decided to firewall
> >>>>> stuff,
> >>>>> but
> >>>>> I
> >>>>> do get the bootstrap script to start and it gets to the wget
> >>>>> part.
> >>>>>
> >>>>> So the question is, do you get a bootstrap log?
> >>>>>
> >>>>> On Thu, 2012-01-12 at 13:45 -0600, Michael Wilde wrote:
> >>>>>> ssh-cl worked for me going from communicado to both login.ci
> >>>>>> and
> >>>>>> bridled.
> >>>>>>
> >>>>>> I *assumed* it used my agent because I did not get a password
> >>>>>> prompt
> >>>>>> from the swift run. And I dont get a password prompt when
> >>>>>> running
> >>>>>> the ssh command line.
> >>>>>>
> >>>>>> It failed when I tried to use coasters with either provider
> >>>>>> staging
> >>>>>> (to login.mcs) or localhost/shared workdir (to login.ci).
> >>>>>>
> >>>>>> The command line and stdout/err for the coaster/local-workdir
> >>>>>> case
> >>>>>> is below. The logs are on ci net under ~wilde/swift/lab. Config
> >>>>>> and
> >>>>>> sites file was:
> >>>>>>
> >>>>>> com$ cat cf
> >>>>>> wrapperlog.always.transfer=true
> >>>>>> sitedir.keep=true
> >>>>>> execution.retries=0
> >>>>>> lazy.errors=false
> >>>>>> status.mode=provider
> >>>>>> use.provider.staging=false
> >>>>>> provider.staging.pin.swiftfiles=false
> >>>>>>
> >>>>>> com$ cat sshcl.xml
> >>>>>> <config>
> >>>>>> <pool handle="localhost">
> >>>>>>  <execution provider="ssh-cl" url="login.ci.uchicago.edu"/>
> >>>>>>  <filesystem provider="local"/>
> >>>>>>  <workdirectory>/home/wilde/swiftwork</workdirectory>
> >>>>>> </pool>
> >>>>>> </config>
> >>>>>> com$
> >>>>>>
> >>>>>> com$ cat sshclcoast.xml
> >>>>>> <config>
> >>>>>> <pool handle="localhost">
> >>>>>>  <execution provider="coaster" url="login.ci.uchicago.edu"
> >>>>>>  jobmanager="ssh-cl:local"/>
> >>>>>>
> >>>>>>  <profile namespace="globus" key="jobsPerNode">8</profile>
> >>>>>>  <profile namespace="globus" key="slots">1</profile>
> >>>>>>  <profile namespace="globus" key="nodeGranularity">1</profile>
> >>>>>>  <profile namespace="globus" key="maxNodes">1</profile>
> >>>>>>  <profile namespace="karajan" key="jobThrottle">.01</profile>
> >>>>>>  <profile namespace="karajan"
> >>>>>>  key="initialScore">10000</profile>
> >>>>>>
> >>>>>>  <filesystem provider="local"/>
> >>>>>>  <workdirectory>/home/wilde/swiftwork</workdirectory>
> >>>>>>
> >>>>>> </pool>
> >>>>>> </config>
> >>>>>> com$
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> - Mike
> >>>>>>
> >>>>>> com$ which swift
> >>>>>> ~/swift/src/trunk/cog/modules/swift/dist/swift-svn/bin/swift
> >>>>>> com$ pwd
> >>>>>> /home/wilde/swift/lab
> >>>>>> com$ swift -tc.file tc -sites.file sshcl.xml -config cf
> >>>>>> catsn.swift
> >>>>>> -n=1
> >>>>>> Swift trunk swift-r5498 cog-r3347
> >>>>>>
> >>>>>> RunID: 20120112-1343-a7mk2zyc
> >>>>>> Progress: time: Thu, 12 Jan 2012 13:43:04 -0600
> >>>>>> Final status: Thu, 12 Jan 2012 13:43:04 -0600 Finished
> >>>>>> successfully:1
> >>>>>> com$ swift -tc.file tc -sites.file sshclcoast.xml -config cf
> >>>>>> catsn.swift -n=1
> >>>>>> Swift trunk swift-r5498 cog-r3347
> >>>>>>
> >>>>>> RunID: 20120112-1343-ql7sn3f7
> >>>>>> Progress: time: Thu, 12 Jan 2012 13:43:20 -0600
> >>>>>> Failed to transfer wrapper log for job cat-ihhm6jlk
> >>>>>> EXCEPTION Exception in cat:
> >>>>>> Arguments: [data.txt]
> >>>>>> Host: localhost
> >>>>>> Directory: catsn-20120112-1343-ql7sn3f7/jobs/i/cat-ihhm6jlk
> >>>>>> stderr.txt:
> >>>>>>
> >>>>>> stdout.txt:
> >>>>>>
> >>>>>> ----
> >>>>>>
> >>>>>> Caused by: null
> >>>>>> Caused by:
> >>>>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
> >>>>>> Could not submit job
> >>>>>> Caused by:
> >>>>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
> >>>>>> Could not start coaster service
> >>>>>> Caused by:
> >>>>>> org.globus.cog.abstraction.impl.common.task.TaskSubmissionException:
> >>>>>> Task ended before registration was received.
> >>>>>> STDOUT: Failed to download bootstrap jar from
> >>>>>> http://communicado.ci.uchicago.edu:45621
> >>>>>>
> >>>>>> STDERR: This machine accepts SSH public key and One Time
> >>>>>> Password
> >>>>>> (OTP) logins only.
> >>>>>> If you do not have a public key set up, you will be prompted
> >>>>>> for
> >>>>>> a
> >>>>>> password.
> >>>>>> This is *not* your CI password, but the One Time Password
> >>>>>> generated
> >>>>>> from your
> >>>>>> OTP token. Do not type your CI password, it will not work. If
> >>>>>> you
> >>>>>> do
> >>>>>> not
> >>>>>> have a token or public key, you will not be able to login.
> >>>>>>
> >>>>>> See http://www.ci.uchicago.edu/faq for more information.
> >>>>>>
> >>>>>> Caused by:
> >>>>>> org.globus.cog.abstraction.impl.common.execution.JobException:
> >>>>>> Job
> >>>>>> failed with an exit code of 1
> >>>>>> Execution failed:
> >>>>>> 	Job failed with an exit code of 1
> >>>>>> com$
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> From: "Jonathan Monette" <jonmon at mcs.anl.gov>
> >>>>>>> To: "Mihael Hategan" <hategan at mcs.anl.gov>
> >>>>>>> Cc: "Swift Devel" <swift-devel at ci.uchicago.edu>, "Michael
> >>>>>>> Wilde"
> >>>>>>> <wilde at mcs.anl.gov>
> >>>>>>> Sent: Thursday, January 12, 2012 1:29:10 PM
> >>>>>>> Subject: Re: [Swift-devel] command line ssh provider...
> >>>>>>> Mike,
> >>>>>>> You mentioned that you were able to use ssh command line
> >>>>>>> provider
> >>>>>>> using catsn this morning. Was it using agents? Mihael did you
> >>>>>>> test
> >>>>>>> using an agent? How do I specify for it to use an agent if
> >>>>>>> available?
> >>>>>>> I can do a simple hostname test from communicado to bridled
> >>>>>>> but
> >>>>>>> it
> >>>>>>> asks for my password instead of using the agent I have set up.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Jan 12, 2012, at 12:21 AM, Mihael Hategan wrote:
> >>>>>>>
> >>>>>>>> ... is in trunk (cog r3347). I was able to start coasters
> >>>>>>>> with
> >>>>>>>> it.
> >>>>>>>> The
> >>>>>>>> provider is called "ssh-cl". It is ssh, so ~/.ssh/config and
> >>>>>>>> agents
> >>>>>>>> will
> >>>>>>>> apply. Please test.
> >>>>>>>>
> >>>>>>>> Mihael
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> Swift-devel mailing list
> >>>>>>>> Swift-devel at ci.uchicago.edu
> >>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>>>>
> >>>>
> >>>> --
> >>>> Michael Wilde
> >>>> Computation Institute, University of Chicago
> >>>> Mathematics and Computer Science Division
> >>>> Argonne National Laboratory
> >>>>
> >>>
> >>> _______________________________________________
> >>> Swift-devel mailing list
> >>> Swift-devel at ci.uchicago.edu
> >>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >>>
> >
> > --
> > Michael Wilde
> > Computation Institute, University of Chicago
> > Mathematics and Computer Science Division
> > Argonne National Laboratory
> >

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory

More information about the Swift-devel mailing list