Alright, I've been playing with this for a few hours, but I can't manage to get any further. The sites.xml file isn't up to date, the one you want to see is sites-pads-pbs-coasters.xml. So I ran it a couple times, saving logs, etc. and noticed that in the .globus/coasters/coasters.log file, the jvm was being started with a -DGLOBUS_HOSTNAME=login.pads.ci.uchicago. So I tried setting GLOBUS_HOSTNAME to login1.pads.ci.uchicago. But even after that, the log file still showed the former. And the log shows an exception being thrown. So my hunch is to figure out how to force GLOBUS_HOSTNAME to get set. Anyone have any thoughts? Am I barking up the wrong tree?<br>
<br>Arjun<br><br><div class="gmail_quote">On Sat, Jun 5, 2010 at 9:53 AM, <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">Looking at your latest logs, in particular coaster.log in your ~/.globus/coasters dir, Swift is still unable to create a secure connection using GSI: it thinks there is not a valid proxy in /tmp/x509/:<br>
<br>
Looking at your sites.xml files, this is because you are telling Swift to run at the hostname "<a href="http://login.ci.uchicago.edu" target="_blank">login.ci.uchicago.edu</a>" - a load balancing virtual DNS host rotors between login1 and login2<br>
<br>
I suspect that the coaster service tried to start on login2 while you made the proxy on login1, or something similar. Its a good exercise for you to examine all the logs involved to confirm or disprove this theory. Look at:<br>
<br>
- the detailed swift .log file<br>
- the $HOME/.globus/coasters/coasters.log file<br>
- the $HOME/.globus/scripts PBS submit file, stdout/err, and exitcode files<br>
- your proxy files in the local /tmp dirs of the machines that grid-proxy-init was run on<br>
- ifconfig (note that pads login hosts have multiple networks)<br>
<br>
---<br>
<div class="im"><a href="http://login1.pads.ci.uchicago.edu" target="_blank">login1.pads.ci.uchicago.edu</a><br>
</div>login1$ ls -lt /tmp/x* | head<br>
-rw------- 1 arjun ci-users 2995 Jun 4 22:01 /tmp/x509up_u1857<br>
---<br>
<br>
I dont have time at the moment to trace this all back for you, but I suggest two steps:<br>
<br>
1) specify login1 everywhere you have "login" in sites.xml and auth.defaults<br>
<br>
2) look at the logs in your ~/.globus/coasters and /scripts directory, perhaps moving the old logs out to a save/ directory each time (save them for debugging till you resolve this). You'll be able to tell from host names and IP addresses<br>
<br>
You may need to set GLOBUS_HOSTNAME, but I am not sure about that (see the users guide and swift-user and devel lists for more info on that, then ask on the list if still not clear).<br>
<br>
If the problem persists after you set everything to use the specific login host login1, then be sure to send the the exact error message your are getting and the locations of all the log files, as even though the top-level error seems the same to you, the logs may indicate that the underlying error changes as you correct various aspects of the configuration and security context.<br>
<br>
- Mike<br>
<br>
<br>
<br>
login1$ grep login.pads *.xml<br>
sites.xml: <filesystem url="<a href="http://login.pads.ci.uchicago.edu" target="_blank">login.pads.ci.uchicago.edu</a>" provider="ssh"/><br>
sites.xml: <execution url="<a href="http://login.pads.ci.uchicago.edu" target="_blank">login.pads.ci.uchicago.edu</a>" provider="ssh"/><br>
testsites.xml: <execution provider="coaster" url="<a href="http://login.pads.ci.uchicago.edu" target="_blank">login.pads.ci.uchicago.edu</a>" jobmanager="ssh:pbs"/><br>
testsites.xml: <filesystem provider="ssh" url="<a href="http://login.pads.ci.uchicago.edu" target="_blank">login.pads.ci.uchicago.edu</a>"/><br>
<div><div></div><div class="h5"><br>
<br>
<br>
----- "Arjun Comar" <<a href="mailto:mandaya@rose-hulman.edu">mandaya@rose-hulman.edu</a>> wrote:<br>
<br>
> Just realized I only sent this to Mike. I'm resending it to<br>
> swift-devel.<br>
><br>
><br>
> On Fri, Jun 4, 2010 at 10:11 PM, Arjun Comar < <a href="mailto:mandaya@rose-hulman.edu">mandaya@rose-hulman.edu</a><br>
> > wrote:<br>
><br>
><br>
> Nope, no luck. Here's grid-proxy-info from both:<br>
><br>
> pads:<br>
> subject : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar<br>
> 693820/CN=53942264<br>
> issuer : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar 693820<br>
> identity : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar 693820<br>
> type : RFC 3820 compliant impersonation proxy<br>
> strength : 512 bits<br>
> path : /tmp/x509up_u1857<br>
> timeleft : 11:52:08<br>
><br>
> bridled:<br>
> subject : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar<br>
> 693820/CN=1363223477<br>
> issuer : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar 693820<br>
> identity : /DC=org/DC=doegrids/OU=People/CN=Arjun Comar 693820<br>
> type : RFC 3820 compliant impersonation proxy<br>
> strength : 512 bits<br>
> path : /tmp/x509up_u1857<br>
> timeleft : 11:57:52<br>
><br>
> Used the same passphrase to get both proxies,and set no options on<br>
> grid-proxy-init.<br>
><br>
> Arjun<br>
><br>
><br>
><br>
><br>
><br>
> On Fri, Jun 4, 2010 at 9:00 PM, <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > wrote:<br>
><br>
><br>
> When you use this configuration for running jobs from a submit host to<br>
> a PBS cluster using ssh to launch the coaster service on the PBS login<br>
> host, you need to create a GSI proxy (using grid-proxy-init) on both<br>
> the client and on the remote login host, using the same certificate.<br>
><br>
> <pool handle="coasterpads"><br>
> <execution provider="coaster" url=" <a href="http://login1.pads.ci.uchicago.edu" target="_blank">login1.pads.ci.uchicago.edu</a> "<br>
> jobmanager="ssh:pbs"/><br>
> <profile namespace="globus" key="maxtime">3000</profile><br>
> <profile namespace="globus" key="workersPerNode">8</profile><br>
> <profile namespace="globus" key="slots">1</profile><br>
> <profile namespace="globus" key="nodeGranularity">1</profile><br>
> <profile namespace="globus" key="maxNodes">1</profile><br>
> <profile namespace="globus" key="queue">fast</profile><br>
> <profile namespace="karajan" key="jobThrottle">0.5</profile><br>
> <profile namespace="karajan" key="initialScore">10000</profile><br>
> <filesystem provider="ssh" url=" <a href="http://login1.pads.ci.uchicago.edu" target="_blank">login1.pads.ci.uchicago.edu</a> "/><br>
> <workdirectory>/home/wilde/swift/lab</workdirectory><br>
> </pool><br>
><br>
> Arjun, this is, I think, what was causing your workflow to fail.<br>
><br>
> I thought, that in the past, we used to get at least a GSI (grid<br>
> security infrastructure) error in the detailed log file. But I don't<br>
> see that in this case.<br>
><br>
> Let me know if creating proxies on both sides works for you. Be sure<br>
> to create it on the right PADS login host.<br>
><br>
> David and Arjun, can you coordinate on integrating this use case into<br>
> the tutorial (and eventually the Users Guide)? I suggested we do a<br>
> series of "profiles" (with diagrams) to show the various ways of<br>
> running Swift locally and remotely, and provide accompanying site file<br>
> entries. Dennis, when you get started next week and try these cases,<br>
> we'll want to find a way to do automated tests for them.<br>
><br>
> Thanks,<br>
><br>
> Mike<br>
><br>
> --<br>
><br>
> Michael Wilde<br>
> Computation Institute, University of Chicago<br>
> Mathematics and Computer Science Division<br>
> Argonne National Laboratory<br>
><br>
><br>
><br>
><br>
> --<br>
> Arjun Comar, Rose-Hulman '12<br>
><br>
><br>
><br>
> --<br>
> Arjun Comar, Rose-Hulman '12<br>
<br>
</div></div>--<br>
<div><div></div><div class="h5">Michael Wilde<br>
Computation Institute, University of Chicago<br>
Mathematics and Computer Science Division<br>
Argonne National Laboratory<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br>Arjun Comar, Rose-Hulman '12<br>