[Swift-user] Error message on Cray XE6

[Lorenzo Pesce]

First thanks a million for all your help.

Does it mean that I should hack the swift script (presumably make my own one), or there is a simpler path that I am missing (Java isn't really my thing).

Thanks again.


---- Original message ----
>Date: Sat, 14 Apr 2012 10:13:40 -0500 (CDT)
>From: Michael Wilde <wilde at mcs.anl.gov>  
>Subject: Re: [Swift-user] Error message on Cray XE6  
>To: Jonathan Monette <jonmon at mcs.anl.gov>
>Cc: Lorenzo Pesce <lpesce at uchicago.edu>, swift-user at ci.uchicago.edu
>
>stackoverflow says this should work:
>
>java -Duser.home=<new_location> <your_program>
>
>Need to get that in via the swift command.
>
>- Mike
>
>
>----- Original Message -----
>> From: "Michael Wilde" <wilde at mcs.anl.gov>
>> To: "Jonathan Monette" <jonmon at mcs.anl.gov>
>> Cc: "Lorenzo Pesce" <lpesce at uchicago.edu>, swift-user at ci.uchicago.edu
>> Sent: Saturday, April 14, 2012 10:10:00 AM
>> Subject: Re: [Swift-user] Error message on Cray XE6
>> I just tried both setting HOME=/lustre/beagle/wilde and setting
>> user.home to the same thing. Neither works. I think user.home is
>> coming from the Java property, and that doesnt seem to be influenced
>> by the HOME env var. I was about to look if Java can be asked to
>> change home. Maybe by setting a command line arg to Java.
>> 
>> - Mike
>> 
>> ----- Original Message -----
>> > From: "Jonathan Monette" <jonmon at mcs.anl.gov>
>> > To: "Michael Wilde" <wilde at mcs.anl.gov>
>> > Cc: "Lorenzo Pesce" <lpesce at uchicago.edu>,
>> > swift-user at ci.uchicago.edu
>> > Sent: Saturday, April 14, 2012 10:02:14 AM
>> > Subject: Re: [Swift-user] Error message on Cray XE6
>> > That is an easy fix I believe. I know where the code is so I will
>> > change and test.
>> >
>> > In the mean time could you try something? Try setting
>> > user.home=<someplace.on.lustre>
>> > in your config file and try again.
>> >
>> > On Apr 14, 2012, at 9:58, Michael Wilde <wilde at mcs.anl.gov> wrote:
>> >
>> > > /home is no longer mounted by the compute nodes, per the
>> > > post-maitenance summary:
>> > >
>> > > "External filesystem dependencies minimized: Compute nodes and the
>> > > scheduler should now continue to process and complete jobs without
>> > > the threat of interference of external filesystem outages.
>> > > /gpfs/pads is only available on login1 through login5; /home is on
>> > > login and mom nodes only."
>> > >
>> > > So we need to (finally) remove Swift's dependence on $HOME/.globus
>> > > and $HOME/.globus/scripts in particular.
>> > >
>> > > I suggest - since the swift command already needs to write to "."
>> > > -
>> > > that we create a scripts/ directory in "." instead of
>> > > $HOME/.globus.
>> > > And this should be used by any provider that would have previously
>> > > created files below .globus.
>> > >
>> > > I'll echo this to swift-devel and start a thread there to discuss.
>> > > Its possible there's already a property to cause scripts/ to be
>> > > created elsewhere. If not, I think we should make one. I think
>> > > grouping the scripts created by a run into the current dir, along
>> > > with the swift log, _concurrent, and (in the conventions I use in
>> > > my
>> > > run scripts) swiftwork/.
>> > >
>> > > Lorenzo, hopefully we can at least get you a workaround for this
>> > > soon.
>> > >
>> > > You *might* be able to trick swift into doing this by setting
>> > > HOME=/lustre/beagle/$USER. I already tried a symlink under .globus
>> > > and that didnt work, as /home is not even readable by the compute
>> > > nodes, which in this case need to run the coaster worker (.pl)
>> > > script.
>> > >
>> > > - Mike
>> > >
>> > >
>> > > ----- Original Message -----
>> > >> From: "Lorenzo Pesce" <lpesce at uchicago.edu>
>> > >> To: "Jonathan Monette" <jonmon at mcs.anl.gov>
>> > >> Cc: swift-user at ci.uchicago.edu
>> > >> Sent: Saturday, April 14, 2012 8:15:39 AM
>> > >> Subject: Re: [Swift-user] Error message on Cray XE6
>> > >> In principle the access to the /home filesystem should still be
>> > >> there.
>> > >>
>> > >> The only thing I did was to chance the cf file to remove some
>> > >> errors I
>> > >> had into it, so that might also be the source of the problem.
>> > >> This
>> > >> is
>> > >> what it looks like now:
>> > >> (BTW, the comments are not mine, I run swift only from lustre)
>> > >>
>> > >>
>> > >> # Whether to transfer the wrappers from the compute nodes
>> > >> # I like to launch from my home dir, but keep everything on
>> > >> # lustre
>> > >> wrapperlog.always.transfer=false
>> > >>
>> > >> #Indicates whether the working directory on the remote site
>> > >> # should be left intact even when a run completes successfully
>> > >> sitedir.keep=true
>> > >>
>> > >> #try only once
>> > >> execution.retries=1
>> > >>
>> > >> # Attempt to run as much as possible, i.g., ignore non-fatal
>> > >> errors
>> > >> lazy.errors=true
>> > >>
>> > >> # to reduce filesystem access
>> > >> status.mode=provider
>> > >>
>> > >> use.provider.staging=false
>> > >>
>> > >> provider.staging.pin.swiftfiles=false
>> > >>
>> > >> foreach.max.threads=100
>> > >>
>> > >> provenance.log=false
>> > >>
>> > >>
>> > >>
>> > >>
>> > >> On Apr 14, 2012, at 12:10 AM, Jonathan Monette wrote:
>> > >>
>> > >>> The perl script is the worker script that is submitted with PBS.
>> > >>> I
>> > >>> have not tried to run on Beagle since the maintenance period has
>> > >>> ended so I am not exactly sure why the error popped up. One
>> > >>> reason
>> > >>> could be that the home file system is no longer mounted on the
>> > >>> compute nodes. I know they spoke about that being a possibility
>> > >>> but
>> > >>> not sure they implemented that during the maintenance period. Do
>> > >>> you
>> > >>> know if the home file system is still mounted on the compute
>> > >>> nodes?
>> > >>>
>> > >>> On Apr 13, 2012, at 17:18, Lorenzo Pesce <lpesce at uchicago.edu>
>> > >>> wrote:
>> > >>>
>> > >>>> Hi --
>> > >>>> I haven't seen this one before:
>> > >>>>
>> > >>>> Can't open perl script
>> > >>>> "/home/lpesce/.globus/coasters/cscript7176272791806289394.pl":
>> > >>>> No
>> > >>>> such file or directory
>> > >>>>
>> > >>>> The config of the cray has changed, might this have anything to
>> > >>>> do
>> > >>>> with it?
>> > >>>> I have no idea what perl script is it talking about and why it
>> > >>>> is
>> > >>>> looking to home.
>> > >>>>
>> > >>>> Thanks a lot,
>> > >>>>
>> > >>>> Lorenzo
>> > >>>>
>> > >>>>
>> > >>>>
>> > >>>> _______________________________________________
>> > >>>> Swift-user mailing list
>> > >>>> Swift-user at ci.uchicago.edu
>> > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> > >>
>> > >> _______________________________________________
>> > >> Swift-user mailing list
>> > >> Swift-user at ci.uchicago.edu
>> > >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user
>> > >
>> > > --
>> > > Michael Wilde
>> > > Computation Institute, University of Chicago
>> > > Mathematics and Computer Science Division
>> > > Argonne National Laboratory
>> > >
>> 
>> --
>> Michael Wilde
>> Computation Institute, University of Chicago
>> Mathematics and Computer Science Division
>> Argonne National Laboratory
>
>-- 
>Michael Wilde
>Computation Institute, University of Chicago
>Mathematics and Computer Science Division
>Argonne National Laboratory
>