[Swift-user] MODIS demo

Wed May 9 19:13:19 CDT 2012

Thank you, David and Jonathan!

It's looking good now: I've run ./demo.local urban -4 and ./demo.local
urban -34 successfully.  Thank you for your help!

--Heather

On Wed, May 9, 2012 at 7:00 PM, Heather Stoller
<heather.stoller at gmail.com>wrote:

> Ah yes, imagemagick.  No I do not.  I shall do that.  Good idea.  Thank
> you, Jonathan.
>
> I predict that half of the jobs will work once I do that.  I seem to
> remember something of the sort from about a month ago.
>
> Thank you!
>
> Heather
>
>
> On Wed, May 9, 2012 at 6:58 PM, Jonathan Monette <jonmon at mcs.anl.gov>wrote:
>
>> Do you have imagemagick installed on your personal computer and in you
>> PATH?
>>
>>
>> On May 9, 2012, at 18:56, Heather Stoller <heather.stoller at gmail.com>
>> wrote:
>>
>> Hi David,
>>
>> Thank you - this is on my local machine.  An ls from home gives:
>>
>> hstoller at ubuntu:~$ ls
>> Desktop  Documents  Downloads  modis  Music  Pictures  Public
>> swift-0.93  swiftwork  Templates  Videos
>>
>> --Heather
>>
>> On Wed, May 9, 2012 at 4:54 PM, David Kelly <davidk at ci.uchicago.edu>wrote:
>>
>>> Thanks - which machine are you testing this on?
>>>
>>> ----- Original Message -----
>>> > From: "Heather Stoller" <heather.stoller at gmail.com>
>>> > To: "David Kelly" <davidk at ci.uchicago.edu>
>>> > Cc: "Jonathan Monette" <jonmon at mcs.anl.gov>,
>>> swift-user at ci.uchicago.edu
>>> > Sent: Wednesday, May 9, 2012 6:46:24 PM
>>> > Subject: Re: [Swift-user] MODIS demo
>>> > Hi David,
>>> >
>>> > Here are the contents of my tc.local file. Jonathan, I have duly
>>> > checked that lazy.errors = false. Thank you both!
>>> >
>>> > # site transformation path obsolete fields for compatibility
>>> >
>>> > localhost echo /bin/echo null null null
>>> > localhost cat /bin/cat null null null
>>> > localhost ls /bin/ls null null null
>>> > localhost grep /bin/grep null null null
>>> > localhost sort /bin/sort null null null
>>> > localhost paste /bin/paste null null null
>>> > localhost pwd /bin/pwd null null null
>>> >
>>> > # For cluster usage
>>> >
>>> > #pbs getlanduse /home/hstoller/swift/demo/modis/bin/getlanduse.sh null
>>> > null null
>>> > #pbs analyzelanduse
>>> > /home/hstoller/swift/demo/modis/bin/analyzelanduse.sh null null null
>>> > #pbs colormodis /home/hstoller/swift/demo/modis/bin/colormodis.sh null
>>> > null null
>>> > #pbs assemble /home/hstoller/swift/demo/modis/bin/assemble.sh null
>>> > null null
>>> >
>>> > # For localhost testing
>>> >
>>> > localhost getlanduse /home/hstoller/modis/bin/getlanduse.sh null null
>>> > null
>>> > localhost analyzelanduse /home/hstoller/modis/bin/analyzelanduse2.sh
>>> > null null null
>>> > localhost colormodis /home/hstoller/modis/bin/colormodis.sh null null
>>> > null
>>> > localhost assemble /home/hstoller/modis/bin/assemble2.sh null null
>>> > null
>>> > localhost markmap /home/hstoller/modis/bin/markmap.sh null null null
>>> >
>>> >
>>> >
>>> > On Wed, May 9, 2012 at 3:56 PM, David Kelly < davidk at ci.uchicago.edu >
>>> > wrote:
>>> >
>>> >
>>> > Heather,
>>> >
>>> > Can you please paste the contents of your tc.local file?
>>> >
>>> > Thanks,
>>> >
>>> > David
>>> >
>>> > ----- Original Message -----
>>> > > From: "Heather Stoller" < heather.stoller at gmail.com >
>>> > > To: swift-user at ci.uchicago.edu
>>> >
>>> >
>>> > > Cc: "David Kelly" < davidk at ci.uchicago.edu >, "Jonathan Monette" <
>>> > > jonmon at mcs.anl.gov >
>>> > > Sent: Wednesday, May 9, 2012 5:30:40 PM
>>> > > Subject: Re: [Swift-user] MODIS demo
>>> > > Hello Swift Group, coming back to MODIS:
>>> > >
>>> > > I followed Jonathan's advice to set retries to 0 and looked at the
>>> > > wrapper section in the files in the *.d directory as per David's
>>> > > advice and now get an error right away, as follows:
>>> > > hstoller at ubuntu:~/modis$ ./demo.local urban 4
>>> > > runid=modis-2012.0509.1527-urban-4-10
>>> > > Swift 0.93 swift-r5483 cog-r3339
>>> > >
>>> > > RunID: 20120509-1527-bsxjpz4g
>>> > > (input): found 0 files
>>> > > Progress: time: Wed, 09 May 2012 15:27:53 -0700
>>> > > Progress: time: Wed, 09 May 2012 15:27:54 -0700 Submitting:7
>>> > > Submitted:1
>>> > > Progress: time: Wed, 09 May 2012 15:27:55 -0700 Active:7 Checking
>>> > > status:1
>>> > > Progress: time: Wed, 09 May 2012 15:27:56 -0700 Stage in:1 Checking
>>> > > status:3 Stage out:1 Finished successfully:4
>>> > > Execution failed:
>>> > > Progress: time: Wed, 09 May 2012 15:27:57 -0700 Checking status:1
>>> > > Failed:4 Finished successfully:4
>>> > > File not found:
>>> > >
>>> /home/hstoller/swiftwork/modis-20120509-1527-bsxjpz4g/shared/landuse/h15v05.color.png
>>> > > Progress: time: Wed, 09 May 2012 15:28:00 -0700 Initializing:1
>>> > > Failed:4 Finished successfully:5
>>> > >
>>> > > I think my question is, where should that shared/landuse directory
>>> > > be
>>> > > getting the missing file?
>>> > >
>>> > > Thank you for your help!
>>> > >
>>> > > Sincerely,
>>> > > Heather
>>> > >
>>> > >
>>> > >
>>> > >
>>> > > On Fri, Apr 13, 2012 at 10:17 PM, Jonathan Monette <
>>> > > jonmon at mcs.anl.gov > wrote:
>>> > >
>>> > >
>>> > > I echo David's suggestion but would like to add another. It looks
>>> > > like
>>> > > soft error handling is being used which may not be the best approach
>>> > > when exploring Swift. In the config file you should set the retry
>>> > > count to 0 and set lazy.errors=false. This will cause Swift to fail
>>> > > as
>>> > > soon as the first error is encountered and will provide an error
>>> > > message. This is useful for when you are exploring Swift behavior.
>>> > >
>>> > >
>>> > >
>>> > > On Apr 13, 2012, at 23:17, David Kelly < davidk at ci.uchicago.edu >
>>> > > wrote:
>>> > >
>>> > > > Heather,
>>> > > >
>>> > > > You might want to check the path names in the tc.local file. Since
>>> > > > about half the tasks fail, I'm guessing either colormodis or
>>> > > > getlanduse is pointing to the wrong place. You can verify this by
>>> > > > looking at the directory called modis-2012<timestamp>.d. In there
>>> > > > is
>>> > > > a list of files that end with -info. Look at the "Wrapper" section
>>> > > > of these files and you should find more information about what is
>>> > > > causing the failures.
>>> > > >
>>> > > > David
>>> > > >
>>> > > > ----- Original Message -----
>>> > > >> From: "Heather Stoller" < heather.stoller at gmail.com >
>>> > > >> To: swift-user at ci.uchicago.edu
>>> > > >> Sent: Thursday, April 5, 2012 9:37:33 AM
>>> > > >> Subject: [Swift-user] MODIS demo
>>> > > >> Hello,
>>> > > >>
>>> > > >> I'm a UC student working with Mike Wilde doing some Swift stuff -
>>> > > >> at
>>> > > >> present, I'm trying to run the demo to see what can be seen. I
>>> > > >> get:
>>> > > >>
>>> > > >> ^Cheather at ubuntu:~/modis$ ./demo.local urban 10
>>> > > >> runid=modis-2012.0405.0704-urban-10-10
>>> > > >> Swift 0.93 swift-r5483 cog-r3339
>>> > > >>
>>> > > >> RunID: 20120405-0704-drh1g0ob
>>> > > >> (input): found 0 files
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:48 -0700
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:49 -0700 Stage in:19
>>> > > >> Submitting:1
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:50 -0700 Stage in:13
>>> > > >> Submitting:1 Active:6
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:53 -0700 Stage in:10
>>> > > >> Submitting:2 Submitted:2 Active:6
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:54 -0700 Stage in:6
>>> > > >> Submitting:1 Submitted:2 Active:9 Checking status:2
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:55 -0700 Stage in:2
>>> > > >> Submitting:2 Submitted:1 Active:9 Checking status:3 Stage out:3
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:57 -0700 Submitting:1
>>> > > >> Submitted:2 Active:9 Checking status:1 Stage out:7
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:58 -0700 Active:3 Checking
>>> > > >> status:2 Stage out:7 Finished successfully:7 Failed but can
>>> > > >> retry:1
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:04:59 -0700 Stage in:2
>>> > > >> Submitting:2 Submitted:1 Active:3 Stage out:1 Finished
>>> > > >> successfully:10
>>> > > >> Failed but can retry:2
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:00 -0700 Active:10 Stage
>>> > > >> out:1
>>> > > >> Finished successfully:10
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:01 -0700 Active:9 Checking
>>> > > >> status:2 Finished successfully:11
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:02 -0700 Submitting:1
>>> > > >> Active:3
>>> > > >> Checking status:3 Stage out:2 Finished successfully:11 Failed but
>>> > > >> can
>>> > > >> retry:2
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:03 -0700 Stage in:2
>>> > > >> Active:6
>>> > > >> Stage out:1 Finished successfully:11 Failed but can retry:2
>>> > > >> Execution failed:
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:04 -0700 Active:7 Checking
>>> > > >> status:2 Stage out:1 Failed:1 Finished successfully:11
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:05 -0700 Active:3 Checking
>>> > > >> status:1 Stage out:3 Failed:4 Finished successfully:11
>>> > > >> Progress: time: Thu, 05 Apr 2012 07:05:06 -0700 Failed:11
>>> > > >> Finished
>>> > > >> successfully:11
>>> > > >>
>>> > > >> lots of "failed but can retry". Does this look right?
>>> > > >>
>>> > > >> --
>>>
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20120509/314a56de/attachment.html>