[Swift-devel] Issues with Montage & Swift-CDM

Thu Mar 22 16:38:03 CDT 2012

Ok.  I may be seeing a transient bug, that sometimes produces:

rect_img:Image = org.griphyn.vdl.mapping.DataDependentException - Closed 
not derived due to errors in data dependencies

Neglecting this bug, I can get the whole thing to run w/o CDM.

With CDM rule:

rule .*raw_dir.* DIRECT 
/home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts/

mProjectBatch works, producing log lines:

CDM_POLICY: raw_dir/2mass-atlas-990214n-j1190021.fits -> DIRECT

and completing those tasks.

I still have some more tests to run...

 	Justin

On Thu, 22 Mar 2012, Jonathan Monette wrote:

> So I was able to run it on surveyor and it completed.  I see those same SetFielfValue lines but I don't think that is an issue.  That is just because I declare an array called projected_images and fill it up inside another function with files I used the regexp mapper on.  However I do see the Swift garbage collector kicking in and throwing exceptions:
>
> 2012-03-22 20:58:03,070+0000 INFO  FileGarbageCollector Failed to clean file://localhost/_concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b
> java.lang.RuntimeException: org.globus.cog.abstraction.impl.file.FileNotFoundException: _concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b not found.
> 	at org.griphyn.vdl.mapping.AbsFile.clean(AbsFile.java:191)
> 	at org.griphyn.vdl.mapping.file.FileGarbageCollector.run(FileGarbageCollector.java:115)
>
>
> I am not sure if that may be causing problems for when Emalayan tries to run with CDM.
>
> Emalayan, it does not look like those scripts expect files to be in a certain location, at least that is not what is intended.  In the main swiftscript, when you call the other functions you pass the directory name you want the intermediate files to be stored in.  Then in the SwiftMontage_Batch functions it uses those directories you passed to map input/output files.  The only things that are expected to be in certain places are the raw_dir and header.hdr.  Those have to be in the pwd.  However I will continue debugging to make sure those assumptions I made are holding.
>
> As to the different cdm setups, I do use the concurrent mapper(files that get dumped to _concurrent) where swift decides on the names.  I did this for a couple files that I did not care what they were named and they were small enough that I didn't care if they were staged in/out or not.  CDM may not like that.  Perhaps Swift expects those _concurrent files in a certain place but you told CDM to put them someplace different.  I am not sure, that is just a hypothesis.  I can always change the scripts to not use the concurrent mapper and use a better mapper  for the CDM rules if that turns out to be the case.
>
> On Mar 22, 2012, at 4:02 PM, Emalayan Vairavanathan wrote:
>
>> Hi Justin,
>>
>> For me the setup in /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts works.
>>
>> I tried on login2.surveyor with the swift binaries located in /home/wozniak/Public/swift/bin/swift
>>
>> May be you are using a different swift version or may be a different login machine.
>>
>> Thank you
>> Emalayan
>>
>> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>> To: Jonathan Monette <jonmon at mcs.anl.gov>
>> Cc: Emalayan Vairavanathan <svemalayan at yahoo.com>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>
>> Sent: Thursday, 22 March 2012 1:49 PM
>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>
>>
>> This run had no CDM file.
>>
>> On Thu, 22 Mar 2012, Jonathan Monette wrote:
>>
>>> Was that with CDM in your run?  I am going to take a look too as to why that is showing up.
>>>
>>>
>>> On Mar 22, 2012, at 3:42 PM, Justin M Wozniak wrote:
>>>
>>>>
>>>> Ok, I can get it started but I get:
>>>>
>>>> 2012-03-22 20:24:06,048+0000 INFO  SetFieldValue Set: projected_images[0]=null
>>>> 2012-03-22 20:24:06,049+0000 INFO  SetFieldValue Set: projected_images[6]=null
>>>> 2012-03-22 20:24:06,050+0000 INFO  SetFieldValue Set: projected_images[7]=null
>>>>
>>>> resulting in:
>>>>
>>>> File not found: /gpfs/home/wozniak/SwiftMontage/scripts/./proj_dir/null
>>>>
>>>> This looks like a Swift bug.  However, do you guys have an existing workaround?
>>>>
>>>>     Thanks
>>>>
>>>> On Thu, 22 Mar 2012, Emalayan Vairavanathan wrote:
>>>>
>>>>> Hi Justin,
>>>>>
>>>>> Please use ./run_local.sh to run the montage without cdm locally on the headnode.
>>>>>
>>>>>
>>>>> The rest of the scripts (run-workers.sh, run.sh, run-swift.sh, main.sh) are written to run experiments in our cluster and wont work in Surveyor.
>>>>> Please let me know if you have questions.
>>>>>
>>>>> Thank you
>>>>> Emalayan
>>>>> ________________________________
>>>>> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>>>>> To: Emalayan Vairavanathan <svemalayan at yahoo.com> Cc: "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>; matei <matei at ece.ubc.ca> Sent: Thursday, 22 March 2012 11:54 AM
>>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
>>>>> On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:
>>>>>
>>>>>>    But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.
>>>>>>  /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>>>>>
>>>>> What is the entry point?
>>>>>
>>>>> Are we missing common.sh?
>>>>>
>>>>> -- Justin M Wozniak
>>>>
>>>> --
>>>> Justin M Wozniak_______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>
>>>
>>
>> --
>> Justin M Wozniak
>>
>>
>
>

-- 
Justin M Wozniak