[Swift-devel] Issues with Montage & Swift-CDM

[Emalayan Vairavanathan]

Hi Jon,

I already tried different combinations too. Some works and some does not work. 

The cdm rules  you provided (fs_1.data) works; But some cdm rules (fs_2.data,  fs_3.data, fs_4.data and fs_5.data) didnt work. 

To run montage with Mosa+Swift we need  to use rules in fs_3.data (input is in GPFS, intemediate results in Mosa, and output should be written again to GPFS). Or may be we can modify montage-swift script to do stage-in and stage-out data. In this case we need some rules as in fs_5.data.

I think the some stage of montage expect the input to be in a specific location. Please let me know if you need any help with debugging. 

Also please correct me if I am wrong.


Thank you
Emalayan


fs_1.data

rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*proj_dir/.* DIRECT /tmp/local1
rule .*diff_dir/.* DIRECT /tmp/local2
rule .*stat_dir/.* DIRECT /tmp/local3
rule .*rect_dir/.* DIRECT /tmp/local4



fs_2.data
rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*proj_dir/.* DIRECT /tmp/local1
rule .*diff_dir/.* DIRECT /tmp/local2
rule .*stat_dir/.* DIRECT /tmp/local3
rule .*rect_dir/.* DIRECT /tmp/local4
rule .*final/.* DIRECT /tmp/local5

fs_3.datarule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*final.* DIRECT  /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*.* DIRECT /tmp/local

fs_4.data

rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*.* DIRECT /tmp/local


fs_5.data (Here I copied all the input data to /tmp/local)

rule .*.* DIRECT /tmp/local


________________________________
 From: Jonathan Monette <jonmon at mcs.anl.gov>
To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> 
Sent: Thursday, 22 March 2012 12:47 PM
Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
 

So here is an old cdm file I used when running this on PADS
rule .*raw_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big
rule .*proj_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*diff_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*stat_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*rect_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012

The on Emalayan is using is 
rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts
rule .*final.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts
rule .*.* DIRECT /tmp/mosa

I explicitly used DIRECT on the different intermediate directories(proj_dir, diff_dir, stat_dir, and rect_dir).  In your however you match anything that is not raw_dir and final and put it into mosa. I would say try to do something like mine where you explicitly map those intermediate directories to /tmp/mosa

On Mar 22, 2012, at 2:35 PM, Emalayan Vairavanathan wrote:

Hi Justin,
>
>
>Please use ./run_local.sh to run the montage without cdm locally on the headnode.
>
>
>
>The rest of the scripts (run-workers.sh, run.sh, run-swift.sh, main.sh) are written to run experiments in our cluster and wont work in Surveyor. 
>
>Please let me know if you have questions.
>
>Thank you
>Emalayan     
>
>________________________________
> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
>Cc: "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>; matei <matei at ece.ubc.ca> 
>Sent: Thursday, 22 March 2012 11:54 AM
>Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
> 
>On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:
>
>>     But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.
>>   /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>
>What is the entry point?
>
>Are we missing
 common.sh?
>
>-- Justin M Wozniak
>
>_______________________________________________
>Swift-devel mailing list
>Swift-devel at ci.uchicago.edu
>https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120322/76a66624/attachment.html>