[Swift-devel] Issues with Montage & Swift-CDM

[Emalayan Vairavanathan]

All these combinations I tried on a local file system in our cluster with Coasters. If you need more detail about my setup please let me know.



________________________________
 From: Emalayan Vairavanathan <svemalayan at yahoo.com>
To: Jonathan Monette <jonmon at mcs.anl.gov> 
Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> 
Sent: Thursday, 22 March 2012 1:47 PM
Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
 

Hi Jon,

I already tried different combinations too. Some works and some does not work. 

The cdm rules  you provided (fs_1.data) works; But some cdm rules (fs_2.data,  fs_3.data, fs_4.data and fs_5.data) didnt work. 

To run montage with Mosa+Swift we need  to use rules in fs_3.data (input is in GPFS, intemediate results in Mosa, and output should be written again to GPFS). Or may be we can modify montage-swift script to do stage-in and stage-out data. In this case we need some rules as in fs_5.data.

I think the some stage of montage expect the input to be in a specific location. Please let me know if you need any help with debugging. 

Also please correct me if I am wrong.


Thank you
Emalayan


fs_1.data

rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*proj_dir/.* DIRECT /tmp/local1
rule .*diff_dir/.* DIRECT /tmp/local2
rule .*stat_dir/.* DIRECT /tmp/local3
rule .*rect_dir/.* DIRECT /tmp/local4



fs_2.data
rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*proj_dir/.* DIRECT /tmp/local1
rule .*diff_dir/.* DIRECT /tmp/local2
rule .*stat_dir/.* DIRECT /tmp/local3
rule .*rect_dir/.* DIRECT /tmp/local4
rule .*final/.* DIRECT /tmp/local5

fs_3.datarule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*final.* DIRECT 
 /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*.* DIRECT /tmp/local

fs_4.data

rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts
rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/
rule .*.* DIRECT /tmp/local


fs_5.data (Here I copied all the input data to /tmp/local)

rule .*.* DIRECT /tmp/local


________________________________
 From: Jonathan Monette <jonmon at mcs.anl.gov>
To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
Cc: Justin M Wozniak <wozniak at mcs.anl.gov>; matei <matei at ece.ubc.ca>; "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> 
Sent: Thursday, 22 March 2012 12:47 PM
Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
 

So here is an old cdm file I used when running this on PADS
rule .*raw_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big
rule .*proj_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*diff_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*stat_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012
rule .*rect_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012

The on Emalayan is using is 
rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts
rule .*final.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts
rule .*.* DIRECT /tmp/mosa

I explicitly used DIRECT on the different intermediate directories(proj_dir, diff_dir, stat_dir, and rect_dir).  In your however you match anything that is not raw_dir and final and put it into mosa. I would say try to do something like mine where you explicitly map those intermediate directories to /tmp/mosa

On Mar 22, 2012, at 2:35 PM, Emalayan Vairavanathan wrote:

Hi Justin,
>
>
>Please use ./run_local.sh to run the montage without cdm locally on the headnode.
>
>
>
>The rest of the scripts (run-workers.sh, run.sh, run-swift.sh, main.sh) are written to run experiments in our cluster and wont work in Surveyor. 
>
>Please let me know if you have questions.
>
>Thank you
>Emalayan     
>
>________________________________
> From: Justin M Wozniak <wozniak at mcs.anl.gov>
>To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
>Cc: "swift-devel at ci.uchicago.edu" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com>; matei <matei at ece.ubc.ca> 
>Sent: Thursday, 22 March 2012 11:54 AM
>Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM
> 
>On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:
>
>>     But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.
>>   /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts
>
>What is the entry point?
>
>Are we missing
 common.sh?
>
>-- Justin M Wozniak
>
>_______________________________________________
>Swift-devel mailing list
>Swift-devel at ci.uchicago.edu
>https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>



-- 
You received this message because you are subscribed to the Google Groups "MosaStore" group.
To post to this group, send email to mosastore at googlegroups.com.
To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120322/aa3f5263/attachment.html>