<html><body><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>All these combinations I tried on a local file system in our cluster with Coasters. If you need more detail about my setup please let me know.<br></span></div><div><br></div> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Emalayan Vairavanathan <svemalayan@yahoo.com><br> <b><span style="font-weight: bold;">To:</span></b> Jonathan Monette <jonmon@mcs.anl.gov> <br><b><span style="font-weight: bold;">Cc:</span></b> Justin M Wozniak <wozniak@mcs.anl.gov>; matei <matei@ece.ubc.ca>; "swift-devel@ci.uchicago.edu" <swift-devel@ci.uchicago.edu>; MosaStore
<mosastore@googlegroups.com> <br> <b><span style="font-weight: bold;">Sent:</span></b> Thursday, 22 March 2012 1:47 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Swift-devel] Issues with Montage & Swift-CDM<br> </font> </div> <br><div id="yiv871800941"><div><div style="color:#000;background-color:#fff;font-family:times new roman, new york, times, serif;font-size:12pt;"><div><span>Hi Jon,</span></div><div><br><span></span></div><div><span>I already tried different combinations too. Some works and some does not work. <br></span></div><div><span style="color:rgb(255, 0, 0);font-weight:bold;">The cdm rules you provided (fs_1.data) works; But some cdm rules </span><span style="color:rgb(255, 0, 0);font-weight:bold;">(fs_2.data, </span><span><span style="color:rgb(255, 0, 0);font-weight:bold;">fs_3.data, fs_4.data and fs_5.data) didnt work. </span></span></div><div><span><br></span></div><div><span>To run
montage with Mosa+Swift we need to use rules in fs_3.data (input is in GPFS, intemediate results in Mosa, and output should be written again to GPFS). Or may be we can modify montage-swift script to do stage-in and stage-out data. In this case we need some rules as in
fs_5.data.</span></div><div><br><span></span></div><div><span>I think the some stage of montage expect the input to be in a specific location. Please let me know if you need any help with debugging. </span></div><div><br><span></span></div><div><span>Also please correct me if I am wrong.<br></span></div><div><br><span></span></div><div><span>Thank you</span></div><div><span>Emalayan<br></span></div><div><br></div><div><span style="color:rgb(0, 0, 191);text-decoration:underline;">fs_1.data</span><br><span></span></div><div><span>rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts<br>rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*proj_dir/.* DIRECT /tmp/local1<br>rule .*diff_dir/.* DIRECT /tmp/local2<br>rule .*stat_dir/.* DIRECT /tmp/local3<br>rule .*rect_dir/.* DIRECT /tmp/local4<br></span></div><div><br></div><div><br></div><div><span style="color:rgb(0, 0,
191);text-decoration:underline;">fs_2.data</span></div><div><span style="color:rgb(0, 0, 0);">rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts<br>rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*proj_dir/.* DIRECT /tmp/local1<br>rule .*diff_dir/.* DIRECT /tmp/local2<br>rule .*stat_dir/.* DIRECT /tmp/local3<br>rule .*rect_dir/.* DIRECT /tmp/local4<br>rule .*final/.* DIRECT /tmp/local5</span><span style="color:rgb(0, 0, 191);text-decoration:underline;"></span></div><div><br><span style="color:rgb(0, 0, 191);text-decoration:underline;"></span></div><div><span style="color:rgb(0, 0, 191);text-decoration:underline;">fs_3.data</span></div>rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts<br>rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*final.* DIRECT
/home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*.* DIRECT /tmp/local<br><br><span style="color:rgb(0, 0, 191);text-decoration:underline;">fs_4.data</span><br><span style="color:rgb(0, 0, 191);text-decoration:underline;"></span><div>rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts<br>rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*.* DIRECT /tmp/local</div><div><br></div><div><br></div><div><span style="color:rgb(0, 0, 191);text-decoration:underline;">fs_5.data </span><span style="text-decoration:underline;">(Here I copied all the input data to /tmp/local)</span></div><div><span style="text-decoration:underline;"></span><br><span style="color:rgb(0, 0, 191);text-decoration:underline;"></span></div><div><span style="color:rgb(0, 0, 0);">rule .*.* DIRECT /tmp/local</span><span style="color:rgb(0, 0, 191);
text-decoration:underline;"></span></div><div><span style="color:rgb(0, 0, 191);text-decoration:underline;"><br></span></div> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Jonathan Monette <jonmon@mcs.anl.gov><br> <b><span style="font-weight:bold;">To:</span></b> Emalayan Vairavanathan <svemalayan@yahoo.com> <br><b><span style="font-weight:bold;">Cc:</span></b> Justin M Wozniak <wozniak@mcs.anl.gov>; matei <matei@ece.ubc.ca>; "swift-devel@ci.uchicago.edu" <swift-devel@ci.uchicago.edu>; MosaStore <mosastore@googlegroups.com> <br> <b><span style="font-weight:bold;">Sent:</span></b> Thursday, 22 March 2012 12:47 PM<br> <b><span style="font-weight:bold;">Subject:</span></b> Re: [Swift-devel]
Issues
with Montage & Swift-CDM<br> </font> </div> <br><div id="yiv871800941"><div>So here is an old cdm file I used when running this on PADS<div><div>rule .*raw_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big</div><div>rule .*proj_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012</div><div>rule .*diff_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012</div><div>rule .*stat_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012</div><div>rule .*rect_dir/.* DIRECT /gpfs/pads/swift/jonmon/Swift/SwiftMontage/big/run.0012</div><div><br></div><div>The on Emalayan is using is </div><div><div>rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts</div><div>rule .*final.* DIRECT /home/emalayan/App/montage-swift/SwiftMontage/scripts</div><div>rule .*.* DIRECT /tmp/mosa</div></div><div><br></div><div>I explicitly used DIRECT on the different intermediate
directories(proj_dir, diff_dir, stat_dir, and rect_dir). In your however you match anything that is not raw_dir and final and put it into mosa. I would say try to do something like mine where you explicitly map those intermediate directories to /tmp/mosa</div><div><br></div><div><div>On Mar 22, 2012, at 2:35 PM, Emalayan Vairavanathan wrote:</div><br class="yiv871800941Apple-interchange-newline"><blockquote type="cite"><div><div style="color:rgb(0, 0, 0);background-color:rgb(255, 255, 255);font-family:times, serif;font-size:12pt;"><div style="color:rgb(0, 0, 255);"><span>Hi Justin,</span></div><div style="color:rgb(0, 0, 255);"><br><span></span></div><div><span><span style="color:rgb(0, 0, 255);">Please use ./run_local.sh to run the montage without cdm locally on the headnode.</span><br></span></div><div><br><span></span></div><div><span>The rest of the scripts (run-workers.sh, run.sh, run-swift.sh, main.sh) are written to run experiments in our
cluster and wont work in Surveyor. </span></div><div><br></div>Please let me know if you have questions.<br><br>Thank you<br>Emalayan<span class="yiv871800941tab"> </span> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1">
<b><span style="font-weight:bold;">From:</span></b> Justin M Wozniak <<a rel="nofollow" ymailto="mailto:wozniak@mcs.anl.gov" target="_blank" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br> <b><span style="font-weight:bold;">To:</span></b> Emalayan Vairavanathan <<a rel="nofollow" ymailto="mailto:svemalayan@yahoo.com" target="_blank" href="mailto:svemalayan@yahoo.com">svemalayan@yahoo.com</a>> <br><b><span style="font-weight:bold;">Cc:</span></b> "<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>" <<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; MosaStore <<a rel="nofollow" ymailto="mailto:mosastore@googlegroups.com" target="_blank" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>>;
matei <<a rel="nofollow" ymailto="mailto:matei@ece.ubc.ca" target="_blank" href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>> <br> <b><span style="font-weight:bold;">Sent:</span></b> Thursday, 22 March 2012 11:54 AM<br> <b><span style="font-weight:bold;">Subject:</span></b> Re: [Swift-devel] Issues with Montage & Swift-CDM<br> </font> </div> <br>On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:<br><br>> But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.<br>> /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts<br><br>What is the entry point?<br><br>Are we missing
common.sh?<br><br>-- Justin M Wozniak<br><br> </div> </div> </div></div>_______________________________________________<br>Swift-devel mailing list<br><a rel="nofollow" ymailto="mailto:Swift-devel@ci.uchicago.edu" target="_blank" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel<br></blockquote></div><br></div></div></div><br><br> </div> </div> </div></div>
-- <br>
You received this message because you are subscribed to the Google Groups "MosaStore" group.<br>
To post to this group, send email to mosastore@googlegroups.com.<br>
To unsubscribe from this group, send email to mosastore+unsubscribe@googlegroups.com.<br>
For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.<br>
</div><br><br> </div> </div> </div></body></html>