<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">I'll run with those rules. I saw the garbage collection exception in the log file. The exception may be causing no harm, but it probably shouldn't be throwing an exception at all so this may help fix it in trunk.<div><br></div><div>But it just occurred to me, which swift are you using again? I am running with the copy I grabbed from Justin's directory. Are you running that as well or with 0.93? If with 0.93 you will not see that exception because that version does not have the swift garbage collector so we should not waste time with that exception.<br><div><div>On Mar 22, 2012, at 4:40 PM, Emalayan Vairavanathan wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><div style="color:#000; background-color:#fff; font-family:times new roman, new york, times, serif;font-size:12pt"><div><span>Hi Jon,</span></div><div><br></div><div>Thank you for you suggestions. By the way c<span>an you try to run motage with the CDM rules below and see whether the problem is with </span>concurrent mapper and not because of location dependency ?<br></div><div><br><span></span></div><div>rule .*raw_dir.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts<br>rule .*header.hdr.* DIRECT /home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*final.* DIRECT
/home/emalayan/App/montage-swift-cdm/SwiftMontage/scripts/<br>rule .*.* DIRECT /tmp/local</div><div><br></div><div>Meantime I will run the montage again and see whether swift garbage-collector throws some error.</div><div><br></div><div>Please let me know if you have a better idea to debug the problem.<br></div><div><br></div><div>Thank you</div><div>Emalayan</div><div><br></div> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div style="font-family: times new roman, new york, times, serif; font-size: 12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Jonathan Monette <<a href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a>><br> <b><span style="font-weight: bold;">To:</span></b> Emalayan Vairavanathan <<a href="mailto:svemalayan@yahoo.com">svemalayan@yahoo.com</a>> <br><b><span style="font-weight: bold;">Cc:</span></b> Justin M Wozniak <<a href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>>; matei <<a href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>>;
"<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; MosaStore <<a href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>> <br> <b><span style="font-weight: bold;">Sent:</span></b> Thursday, 22 March 2012 2:20 PM<br> <b><span style="font-weight: bold;">Subject:</span></b> Re: [Swift-devel] Issues with Montage & Swift-CDM<br> </font> </div> <br><div id="yiv1816547994"><div>So I was able to run it on surveyor and it completed. I see those same SetFielfValue lines but I don't think that is an issue. That is just because I declare an array called projected_images and fill it up inside another function with files I used the regexp mapper on. However I do see the Swift garbage collector kicking in and throwing exceptions:<div><div><div><br></div><div>2012-03-22 20:58:03,070+0000 INFO FileGarbageCollector Failed to clean <a rel="nofollow">file://localhost/_concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b</a></div><div>java.lang.RuntimeException: org.globus.cog.abstraction.impl.file.FileNotFoundException: _concurrent/back_struct-d10301cf-5be0-4918-b0c2-49be758cf53a-7-array//elt-2.-field/b not found.</div><div><span class="yiv1816547994Apple-tab-span" style="white-space:pre;"> </span>at org.griphyn.vdl.mapping.AbsFile.clean(AbsFile.java:191)</div><div><span class="yiv1816547994Apple-tab-span" style="white-space:pre;"> </span>at org.griphyn.vdl.mapping.file.FileGarbageCollector.run(FileGarbageCollector.java:115)</div><div><br></div><div><br></div><div>I am not sure if that may be causing problems for when Emalayan tries to run with CDM.</div><div><br></div><div>Emalayan, it does not look like those scripts expect files to be in a certain location, at least that is not what is intended. In the main swiftscript, when you call the other
functions you pass the directory name you want the intermediate files to be stored in. Then in the SwiftMontage_Batch functions it uses those directories you passed to map input/output files. The only things that are expected to be in certain places are the raw_dir and header.hdr. Those have to be in the pwd. However I will continue debugging to make sure those assumptions I made are holding.</div><div><br></div><div>As to the different cdm setups, I do use the concurrent mapper(files that get dumped to _concurrent) where swift decides on the names. I did this for a couple files that I did not care what they were named and they were small enough that I didn't care if they were staged in/out or not. CDM may not like that. Perhaps Swift expects those _concurrent files in a certain place but you told CDM to put them someplace different. I am not sure, that is just a hypothesis. I can always change the
scripts to not use the concurrent mapper and use a better mapper for the CDM rules if that turns out to be the case.</div><div><br></div><div><div><div>On Mar 22, 2012, at 4:02 PM, Emalayan Vairavanathan wrote:</div><br class="yiv1816547994Apple-interchange-newline"><blockquote type="cite"><div><div style="color:#000;background-color:#fff;font-family:times new roman, new york, times, serif;font-size:12pt;"><div><span>Hi Justin,</span></div><div><br><span></span></div><div><span>For me the setup in </span><span><span style="color:rgb(128, 0, 0);">/home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts</span> works.<br></span></div><div><br><span></span></div><div><span><span style="color:rgb(0, 0, 0);">I tried on<span style="color:rgb(128, 0, 0);"> <span style="font-weight:bold;">login2.surveyor</span></span> with the s<span style="font-weight:bold;color:rgb(127, 0, 63);">wift binaries located in
/home/wozniak/Public/swift/bin/swift</span></span></span></div><div><br><span></span></div><div><span>May be you are using a different swift version or may be a different login machine. <br></span></div><div><br><span></span></div><div><span>Thank you</span></div><div><span>Emalayan<br style="font-weight:bold;color:rgb(127, 0,
63);"></span></div><div><br></div> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div style="font-family:times new roman, new york, times, serif;font-size:12pt;"> <div dir="ltr"> <font face="Arial" size="2"> <hr size="1"> <b><span style="font-weight:bold;">From:</span></b> Justin M Wozniak <<a rel="nofollow" ymailto="mailto:wozniak@mcs.anl.gov" target="_blank" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br> <b><span style="font-weight:bold;">To:</span></b> Jonathan Monette <<a rel="nofollow" ymailto="mailto:jonmon@mcs.anl.gov" target="_blank" href="mailto:jonmon@mcs.anl.gov">jonmon@mcs.anl.gov</a>> <br><b><span style="font-weight:bold;">Cc:</span></b> Emalayan Vairavanathan <<a rel="nofollow" ymailto="mailto:svemalayan@yahoo.com" target="_blank" href="mailto:svemalayan@yahoo.com">svemalayan@yahoo.com</a>>; matei <<a rel="nofollow" ymailto="mailto:matei@ece.ubc.ca" target="_blank" href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>>; "<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>" <<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; MosaStore <<a rel="nofollow" ymailto="mailto:mosastore@googlegroups.com" target="_blank" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>> <br> <b><span style="font-weight:bold;">Sent:</span></b> Thursday, 22 March 2012 1:49 PM<br> <b><span style="font-weight:bold;">Subject:</span></b> Re: [Swift-devel] Issues with Montage & Swift-CDM<br> </font> </div> <br><br>This run had no CDM
file.<br><br>On Thu, 22 Mar 2012, Jonathan Monette wrote:<br><br>> Was that with CDM in your run? I am going to take a look too as to why that is showing up.<br>><br>><br>> On Mar 22, 2012, at 3:42 PM, Justin M Wozniak wrote:<br>><br>>><br>>> Ok, I can get it started but I get:<br>>><br>>> 2012-03-22 20:24:06,048+0000 INFO SetFieldValue Set: projected_images[0]=null<br>>> 2012-03-22 20:24:06,049+0000 INFO SetFieldValue Set: projected_images[6]=null<br>>> 2012-03-22 20:24:06,050+0000 INFO SetFieldValue Set: projected_images[7]=null<br>>><br>>> resulting in:<br>>><br>>> File not found: /gpfs/home/wozniak/SwiftMontage/scripts/./proj_dir/null<br>>><br>>> This looks like a Swift bug. However, do you guys have an existing workaround?<br>>><br>>> Thanks<br>>><br>>> On Thu, 22 Mar 2012, Emalayan
Vairavanathan wrote:<br>>><br>>>> Hi Justin,<br>>>><br>>>> Please use ./run_local.sh to run the montage without cdm locally on the headnode.<br>>>><br>>>><br>>>> The rest of the scripts (run-workers.sh, run.sh, run-swift.sh, main.sh) are written to run experiments in our cluster and wont work in Surveyor.<br>>>> Please let me know if you have questions.<br>>>><br>>>> Thank you<br>>>> Emalayan<br>>>> ________________________________<br>>>> From: Justin M Wozniak <<a rel="nofollow" ymailto="mailto:wozniak@mcs.anl.gov" target="_blank" href="mailto:wozniak@mcs.anl.gov">wozniak@mcs.anl.gov</a>><br>>>> To: Emalayan Vairavanathan <<a rel="nofollow" ymailto="mailto:svemalayan@yahoo.com" target="_blank" href="mailto:svemalayan@yahoo.com">svemalayan@yahoo.com</a>> Cc: "<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>"
<<a rel="nofollow" ymailto="mailto:swift-devel@ci.uchicago.edu" target="_blank" href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>>; MosaStore <<a rel="nofollow" ymailto="mailto:mosastore@googlegroups.com" target="_blank" href="mailto:mosastore@googlegroups.com">mosastore@googlegroups.com</a>>; matei <<a rel="nofollow" ymailto="mailto:matei@ece.ubc.ca" target="_blank" href="mailto:matei@ece.ubc.ca">matei@ece.ubc.ca</a>> Sent: Thursday, 22 March 2012 11:54 AM<br>>>> Subject: Re: [Swift-devel] Issues with Montage & Swift-CDM<br>>>> On Wed, 21 Mar 2012, Emalayan Vairavanathan wrote:<br>>>><br>>>>> But I just setup everything on Surveyor and it works locally on the head node. You can find the setup here.<br>>>>> /home/emalayan/app/montage-swift-cdm/SwiftMontage/scripts<br>>>><br>>>> What is the entry
point?<br>>>><br>>>> Are we missing common.sh?<br>>>><br>>>> -- Justin M
Wozniak<br>>><br>>> --<br>>> Justin M Wozniak_______________________________________________<br>>> Swift-devel mailing list<br>>> <a rel="nofollow" ymailto="mailto:Swift-devel@ci.uchicago.edu" target="_blank" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>>> <a rel="nofollow" target="_blank" href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>><br>><br><br>-- <br>Justin M Wozniak<br><br><br> </div> </div> </div></div></blockquote></div><br></div></div></div></div></div><br><br> </div> </div> </div></div></blockquote></div><br></div></body></html>