[Swift-devel] Mosaswift update

[Jonathan Monette]

So here is an update on the progress of the mosaswift test scripts:
I have a working CDM script and catsn(well semi working).  The issue was that the rule for CDM direct was matching "data_files" and using an absolute path with "data_files" at the end.  This caused the path to look like /path/to/data_files/data_files since CDM takes what it matched and appends it to the absolute path that was provided.  This will documented better in the CDM section of the user guide.  The other error that was brought to light was that the worker nodes do not share /tmp so they couldn't find error files when checking to see if they jobs completed.  This was resolved by setting the number of workers from 1 to 2.  Once mosastore is set up on the worker nodes this number can vary since all the workers will share /tmp/mosa(at least that is my assumption).

I say semi working because I intended the script to build up output during the stages on the swift run and the final output would show that it was stored temporarily in /tmp.  This is not happening as the final output for the file contains only the last line(a Hello from <id> line).  I believe this is due to how CDM using symlinks for the files that is matched, so a symlink is being overwritten instead of at the file it points to.  I ran into this problem in the past(but this is documented in the user guide) and just need verify that this is indeed the issue.

I have also fixed the parameter issue  so we get a better error message for the problem.  It provides a bit better error message as to what the issue was.

On Mar 13, 2012, at 1:02 PM, Emalayan Vairavanathan wrote:

> Hi Jon,
> 
> Great news and thank you for the update. I think we are getting very closer. Please let me know if I can help you.
> 
> I am cc-ing MosaStore group to inform about the progress.
> 
> Thank you
> Emalayan
> 
> From: Jonathan Monette <jonmon at mcs.anl.gov>
> To: "svemalayan at yahoo.com" <svemalayan at yahoo.com> 
> Sent: Tuesday, 13 March 2012 10:03 AM
> Subject: Mosaswift update
> 
> Emalayan,
>   I may have figured out how to use the intermediate file set up that moss needs. We will need to use something called provider staging which is a feature Swift has. Provider staging copies data directly to the compute node. I will experiment with this and see if this is the missing piece we need. 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120314/cd24e1c1/attachment.html>