[Swift-devel] Mosaswift update

[Jonathan Monette]

I do not think it will matter since all the intermediate files will be mapped onto mosa but you can change the paths in those generated files to point to whatever you want.  Then you can experiment with having the starting and final files in mosa if you want. 

On Mar 15, 2012, at 3:30, Emalayan Vairavanathan <svemalayan at yahoo.com> wrote:

> Hi Jon and All,
> 
> MosaStore+Swift integration is done successfully. This is really a great news.
> 
> I successfully ran the setup provided by Jon with MosaStore+Swift on Surveyor (on 1, 3 , 5 and 64 nodes with different number of  jobs varying from 1 to 1024).
> 
> Jon and Mike thank you again for your time in helping me to get this far.
> 
> A quick question: The setup creates the swift working directory in GPFS but stores the intermediate files in MosaStore. Where should we actually have the swift-working directory (in GPFS / Mosa)? and Does this matter at all in case if I am going to use Mosa as an intermediate storage? 
> 
> (As a next step I will be scripting our pipeline benchmark on swift and run at large scale to get some initial performance numbers. Also I will be parallely working to get Montage and ModFTDock running on BG/P with swift-CDM and MosaStore.)
> 
> Regards
> Emalayan
> 
> From: Jonathan Monette <jonmon at mcs.anl.gov>
> To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
> Cc: "mosastore at googlegroups.com" <mosastore at googlegroups.com>; Justin M Wozniak <wozniak at mcs.anl.gov>; "swift-devel at ci.uchicago.edu Devel" <swift-devel at ci.uchicago.edu> 
> Sent: Wednesday, 14 March 2012 2:29 PM
> Subject: Re: [Swift-devel] Mosaswift update
> 
> The working example is located in /home/jonmon/Workspace/Swift/mosaswift-test
> 
> There is a README there that provides a little bit of information.
> 
> Quick version to run it:
> 1) start your workers
> 2) ./setup.sh
> 3) ./run.sh
> 
> The directory final will have 5 files if it completed without error.  The content of the files will look like:
> Hello world
> Hello from <number>
> 
> On Mar 14, 2012, at 4:03 PM, Emalayan Vairavanathan wrote:
> 
>> Hi Jon,
>> 
>> Great, Thank you very much. Could you please send me the setup ? 
>> 
>> Thank you
>> Emalayan
>> 
>> From: Jonathan Monette <jonmon at mcs.anl.gov>
>> To: Justin M Wozniak <wozniak at mcs.anl.gov> 
>> Cc: "swift-devel at ci.uchicago.edu Devel" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> 
>> Sent: Wednesday, 14 March 2012 2:00 PM
>> Subject: Re: [Swift-devel] Mosaswift update
>> 
>> So that actually fixed the issue.  So we have a working example of using CDM and intermediate file storage that Emalayan can use for testing his mosa store set up.
>> 
>> On Mar 14, 2012, at 3:05 PM, Jonathan Monette wrote:
>> 
>> > Oh, thanks, didn't catch that.  Trying another run right now but I think someone is using all of surveyor right now, but manual coaster service job is stuck in the queue state.
>> > 
>> > 
>> > On Mar 14, 2012, at 3:02 PM, Justin M Wozniak wrote:
>> > 
>> >> 
>> >> Ok, then it's a typo on readlinE .
>> >> 
>> >> On Wed, 14 Mar 2012, Jonathan Monette wrote:
>> >> 
>> >>> I am not sure.  I thought readlink returned the absolute path of the link which is what I wanted to use.  I am not sure how readlink behaves on the compute nodes.  This was my first attempt at getting around the link but I do not think this is working.
>> >>> 
>> >>> On Mar 14, 2012, at 2:43 PM, Justin M Wozniak wrote:
>> >>> 
>> >>>> On Wed, 14 Mar 2012, Jonathan Monette wrote:
>> >>>> 
>> >>>>> I say semi working because I intended the script to build up output during the stages on the swift run and the final output would show that it was stored temporarily in /tmp.  This is not happening as the final output for the file contains only the last line(a Hello from <id> line). I believe this is due to how CDM using symlinks for the files that is matched, so a symlink is being overwritten instead of at the file it points to.  I ran into this problem in the past(but this is documented in the user guide) and just need verify that this is indeed the issue.
>> >>>> 
>> >>>> In the past, the application unlinked the link.  It looks like you are appending to the link.
>> >>>> 
>> >>>> infile=`readlink $1`
>> >>>> outfile=`readline $2`
>> >>>> cat $infile >> $outfile
>> >>>> 
>> >>>> Is there a case in which the readlink affects the behavior of this sequence?
>> >>>> 
>> >>>> Does readline work correctly on the BG/P compute node?
>> >>>> 
>> >>>>     Justin
>> >>>> 
>> >>>> --
>> >>>> Justin M Wozniak
>> >>> 
>> >>> 
>> >> 
>> >> -- 
>> >> Justin M Wozniak
>> > 
>> > _______________________________________________
>> > Swift-devel mailing list
>> > Swift-devel at ci.uchicago.edu
>> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups "MosaStore" group.
>> To post to this group, send email to mosastore at googlegroups.com.
>> To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.
>> 
>> 
>> 
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> -- 
> You received this message because you are subscribed to the Google Groups "MosaStore" group.
> To post to this group, send email to mosastore at googlegroups.com.
> To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120315/4c7461d3/attachment.html>