[Swift-devel] Mosaswift update

[Jonathan Monette]

No problem.  If there is any problems on the Swift side please do not hesitate in contacting me.  I will help in any way I can.

On Mar 19, 2012, at 10:31 AM, Matei Ripeanu wrote:

> Jon,
>  
> I wanted to thank you for your huge effort making this work.  For us this is big progress – we can start thinking about the experiments we’d like to have for the SC paper.
>  
> -Matei
>  
> From: mosastore at googlegroups.com [mailto:mosastore at googlegroups.com] On Behalf Of Emalayan Vairavanathan
> Sent: March-15-12 10:30 AM
> To: swift-devel at ci.uchicago.edu Devel; Jonathan Monette
> Cc: MosaStore; Michael Wilde; matei
> Subject: Re: [Swift-devel] Mosaswift update
>  
> Hi Jon and All,
>  
> MosaStore+Swift integration is done successfully. This is really a great news.
>  
> I successfully ran the setup provided by Jon with MosaStore+Swift on Surveyor (on 1, 3 , 5 and 64 nodes with different number of  jobs varying from 1 to 1024).
>  
> Jon and Mike thank you again for your time in helping me to get this far.
>  
> A quick question: The setup creates the swift working directory in GPFS but stores the intermediate files in MosaStore. Where should we actually have the swift-working directory (in GPFS / Mosa)? and Does this matter at all in case if I am going to use Mosa as an intermediate storage?
> 
> (As a next step I will be scripting our pipeline benchmark on swift and run at large scale to get some initial performance numbers. Also I will be parallely working to get Montage and ModFTDock running on BG/P with swift-CDM and MosaStore.)
> 
> Regards
> Emalayan
>  
> From: Jonathan Monette <jonmon at mcs.anl.gov>
> To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
> Cc: "mosastore at googlegroups.com" <mosastore at googlegroups.com>; Justin M Wozniak <wozniak at mcs.anl.gov>; "swift-devel at ci.uchicago.edu Devel" <swift-devel at ci.uchicago.edu> 
> Sent: Wednesday, 14 March 2012 2:29 PM
> Subject: Re: [Swift-devel] Mosaswift update
>  
> The working example is located in /home/jonmon/Workspace/Swift/mosaswift-test
>  
> There is a README there that provides a little bit of information.
>  
> Quick version to run it:
> 1) start your workers
> 2) ./setup.sh
> 3) ./run.sh
>  
> The directory final will have 5 files if it completed without error.  The content of the files will look like:
> Hello world
> Hello from <number>
>  
> On Mar 14, 2012, at 4:03 PM, Emalayan Vairavanathan wrote:
> 
> 
> Hi Jon,
>  
> Great, Thank you very much. Could you please send me the setup ?
>  
> Thank you
> Emalayan
>  
> From: Jonathan Monette <jonmon at mcs.anl.gov>
> To: Justin M Wozniak <wozniak at mcs.anl.gov> 
> Cc: "swift-devel at ci.uchicago.edu Devel" <swift-devel at ci.uchicago.edu>; MosaStore <mosastore at googlegroups.com> 
> Sent: Wednesday, 14 March 2012 2:00 PM
> Subject: Re: [Swift-devel] Mosaswift update
> 
> So that actually fixed the issue.  So we have a working example of using CDM and intermediate file storage that Emalayan can use for testing his mosa store set up.
> 
> On Mar 14, 2012, at 3:05 PM, Jonathan Monette wrote:
> 
> > Oh, thanks, didn't catch that.  Trying another run right now but I think someone is using all of surveyor right now, but manual coaster service job is stuck in the queue state.
> > 
> > 
> > On Mar 14, 2012, at 3:02 PM, Justin M Wozniak wrote:
> > 
> >> 
> >> Ok, then it's a typo on readlinE .
> >> 
> >> On Wed, 14 Mar 2012, Jonathan Monette wrote:
> >> 
> >>> I am not sure.  I thought readlink returned the absolute path of the link which is what I wanted to use.  I am not sure how readlink behaves on the compute nodes.  This was my first attempt at getting around the link but I do not think this is working.
> >>> 
> >>> On Mar 14, 2012, at 2:43 PM, Justin M Wozniak wrote:
> >>> 
> >>>> On Wed, 14 Mar 2012, Jonathan Monette wrote:
> >>>> 
> >>>>> I say semi working because I intended the script to build up output during the stages on the swift run and the final output would show that it was stored temporarily in /tmp.  This is not happening as the final output for the file contains only the last line(a Hello from <id> line). I believe this is due to how CDM using symlinks for the files that is matched, so a symlink is being overwritten instead of at the file it points to.  I ran into this problem in the past(but this is documented in the user guide) and just need verify that this is indeed the issue.
> >>>> 
> >>>> In the past, the application unlinked the link.  It looks like you are appending to the link.
> >>>> 
> >>>> infile=`readlink $1`
> >>>> outfile=`readline $2`
> >>>> cat $infile >> $outfile
> >>>> 
> >>>> Is there a case in which the readlink affects the behavior of this sequence?
> >>>> 
> >>>> Does readline work correctly on the BG/P compute node?
> >>>> 
> >>>>     Justin
> >>>> 
> >>>> --
> >>>> Justin M Wozniak
> >>> 
> >>> 
> >> 
> >> -- 
> >> Justin M Wozniak
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> 
> -- 
> You received this message because you are subscribed to the Google Groups "MosaStore" group.
> To post to this group, send email to mosastore at googlegroups.com.
> To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.
> 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>  
> -- 
> You received this message because you are subscribed to the Google Groups "MosaStore" group.
> To post to this group, send email to mosastore at googlegroups.com.
> To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.
>  
> 
> -- 
> You received this message because you are subscribed to the Google Groups "MosaStore" group.
> To post to this group, send email to mosastore at googlegroups.com.
> To unsubscribe from this group, send email to mosastore+unsubscribe at googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/mosastore?hl=en.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120319/c6121f8a/attachment.html>