[Swift-devel] Mosaswift update

Mon Mar 19 10:31:13 CDT 2012

Jon, 

I wanted to thank you for your huge effort making this work.  For us this is
big progress - we can start thinking about the experiments we'd like to have
for the SC paper.

-Matei 

From: mosastore at googlegroups.com [mailto:mosastore at googlegroups.com] On
Behalf Of Emalayan Vairavanathan
Sent: March-15-12 10:30 AM
To: swift-devel at ci.uchicago.edu Devel; Jonathan Monette
Cc: MosaStore; Michael Wilde; matei
Subject: Re: [Swift-devel] Mosaswift update

Hi Jon and All,

MosaStore+Swift integration is done successfully. This is really a great
news.

I successfully ran the setup provided by Jon with MosaStore+Swift on
Surveyor (on 1, 3 , 5 and 64 nodes with different number of  jobs varying
from 1 to 1024).

Jon and Mike thank you again for your time in helping me to get this far.

A quick question: The setup creates the swift working directory in GPFS but
stores the intermediate files in MosaStore. Where should we actually have
the swift-working directory (in GPFS / Mosa)? and Does this matter at all in
case if I am going to use Mosa as an intermediate storage? 

(As a next step I will be scripting our pipeline benchmark on swift and run
at large scale to get some initial performance numbers. Also I will be
parallely working to get Montage and ModFTDock running on BG/P with
swift-CDM and MosaStore.)

Regards
Emalayan

  _____  

From: Jonathan Monette <jonmon at mcs.anl.gov>
To: Emalayan Vairavanathan <svemalayan at yahoo.com> 
Cc: "mosastore at googlegroups.com" <mosastore at googlegroups.com>; Justin M
Wozniak <wozniak at mcs.anl.gov>; "swift-devel at ci.uchicago.edu Devel
<mailto:swift-devel at ci.uchicago.edu%20Devel> " <swift-devel at ci.uchicago.edu>

Sent: Wednesday, 14 March 2012 2:29 PM
Subject: Re: [Swift-devel] Mosaswift update

The working example is located in
/home/jonmon/Workspace/Swift/mosaswift-test

There is a README there that provides a little bit of information.

Quick version to run it:

1) start your workers

2) ./setup.sh

3) ./run.sh

The directory final will have 5 files if it completed without error.  The
content of the files will look like:

Hello world

Hello from <number>

On Mar 14, 2012, at 4:03 PM, Emalayan Vairavanathan wrote:

Hi Jon,

Great, Thank you very much. Could you please send me the setup ? 

Thank you

Emalayan

  _____  

From: Jonathan Monette <jonmon at mcs.anl.gov>
To: Justin M Wozniak <wozniak at mcs.anl.gov> 
Cc: "swift-devel at ci.uchicago.edu Devel" <swift-devel at ci.uchicago.edu>;
MosaStore <mosastore at googlegroups.com> 
Sent: Wednesday, 14 March 2012 2:00 PM
Subject: Re: [Swift-devel] Mosaswift update

So that actually fixed the issue.  So we have a working example of using CDM
and intermediate file storage that Emalayan can use for testing his mosa
store set up.

On Mar 14, 2012, at 3:05 PM, Jonathan Monette wrote:

> Oh, thanks, didn't catch that.  Trying another run right now but I think
someone is using all of surveyor right now, but manual coaster service job
is stuck in the queue state.
> 
> 
> On Mar 14, 2012, at 3:02 PM, Justin M Wozniak wrote:
> 
>> 
>> Ok, then it's a typo on readlinE .
>> 
>> On Wed, 14 Mar 2012, Jonathan Monette wrote:
>> 
>>> I am not sure.  I thought readlink returned the absolute path of the
link which is what I wanted to use.  I am not sure how readlink behaves on
the compute nodes.  This was my first attempt at getting around the link but
I do not think this is working.
>>> 
>>> On Mar 14, 2012, at 2:43 PM, Justin M Wozniak wrote:
>>> 
>>>> On Wed, 14 Mar 2012, Jonathan Monette wrote:
>>>> 
>>>>> I say semi working because I intended the script to build up output
during the stages on the swift run and the final output would show that it
was stored temporarily in /tmp.  This is not happening as the final output
for the file contains only the last line(a Hello from <id> line). I believe
this is due to how CDM using symlinks for the files that is matched, so a
symlink is being overwritten instead of at the file it points to.  I ran
into this problem in the past(but this is documented in the user guide) and
just need verify that this is indeed the issue.
>>>> 
>>>> In the past, the application unlinked the link.  It looks like you are
appending to the link.
>>>> 
>>>> infile=`readlink $1`
>>>> outfile=`readline $2`
>>>> cat $infile >> $outfile
>>>> 
>>>> Is there a case in which the readlink affects the behavior of this
sequence?
>>>> 
>>>> Does readline work correctly on the BG/P compute node?
>>>> 
>>>>     Justin
>>>> 
>>>> --
>>>> Justin M Wozniak
>>> 
>>> 
>> 
>> -- 
>> Justin M Wozniak
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
You received this message because you are subscribed to the Google Groups
"MosaStore" group.
To post to this group, send email to mosastore at googlegroups.com.
To unsubscribe from this group, send email to
mosastore+unsubscribe at googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/mosastore?hl=en.

_______________________________________________
Swift-devel mailing list
Swift-devel at ci.uchicago.edu
https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
You received this message because you are subscribed to the Google Groups
"MosaStore" group.
To post to this group, send email to mosastore at googlegroups.com.
To unsubscribe from this group, send email to
mosastore+unsubscribe at googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/mosastore?hl=en.

-- 
You received this message because you are subscribed to the Google Groups
"MosaStore" group.
To post to this group, send email to mosastore at googlegroups.com.
To unsubscribe from this group, send email to
mosastore+unsubscribe at googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/mosastore?hl=en.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20120319/22c46a4d/attachment.html>