[Swift-user] sort on large data

[Yadu Nand Babuji]

Hi Jiada Tu,

1) Here's an example for returning an array of files :

type file;
app (file outs[]) make_outputs (file script)
{
     bash @script;
}

file outputs[] <filesys_mapper; prefix="outputs">;
file script       <"make_outputs.sh">; # This script creates a few files 
with outputs as prefix
(outputs) = make_outputs(script);

2) The products of a successful task execution, must be visible to the 
headnode (where swift runs) either through a
- shared filesystem (NFS, S3 mounted over s3fs etc)  or
- must be brought back over the network.
But, we can reduce the overhead in moving the results to the headnode 
and then to the workers for the reduce stage.

I understand that this is part of your assignment, so I will try to 
answer without getting too specific, at the same time,
concepts from hadoop do not necessarily work directly in this context. 
So here are some things to consider to get
the best performance possible:

- Assuming that the texts contain 10K unique words, your sort program 
will generate a file containing atmost 10K lines
  (which would be definitely under an MB). Is there any advantage into 
splitting this into smaller files ?

- Since the final merge involves tiny files, you could very well do the 
reduce stage on the headnode and be quite efficient
   (you can define the reduce app only for site:local)

   sites : [local, cloud-static]
   site.local {
                 ....
                 app.reduce {
                         executable : ${env.PWD}/reduce.py
                 }
   }

   site.cloud-static {
                 ....
                 app.python {
                         executable : /usr/bin/python
                 }

  }

  This assumes that you are going to define your sorting app like this :

   app (file freqs) sort (file sorting_script, file input ) {
        python @sorting_script @input;
  }


- The real cost is in having the original text reach the workers, this 
can be made faster by :
     - A better headnode with better network/disk IO (I've measured 
140Mbit/s between m1.medium nodes, c3.8xlarge comes with 975Mbits/s)
     - Use S3 with S3fs and have swift-workers pull data from S3 which 
is pretty scalable, and remove the IO load from the headnode.

- Identify the optimal size for data chunks for your specific problem. 
Each chunk of data in this case comes with the overhead of starting
   a new remote task, sending the data and bringing results back. Note 
that the result of a wordcount on a file whether it is 1Mb or 10Gb
   is still the atmost 1Mb (with earlier assumptions)

- Ensure that the data with the same datacenter, for cost as well as 
performance. By limiting the cluster to US-Oregon we already do this.

If you would like to attempt this using S3FS, let me know, I'll be happy 
to explain that in detail.

Thanks,
Yadu


On 10/18/2014 04:18 PM, Jiada Tu wrote:
> I am doing an assignment with swift to sort large data. The data 
> contains one record (string) each line. We need to sort the records 
> base on ascii code. The data is too large to fit in the memory.
>
> The large data file is in head node, and I run the swift script 
> directly on head node.
>
> Here's what I plan to do:
>
> 1) split the big file into 64MB files
> 2) let each worker task sort one 64MB files. Say, each task will call 
> a "sort.py" (written by me). sort.py will output a list of files, 
> say:"sorted-worker1-001; sorted-worker1-002; ......". The first file 
> contains the records started with 'a', the second started with 'b', etc.
> 3) now we will have all records started with 'a' in 
> (sorted-worker1-001;sorted-worker2-001;...); 'b' in 
>  (sorted-worker1-002;sorted-worker2-002; ......); ...... Then I send 
> all the files contains records 'a' to a "reduce" worker task and let 
> it merge these files into one single file. Same to 'b', 'c', etc.
> 4) now we get 26 files (a-z) with each sorted inside.
>
> Basically what I am doing is simulate Map-reduce. step 2 is map and 
> step 3 is reduce
>
> Here comes some problems:
> 1) for step 2, sort.py need to output a list of files. How can swift 
> app function handles list of outputs?
>     app (file[] outfiles) sort (file[] infiles) {
>           sort.py // how to put out files here?
>     }
>
> 2) As I know (may be wrong), swift will stage all the output file back 
> to the local disk (here is the head node since I run the swift script 
> directly on headnode). So the output files in step 2 will be staged 
> back to head node first, then stage from head node to the worker nodes 
> to do the step 3, then stage the 26 files in step 4 back to head node. 
> I don't want it because the network will be a huge bottleneck. Is 
> there any way to tell the "reduce" worker to get data directly from 
> "map" worker? Maybe a shared file system will help, but is there any 
> way that user can control the data staging between workers without 
> using the shared file system?
>
> Since I am new to the swift, I may be totally wrong and 
> misunderstanding what swift do. If so, please correct me.
>
>
>
>
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-user/attachments/20141018/846551a8/attachment.html>