<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
At the Microsoft workshop I just attended, someone had a 25 million
task application that dealt with AIDS research :)<br>
<br>
Mihael Hategan wrote:
<blockquote cite="mid:1193610846.28024.12.camel@blabla.mcs.anl.gov"
type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">Well lg(37^9) =~ 14, so you need about 14 digits to cover the same range
of values:
00000000000000/angle4-00000000000001-kickstart.xml
</pre>
</blockquote>
<pre wrap=""><!---->
Although that's silly. We'll never have more than 10 million jobs of a
kind (pretty much like 640K should be enough for everybody).
</pre>
<blockquote type="cite">
<pre wrap="">
</pre>
<blockquote type="cite">
<pre wrap="">000000/angle4-00000002-kickstart.xml
...
000000/angle4-00000099-kickstart.xml
...
000020/angle4-00002076-kickstart.xml
etc.
This makes splitting based on powers of 10 (or 26 or 36) trivial. Other
splits can be done with mod() functions.
Can we start heading in this or some similar direction?
We need to coordinate a plan for this, I suspect, to make Andrew's
workflows perform acceptably.
- Mike
On 10/27/07 2:08 PM, Ben Clifford wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Sat, 27 Oct 2007, Mihael Hategan wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Quickly before I leave the house:
Perhaps we could try copying to local FS instead of linking from shared
dir and hence running the jobs on the local FS.
</pre>
</blockquote>
<pre wrap="">Maybe. I'd be suspicious that doesn't reduce access to the directory too
much.
I think the directories where there are lots of files being read/written
by lots of hosts are:
the top directory (one job directory per job)
the info directory
the kickstart directory
the file cache
In the case where directories get too many files in them because of
directory size constraints, its common to split that directory into many
smaller directories (eg. how squid caching, or git object storage works).
eg, given a file fubar.txt store it in fu/fubar.txt, with 'fu' being some
short hash of the filename (with the hash here being 'extract the first
two characters).
Pretty much I think Andrew wanted to do that for his data files anyway,
which would then reflect in the layout of the data cache directory
structure.
For job directories, it may not be too hard to split the big directories
into smaller ones. There will still be write-lock conflicts, but this
might mean the contention for each directories write-lock is lower.
</pre>
</blockquote>
</blockquote>
<pre wrap="">_______________________________________________
Swift-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
</pre>
</blockquote>
<pre wrap=""><!---->
_______________________________________________
Swift-devel mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.cs.uchicago.edu/~iraicu">http://www.cs.uchicago.edu/~iraicu</a>
<a class="moz-txt-link-freetext" href="http://dsl.cs.uchicago.edu/">http://dsl.cs.uchicago.edu/</a>
============================================
============================================</pre>
</body>
</html>