<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
One of the 2 main motivations for Falkon was the data management. We
saw early on that we need to couple the compute and data resource
management, and that is what we are doing as we push forward with
Falkon. Falkon should be something that could be usable by other
applications, that don't have all the smarts of Swift, that simply want
to run jobs efficiently and have the data management abstracted away. <br>
<br>
The main idea is that Swift's data management will likely still be
needed (at a site level), but Falkon can push that further to the
physical node level. Swift and Falkon will likely evolve
independently, but if we work together, we can ensure that they can
inter-operate, as thy do today!<br>
<br>
Ioan<br>
<br>
Mihael Hategan wrote:
<blockquote cite="mid:1179306421.2402.12.camel@blabla.mcs.anl.gov"
type="cite">
<pre wrap="">I think we're moving towards a scenario in which Falkon does
increasingly more things that it wasn't supposed to do. That includes
scheduling and data management (which, is a tricky business if we look
at the necessity for throttling, error handling and other management
issues).
I'm not sure if this is a good idea from an engineering standpoint.
Mihael
On Tue, 2007-05-15 at 23:24 +0000, Ben Clifford wrote:
</pre>
<blockquote type="cite">
<pre wrap="">On Tue, 15 May 2007, Ioan Raicu wrote:
</pre>
<blockquote type="cite">
<pre wrap="">If we can get the data caching working in Falkon, we might be able to
run Swift over Falkon without a shared file system. This is still work
in progress, but we might be closer to achieving this that not. BTW,
the data caching would mean that Swift does not stage in any data
anymore, but wold essentially stand up a GridFTP server from where
Falkon workers would get the needed data just when they need it. We are
still ironing out all this stuff, but it could potentially do away with
the shared file sytem assumption.
</pre>
</blockquote>
<pre wrap="">In the longer term, Swift possibly won't have its input data on the
submitting system - for example, if data is mapped from remote gridftp
servers, then it should be transferred directly from those ftp servers to
the execute side (perhaps to a shared filesystem, perhaps direct to a
worker node), and output data should be transferred back fairly directly,
rather than going via the submit system.
If Falkon is doing its own 'interesting' data movement stuff, then it
would probably be a good idea for it to mesh in with what Swift (eg. swift
provides a list of stage-these-in and stage-these-out URLs or something
like that and has various ways of performing that, such as submitting a
transfer job, or passing that information onto falkon)
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.cs.uchicago.edu/~iraicu">http://www.cs.uchicago.edu/~iraicu</a>
<a class="moz-txt-link-freetext" href="http://dsl.cs.uchicago.edu/">http://dsl.cs.uchicago.edu/</a>
============================================
============================================
</pre>
</body>
</html>