<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Nika, all,<br>
I have just committed my changes for the last week or two (R1221):<br>
- added a GUI monitor to view the service state remotely<br>
- added support for data caching, including modifying the service WSDL<br>
to support the meta-data associated with the data caching... if data<br>
caching is not needed, the old clients using the old WSDL should still<br>
work just fine; the data-aware scheduler has not been tested much yet,
use<br>
with caution<br>
- added some new options to the two config files from the service and<br>
workers<br>
- added some new scripts to all, clients, service, workers...<br>
- added various sample workloads to test the data caching code<br>
<b>- addressed BUG53 from Swift... </b>added support for capturing and<br>
identifying certain hardware errors (i.e. stale NFS file handle), and<br>
retrying those tasks up to a certain number of retry attempts; also<br>
added support for the scheduler to suspend workers for some time that<br>
have failed some number of tasks (for now, only the known hardware<br>
errors actually affect this, and application errors should not suspend<br>
any workers)<br>
- other minor stuff<br>
<br>
Regarding BUG53, here are more details (service/etc/Falkon.config):<br>
#settings for task retries upon known errors (i.e. stale NFS file
handle) and max number of known failures per node before suspending the
corresponding node<br>
maxNumErrorsPerTask=10<br>
maxNumErrorsPerExecutor=3<br>
suspendTimeoutInterval_ms=15000<br>
<br>
These setting is probably a good place to start, but we could change
them as we see fit as we learn more about how this new feature behaves
when there are errors. For example, we might want to increase the
suspend timeout value from 15sec to say 60 sec, and possibly reduce the
number of errors per executor from 3 to 1 as its likely that when 1
error occurs, another will happen on the next 2 as well. <br>
<br>
Nika, feel free to update Falkon (the provider should not need an
update), and try another 244 mol run with MolDyn. You have an account
to charge against at ANL/UC, right? You can try to do the whole run
yourself, and see how it works. Let us know if the workflow
completes! I am also still interested for some comparison numbers if
MolDyn would run over GRAM directly for the larger number of molecules,
so keep us posted with that progress as well.<br>
<br>
BTW, Catalin is still waiting on instructions on how to compile the
entire MolDyn app. Do you not have time for this now, and we should
move to a different app for running Swift+Falkon in a virtual cluster?<br>
<br>
Ioan<br>
<br>
Veronika Nefedova wrote:
<blockquote cite="mid:D7BE0424-C7DC-42BE-9A7B-09EA541DF305@mcs.anl.gov"
type="cite">Ioan, how your work on that 'avoiding bad node' thing is
progressing? You seem to be more interested in running my workflow on a
virtual cluster rather then working on a new feature that would
enable MolDyn to run reliably on TG... I apologize if I am wrong - the
lack of information made me to come to this conclusion; please provide
me with a relevant information and an estimate on when I can expect
Falcon to be ready for a new rounds of tests.
<br>
<br>
Thanks,
<br>
<br>
Nika
<br>
<br>
<br>
<br>
On Sep 13, 2007, at 5:48 PM, Ioan Raicu wrote:
<br>
<br>
<blockquote type="cite">It would be good to have some comparison
numbers, so I think its worth doing to see if the workflow will
complete, and to see what performance it gets!
<br>
Ioan
<br>
<br>
Veronika Nefedova wrote:
<br>
<blockquote type="cite">Thanks, Mihael! I could try submitting now
some 20 molecules to tg-uc (directly to GRAM) -- just to be on a safe
side. If no GRAM problems will be reported, I'll increase the number to
244.
<br>
Of, course the performance will suffer greatly -- but I hope it would
enable to get the whole workflow to go throw. Are there any throttles
that could be set to increase a bit the performance (given that I set
the maxSubmitRate to 0.2) ?'
<br>
<br>
Nika
<br>
<br>
On Sep 13, 2007, at 4:41 PM, Mihael Hategan wrote:
<br>
<br>
<blockquote type="cite">Ok, so there's something in.
<br>
There are some discussions that can be had on certain aesthetic topics.
<br>
In any event, in sites.xml, you can add, for a site, something like
<br>
this:
<br>
<br>
<profile namespace="karajan"
key="maxSubmitRate">0.1</profile>
<br>
<br>
The rate is in jobs per second. The above would mean one job every ten
<br>
seconds.
<br>
<br>
Mihael
<br>
<br>
On Thu, 2007-09-13 at 15:23 +0000, Ben Clifford wrote:
<br>
<blockquote type="cite">Yes?
<br>
<br>
On Thu, 13 Sep 2007, Mihael Hategan wrote:
<br>
<br>
<blockquote type="cite">May I still fix that bug though?
<br>
<br>
On Thu, 2007-09-13 at 09:54 -0500, Ioan Raicu wrote:
<br>
<blockquote type="cite">Hi,
<br>
I am still working on the new feature for Falkon to avoid submitting
<br>
tasks to known bad nodes, and to perhaps do its own retries for failed
<br>
jobs with certain known errors (i.e. stale NFS handle). I should have
<br>
that ready for next week to try out. Once this new feature is in, we
<br>
could try MolDyn again to see how it behaves.
<br>
<br>
About avoiding Falkon of MolDyn, I recall something about the
<br>
scalability/policies of GRAM/PBS to handle many con current jobs,
<br>
having to throttle job submissions to something around 1 job every 10
<br>
seconds (for sustained periods of time, short bursts could send
<br>
faster), and the fact that only a few 10s of nodes would be used
<br>
concurrently, even though the sites that it was running on had more
<br>
free nodes. I also think that MolDyn through GRAM/PBS was running
<br>
only 1 job per node, in essence only using 1 processor of the 2 per
<br>
node. I think the largest workflow Nika was able to run over GRAM/PBS
<br>
was 5 molecules, 421 jobs (but only 340 jobs in the large stage).
<br>
Nika, were there other problems you encountered?
<br>
<br>
Ioan
<br>
<br>
Mihael Hategan wrote:
<br>
<blockquote type="cite">Very well Sir. I shall see to the
priority of the issue being raised.
<br>
<br>
On Thu, 2007-09-13 at 14:09 +0000, Ben Clifford wrote:
<br>
<br>
<blockquote type="cite">I think one of the main
impediments to moldyn running with GRAM directly
<br>
is bug 53 which is a request for sumission rate limiting.
<br>
<br>
It might be relatively easy to implement that and see how the MolDyn
<br>
workflow behaves then.
<br>
<br>
I'm interested to see if Falkon can be avoided for this workflow.
<br>
<br>
<br>
</blockquote>
<br>
_______________________________________________
<br>
Swift-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
<br>
<br>
<br>
</blockquote>
<br>
-- <br>
============================================
<br>
Ioan Raicu
<br>
Ph.D. Student
<br>
============================================
<br>
Distributed Systems Laboratory
<br>
Computer Science Department
<br>
University of Chicago
<br>
1100 E. 58th Street, Ryerson Hall
<br>
Chicago, IL 60637
<br>
============================================
<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
<br>
Web: <a class="moz-txt-link-freetext" href="http://www.cs.uchicago.edu/~iraicu">http://www.cs.uchicago.edu/~iraicu</a>
<br>
<a class="moz-txt-link-freetext" href="http://dsl.cs.uchicago.edu/">http://dsl.cs.uchicago.edu/</a>
<br>
============================================
<br>
============================================
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
</blockquote>
<br>
_______________________________________________
<br>
Swift-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
<br>
<br>
</blockquote>
<br>
_______________________________________________
<br>
Swift-devel mailing list
<br>
<a class="moz-txt-link-abbreviated" href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a>
<br>
<a class="moz-txt-link-freetext" href="http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel">http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel</a>
<br>
<br>
</blockquote>
<br>
-- <br>
============================================
<br>
Ioan Raicu
<br>
Ph.D. Student
<br>
============================================
<br>
Distributed Systems Laboratory
<br>
Computer Science Department
<br>
University of Chicago
<br>
1100 E. 58th Street, Ryerson Hall
<br>
Chicago, IL 60637
<br>
============================================
<br>
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
<br>
Web: <a class="moz-txt-link-freetext" href="http://www.cs.uchicago.edu/~iraicu">http://www.cs.uchicago.edu/~iraicu</a>
<br>
<a class="moz-txt-link-freetext" href="http://dsl.cs.uchicago.edu/">http://dsl.cs.uchicago.edu/</a>
<br>
============================================
<br>
============================================
<br>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
============================================
Ioan Raicu
Ph.D. Student
============================================
Distributed Systems Laboratory
Computer Science Department
University of Chicago
1100 E. 58th Street, Ryerson Hall
Chicago, IL 60637
============================================
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@cs.uchicago.edu">iraicu@cs.uchicago.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.cs.uchicago.edu/~iraicu">http://www.cs.uchicago.edu/~iraicu</a>
<a class="moz-txt-link-freetext" href="http://dsl.cs.uchicago.edu/">http://dsl.cs.uchicago.edu/</a>
============================================
============================================</pre>
</body>
</html>