[Swift-user] Swift/PBS Scheduler Slow to Report "Finished"?

[Michael Wilde]

On 3/19/09 5:41 PM, Mihael Hategan wrote:
> On Thu, 2009-03-19 at 17:35 -0500, Andrew Boyce wrote:
>> I will definitely have to try that. Now I think I understand the
>> benefits of that approach. Sending the data back is important.
>> I do have a question though. In the Swift user guide, it states: "CoG
>> coasters provide a low-overhead job submission and 
>> file transfer mechanism suited for the execution of short jobs (on the
>> order of a few seconds) and the transfer of small files 
>> (on the order of a few kilobytes)."
>>
>>
>> If I want to use coasters with longer jobs, with the transfer of large
>> files, will that be a problem/be ill-suited with this approach?
>> I know that our goal is to work with much, much larger files down the
>> road, and much longer jobs.
> 
> You may not want to use the coaster file provider for large files, but
> you could still use the coaster execution provider for running the jobs.
> It should support larger jobs, but the benefit of using coasters for
> such jobs may not be worth the extra layers of code (though there is
> this specific case where coasters can help running multiple jobs per
> node that has multiple cores/CPUs).

Let me offer a slight variation on this. As the implementation 
stabilizes, coaster-providers would ideally become the predominant way 
of running all jobs on all sites. True, they involve more code layers 
than the "plain" job providers, but that will be transparent once they 
are solid. And they seem have reached a level of stability where they 
are quite usable, and readily fixable when new, unexpected cases are 
encountered. Once they work for a given site, they tend to be reliable. 
Making their startup transparent across many environments has been 
devilishly hard, though. This makes the trunk version prone to breakage 
as fixes are applied.

So, Andrew, I would suggest you use them for all configurations where 
they work, and any help you can provide in testing and hardening them 
will be greatly appreciated.

As Mihael suggests, for data, there is likely to be a crossover point 
where as file size grows, gridftp becomes more efficient. I would hazard 
a guess that if you file is <10Kbytes, try coasters, >1MB, use gridftp.