[Swift-devel] stageout ordering vs restarts

[Ian Foster]

Ben:

That's an interesting observation.

Can we control the relative priorities of stage outs and stage ins?

Ian.

On Jul 6, 2008, at 11:41 AM, Ben Clifford wrote:

>
> At present, stageouts for jobs tend to execute quite late in a run,  
> in as
> much as when there are other jobs to run, the stageins for those  
> jobs will
> usually use available file transfer rate-limit load before stageouts
> happen.
>
> I've noticed this before as a user interface quirk - users see GRAM  
> jobs
> complete on remote sites, but do not see output files appear on the  
> submit
> side until much much later and sometimes misinterpret that as a  
> failure.
>
> However, I think there is an issue here with how restarts work too.  
> Jobs
> are not recorded as done for the purposes of restart (i.e. will not be
> re-executed) until stageout has finished.
>
> When stageout is happening late, that means in late-stageout  
> situations,
> lots of work will be done but to the extent that it can be ignored on
> restarts.
>
> So that makes early-stageout behaviour more appealing in some  
> situations -
> situations in which it is expected that a restart will be necessary,  
> or
> where it is preferable to have slower job execution in exchange for  
> more
> stuff marked as done in the restart logs.
>
> That is perhaps worth thinking about as part of the project that  
> Ragib is
> working on.
>
> -- 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel