[Swift-devel] user guide work

[Mihael Hategan]

On Tue, 2015-05-19 at 14:16 -0500, Lorenzo Pesce wrote:
> > On May 19, 2015, at 1:34 PM, Mihael Hategan <hategan at mcs.anl.gov> wrote:
> > [...]
> > 
> > Thanks! That is very useful. Do you have specific feedback on the
> > documentation? Such as #1 and #2 most annoying things about it?
> 
> Do you want an honest answer? ;-)

What good is a dishonest answer?

> 
> My first though were, bear with the humor I mean it in a positive way
> (all what sounds obnoxious is just my devious sense of humor,
> promised):
> 
> -Awesome, now I know another language in which to write “Hello World”,
> gee, I can even have a file called hello world or use tr to make it
> caps!!!! How do I run 300 genomes which are going to clog up half of
> my Cray, break my disks and drive my users insane? 
> 
> -Hmm… my scheduler choked and returned me errors that are
> incomprehensible and I am left with a 10 GB log file that requires tea
> leaves to be sorted out. If I wanted to write something that I can run
> on 10 nodes I would have used bash. OK, how do I sort out the mess? Is
> there any procedure to try and figure out what did I do wrong?
> 
> -Oh shit, I did not realize that I was making swift stage 10 TB of
> data during each run. Luckily my admin missed me when he tried to
> shoot me. :-)
> 
> - Can anyone tell me how does swift control the scheduler and keep
> track of what is running and what should be run? How do I help it sort
> out what to stage and when to start doing it? Can one help me sort
> that out without having to read the entire guide? It really looks like
> my scheduler is comatose and I am being blamed for anything that
> happens there. ;-)

Right. The eternal problem of conceptually similar environments/clusters
all slightly different in devious ways. You can abstract away the
functionality. You cannot abstract away all the ways in which things can
go wrong.

> 
> 
> >> I can try to be more helpful, if you want me to. 
> >> 
> >> I have worked on a number of parallel projects, where essentially the
> >> same workflow was implemented with and without swift and both teams
> >> played to win. That is one paper I plan to write some day…
> > 
> > Wait, who won? :)
> 
> 
> Actually the jury is still out and my inclination would be to say that
> there is a lot to learn from it on how to make using Swift easy on the
> users who don’t want to learn how to use swift.
> I like the swift implementation better (honestly, I am not trying to
> flatter you), but the other paper was published a year ago.

I think we would all appreciate details here.

Mihael