[Swift-devel] LQCD meeting at Fermi

[Veronika Nefedova]

Hi,

Yong and I have met with Xian-He and his team today to talk over  
their current problems with the production swift code.

Some of the major issues we talked about:

- Sperate of concern: SwiftScript could be made to just describe the
abstract interfaces and data flows, and the app blocks could be pushed
into some separate specifications ( in a repository or something ), in
which other scripting lanugages can be used (e.g. python) to specify how
to invoke an actual application.

- Dealing with absolute path:
   LQCD uses dcache, which requires copying to/from some absolute path.

- Run clean up jobs outside pbs (i.e. using the fork manager instead)

- parameter problem: need to override things in tc.data, sites.xml, like
number of nodes for MPI jobs
   possible solution: put profile specification back in. (but we do not
have derivations, in which we were able to put some profiles).
   template based sites.xml and tc.data (generate the actual config  
files
using some templates and user supplied values at runtime)

- DB-mapper: users have an elaborate input data structures, keep it  
in the DB, so it would be nice to have a mapper that would read the  
input from the DB. This feature is in the works (?)

-intermediate results problem -- the same as MolDyn: need to have an  
ability to specify which file to keep and which file not.

- quoting problem:
   MPIrun does not deal correctly with "" that are passed to wrapper.sh
I remember there was also quoting issue with condor queues.

We also talked about using Falkon. But since LQCD uses dedicated  
resources
(600 or more nodes) and pbs queue checking time is set to around 10s, it
is not a big issue for them to run large number of jobs.

None of these except for the absolute path problem is a show- 
stoppers, next
we'll try to get their swiftscript running, and push some of the  
requests
into 0.3 features.

Yong and Nika