[Swift-user] A question about design choice

[Mihael Hategan]

The swift script you have for the parallel jobs, is it a simple foreach
loop?

Mihael

On Mon, 2012-10-08 at 14:29 -0500, Jonathan Margoliash wrote:
> Hello swift-user,
> 
> 
> I had a question about the design of my program. My program involves a
> fair bit of matrix manipulations, so instead of writing the control
> structure of the program in swift, I've written most of the logic and
> data management in matlab scripts. The only functionality of swift
> that I use is to parallelize large for-loops, while the code that runs
> my swift scripts, and the apps that swift distributes inside each
> for-loop, are written in matlab. In essence, I've created a program in
> matlab which calls a swift script many times in serial, using swift to
> distributing many jobs in parallel.
> 
> 
> The issue I've come across is that after a certain number of serial
> calls to swift, I need to do a significant amount of data crunching
> before I can resume making serial swift calls. This data crunching is
> resource-intensive enough that it can't be done on a log in node, but
> not nearly intensive enough that it needs to be broken down into
> component parts and parallelized. My question is, what is the correct
> way of handling this situation? Does it make sense to write a separate
> swift script whose body only contains a single app call, so I can get
> this minor job run on one of the compute nodes? If I do it that way,
> will there likely be a significant delay while the scheduler processes
> my request, or is it likely to go through quickly because I'm only
> asking for one node for a short period of time?
> 
> 
> Thanks for your comments,
> 
> 
> Jonathan
> _______________________________________________
> Swift-user mailing list
> Swift-user at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-user