[Swift-user] About packing jobs on a supercomputer

[Lorenzo Pesce]

Hi All --

I have two (more) questions:

1) Listening to Justin and Mike I inferred that you could help me figure out how to pack jobs "as much as possible", when memory is a limiting factor, but not entirely predictable (for various reasons Matlab seems to be sloppy in taking care of its memory usage and I don't seem to be able to accurately predict how large a job will be). The project is made of may thousands of simulations of different sizes. The ones of the same size and the same type behave similarly. The only reliable predictor of maximum packing is the value before I get the out of memory (OOM) message. :-)

2) When I log in and check the cluster status, usually there is some backfill space available (my calculations comprise many many small calculations in addition to larger ones). Do you have any crafty ways to find out what type of jobs will run immediately as backfill and which one will have to wait? (BTW, this would make our boss happy too because we will increase utilization at no real cost).

To give you an idea every campaign churns a few hundred thousands core hours and the current plans involved many campaigns. 

Thanks!

Lorenzo