[Swift-devel] disk space requirement

[Mihael Hategan]

On Fri, 2007-06-01 at 08:55 -0500, Mike Wilde wrote:
> Nika, regarding where you can get 244GB, we should send in a request 
> to CI Support to get 1TB or so from the CI SAN mounted on a Swift 
> submit host.
> 
> In the interim:
> 
> - you can get 100GB or so on NFS on terminable
> 
> - we can try to quickly add a 300GB drive (or two) to terminable
> 
> - One of the gridlab hosts (4 I think) should be available with that 
> storage now. I am not sure if we feel this lab host is stable yet 
> but we should make sure.
> 
> We need to be gearing up quickly to this scale, so the lab 
> environment that Ive been pushing to get set up is going to be 
> increasingly important.
> 
> Regarding the core limitation that Nika points out - this seems to 
> dictate that we take the step of allowing workflow inputs, outputs 
> and intermediate results to be located on any gridftp server and 
> tracked via a replica catalog, as VDS did.

I'd say tracking where intermediate files are for a run should not
necessarily imply a "replica catalog" as implemented by VDS (in
particular RLS).

> 
> Ben, Mihael, do you have a feel for where we can slot this into 
> development? v3 - say circa September?

Sounds reasonable with the limited knowledge I have.

Mihael

> 
> Mike
> 
> 
> Veronika Nefedova wrote, On 6/1/2007 8:19 AM:
> > Hi,
> > 
> > I know I raised this question many times before, but I think I need a 
> > solution to it very soon (=now). I have a machinery in place to do a 
> > really big runs for MolDyn. Currently, the workflow produces about 0.9GB 
> > of data for each molecule, but this is all the intermediate data, all I 
> > need is *one* 300K file as a result (per molecule). The rest is 
> > intermediate data that I do not need.  So my questions are:
> > 
> > 1. How can I eliminate the intermediate results staging back to my 
> > submit host? I do not need it in case of one remote compute pool. VDS 
> > had this feature...
> > 
> > 2. If implementing this feature is very hard and time-consuming -- what 
> > submit host would you recommend for a 244-molecule run ? Roughly 244 GB 
> > of space is needed.
> > 
> > A 20-molecule run is all I can do on wiggum (where the application code 
> > is currently compiled).
> > 
> > Thanks!
> > 
> > Nika
> > 
> > 
> > 
> > _______________________________________________
> > Swift-devel mailing list
> > Swift-devel at ci.uchicago.edu
> > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > 
> > 
>