[Swift-devel] ram disk with swift

[Michael Wilde]

> We were talking about putting the input files for Taka's calculations
> (the X.mat files), which are rather small ~ 60Kb or so, in ram disk
> and try to use them for all the apps (that use the same file). giving
> the size, we could even load up all of them for each node
> (7*3*60=1.2MB, that is zero) to reduce traffic towards /lustre and
> thus avoid the meta-data slow downs that Beagle seems to have.
> 
> If we copy files in /tmp (as far as I understand it would be ramdisk)
> on the each node through swift, would it be there for all the apps?

One way to do this, which has worked well for us is this:

- in the app wrapper script, create a function that loads the constant input data

- in this function, using directory-based locking logic to:

  -- create a data dir under /tmp
  -- fill the dir from a tarball

- in the app wrapper logic, if the /tmp/data dir is there, use it
  else call the function to create it.

This logic uses the fact that directory creation is atomic, so you can use that to prevent races where two apps running on the same node would each try to create the dir.

David, can you post the logic you developed for the above?

Thanks,

- Mike