[Swift-user] Issues with swiftR

[Lorenzo Pesce]

Hi --

We need to run a set of simulations to test the statistical properties of an estimator.

This creates two sets of problems:

1) Our calculations involve exploring a number of parameter combinations, about 350 parameter combinations. For each combination we have 10,000 calculations or 3.5 million calculations. How am I going to use SwiftR for that? I can't apply to a 3.5 million element array and it makes little sense because the difference between each block of 10,000 is the random dataset, which is created following the same rules. Note that 10,000 calculations take about 160 hours on a single core, varying a little bit between dataset types.

2) Moreover, we need to generate random numbers, each of our calculations needs a randomly generate set of 50 to 200 numbers. We also need random numbers for the estimator. These random numbers ideally would be reproducible and not on disk. We need to use seeds, but the seeds need to be associated.

We can't split it into 350 instances of SwiftR of the cluster will die (probably). 


Suggestions?