[Swift-devel] foreach.max.threads question

[Mihael Hategan]

How many cores do you run this on?

On Tue, 2012-10-16 at 13:56 -0500, David Kelly wrote:
> Previously it was not explicitly set, so I am assuming it would have been 1024. As a test I tried setting it to 520 (the maximum number of available cores), but that did not seem to help.
> 
> ----- Original Message -----
> > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > To: "David Kelly" <davidk at ci.uchicago.edu>
> > Cc: "swift-devel Devel" <swift-devel at ci.uchicago.edu>
> > Sent: Tuesday, October 16, 2012 1:44:47 PM
> > Subject: Re: [Swift-devel] foreach.max.threads question
> > What was foreach.max.threads set to before?
> > 
> > On Tue, 2012-10-16 at 11:04 -0500, David Kelly wrote:
> > > Hello,
> > >
> > > I have noticed that since the foreach.max.threads changes, the DSSAT
> > > script is now running out of memory. I have the heap size set to 4
> > > gigabytes. There are 120K items in gridLists.
> > > The main foreach loop of the script looks like this:
> > >
> > > foreach g,i in gridLists {
> > >    file tar_output <single_file_mapper; file=@strcat("output/",
> > >    gridLists[i], "output.tar.gz")>;
> > >    file part_output <single_file_mapper; file=@strcat("parts/",
> > >    gridLists[i], ".part")>;
> > >
> > >    file in1[] <filesys_mapper; location=@strcat(@arg("scenarios"),
> > >    "/", gridLists[i]), pattern="*">; // Scenario files
> > >    file in2[] <filesys_mapper; location=@strcat(@arg("weather"),
> > >    "/", gridLists[i]), pattern="*">; // Weather files
> > >    file in3[] <filesys_mapper; location=@arg("refdata"),
> > >    pattern="*">; // Common data
> > >    file in4[] <filesys_mapper; location=@arg("bindata"),
> > >    pattern="*.EXE">; // Binaries
> > >    file in5[] <filesys_mapper; location=@arg("bindata"),
> > >    pattern="*.pl">; // Perl scripts
> > >    file wrapper <single_file_mapper; file="RunDSSAT.sh">; //
> > >    RunDSSAT wrapper
> > >
> > >    (tar_output, part_output) = RunDSSAT(xfile, in1, in2, in3, in4,
> > >    in5, wrapper);
> > > }
> > >
> > > Is there any way to throttle foreach again, or any other workarounds
> > > I could use to avoid this?
> > >
> > > Thanks,
> > > David
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel