[Swift-devel] foreach.max.threads question

[David Kelly]

I set the max threads to 520 and saw the same behavior. The only difference is that I had to do the work on OSG rather than UC3 since all nodes were in use. Is it possible that since there is no foreach throttle, all the files and file arrays needed for each of the 120K tasks are getting set at once? 

All of the log files, configuration files, and a heap dump are at /scratch/local/davidk/DSSAT/run067 on communicado.

Thanks,
David

----- Original Message -----
> From: "Mihael Hategan" <hategan at mcs.anl.gov>
> To: "David Kelly" <davidk at ci.uchicago.edu>
> Cc: "swift-devel Devel" <swift-devel at ci.uchicago.edu>
> Sent: Tuesday, October 16, 2012 2:44:59 PM
> Subject: Re: [Swift-devel] foreach.max.threads question
> The default is 16384. Can you set foreach.max.threads to 520 and try
> again?
> 
> On Tue, 2012-10-16 at 14:25 -0500, David Kelly wrote:
> > Swift and coaster-service are running on communicado/bridled and the
> > work is being done on UC3 via persistent coasters. UC3 has 520
> > cores, 1 core per node. In most cases I will get all 520 cores.
> >
> > ----- Original Message -----
> > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > To: "David Kelly" <davidk at ci.uchicago.edu>
> > > Cc: "swift-devel Devel" <swift-devel at ci.uchicago.edu>
> > > Sent: Tuesday, October 16, 2012 2:17:06 PM
> > > Subject: Re: [Swift-devel] foreach.max.threads question
> > > How many cores do you run this on?
> > >
> > > On Tue, 2012-10-16 at 13:56 -0500, David Kelly wrote:
> > > > Previously it was not explicitly set, so I am assuming it would
> > > > have
> > > > been 1024. As a test I tried setting it to 520 (the maximum
> > > > number
> > > > of available cores), but that did not seem to help.
> > > >
> > > > ----- Original Message -----
> > > > > From: "Mihael Hategan" <hategan at mcs.anl.gov>
> > > > > To: "David Kelly" <davidk at ci.uchicago.edu>
> > > > > Cc: "swift-devel Devel" <swift-devel at ci.uchicago.edu>
> > > > > Sent: Tuesday, October 16, 2012 1:44:47 PM
> > > > > Subject: Re: [Swift-devel] foreach.max.threads question
> > > > > What was foreach.max.threads set to before?
> > > > >
> > > > > On Tue, 2012-10-16 at 11:04 -0500, David Kelly wrote:
> > > > > > Hello,
> > > > > >
> > > > > > I have noticed that since the foreach.max.threads changes,
> > > > > > the
> > > > > > DSSAT
> > > > > > script is now running out of memory. I have the heap size
> > > > > > set to
> > > > > > 4
> > > > > > gigabytes. There are 120K items in gridLists.
> > > > > > The main foreach loop of the script looks like this:
> > > > > >
> > > > > > foreach g,i in gridLists {
> > > > > >    file tar_output <single_file_mapper;
> > > > > >    file=@strcat("output/",
> > > > > >    gridLists[i], "output.tar.gz")>;
> > > > > >    file part_output <single_file_mapper;
> > > > > >    file=@strcat("parts/",
> > > > > >    gridLists[i], ".part")>;
> > > > > >
> > > > > >    file in1[] <filesys_mapper;
> > > > > >    location=@strcat(@arg("scenarios"),
> > > > > >    "/", gridLists[i]), pattern="*">; // Scenario files
> > > > > >    file in2[] <filesys_mapper;
> > > > > >    location=@strcat(@arg("weather"),
> > > > > >    "/", gridLists[i]), pattern="*">; // Weather files
> > > > > >    file in3[] <filesys_mapper; location=@arg("refdata"),
> > > > > >    pattern="*">; // Common data
> > > > > >    file in4[] <filesys_mapper; location=@arg("bindata"),
> > > > > >    pattern="*.EXE">; // Binaries
> > > > > >    file in5[] <filesys_mapper; location=@arg("bindata"),
> > > > > >    pattern="*.pl">; // Perl scripts
> > > > > >    file wrapper <single_file_mapper; file="RunDSSAT.sh">; //
> > > > > >    RunDSSAT wrapper
> > > > > >
> > > > > >    (tar_output, part_output) = RunDSSAT(xfile, in1, in2,
> > > > > >    in3,
> > > > > >    in4,
> > > > > >    in5, wrapper);
> > > > > > }
> > > > > >
> > > > > > Is there any way to throttle foreach again, or any other
> > > > > > workarounds
> > > > > > I could use to avoid this?
> > > > > >
> > > > > > Thanks,
> > > > > > David
> > > > > > _______________________________________________
> > > > > > Swift-devel mailing list
> > > > > > Swift-devel at ci.uchicago.edu
> > > > > > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel