[Swift-user] foreach dilemma (was Re: Heap space being exhausted)

[Allan Espinosa]

I looked again at my mainloop to figure out why are there not enough
jobs from being called in the inner  loop despite the huge number of
availble workers.

Answer: not enough ready jobs! :(

I set my foreach.max.threads=100 so there can only be 100 outerloop
jobs (6k total)  at a time.  these outerloop jobs is a dependency for
the innerloop jobs so there will be a slow start of the number of
ready jobs.

Another bottleneck is my data mapper wrapped as an app.  since it's
only a local job, the default throttle is set to 4 at a time. I made
this work around because of synchronization issues on mappers a few
months ago. I think it would be good to look at the previous version
of my workflow again.

-Allan


2010/6/2 Mihael Hategan <hategan at mcs.anl.gov>:
> On Wed, 2010-06-02 at 19:03 -0500, Allan Espinosa wrote:
>> My Cybershake workflow.  its basically a 2 level for loop with varying
>> inner  loop sizes.
>>
>> foreach i in (~4k elements) {
>>   x = f();
>>   foreach (20-2k elements) {
>>      ...
>>  }
>> }
>
> Yep. You have a winner. Max threads = 1024 * 1024.
>
> You should adjust that parameter accordingly. I.e. foreach.max.threads =
> sqrt(maxTotalThreads).
>
>>
>> 2010/6/2 Mihael Hategan <hategan at mcs.anl.gov>:
>> > On Wed, 2010-06-02 at 18:40 -0500, Allan Espinosa wrote:
>> >> I tried a HEAPMAX of 4GB.
>> >>
>> >> No memory problems so far :)
>> >
>> > Still odd. What's the swift script?
>> >
>> > I'm asking because foreach.max.threads should work, but it applies to
>> > each individual foreach rather than globally.
>> >
>> >>
>> >> 2010/6/2 Mihael Hategan <hategan at mcs.anl.gov>:
>> >> > On Wed, 2010-06-02 at 17:47 -0500, Mihael Hategan wrote:
>> >> >> On Wed, 2010-06-02 at 17:32 -0500, Allan Espinosa wrote:
>> >> >> > btw
>> >> >> >
>> >> >> > foreach.maxthreads=1024
>> >> >>
>> >> >> or more heapmax.
>> >> >
>> >> > Ehm, I though you found a solution :)
>> >> >
>> >> > What's the swift script?
>> >
>
>
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>