[Swift-user] throttling mapping threads
Allan Espinosa
aespinosa at cs.uchicago.edu
Wed Apr 14 02:41:07 CDT 2010
Moving the thread to swift-devel.
Ah you're right. i did a thread trace in my old setup:
$ jstack 8026 | grep "state =" | cat -n
1 Thread 8051: (state = BLOCKED)
2 Thread 8050: (state = BLOCKED)
3 Thread 8047: (state = BLOCKED)
4 Thread 8045: (state = BLOCKED)
5 Thread 8043: (state = BLOCKED)
6 Thread 8042: (state = BLOCKED)
7 Thread 8041: (state = BLOCKED)
8 Thread 8040: (state = BLOCKED)
9 Thread 8034: (state = BLOCKED)
10 Thread 8033: (state = BLOCKED)
11 Thread 8032: (state = BLOCKED)
12 Thread 8026: (state = BLOCKED)
Also I added a line in my ext mapper that creates a files. after 13
minutes from the time i started the workflow, only 12 files were
created. i'll check with the new version's swift.log on how long
these files are expected to come.
-Alla
2010/4/14 Mihael Hategan <hategan at mcs.anl.gov>:
> So I looked at the problem in a bit more detail.
>
> The part that starts the external process is run from the karajan worker
> threads which are limited in number (somewhere between 1 and 8)*. So
> that's the maximum number of concurrent invocations of an external
> mapper. Do you actually see all 400 of them run at once?
>
> Mihael
>
> (*) Funny thing that for cooperative multi-tasking such non-cooperating
> tasks as the example above were the thing that prompted the development
> of preemptive multi-tasking. And yet here's an example where, due to
> lazy coding, it turns out to be helpful.
>
> On Tue, 2010-04-13 at 11:52 -0500, Allan Espinosa wrote:
>> oh i was referring to my ext mapper script
>> 1 swift script invoked once, inside of it is a foreach doing 400+ ext mappings.
>>
>> -Allan
>>
>> 2010/4/13 Ben Clifford <benc at hawaga.org.uk>:
>> >> one for the whole script which is invoked a lot of times
>> >
>> >
>> > you're invoking the same swiftscript a whole lot of times?
More information about the Swift-user
mailing list