[Swift-user] throttling mapping threads

[Allan Espinosa]

Moving the thread to swift-devel.

Ah you're right.  i did a thread trace in my old setup:

$ jstack 8026 | grep "state =" | cat -n
     1  Thread 8051: (state = BLOCKED)
     2  Thread 8050: (state = BLOCKED)
     3  Thread 8047: (state = BLOCKED)
     4  Thread 8045: (state = BLOCKED)
     5  Thread 8043: (state = BLOCKED)
     6  Thread 8042: (state = BLOCKED)
     7  Thread 8041: (state = BLOCKED)
     8  Thread 8040: (state = BLOCKED)
     9  Thread 8034: (state = BLOCKED)
    10  Thread 8033: (state = BLOCKED)
    11  Thread 8032: (state = BLOCKED)
    12  Thread 8026: (state = BLOCKED)

Also I added a line in my ext mapper that creates a files.  after 13
minutes from the time i started the workflow, only 12 files were
created.  i'll check with the new version's swift.log on how long
these files are expected to come.

-Alla

2010/4/14 Mihael Hategan <hategan at mcs.anl.gov>:
> So I looked at the problem in a bit more detail.
>
> The part that starts the external process is run from the karajan worker
> threads which are limited in number (somewhere between 1 and 8)*. So
> that's the maximum number of concurrent invocations of an external
> mapper. Do you actually see all 400 of them run at once?
>
> Mihael
>
> (*) Funny thing that for cooperative multi-tasking such non-cooperating
> tasks as the example above were the thing that prompted the development
> of preemptive multi-tasking. And yet here's an example where, due to
> lazy coding, it turns out to be helpful.
>
> On Tue, 2010-04-13 at 11:52 -0500, Allan Espinosa wrote:
>> oh i was referring to my ext mapper script
>> 1 swift script invoked once, inside of it is a foreach doing 400+ ext mappings.
>>
>> -Allan
>>
>> 2010/4/13 Ben Clifford <benc at hawaga.org.uk>:
>> >> one for the whole script which is invoked a lot of times
>> >
>> >
>> > you're invoking the same swiftscript a whole lot of times?