[Swift-devel] process hogging memory on ranger login

[skenny at uchicago.edu]

>> so, my (~1 million job) workflow, submitted to ranger hangs in
>> this state:
>> 
>> Progress:  Submitted:16383  Finished successfully:55681
>> Progress:  Submitted:16383  Finished successfully:55681
>> 
>> on ranger i have nothing in the queue. but i am showing a
>> process still running on login3:
>>  
>>  8825 tg457040  28  12  472m 232m 5660 S 15.8  0.7
130:56.41 java
>> 
>> i am showing some errors in the stderr.txt of the jobs that
>> were running (they access our database which apparently went
>> down at some point). however, it seems troubling that when the
>> app fails that coaster job is still running on the remote site
>> and the workflow hangs w/o reporting anything...
>> 
>> the log is too large to attach, but is here on ci:
>> 
>>
/ci/projects/cnari/logs/skenny/importDTI-20091124-1655-agj0mze1.log
>> 
>> let me know if you need the coaster log as well.
>
>That may occur at times, such as when the service runs out of
memory. So
>yes, I do need the coaster log.
>
>Regardless of the exact reason, I think that there needs to
be extra
>logic in there to ensure liveness. In other words a lost
state should
>not be interpreted as "still in last state" but "failure".

ok, just added the coasters.log to that same dir:

/ci/projects/cnari/logs/skenny/coasters.log