[Swift-devel] Deadlocks running ParVis script under 0.93

[Mihael Hategan]

This is a rather old 0.93. One of the deadlocks was fixed a while ago.
I'm investigating the other one.

On Mon, 2011-09-12 at 18:04 -0500, Michael Wilde wrote:
> Mihael, Im getting Java-level deadlocks in running a ParVis script. The user is seeming this as well (on at least one instance).
> 
> Im running on Fusion in the directory /home/wilde/amwg/run01.
> 
> The script is complex, it should run >325 app calls (Im not yet sure how many - Im guessing at least 100 more).
> 
> The two runs (swift work dirs) that show the deadlocks are:
> 
> fusion$ ls -lt */jstack.out
> -rw-r--r-- 1 wilde mcsz  80249 Sep 12 17:39 amwg_stats-20110912-1546-5aqzkvhe/jstack.out
> -rw-r--r-- 1 wilde mcsz 135539 Sep 11 10:38 amwg_stats-20110911-1033-fd1brig2/jstack.out
> fusion$ 
> 
> The log files are in the run01 directory. The Swift stdout progress logs are in the top of the respective swift work dirs.
> 
> One of my runs, amwg_stats-20110912-1546-5aqzkvhe, (as well as one of the user's runs) hung after 323 app calls with this Java deadlock:
> 
> Found one Java-level deadlock:
> =============================
> "pool-1-thread-32":
>   waiting to lock monitor 0x000000005ccf97b0 (object 0x00002aaab56f0e30, a org.griphyn.vdl.mapping.RootDataNode),
>   which is held by "pool-1-thread-11"
> "pool-1-thread-11":
>   waiting to lock monitor 0x000000005ce1cad8 (object 0x00002aaab5a50768, a org.griphyn.vdl.karajan.DSHandleFutureWrapper),
>   which is held by "pool-1-thread-32"
> 
> Java stack information for the threads listed above:
> ===================================================
> "pool-1-thread-32":
>         at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:52)
>         - waiting to lock <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
>         at org.griphyn.vdl.karajan.lib.SwiftArg$Vargs.asArray(SwiftArg.java:177)
>         at org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_strcat(Misc.java:82)
>         at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
>         at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82)
>         at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
> ...
> "pool-1-thread-11":
>         at org.griphyn.vdl.karajan.DSHandleFutureWrapper.notifyListeners(DSHandleFutureWrapper.java:68)
>         - waiting to lock <0x00002aaab5a50768> (a org.griphyn.vdl.karajan.DSHandleFutureWrapper)
>         at org.griphyn.vdl.karajan.DSHandleFutureWrapper.handleClosed(DSHandleFutureWrapper.java:122)
>         at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:605)
>         at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:408)
>         - locked <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
>         at org.griphyn.vdl.mapping.AbstractDataNode.setValue(AbstractDataNode.java:358)
>         at org.griphyn.vdl.mapping.RootDataNode.setValue(RootDataNode.java:227)
>         at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:90)
>         at org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:49)
>         - locked <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
>         at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
>         at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
>         at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)
> 
> 
> ---
> The other script hung after only about 20 app calls with two deadlocks, one of which is:
> 
> Found one Java-level deadlock:
> =============================
> "pool-1-thread-32":
>   waiting to lock monitor 0x000000005b7a1da8 (object 0x00002aaab46d7490, a org.griphyn.vdl.mapping.RootDataNode),
>   which is held by "pool-1-thread-26"
> "pool-1-thread-26":
>   waiting to lock monitor 0x000000005b5e6620 (object 0x00002aaab46d63d0, a org.griphyn.vdl.karajan.WrapperMap),
>   which is held by "pool-1-thread-10"
> "pool-1-thread-10":
>   waiting to lock monitor 0x000000005b4e47e0 (object 0x00002aaac2015f90, a org.griphyn.vdl.mapping.RootArrayDataNode),
>   which is held by "pool-1-thread-15"
> "pool-1-thread-15":
>   waiting to lock monitor 0x000000005b5e6620 (object 0x00002aaab46d63d0, a org.griphyn.vdl.karajan.WrapperMap),
>   which is held by "pool-1-thread-10"
> 
> ---
> 
>