[Swift-devel] Deadlocks running ParVis script under 0.93

[Michael Wilde]

Mihael, Im getting Java-level deadlocks in running a ParVis script. The user is seeming this as well (on at least one instance).

Im running on Fusion in the directory /home/wilde/amwg/run01.

The script is complex, it should run >325 app calls (Im not yet sure how many - Im guessing at least 100 more).

The two runs (swift work dirs) that show the deadlocks are:

fusion$ ls -lt */jstack.out
-rw-r--r-- 1 wilde mcsz  80249 Sep 12 17:39 amwg_stats-20110912-1546-5aqzkvhe/jstack.out
-rw-r--r-- 1 wilde mcsz 135539 Sep 11 10:38 amwg_stats-20110911-1033-fd1brig2/jstack.out
fusion$ 

The log files are in the run01 directory. The Swift stdout progress logs are in the top of the respective swift work dirs.

One of my runs, amwg_stats-20110912-1546-5aqzkvhe, (as well as one of the user's runs) hung after 323 app calls with this Java deadlock:

Found one Java-level deadlock:
=============================
"pool-1-thread-32":
  waiting to lock monitor 0x000000005ccf97b0 (object 0x00002aaab56f0e30, a org.griphyn.vdl.mapping.RootDataNode),
  which is held by "pool-1-thread-11"
"pool-1-thread-11":
  waiting to lock monitor 0x000000005ce1cad8 (object 0x00002aaab5a50768, a org.griphyn.vdl.karajan.DSHandleFutureWrapper),
  which is held by "pool-1-thread-32"

Java stack information for the threads listed above:
===================================================
"pool-1-thread-32":
        at org.griphyn.vdl.karajan.lib.SwiftArg.unwrap(SwiftArg.java:52)
        - waiting to lock <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
        at org.griphyn.vdl.karajan.lib.SwiftArg$Vargs.asArray(SwiftArg.java:177)
        at org.griphyn.vdl.karajan.lib.swiftscript.Misc.swiftscript_strcat(Misc.java:82)
        at sun.reflect.GeneratedMethodAccessor9.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.globus.cog.karajan.workflow.nodes.functions.FunctionsCollection.function(FunctionsCollection.java:82)
        at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:27)
...
"pool-1-thread-11":
        at org.griphyn.vdl.karajan.DSHandleFutureWrapper.notifyListeners(DSHandleFutureWrapper.java:68)
        - waiting to lock <0x00002aaab5a50768> (a org.griphyn.vdl.karajan.DSHandleFutureWrapper)
        at org.griphyn.vdl.karajan.DSHandleFutureWrapper.handleClosed(DSHandleFutureWrapper.java:122)
        at org.griphyn.vdl.mapping.AbstractDataNode.notifyListeners(AbstractDataNode.java:605)
        at org.griphyn.vdl.mapping.AbstractDataNode.closeShallow(AbstractDataNode.java:408)
        - locked <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
        at org.griphyn.vdl.mapping.AbstractDataNode.setValue(AbstractDataNode.java:358)
        at org.griphyn.vdl.mapping.RootDataNode.setValue(RootDataNode.java:227)
        at org.griphyn.vdl.karajan.lib.SetFieldValue.deepCopy(SetFieldValue.java:90)
        at org.griphyn.vdl.karajan.lib.SetFieldValue.function(SetFieldValue.java:49)
        - locked <0x00002aaab56f0e30> (a org.griphyn.vdl.mapping.RootDataNode)
        at org.griphyn.vdl.karajan.lib.VDLFunction.post(VDLFunction.java:67)
        at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.completed(AbstractSequentialWithArguments.java:194)
        at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:214)


---
The other script hung after only about 20 app calls with two deadlocks, one of which is:

Found one Java-level deadlock:
=============================
"pool-1-thread-32":
  waiting to lock monitor 0x000000005b7a1da8 (object 0x00002aaab46d7490, a org.griphyn.vdl.mapping.RootDataNode),
  which is held by "pool-1-thread-26"
"pool-1-thread-26":
  waiting to lock monitor 0x000000005b5e6620 (object 0x00002aaab46d63d0, a org.griphyn.vdl.karajan.WrapperMap),
  which is held by "pool-1-thread-10"
"pool-1-thread-10":
  waiting to lock monitor 0x000000005b4e47e0 (object 0x00002aaac2015f90, a org.griphyn.vdl.mapping.RootArrayDataNode),
  which is held by "pool-1-thread-15"
"pool-1-thread-15":
  waiting to lock monitor 0x000000005b5e6620 (object 0x00002aaab46d63d0, a org.griphyn.vdl.karajan.WrapperMap),
  which is held by "pool-1-thread-10"

---


-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory