[Swift-devel] Coaster test failed at 86K of 100K jobs

[Michael Wilde]

Mihael,

I was running catsn.swift with 100K jobs (-n=100000) to a single-server persistent coaster pool to about 50 OSG worker nodes. Using 0.93.

It failed after about 86K jobs with this error:

Submitted:82  Active:2  Finished successfully:86521
Progress:  time: Fri, 05 Aug 2011 22:15:50 -0500  Selecting site:921  Submitting:16  Submitted:83  Active:2  Finished successfully:86531
Progress:  time: Fri, 05 Aug 2011 22:15:51 -0500  Selecting site:922  Submitting:12  Submitted:76  Active:13  Finished successfully:86534
Progress:  time: Fri, 05 Aug 2011 22:15:54 -0500  Selecting site:918  Submitting:16  Submitted:83  Active:1  Finished successfully:86548
Execution failed:
        java.util.ConcurrentModificationException


The first exception in the logs shows:

2011-08-05 22:15:54,845-0500 DEBUG vdl:mains FOREACH_IT_END line=9 thread=0-3-87187
2011-08-05 22:15:54,845-0500 DEBUG VDL2ExecutionContext java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
Caused by: java.util.ConcurrentModificationException
        at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
        at java.util.AbstractList$Itr.next(AbstractList.java:343)
        at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.getSummary(RuntimeStats.java:177)
        at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.printStates(RuntimeStats.java:194)
        at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.dumpState(RuntimeStats.java:159)
        at org.griphyn.vdl.karajan.lib.RuntimeStats.setProgress(RuntimeStats.java:88)
        at org.griphyn.vdl.karajan.lib.RuntimeStats.vdl_setprogress(RuntimeStats.java:82)
        at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)


Ive moved the logs to: /home/wilde/swiftgrid/test.swift-workers/logs.05

- Mike