[Swift-devel] Coaster test failed at 86K of 100K jobs

[Mihael Hategan]

Amazing how that bug in what would otherwise be a relatively simple
class (CopyOnWriteArrayList) has managed to survive so long. Concurrency
ain't easy!

I'll have a fix committed after I do a bit of testing.

On Fri, 2011-08-05 at 23:27 -0500, Michael Wilde wrote:
> Mihael,
> 
> I was running catsn.swift with 100K jobs (-n=100000) to a single-server persistent coaster pool to about 50 OSG worker nodes. Using 0.93.
> 
> It failed after about 86K jobs with this error:
> 
> Submitted:82  Active:2  Finished successfully:86521
> Progress:  time: Fri, 05 Aug 2011 22:15:50 -0500  Selecting site:921  Submitting:16  Submitted:83  Active:2  Finished successfully:86531
> Progress:  time: Fri, 05 Aug 2011 22:15:51 -0500  Selecting site:922  Submitting:12  Submitted:76  Active:13  Finished successfully:86534
> Progress:  time: Fri, 05 Aug 2011 22:15:54 -0500  Selecting site:918  Submitting:16  Submitted:83  Active:1  Finished successfully:86548
> Execution failed:
>         java.util.ConcurrentModificationException
> 
> 
> The first exception in the logs shows:
> 
> 2011-08-05 22:15:54,845-0500 DEBUG vdl:mains FOREACH_IT_END line=9 thread=0-3-87187
> 2011-08-05 22:15:54,845-0500 DEBUG VDL2ExecutionContext java.util.ConcurrentModificationException
> java.util.ConcurrentModificationException
> Caused by: java.util.ConcurrentModificationException
>         at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
>         at java.util.AbstractList$Itr.next(AbstractList.java:343)
>         at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.getSummary(RuntimeStats.java:177)
>         at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.printStates(RuntimeStats.java:194)
>         at org.griphyn.vdl.karajan.lib.RuntimeStats$ProgressTicker.dumpState(RuntimeStats.java:159)
>         at org.griphyn.vdl.karajan.lib.RuntimeStats.setProgress(RuntimeStats.java:88)
>         at org.griphyn.vdl.karajan.lib.RuntimeStats.vdl_setprogress(RuntimeStats.java:82)
>         at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>         at java.lang.reflect.Method.invoke(Method.java:597)
> 
> 
> Ive moved the logs to: /home/wilde/swiftgrid/test.swift-workers/logs.05
> 
> - Mike
> 
> 
>