[Swift-user] GC overhead limit exceeded

[Allan Espinosa]

I do have a large workflow ~400k jobs and tons of mappers and
primitive types that go along each one.

I have no trace.  my workflow is running on the latest trunk:
swift-r3728 cog-r2948


Setting the heap to 8GB somehow solves the problem.  Are there any
tips to estimate heap usage? # of local variables, array sizes, etc.?

2010/12/1 Mihael Hategan <hategan at mcs.anl.gov>:
> Something is very likely wrong if things can't fit in 4G.
>
> Usual suspects:
> -very large workflows.
> -lots of trace/tracef
> -Justin reported some problems with coaster I/O on the BGP, so it might
> be that (update to latest trunk and see if that solves it).
>
> Mihael
>
> On Wed, 2010-12-01 at 20:38 -0600, Allan Espinosa wrote:
>> I have my heap set to 4GB does this mean i still need more memory?
>>
>> HEAPMAX=4096M
>>
>>
>> -Allan
>>
>> Progress:  Finished in previous run:4002
>> Uncaught exception: java.lang.OutOfMemoryError: GC overhead limit
>> exceeded in swiftscript:strcat @ postproc.kml, line: 407
>> java.lang.OutOfMemoryError: GC overhead limit exceeded
>>         at edu.emory.mathcs.backport.java.util.concurrent.AbstractExecutorService.newTaskFor(AbstractExecutorService.java:59)
>>         at edu.emory.mathcs.backport.java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:78)
>>         at org.globus.cog.karajan.workflow.events.EventBus._post(EventBus.java:74)
>>         at org.globus.cog.karajan.workflow.events.EventBus.post(EventBus.java:97)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.fireControlEvent(FlowNode.java:187)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.startElement(FlowNode.java:467)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.startElement(Sequential.java:84)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:57)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.childCompleted(Sequential.java:44)
>>         at org.globus.cog.karajan.workflow.nodes.AbstractSequentialWithArguments.childCompleted(AbstractSequentialWithArguments.java:195)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.notificationEvent(Sequential.java:32)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:340)
>>         at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.fireNotificationEvent(FlowNode.java:181)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.complete(FlowNode.java:309)
>>         at org.globus.cog.karajan.workflow.nodes.FlowContainer.post(FlowContainer.java:58)
>>         at org.globus.cog.karajan.workflow.nodes.functions.AbstractFunction.post(AbstractFunction.java:28)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.startNext(Sequential.java:50)
>>         at org.globus.cog.karajan.workflow.nodes.Sequential.executeChildren(Sequential.java:26)
>>         at org.globus.cog.karajan.workflow.nodes.FlowContainer.execute(FlowContainer.java:63)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.restart(FlowNode.java:238)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.start(FlowNode.java:289)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.controlEvent(FlowNode.java:402)
>>         at org.globus.cog.karajan.workflow.nodes.FlowNode.event(FlowNode.java:343)
>>         at org.globus.cog.karajan.workflow.FlowElementWrapper.event(FlowElementWrapper.java:230)
>>         at org.globus.cog.karajan.workflow.events.EventBus.send(EventBus.java:173)
>>         at org.globus.cog.karajan.workflow.events.EventTargetPair.run(EventTargetPair.java:44)
>>         at edu.emory.mathcs.backport.java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:431)
>>         at edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:166)
>>         at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
>>         at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
>>         at java.lang.Thread.run(Thread.java:619)
>> Event was NotificationEvent:EXECUTION_COMPLETED
>> Exception is: java.lang.OutOfMemoryError: GC overhead limit exceeded
>> Near Karajan line: swiftscript:strcat @ postproc.kml, line: 407
>> Execution failed:
>>         Uncaught exception: java.lang.OutOfMemoryError: GC overhead
>> limit exceeded
>> Exception in thread "Overloaded Host Monitor"
>> java.lang.OutOfMemoryError: Java heap space
>>         at java.util.HashMap.newKeyIterator(HashMap.java:840)
>>         at java.util.HashMap$KeySet.iterator(HashMap.java:874)
>>         at java.util.HashSet.iterator(HashSet.java:153)
>>         at org.globus.cog.karajan.scheduler.OverloadedHostMonitor.run(OverloadedHostMonitor.java:50)
>> Progress:  Finished in previous run:4002