[Swift-devel] [swift-support] Channel multiplexer error

[Mihael Hategan]

I spoke to Mike on the phone on Friday, and we agreed that
foreach.max.threads is a bit difficult to use.

So I removed the throttling for foreach and added it to app invocations.
It might help with memory consumption if you set it to around the number
of cpus you have access to.

I also did some small optimizations to improve memory use. I can run
about 50K jobs with your script on a 32 bit system with 1GB of heap
space. I suspect that on 64 bit systems this might require more heap.

I also added an option to the swift executable to automatically dump a
copy of the heap when an out of memory condition occurs. Hopefully that
will help us troubleshoot such problems in the future.

This is all in trunk.

Mihael

On Sat, 2012-10-13 at 04:47 -0500, Kazutaka Takahashi wrote:
> Hi All, 
> 
> Sorry for a late reply, but I am already at a conference and had not
> had a chance to try what Mike proposed. I will try later during the
> second half of the conference, if not after the conference ending next
> wed. 
> Taka
> 
> 
> On Thu, Oct 11, 2012 at 7:12 PM, Mihael Hategan <hategan at mcs.anl.gov>
> wrote:
>         Did you try what Mike proposed?
>         
>         Mihael
>         
>         On Thu, 2012-10-11 at 17:49 -0500, Kazutaka Takahashi wrote:
>         > OK...
>         > The last one died with the same error msg... Please check
>         the both
>         > directories...
>         >
>         >
>         taka at login4:/lustre/beagle/GCNet/RG/Athena/a080521_BS_new/Cond2>
>         >
>         taka at login4:/lustre/beagle/GCNet/RG/Athena/a080521_BS_new/Cond3>
>         >
>         > /lustre/beagle/GCNet/bin/Swift/swift: line 164:  8527 Killed
>         > java -Xmx5120M
>         -Djava.endorsed.dirs=/soft/swift/0.93//lib/endorsed
>         > -DUID=3023 -DGLOBUS_HOSTNAME=login4.beagle.ci.uchicago.edu
>         > -DCOG_INSTALL_PATH=/soft/swift/0.93/
>         -Dswift.home=/soft/swift/0.93/
>         > -Duser.home=/lustre/beagle/GCNet
>         > -Djava.security.egd=file:///dev/urandom -XX:+UseParallelGC
>         > -XX:ParallelGCThreads=1
>         >
>         -classpath /soft/swift/0.93//etc:/soft/swift/0.93//libexec:/soft/swift/0.93//lib/addressing-1.0.jar:/soft/swift/0.93//lib/ant.jar:/soft/swift/0.93//lib/antlr-2.7.5.jar:/soft/swift/0.93//lib/axis.jar:/soft/swift/0.93//lib/axis-url.jar:/soft/swift/0.93//lib/castor-0.9.6.jar:/soft/swift/0.93//lib/coaster-bootstrap.jar:/soft/swift/0.93//lib/cog-abstraction-common-2.4.jar:/soft/swift/0.93//lib/cog-axis.jar:/soft/swift/0.93//lib/cog-grapheditor-0.47.jar:/soft/swift/0.93//lib/cog-jglobus-1.7.0.jar:/soft/swift/0.93//lib/cog-karajan-0.36-dev.jar:/soft/swift/0.93//lib/cog-provider-clref-gt4_0_0.jar:/soft/swift/0.93//lib/cog-provider-coaster-0.3.jar:/soft/swift/0.93//lib/cog-provider-dcache-0.1.jar:/soft/swift/0.93//lib/cog-provider-gt2-2.4.jar:/soft/swift/0.93//lib/cog-provider-gt4_0_0-2.5.jar:/soft/swift/0.93//lib/cog-provider-local-2.2.jar:/soft/swift/0.93//lib/cog-provider-localscheduler-0.4.jar:/soft/swift/0.93//lib/cog-provider-ssh-2.4.jar:/soft/swift/0.93//lib/cog-provider-webdav-2.1.jar:/soft/swift/0.93//lib/cog-resources-1.0.jar:/soft/swift/0.93//lib/cog-swift-svn.jar:/soft/swift/0.93//lib/cog-trap-1.0.jar:/soft/swift/0.93//lib/cog-url.jar:/soft/swift/0.93//lib/cog-util-0.92.jar:/soft/swift/0.93//lib/commonj.jar:/soft/swift/0.93//lib/commons-beanutils.jar:/soft/swift/0.93//lib/commons-collections-3.0.jar:/soft/swift/0.93//lib/commons-digester.jar:/soft/swift/0.93//lib/commons-discovery.jar:/soft/swift/0.93//lib/commons-httpclient.jar:/soft/swift/0.93//lib/commons-logging-1.1.jar:/soft/swift/0.93//lib/concurrent.jar:/soft/swift/0.93//lib/cryptix32.jar:/soft/swift/0.93//lib/cryptix-asn1.jar:/soft/swift/0.93//lib/cryptix.jar:/soft/swift/0.93//lib/globus_delegation_service.jar:/soft/swift/0.93//lib/globus_delegation_stubs.jar:/soft/swift/0.93//lib/globus_wsrf_mds_aggregator_stubs.jar:/soft/swift/0.93//lib/globus_wsrf_rendezvous_service.jar:/soft/swift/0.93//lib/globus_wsrf_rendezvous_stubs.jar:/soft/swift/0.93//lib/globus_wsrf_rft_stubs.jar:/soft/swift/0.93//lib/gram-client.jar:/soft/swift/0.93//lib/gram-stubs.jar:/soft/swift/0.93//lib/gram-utils.jar:/soft/swift/0.93//lib/j2ssh-common-0.2.2.jar:/soft/swift/0.93//lib/j2ssh-core-0.2.2-patch-b.jar:/soft/swift/0.93//lib/jakarta-regexp-1.2.jar:/soft/swift/0.93//lib/jakarta-slide-webdavlib-2.0.jar:/soft/swift/0.93//lib/jaxrpc.jar:/soft/swift/0.93//lib/jce-jdk13-131.jar:/soft/swift/0.93//lib/jgss.jar:/soft/swift/0.93//lib/jline-0.9.94.jar:/soft/swift/0.93//lib/jsr173_1.0_api.jar:/soft/swift/0.93//lib/jug-lgpl-2.0.0.jar:/soft/swift/0.93//lib/junit.jar:/soft/swift/0.93//lib/log4j-1.2.16.jar:/soft/swift/0.93//lib/naming-common.jar:/soft/swift/0.93//lib/naming-factory.jar:/soft/swift/0.93//lib/naming-java.jar:/soft/swift/0.93//lib/naming-resources.jar:/soft/swift/0.93//lib/opensaml.jar:/soft/swift/0.93//lib/puretls.jar:/soft/swift/0.93//lib/resolver.jar:/soft/swift/0.93//lib/saaj.jar:/soft/swift/0.93//lib/stringtemplate.jar:/soft/swift/0.93//lib/vdldefinitions.jar:/soft/swift/0.93//lib/wsdl4j.jar:/soft/swift/0.93//lib/wsrf_core.jar:/soft/swift/0.93//lib/wsrf_core_stubs.jar:/soft/swift/0.93//lib/wsrf_mds_index_stubs.jar:/soft/swift/0.93//lib/wsrf_mds_usefulrp_schema_stubs.jar:/soft/swift/0.93//lib/wsrf_provider_jce.jar:/soft/swift/0.93//lib/wsrf_tools.jar:/soft/swift/0.93//lib/wss4j.jar:/soft/swift/0.93//lib/xalan.jar:/soft/swift/0.93//lib/xbean.jar:/soft/swift/0.93//lib/xbean_xpath.jar:/soft/swift/0.93//lib/xercesImpl.jar:/soft/swift/0.93//lib/xml-apis.jar:/soft/swift/0.93//lib/xmlsec.jar:/soft/swift/0.93//lib/xpp3-1.1.3.4d_b4_min.jar:/soft/swift/0.93//lib/xstream-1.1.1-patched.jar: org.griphyn.vdl.karajan.Loader '-config' 'demo_realcf.cf' '-sites.file' 'demo_realSites.xml' '-tc.file' 'demo_realtc.tc' 'demo_real.swift'
>         >
>         >
>         
>         
>         
> 
> 
> 
> -- 
> What is essential is invisible to the eye
>