[Swift-devel] Re: Memory in Swift

[Mihael Hategan]

A side note: I think that 32 bit JVMs have a limit of around 3GB (the
limit for a 32 bit process).

You should also be wary of the following possibility: when a process is
forked in linux, it first inherits the parent's heap size (so total VM
is double of the original process). That gets quickly dropped by the
System.exec() implementation as soon as the process is started, but for
a small window of time (and with multiple concurrent forks the problem
may get worse), there is a chance that you will run out of virtual
memory.

Mihael

On Mon, 2010-10-18 at 20:22 -0500, Jonathan Monette wrote:
> Ok.  That seemed to fix my out of memory problem.
> 
> On 10/18/2010 05:14 PM, Justin M Wozniak wrote:
> >
> > Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
> > environment in which you run bin/swift .  So, in the shell you could do:
> >
> > SWIFT_HEAP_MAX=8192M swift ...
> >
> > The namespace env is there to let you set the environment for the 
> > remote user job.
> >
> > On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >
> >> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
> >> -Xmx256M
> >>
> >> I am assuming that this means the heap space is 256 Megabytes.  Here 
> >> is how I set the SWIFT_HEAP_MAX variable:
> >>
> >> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
> >>
> >> Is this how you would set environment variables in the sites.xml 
> >> file?  I want to make sure the variable is also set for the worker 
> >> nodes.
> >>
> >> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
> >>>
> >>> What do you get if you set it to 8192M?
> >>>
> >>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >>>
> >>>> I am on PADS.
> >>>>
> >>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
> >>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
> >>>>>
> >>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the 
> >>>>>> maximum memory available on the machine?  If I ask for 8Gb will 
> >>>>>> the Java heap space be 8Gb?
> >>>>>
> >>>>> In bin/swift, that value is passed directly to the Java VM, so 
> >>>>> it's implementation-specific.  What kind of machine are you on?
> >>>>>
> >>>>>>    2) Assuming that the above question does not fix my problem, 
> >>>>>> what can be done to fix memory problems in Swift?  How should a 
> >>>>>> Swift script be programmed to avoid these issues or is how Swift 
> >>>>>> handles memory not optimal. I have a set of about 4100 images 
> >>>>>> that will be stitched together to form a mosaic.  The size of 
> >>>>>> this set is 8.1Gb.  Right now in my sites file I set 
> >>>>>> SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting out of memory 
> >>>>>> errors with the Java heap space.  Is Swift just not able to scale 
> >>>>>> up to this size of data yet?
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >
>