[Swift-devel] Re: Memory in Swift

[jon.monette at gmail.com]

Ok. so I only have to make sure it is set in the shell in which I call swift? It doesn't have to be set for the remote jobs?
Sent on the Sprint® Now Network from my BlackBerry®

-----Original Message-----
From: Justin M Wozniak <wozniak at mcs.anl.gov>
Date: Mon, 18 Oct 2010 17:14:52 
To: Jonathan Monette<jon.monette at gmail.com>
Cc: Swift Devel<swift-devel at ci.uchicago.edu>
Subject: Re: Memory in Swift


Actually, to set SWIFT_HEAP_MAX for Swift you should export it in the 
environment in which you run bin/swift .  So, in the shell you could do:

SWIFT_HEAP_MAX=8192M swift ...

The namespace env is there to let you set the environment for the remote 
user job.

On Mon, 18 Oct 2010, Jonathan Monette wrote:

> jonmon   22946 54.6  1.2 788800 318496 pts/17  Sl+  16:42   0:39 java 
> -Xmx256M
>
> I am assuming that this means the heap space is 256 Megabytes.  Here is how I 
> set the SWIFT_HEAP_MAX variable:
>
> <profile key="SWIFT_HEAP_MAX" namespace="env">8192M</profile>
>
> Is this how you would set environment variables in the sites.xml file?  I 
> want to make sure the variable is also set for the worker nodes.
>
> On 10/18/2010 04:36 PM, Justin M Wozniak wrote:
>> 
>> What do you get if you set it to 8192M?
>> 
>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>> 
>>> I am on PADS.
>>> 
>>> On 10/18/2010 01:21 PM, Justin M Wozniak wrote:
>>>> On Mon, 18 Oct 2010, Jonathan Monette wrote:
>>>>
>>>>>    1) What is the maximum value for SWIFT_HEAP_MAX?  Is it the maximum 
>>>>> memory available on the machine?  If I ask for 8Gb will the Java heap 
>>>>> space be 8Gb?
>>>> 
>>>> In bin/swift, that value is passed directly to the Java VM, so it's 
>>>> implementation-specific.  What kind of machine are you on?
>>>>
>>>>>    2) Assuming that the above question does not fix my problem, what can 
>>>>> be done to fix memory problems in Swift?  How should a Swift script be 
>>>>> programmed to avoid these issues or is how Swift handles memory not 
>>>>> optimal. I have a set of about 4100 images that will be stitched 
>>>>> together to form a mosaic.  The size of this set is 8.1Gb.  Right now in 
>>>>> my sites file I set SWIFT_HEAP_MAX to 4096M(4Gb) but I am still getting 
>>>>> out of memory errors with the Java heap space.  Is Swift just not able 
>>>>> to scale up to this size of data yet?
>>>> 
>>> 
>>> 
>> 
>
>

-- 
Justin M Wozniak