[Swift-devel] java.lang.OutOfMemoryError when running grep on 10k files

[Gabri Turcu]

Hi Glen,

Thanks a lot! I can now see in the log file that what I had was not working.
It seems to be on its way now.

Best,
Gabri

On Mon, Apr 13, 2009 at 6:45 PM, Glen Hocky <hockyg at uchicago.edu> wrote:

> I did the same but in COG_OPTS, i.e.
> export COG_OPTS="-Xmx1024m"
> or 2048 for really big jobs
>
> Gabri Turcu wrote:
>
>> Hi,
>>
>> I am trying to run grep on newslab data on teraport. While everything
>> works fine for a small number of patterns and files(e.g. ~20 patterns, ~1000
>> files), I get errors for larger numbers of files (~10k). I would be very
>> grateful for any help.
>>
>> The main files I'm using are (at CI) :
>> /home/gabri/swift-0.8/examples/swift/newslabex/count/tc.data
>> /home/gabri/swift-0.8/examples/swift/newslabex/count/sites.xml  (-using
>> the fast queue)
>> /home/gabri/swift-0.8/examples/swift/newslabex/count/count.swift
>> /home/gabri/swift-0.8/examples/swift/newslabex/count/grp
>>
>> For number of files=10k and number of patterns=2
>> - I'm getting an "java.lang.OutOfMemoryError". I have tried increasing the
>> heap size by runnig Swift with (-Xms1536m -Xmx4096m) in the command line,
>> but that seemed to just push the failure point a little further. Is this at
>> all the way to go?
>> - The corresponding logs are at:
>> /home/gabri/swift-0.8/examples/swift/newslabex/count/errmanyfiles/
>>
>> Thank you very much for any suggestions.
>> Best,
>> Gabri
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> Swift-devel mailing list
>> Swift-devel at ci.uchicago.edu
>> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20090413/2f5f76a6/attachment.html>