[Swift-devel] java.lang.OutOfMemoryError when running grep on 10k files

[Gabri Turcu]

Hi,

I am trying to run grep on newslab data on teraport. While everything works
fine for a small number of patterns and files(e.g. ~20 patterns, ~1000
files), I get errors for larger numbers of files (~10k). I would be very
grateful for any help.

The main files I'm using are (at CI) :
/home/gabri/swift-0.8/examples/swift/newslabex/count/tc.data
/home/gabri/swift-0.8/examples/swift/newslabex/count/sites.xml  (-using the
fast queue)
/home/gabri/swift-0.8/examples/swift/newslabex/count/count.swift
/home/gabri/swift-0.8/examples/swift/newslabex/count/grp

For number of files=10k and number of patterns=2
- I'm getting an "java.lang.OutOfMemoryError". I have tried increasing the
heap size by runnig Swift with (-Xms1536m -Xmx4096m) in the command line,
but that seemed to just push the failure point a little further. Is this at
all the way to go?
- The corresponding logs are at:
/home/gabri/swift-0.8/examples/swift/newslabex/count/errmanyfiles/

Thank you very much for any suggestions.
Best,
Gabri
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/swift-devel/attachments/20090413/bcc1ba08/attachment.html>