[Swift-devel] java.lang.OutOfMemoryError when running grep on 10k files

[Glen Hocky]

I did the same but in COG_OPTS, i.e.
export COG_OPTS="-Xmx1024m"
or 2048 for really big jobs

Gabri Turcu wrote:
> Hi,
>
> I am trying to run grep on newslab data on teraport. While everything 
> works fine for a small number of patterns and files(e.g. ~20 patterns, 
> ~1000 files), I get errors for larger numbers of files (~10k). I would 
> be very grateful for any help.
>
> The main files I'm using are (at CI) :
> /home/gabri/swift-0.8/examples/swift/newslabex/count/tc.data
> /home/gabri/swift-0.8/examples/swift/newslabex/count/sites.xml  
> (-using the fast queue)
> /home/gabri/swift-0.8/examples/swift/newslabex/count/count.swift
> /home/gabri/swift-0.8/examples/swift/newslabex/count/grp
>
> For number of files=10k and number of patterns=2
> - I'm getting an "java.lang.OutOfMemoryError". I have tried increasing 
> the heap size by runnig Swift with (-Xms1536m -Xmx4096m) in the 
> command line, but that seemed to just push the failure point a little 
> further. Is this at all the way to go?
> - The corresponding logs are at: 
> /home/gabri/swift-0.8/examples/swift/newslabex/count/errmanyfiles/
>
> Thank you very much for any suggestions.
> Best,
> Gabri
> ------------------------------------------------------------------------
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>