[Swift-devel] Reducing swift log size

[Mihael Hategan]

On Wed, 2014-06-11 at 14:46 -0500, Yadu Nand Babuji wrote:
> Hi,
> 
> I'm running a proxy app for a user with 6000 tasks each taking a few 
> milliseconds, and the log sizes are unusually large. When the tasks are 
> set to take 20s, the total log size reaches ~7Gb.

7GB? Wow. I'd like to see that. Can you upload and post link?

> 
> I tried setting -minimal.logging and -reduced.logging but still see 
> debug lines in the log like:
> 2014-06-11 19:15:06,961+0000 DEBUG Cpu worker started: 
> block=0611-1507050-000000 host=midway461 id=10
> 2014-06-11 19:15:06,961+0000 DEBUG Cpu ready for work: 
> block=0611-1507050-000000 id=10
> 
> Do we need DEBUG lines such as the ones listed above ? Is it reasonable 
> to have these set by default to WARN ?

It is until there's a problem and then people ask for the opposite. We
should evaluate whether this belongs in reduced logging or not. But does
that really account for most of the 7G?

> Secondly, setting -minimal.logging did not turn off these DEBUG lines 
> and I had to set the following log4j.properties from DEBUG to WARN to 
> remove most of the offending lines:

That sounds like a bug.

> 
> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=WARN
> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=WARN
> 
> With 6001 tasks, each taking 2 ms or so:
> Swift without any changes to logging ->  440879 lines and  51Mb
> Swift with -minimal.logging                ->    83350  lines and 9.5Mb
> Swift with -minimal.logging and         ->      7625  lines and 791Kb
> Cpu, Block log4j properties set
> 
> Thanks,
> Yadu
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel