[Swift-devel] Reducing swift log size

[Mihael Hategan]

Sorry, I should have mentioned this, but can you please gzip them? It's
a bit much otherwise.

Mihael

On Wed, 2014-06-11 at 17:57 -0500, Yadu Nand Babuji wrote:
> Hi Mihael,
> 
> I've got the logs for you.
> 
> This time, i've run the 6001 tasks with 20s delay added, and was all run 
> with swift-0.95-RC6 (from our website) :
> 
> Normal run                 -> 
> http://users.rcc.uchicago.edu/~yadunand/run010_swift_normal/run010.log ( 
> 6.1gb )
> With minimal logging -> 
> http://users.rcc.uchicago.edu/~yadunand/run011_minimal_logging/run011.log ( 
> 7.5 gb )
> With minimal logging -> 
> http://users.rcc.uchicago.edu/~yadunand/run012_minimal_plus_log4j/run012.log 
> ( 845.4 kb )
> & log4j properties modified
> 
> The run with minimal logging ran for ~15mins while the first run took 
> ~12mins. That *might* explain why the one
> with minimal logging is larger.
> 
> Thanks,
> Yadu
> 
> On 06/11/2014 02:54 PM, Mihael Hategan wrote:
> > On Wed, 2014-06-11 at 14:46 -0500, Yadu Nand Babuji wrote:
> >> Hi,
> >>
> >> I'm running a proxy app for a user with 6000 tasks each taking a few
> >> milliseconds, and the log sizes are unusually large. When the tasks are
> >> set to take 20s, the total log size reaches ~7Gb.
> > 7GB? Wow. I'd like to see that. Can you upload and post link?
> >
> >> I tried setting -minimal.logging and -reduced.logging but still see
> >> debug lines in the log like:
> >> 2014-06-11 19:15:06,961+0000 DEBUG Cpu worker started:
> >> block=0611-1507050-000000 host=midway461 id=10
> >> 2014-06-11 19:15:06,961+0000 DEBUG Cpu ready for work:
> >> block=0611-1507050-000000 id=10
> >>
> >> Do we need DEBUG lines such as the ones listed above ? Is it reasonable
> >> to have these set by default to WARN ?
> > It is until there's a problem and then people ask for the opposite. We
> > should evaluate whether this belongs in reduced logging or not. But does
> > that really account for most of the 7G?
> >
> >> Secondly, setting -minimal.logging did not turn off these DEBUG lines
> >> and I had to set the following log4j.properties from DEBUG to WARN to
> >> remove most of the offending lines:
> > That sounds like a bug.
> >
> >> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=WARN
> >> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=WARN
> >>
> >> With 6001 tasks, each taking 2 ms or so:
> >> Swift without any changes to logging ->  440879 lines and  51Mb
> >> Swift with -minimal.logging                ->    83350  lines and 9.5Mb
> >> Swift with -minimal.logging and         ->      7625  lines and 791Kb
> >> Cpu, Block log4j properties set
> >>
> >> Thanks,
> >> Yadu
> >> _______________________________________________
> >> Swift-devel mailing list
> >> Swift-devel at ci.uchicago.edu
> >> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> >
>