[Swift-devel] Reducing swift log size

[Mihael Hategan]

Allright, Can you do me the following favor?

in coasters/src/......./Cpu.java, change line 183 from
" Cpus sleeping: " + cpus);
to
" Cpus sleeping: " + cpus + ", qseq: " + lastseq + ", myseq: " +
this.getLastSeq());

Then re-run and send me the log. You don't have to run the full thing.
When you see the very frequent sleeping/requesting work craziness in the
log, you can kill the run.

I'm asking this because I have not seen this problem occurring, and it
shouldn't be happening, but it clearly is and your version holds the
key.

Mihael

On Wed, 2014-06-11 at 23:45 -0700, Mihael Hategan wrote:
> Ok, the worker CPU stuff is indeed generating a lot of messages. But
> that isn't supposed to happen. It's supposed to do nothing if nothing
> happens. So I need to check what's going on.
> 
> Mihael
> 
> On Wed, 2014-06-11 at 19:45 -0500, Yadu Nand Babuji wrote:
> > Okay, here you go:
> > http://users.rcc.uchicago.edu/~yadunand/run010_swift_normal.tar.gz
> > http://users.rcc.uchicago.edu/~yadunand/run011_minimal_logging.tar.gz
> > http://users.rcc.uchicago.edu/~yadunand/run012_minimal_plus_log4j.tar.gz
> > 
> > -Yadu
> > On 06/11/2014 07:03 PM, Mihael Hategan wrote:
> > > Sorry, I should have mentioned this, but can you please gzip them? It's
> > > a bit much otherwise.
> > >
> > > Mihael
> > >
> > > On Wed, 2014-06-11 at 17:57 -0500, Yadu Nand Babuji wrote:
> > >> Hi Mihael,
> > >>
> > >> I've got the logs for you.
> > >>
> > >> This time, i've run the 6001 tasks with 20s delay added, and was all run
> > >> with swift-0.95-RC6 (from our website) :
> > >>
> > >> Normal run                 ->
> > >> http://users.rcc.uchicago.edu/~yadunand/run010_swift_normal/run010.log (
> > >> 6.1gb )
> > >> With minimal logging ->
> > >> http://users.rcc.uchicago.edu/~yadunand/run011_minimal_logging/run011.log (
> > >> 7.5 gb )
> > >> With minimal logging ->
> > >> http://users.rcc.uchicago.edu/~yadunand/run012_minimal_plus_log4j/run012.log
> > >> ( 845.4 kb )
> > >> & log4j properties modified
> > >>
> > >> The run with minimal logging ran for ~15mins while the first run took
> > >> ~12mins. That *might* explain why the one
> > >> with minimal logging is larger.
> > >>
> > >> Thanks,
> > >> Yadu
> > >>
> > >> On 06/11/2014 02:54 PM, Mihael Hategan wrote:
> > >>> On Wed, 2014-06-11 at 14:46 -0500, Yadu Nand Babuji wrote:
> > >>>> Hi,
> > >>>>
> > >>>> I'm running a proxy app for a user with 6000 tasks each taking a few
> > >>>> milliseconds, and the log sizes are unusually large. When the tasks are
> > >>>> set to take 20s, the total log size reaches ~7Gb.
> > >>> 7GB? Wow. I'd like to see that. Can you upload and post link?
> > >>>
> > >>>> I tried setting -minimal.logging and -reduced.logging but still see
> > >>>> debug lines in the log like:
> > >>>> 2014-06-11 19:15:06,961+0000 DEBUG Cpu worker started:
> > >>>> block=0611-1507050-000000 host=midway461 id=10
> > >>>> 2014-06-11 19:15:06,961+0000 DEBUG Cpu ready for work:
> > >>>> block=0611-1507050-000000 id=10
> > >>>>
> > >>>> Do we need DEBUG lines such as the ones listed above ? Is it reasonable
> > >>>> to have these set by default to WARN ?
> > >>> It is until there's a problem and then people ask for the opposite. We
> > >>> should evaluate whether this belongs in reduced logging or not. But does
> > >>> that really account for most of the 7G?
> > >>>
> > >>>> Secondly, setting -minimal.logging did not turn off these DEBUG lines
> > >>>> and I had to set the following log4j.properties from DEBUG to WARN to
> > >>>> remove most of the offending lines:
> > >>> That sounds like a bug.
> > >>>
> > >>>> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=WARN
> > >>>> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=WARN
> > >>>>
> > >>>> With 6001 tasks, each taking 2 ms or so:
> > >>>> Swift without any changes to logging ->  440879 lines and  51Mb
> > >>>> Swift with -minimal.logging                ->    83350  lines and 9.5Mb
> > >>>> Swift with -minimal.logging and         ->      7625  lines and 791Kb
> > >>>> Cpu, Block log4j properties set
> > >>>>
> > >>>> Thanks,
> > >>>> Yadu
> > >>>> _______________________________________________
> > >>>> Swift-devel mailing list
> > >>>> Swift-devel at ci.uchicago.edu
> > >>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
> > >
> > 
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel