[Swift-devel] Reducing swift log size

[Michael Wilde]

In general we're moving to running coasters in all configurations (in 
part to reduce the number of configurations to explain and test).

Yadu's also looking at using provider staging shared-filesystem mode to 
avoid un-necessary staging for local filesystems.

Can you explain the connection between this and the excessive logging? 
Can that be fixed rather than resorting to an alternate provider?

- Mike

On 6/12/14, 12:58 PM, Mihael Hategan wrote:
> Ah!
>
> Well, so the trick with two coaster services on localhost doesn't really
> work well unless you use "ssh:", and this is a good example why.
>
> In your case you can avoid it easily if you change your first pool to
> use the local provider instead of the coaster provider, since you don't
> really need coasters to run locally.
>
> Mihael
>
>
> On Thu, 2014-06-12 at 12:11 -0500, Yadu Nand Babuji wrote:
>> Hi Mihael,
>>
>> Here's the package I'm running:
>> http://users.rcc.uchicago.edu/~yadunand/swift-0.95-modded.tar.gz
>> (It has Cpu.java modified, as well as as a debugging line in
>> libexec/swift-int-staging.k)
>>
>> I shutdown the run once the logs were past 4gb, here's the link to the
>> log :
>> http://users.rcc.uchicago.edu/~yadunand/run013_with_mods_to_Cpujava.tar.gz
>>
>> Thanks,
>> -Yadu
>>
>> On 06/11/2014 07:03 PM, Mihael Hategan wrote:
>>>>>> Sorry, I should have mentioned this, but can you please gzip them? It's
>>>>>> a bit much otherwise.
>>>>>>
>>>>>> Mihael
>>>>>>
>>>>>> On Wed, 2014-06-11 at 17:57 -0500, Yadu Nand Babuji wrote:
>>>>>>> Hi Mihael,
>>>>>>>
>>>>>>> I've got the logs for you.
>>>>>>>
>>>>>>> This time, i've run the 6001 tasks with 20s delay added, and was all run
>>>>>>> with swift-0.95-RC6 (from our website) :
>>>>>>>
>>>>>>> Normal run                 ->
>>>>>>> http://users.rcc.uchicago.edu/~yadunand/run010_swift_normal/run010.log (
>>>>>>> 6.1gb )
>>>>>>> With minimal logging ->
>>>>>>> http://users.rcc.uchicago.edu/~yadunand/run011_minimal_logging/run011.log (
>>>>>>> 7.5 gb )
>>>>>>> With minimal logging ->
>>>>>>> http://users.rcc.uchicago.edu/~yadunand/run012_minimal_plus_log4j/run012.log
>>>>>>> ( 845.4 kb )
>>>>>>> & log4j properties modified
>>>>>>>
>>>>>>> The run with minimal logging ran for ~15mins while the first run took
>>>>>>> ~12mins. That *might* explain why the one
>>>>>>> with minimal logging is larger.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Yadu
>>>>>>>
>>>>>>> On 06/11/2014 02:54 PM, Mihael Hategan wrote:
>>>>>>>> On Wed, 2014-06-11 at 14:46 -0500, Yadu Nand Babuji wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I'm running a proxy app for a user with 6000 tasks each taking a few
>>>>>>>>> milliseconds, and the log sizes are unusually large. When the tasks are
>>>>>>>>> set to take 20s, the total log size reaches ~7Gb.
>>>>>>>> 7GB? Wow. I'd like to see that. Can you upload and post link?
>>>>>>>>
>>>>>>>>> I tried setting -minimal.logging and -reduced.logging but still see
>>>>>>>>> debug lines in the log like:
>>>>>>>>> 2014-06-11 19:15:06,961+0000 DEBUG Cpu worker started:
>>>>>>>>> block=0611-1507050-000000 host=midway461 id=10
>>>>>>>>> 2014-06-11 19:15:06,961+0000 DEBUG Cpu ready for work:
>>>>>>>>> block=0611-1507050-000000 id=10
>>>>>>>>>
>>>>>>>>> Do we need DEBUG lines such as the ones listed above ? Is it reasonable
>>>>>>>>> to have these set by default to WARN ?
>>>>>>>> It is until there's a problem and then people ask for the opposite. We
>>>>>>>> should evaluate whether this belongs in reduced logging or not. But does
>>>>>>>> that really account for most of the 7G?
>>>>>>>>
>>>>>>>>> Secondly, setting -minimal.logging did not turn off these DEBUG lines
>>>>>>>>> and I had to set the following log4j.properties from DEBUG to WARN to
>>>>>>>>> remove most of the offending lines:
>>>>>>>> That sounds like a bug.
>>>>>>>>
>>>>>>>>> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Cpu=WARN
>>>>>>>>> log4j.logger.org.globus.cog.abstraction.coaster.service.job.manager.Block=WARN
>>>>>>>>>
>>>>>>>>> With 6001 tasks, each taking 2 ms or so:
>>>>>>>>> Swift without any changes to logging ->  440879 lines and  51Mb
>>>>>>>>> Swift with -minimal.logging                ->    83350  lines and 9.5Mb
>>>>>>>>> Swift with -minimal.logging and         ->      7625  lines and 791Kb
>>>>>>>>> Cpu, Block log4j properties set
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Yadu
>>>>>>>>> _______________________________________________
>>>>>>>>> Swift-devel mailing list
>>>>>>>>> Swift-devel at ci.uchicago.edu
>>>>>>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>>>> _______________________________________________
>>>> Swift-devel mailing list
>>>> Swift-devel at ci.uchicago.edu
>>>> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel

-- 
Michael Wilde
Mathematics and Computer Science          Computation Institute
Argonne National Laboratory               The University of Chicago