[Swift-devel] finding the execution sites from swift logs

[Allan Espinosa]

Right. the execute2.event target in swift-plot-log gets the time
duration between JOB_START and JOB_END (correct me if i'm wrong Ben).
 With that you could get how many jobs were sent to each site.

-Allan

2011/6/30 Ketan Maheshwari <ketancmaheshwari at gmail.com>:
> In addition to JOB_START, vdl:execute2 is associated with many events:
>
> THREAD_ASSOCIATION
> APPLICATION_EXCEPTION
> STAGING_OUT
> ..
>
> Ketan
>
> On Thu, Jun 30, 2011 at 5:39 PM, Allan Espinosa <aespinosa at cs.uchicago.edu>
> wrote:
>>
>> I think JOB_START are the vdl:execute2 ones if i remember correctly.
>>
>> 2011/6/30 Ketan Maheshwari <ketancmaheshwari at gmail.com>:
>> > Thanks Allan.
>> >
>> >  In some log files, I do not see any execute2 lines. However, I do see
>> > JOB_START lines ending with host=<hostname>. I confirmed that the logs
>> > indeed belong to successful complete runs. While in other logs
>> > corresponding
>> > to other successful runs, I do see the execute2 lines.
>> >
>> > All the runs were carried out using the same version of Swift. Am I
>> > missing
>> > something here?
>> >
>> > Ketan
>> >
>> >
>> > On Thu, Jun 30, 2011 at 11:45 AM, Allan Espinosa
>> > <aespinosa at cs.uchicago.edu>
>> > wrote:
>> >>
>> >> Hi Ketan,
>> >>
>> >> What I do is match the jobnames with the sites by matching execute2
>> >> log information.  I have a set of R and ruby scripts in
>> >> ~aespinosa/Documents/swift that analyzes the number of transfers per
>> >> site.  You can modify them to look for job execution.
>> >>
>> >> If you look at one of the makefile targets in libexec/log-processing,
>> >> you can find files with the name 'color'.  these targets to swift plot
>> >> logs colors the plot per site.
>> >>
>> >> To just explore the general statistics, I obtain the execute2.event
>> >> file from the log using
>> >>
>> >> $swift-plot-log logfile execute2.event
>> >>
>> >> And then just use R to analyze the *.event file.
>> >>
>> >> 2011/6/30 Ketan Maheshwari <ketancmaheshwari at gmail.com>:
>> >> > Hello,
>> >> >
>> >> > Does anyone knows from swift log, how to find how many jobs executed
>> >> > on
>> >> > a
>> >> > given site when there is a mix of localhost and osg sites?
>> >> >
>> >> > I have logs of many runs each with about 3400 app tasks ran on
>> >> > localhost
>> >> > +
>> >> > osg sites.
>> >> >
>> >> > Trying to look into log and find that there are multiple messages
>> >> > corresponding to staging in-out, run and other events (change in
>> >> > score,
>> >> > thread associations, etc.).
>> >> >
>> >> > What should I be looking to identify each job uniquely?
>> >> >
>> >> > Thanks for any help on this.
>> >> >
>> >> > Regards,
>> >> > --
>> >> > Ketan
>> >> >
>> >> >
>> >> >
>> >> > _______________________________________________
>> >> > Swift-devel mailing list
>> >> > Swift-devel at ci.uchicago.edu
>> >> > https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Allan M. Espinosa <http://amespinosa.wordpress.com>
>> >> PhD student, Computer Science
>> >> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
>> >
>> >
>> >
>> > --
>> > Ketan
>> >
>> >
>> >
>>
>>
>>
>> --
>> Allan M. Espinosa <http://amespinosa.wordpress.com>
>> PhD student, Computer Science
>> University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
>
>
>
> --
> Ketan
>
>
>



-- 
Allan M. Espinosa <http://amespinosa.wordpress.com>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>