[Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P

[Allan Espinosa]

I was using 1000 files  (or was it 3000?) per directory. it looks like
i need to lower my ratio...

-Allan

2009/10/16 Mihael Hategan <hategan at mcs.anl.gov>:
> On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote:
>> Progress  2009-10-16 18:00:33.756364000-0500  COPYING_OUTPUTS
>> Progress  2009-10-16 18:08:19.970449000-0500  RM_JOBDIR
>
> Grr. 8 minutes spent COPYING_OUTPUTS.
>
> What would be useful is to aggregate all the access that happened on
> that FS from all the relevant jobs, to see the exact thing that causes
> contention. I strongly suspect it's
> home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/
>
> Pretty much all the outputs seem to go to that directory.
>
> I'm afraid however that the information in the logs is insufficient.
> Strace with relevant options (for fs calls only) may be useful if you
> want to try.
>
> Alternatively, you could try to spread your output over multiple
> directories and see what the difference is.
>
> Also, it may be interesting to see the dependence between the delay and
> the number of contending processes. That is so that we know the limit of
> how many processes we can allow to compete for a shared resource without
> causing too much trouble.
>
> Mihael
>
>
>



-- 
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>