[Swift-devel] IO overheads of swift wrapper scripts on BlueGene/P

[Mihael Hategan]

On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote:
> Progress  2009-10-16 18:00:33.756364000-0500  COPYING_OUTPUTS
> Progress  2009-10-16 18:08:19.970449000-0500  RM_JOBDIR

Grr. 8 minutes spent COPYING_OUTPUTS.

What would be useful is to aggregate all the access that happened on
that FS from all the relevant jobs, to see the exact thing that causes
contention. I strongly suspect it's
home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/

Pretty much all the outputs seem to go to that directory.

I'm afraid however that the information in the logs is insufficient.
Strace with relevant options (for fs calls only) may be useful if you
want to try.

Alternatively, you could try to spread your output over multiple
directories and see what the difference is.

Also, it may be interesting to see the dependence between the delay and
the number of contending processes. That is so that we know the limit of
how many processes we can allow to compete for a shared resource without
causing too much trouble.

Mihael