<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<meta content="text/html;charset=ISO-8859-1" http-equiv="Content-Type">
</head>
<body bgcolor="#ffffff" text="#000000">
Hi Allan,<br>
I don't remember, but your Falkon only run seemed to run OK, right?
Didn't that also produce the output files Swift is producing? Or is
Swift doing an extra step, to copy/move files from one place to another
after the computation terminates, which is the thing that takes so
long? Just trying to understand the difference between the Falkon only
run and Swift run.<br>
<br>
Ioan<br>
<pre class="moz-signature" cols="72">--
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384
Evanston, IL 60208-3118
=================================================================
Cel: 1-847-722-0876
Tel: 1-847-491-8163
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@eecs.northwestern.edu">iraicu@eecs.northwestern.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.eecs.northwestern.edu/~iraicu/">http://www.eecs.northwestern.edu/~iraicu/</a>
<a class="moz-txt-link-freetext" href="https://wiki.cucis.eecs.northwestern.edu/">https://wiki.cucis.eecs.northwestern.edu/</a>
=================================================================
=================================================================
</pre>
<br>
<br>
Allan Espinosa wrote:
<blockquote
cite="mid:50b07b4b0910171957of897b9dy90c349d1069004be@mail.gmail.com"
type="cite">
<pre wrap="">Here I tried one directory per job (Q0000130). 3 output files are
expected per directory which are produced by a single job:
Progress 2009-10-17 20:53:56.943503000-0500 LOG_START
_____________________________________________________________________________
Wrapper
_____________________________________________________________________________
Job directory mode is: link on shared filesystem
DIR=jobs/7/blastall-715ul5ij
EXEC=/home/espinosa/workflows/jgi_blastp/blastall_wrapper
STDIN=
STDOUT=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout
STDERR=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr
DIRS=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130
INF=
OUTF=home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.serr^home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.sout
KICKSTART=
ARGS=-p blastp -m 8 -e 1.0e-5 -FF -d /dataifs/nr -i
/intrepid-fs0/users/espinosa/persistent/datasets/nr_bob/queries/mock_2seq/D0000000/SEQ0000130.fasta
-o home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130/out_Q0000130.out
ARGC=13
Progress 2009-10-17 20:53:58.656335000-0500 CREATE_JOBDIR
Created job directory: jobs/7/blastall-715ul5ij
Progress 2009-10-17 20:54:05.204962000-0500 CREATE_INPUTDIR
Created output directory:
jobs/7/blastall-715ul5ij/home/espinosa/workflows/jgi_blastp/oldtests/test3.2.7_3cpn.64ifs.192cpu/output/D0000000/Q0000130
Progress 2009-10-17 20:54:15.498666000-0500 LINK_INPUTS
Progress 2009-10-17 20:54:19.900786000-0500 EXECUTE
Moving back to workflow directory
/fuse/intrepid-fs0/users/espinosa/scratch/jgi-blastp_runs/blastp-test3.2.7_3cpn.64ifs.192cpu
Progress 2009-10-17 21:20:23.390800000-0500 EXECUTE_DONE
Job ran successfully
Progress 2009-10-17 21:31:11.179664000-0500 COPYING_OUTPUTS
Progress 2009-10-17 21:37:14.539569000-0500 RM_JOBDIR
Progress 2009-10-17 21:38:24.220130000-0500 END
COPYING_OUTPUTS still take time.
2009/10/17 Michael Wilde <a class="moz-txt-link-rfc2396E" href="mailto:wilde@mcs.anl.gov"><wilde@mcs.anl.gov></a>:
</pre>
<blockquote type="cite">
<pre wrap="">Remember that any situation in which multiple IONs modify the same file or
directory (ie by creating files or directories in the same parent directory)
will cause severe contention and performance degradation on any GPFS
filesystem.
In addition to creating many directories, you need to ensure that no single
file or directories is likely to ever be written to from multiple client
nodes (eg IONs on the BG/P) concurrently.
</pre>
</blockquote>
<pre wrap=""><!---->
This workload is just over 1 PSET so there are no other IONs
contending over the directories.
</pre>
<blockquote type="cite">
<pre wrap="">Have you done that in this workload, Allan?
- Mike
On 10/17/09 2:59 AM, Allan Espinosa wrote:
</pre>
<blockquote type="cite">
<pre wrap="">I was using 1000 files (or was it 3000?) per directory. it looks like
i need to lower my ratio...
-Allan
2009/10/16 Mihael Hategan <a class="moz-txt-link-rfc2396E" href="mailto:hategan@mcs.anl.gov"><hategan@mcs.anl.gov></a>:
</pre>
<blockquote type="cite">
<pre wrap="">On Fri, 2009-10-16 at 21:07 -0500, Allan Espinosa wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Progress 2009-10-16 18:00:33.756364000-0500 COPYING_OUTPUTS
Progress 2009-10-16 18:08:19.970449000-0500 RM_JOBDIR
</pre>
</blockquote>
<pre wrap="">Grr. 8 minutes spent COPYING_OUTPUTS.
What would be useful is to aggregate all the access that happened on
that FS from all the relevant jobs, to see the exact thing that causes
contention. I strongly suspect it's
home/espinosa/workflows/jgi_blastp/test3.4.7_3cpn.32ifs.192cpu/output/
Pretty much all the outputs seem to go to that directory.
I'm afraid however that the information in the logs is insufficient.
Strace with relevant options (for fs calls only) may be useful if you
want to try.
Alternatively, you could try to spread your output over multiple
directories and see what the difference is.
Also, it may be interesting to see the dependence between the delay and
the number of contending processes. That is so that we know the limit of
how many processes we can allow to compete for a shared resource without
causing too much trouble.
Mihael
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap="">
</pre>
</blockquote>
<pre wrap=""><!---->
</pre>
</blockquote>
<br>
<pre class="moz-signature" cols="72">--
=================================================================
Ioan Raicu, Ph.D.
NSF/CRA Computing Innovation Fellow
=================================================================
Center for Ultra-scale Computing and Information Security (CUCIS)
Department of Electrical Engineering and Computer Science
Northwestern University
2145 Sheridan Rd, Tech M384
Evanston, IL 60208-3118
=================================================================
Cel: 1-847-722-0876
Tel: 1-847-491-8163
Email: <a class="moz-txt-link-abbreviated" href="mailto:iraicu@eecs.northwestern.edu">iraicu@eecs.northwestern.edu</a>
Web: <a class="moz-txt-link-freetext" href="http://www.eecs.northwestern.edu/~iraicu/">http://www.eecs.northwestern.edu/~iraicu/</a>
<a class="moz-txt-link-freetext" href="https://wiki.cucis.eecs.northwestern.edu/">https://wiki.cucis.eecs.northwestern.edu/</a>
=================================================================
=================================================================
</pre>
</body>
</html>