[Swift-commit] r6504 - branches/release-0.94/docs/userguide

[davidk at ci.uchicago.edu]

Author: davidk
Date: 2013-05-06 14:50:31 -0500 (Mon, 06 May 2013)
New Revision: 6504

Modified:
   branches/release-0.94/docs/userguide/howto_tips
Log:
More details about running MPI jobs, what needs to be set in sites.xml, how to call mpiexec, etc.


Modified: branches/release-0.94/docs/userguide/howto_tips
===================================================================
--- branches/release-0.94/docs/userguide/howto_tips	2013-05-06 18:51:27 UTC (rev 6503)
+++ branches/release-0.94/docs/userguide/howto_tips	2013-05-06 19:50:31 UTC (rev 6504)
@@ -28,53 +28,72 @@
 Launching MPI jobs from Swift
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-There are several ways to run MPI jobs under Swift.
-The simplest is to simply launch the desired +mpiexec+ command line
-locally by using the local provider.  Two additional use cases are
-noted here.
+There are several ways to run MPI jobs under Swift. Two will be discussed here -
+calling mpiexec from a wrapper script, and using the MPICH/coasters interface.
 
-==== Submission of MPI jobs to remote queues
+Calling mpiexec
+^^^^^^^^^^^^^^^
+In this example, a single MPI program will run across two nodes. For this to
+happen, sites.xml must be configured to allocate two nodes but only run a
+single job on them. A wrapper script must then be used to call mpiexec.
 
-// TODO: This seems to require appropriate PBS profile settings
+sites.xml
++++++++++
+First, we need to make sure that Swift will allocate exactly two nodes. This
+can be done with the maxnodes and nodegranularity settings.
 
-This use case allows the user to submit an MPI job to a remote queue.
+-----
+<profile namespace="globus" key="nodeGranularity">2</profile>
+<profile namespace="globus" key="maxnodes">2</profile>
+-----
 
-Swift makes an invocation that does not look any different
-from any other invocation. In the code below, we do not have any input
-files, and have two output files on stdout and stderr:
+Next, we want to make sure that the MPI program is called only once on those
+nodes. There are two settings we must set to get this behavior:
+-----
+<profile namespace="globus" key="jobsPerNode">1</profile>
+<profile namespace="globus" key="jobtype">single</profile>
+-----
 
-----
-type file;
+tc.data
++++++++
+The app defined in tc.data should be a shell script wrapper to the actual
+program that is being called. Let's assume in this example that the MPI program
+we are using is called "mpitest", and the wrapper script will be called 
+"mpitest.sh". The tc.data will look like this then:
 
-(file o, file e) p() {
-    app {
-        mpi stdout=@filename(o) stderr=@filename(e);
-    }
-}
+-----
+host mpitest /path/to/mpitest.sh
+-----
 
-file mpiout <"mpi.out">;
-file mpierr <"mpi.err">;
+Wrapper script
+++++++++++++++
+The wrapper script in this example, mpitest.sh, will call mpiexec and launch
+the real MPI program. Here is an example:
 
-(mpiout, mpierr) = p();
-----
+-----
+#!/bin/bash 
 
-Now we define how 'mpi' will run in tc.data:
+mpiexec /path/to/mpitest "$@"
+-----
+
+Swift then makes an invocation that does not look any different from any other
+invocation. In the code below, we pass one input file and get back one output
+file. 
+
 ----
-tguc    mpi             /home/benc/mpi/mpi.sh   INSTALLED       INTEL32::LINUX GLOBUS::host_xcount=3
-----
+type file;
 
-mpi.sh is a wrapper script that launches the MPI program. It must be
-installed on the remote site:
+app (file output_file) mpitest (file input_file)
+{
+   mpitest @input_file @output_file; 
+}
 
+file input <"input.txt">;
+file output <"output.txt">;
+
+output = mpitest(input);
 ----
-#!/bin/bash
-mpirun -np 3 -machinefile $PBS_NODEFILE /home/benc/mpi/a.out
-----
 
-Because of the way that Swift runs its server side code,
-provider-specific MPI modes (such as GRAM jobType=mpi) should not be
-used. Instead, the mpirun command should be explicitly invoked.
-
 ==== MPICH/Coasters
 
 In this case, the user desires to launch many MPI jobs within a single