[Swift-devel] Re: Persistent coasters

[Michael Wilde]

You should configure the tools for this mode of operation on PADS (and any PBS system):

- run the commands on a login node (but should work on any PADS node that you are ssh'ed into)

- use qsub to obtain nodes
  -- mode 1: 1 N-node M-core job
  -- mode 2: N 1-core jobs

Do mode 1 first:

Job script (the script you use as an arg to qsub) should use a foreach loop to start one worker.pl on each node of the job. You can adapt the code below from Swift R start-swift:

make-pbs-submit-file()
{
  if [ $queue != default ]; then
    queueDirective="#PBS -q $queue"
  else
    queueDirective=""
  fi
cat >pbs.sub <<END
#PBS -S /bin/sh
#PBS -N SwiftR-workers
#PBS -m n
#PBS -l nodes=$nodes
#PBS -l walltime=$time
#PBS -o $HOME
#PBS -e $HOME
$queueDirective
WORKER_LOGGING_ENABLED=true # FIXME: parameterize; fix w PBS -v
#cd / && /usr/bin/perl $SWIFTBIN/worker.pl $CONTACT SwiftR-workers $HOME/.globus/coasters $IDLETIMEOUT
HOST=\$(echo $CONTACT | sed -e 's,^http://,,' -e 's/:.*//')
PORT=\$(echo $CONTACT | sed -e 's,^.*:,,')
echo '***' PBS_NODEFILE file: \$PBS_NODEFILE CONTACT:$CONTACT
cat \$PBS_NODEFILE
echo '***' unique nodes are:
sort < \$PBS_NODEFILE|uniq
for h in \$(sort < \$PBS_NODEFILE|uniq); do
  ssh \$h "echo Swift R startup running on host; hostname; cd /; /usr/bin/perl $SWIFTBIN/worker.pl $CONTACT SwiftR-\$h $HOME/.globus/\
coasters $IDLETIMEOUT" &
done
wait
END
}

then:

  make-${server}-submit-file
  qsub pbs.sub >$pbsjobidfile


Mike

----- Original Message -----
> How should I proceed in testing the persistent coasters scripts on
> PADS? Should I use workers-ssh from the login node to pads? Should I
> copy the format of workers-cobalt and modify it to use qsub parameters
> that work with pbs?
> 
> David

-- 
Michael Wilde
Computation Institute, University of Chicago
Mathematics and Computer Science Division
Argonne National Laboratory