This fix works for me. I tested with one catsn job on Beagle.<br><br><div class="gmail_quote">On Sun, Nov 13, 2011 at 7:48 PM, Michael Wilde <span dir="ltr"><<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">OK, here is a simple fix for this problem. Just add the variable "SWIFT_USERHOME" to your swift command; then do:<br>
<br>
export SWIFT_USERHOME=/lustre/beagle/wilde<br>
swift etc<br>
<br>
This makes swift use $SWIFT_USERHOME instead of $HOME to locate the .globus directory.<br>
<br>
This will of course mess up if a swift run needs to locate your certificates; possibly you can get around that with a symlink. But I suspect most uses of this will be for local execution on systems like Beagle with non-writeable home dirs.<br>
<br>
Here's the 1-line fix:<br>
<br>
login$ pwd<br>
/home/wilde/swift/src/0.93/cog/modules/swift/bin<br>
login$ svn diff<br>
Index: swift<br>
===================================================================<br>
--- swift (revision 5284)<br>
+++ swift (working copy)<br>
@@ -86,6 +86,7 @@<br>
updateOptions "$X509_USER_PROXY" "X509_USER_PROXY"<br>
updateOptions "$SWIFT_HOME" "COG_INSTALL_PATH"<br>
updateOptions "$SWIFT_HOME" "swift.home"<br>
+updateOptions "$SWIFT_USERHOME" "user.home"<br>
#Use /dev/urandom instead of /dev/random for seeding RNGs<br>
#This will lower the randomness of the seed, but avoid<br>
#large delays if /dev/random does not have enough entropy collected<br>
login$<br>
<br>
If others can confirm that this works, I'll check it in.<br>
<div class="im"><br>
- Mike<br>
<br>
<br>
<br>
----- Original Message -----<br>
> From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> To: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
</div><div><div></div><div class="h5">> Sent: Sunday, November 13, 2011 3:08:36 PM<br>
> Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> OK, as some of you can see in the mesg I just send to beagle-support:<br>
> it now looks ot me like the root problem of the swift jobs failing is<br>
> that our home dirs are not beng seen on the compute nodes, hance the<br>
> swift-generated PBS script to launch the coaster workers cant find the<br>
> <a href="http://worker.pl" target="_blank">worker.pl</a> script that swift copied to $HOME/.globus/coasters.<br>
><br>
> This is what I see:<br>
><br>
> The following was run under qsub -I; the line "total 0" shows that<br>
> /home/wilde was empty as seen by the compute node.<br>
><br>
> login1$ aprun /bin/sh -c 'hostname; ls -l /home/wilde/; mount | grep<br>
> home; '<br>
> nid00466<br>
> total 0<br>
> /autonfs/home on /autonfs/home type dvs<br>
> (ro,blksize=16384,nodename=c1-0c0s7n3:c4-0c0s2n0:c4-0c0s2n1:c4-0c0s2n2:c4-0c0s2n3,attrcache_timeout=14400,cache,nodatasync,noclosesync,retry,failover,userenv,clusterfs,killprocess,nobulk_rw,noatomic,nodeferopens,loadbalance,maxnodes=1,nnodes=5)<br>
> /autonfs/home on /autonfs/home type dvs<br>
> (ro,blksize=16384,nodename=c1-0c0s7n3:c4-0c0s2n0:c4-0c0s2n1:c4-0c0s2n2:c4-0c0s2n3,attrcache_timeout=14400,cache,nodatasync,noclosesync,retry,failover,userenv,clusterfs,killprocess,nobulk_rw,noatomic,nodeferopens,loadbalance,maxnodes=1,nnodes=5)<br>
> Application 863284 resources: utime ~0s, stime ~0s<br>
> login1$<br>
><br>
> Can anyone verify that they are seeing the same symptom?<br>
><br>
> Thanks,<br>
><br>
> - Mike<br>
><br>
><br>
> ----- Original Message -----<br>
> > From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> > To: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > Sent: Sunday, November 13, 2011 2:41:36 PM<br>
> > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > I tracked the message below down to the fact that aprun doesnt like<br>
> > "&" in its command string. I vaguely recall reporting something<br>
> > similar to Cray way back and they agreed its a bug.<br>
> ><br>
> > But it seems that the *original* Swift command string did not have a<br>
> > "&" in it, so Im back to square one.<br>
> ><br>
> > - Mike<br>
> ><br>
> > ----- Original Message -----<br>
> > > From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> > > To: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > > Sent: Sunday, November 13, 2011 1:52:58 PM<br>
> > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > Its starting to look like some kind of aprun-based failure. I see<br>
> > > this<br>
> > > from more detailed logging I put into the generated script:<br>
> > ><br>
> > > IN .submit script<br>
> > > aprun: Unexpected close of the apsys control connection<br>
> > > aprun: Exiting due to errors. Application aborted<br>
> > > aprun rc 1<br>
> > ><br>
> > > I was led off track by the fact that the exitcode file is missing.<br>
> > > Seems that its generated but then removed before we can see it. I<br>
> > > suspect one part of the provider thinks the worker-launch job<br>
> > > succeeded, and hence removes the exitcode file, but another part<br>
> > > realizes that the job failed. (conjecture...)<br>
> > ><br>
> > > Now that that part is partially explained, I think I can go back<br>
> > > to<br>
> > > debugging this from manual qsubs which should go faster.<br>
> > ><br>
> > > Im still unsure if the missing stdout/err files is due to a Beagle<br>
> > > issue; starting to look more like maybe due to the weird way in<br>
> > > which<br>
> > > the aprun dies.<br>
> > ><br>
> > > Digging deeper...<br>
> > ><br>
> > > - Mike<br>
> > ><br>
> > > ----- Original Message -----<br>
> > > > From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> > > > To: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > > > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > > > Sent: Sunday, November 13, 2011 7:51:57 AM<br>
> > > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > > Ive backed up and just did a test from swift (automatic)<br>
> > > ><br>
> > > > I see that in that case I am *not* getting an exitcode file.<br>
> > > > Are you getting one?<br>
> > > ><br>
> > > > - Mike<br>
> > > ><br>
> > > > ----- Original Message -----<br>
> > > > > From: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> > > > > To: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > > > > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > > > > Sent: Sunday, November 13, 2011 7:45:05 AM<br>
> > > > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > > > But if you put an explicit output redirection in the /bin/sh<br>
> > > > > -c<br>
> > > > > command, you will see that those commands are indeed executing<br>
> > > > > and<br>
> > > > > generating output.<br>
> > > > ><br>
> > > > > So like I mentioned earlier, I dont know if the qsub -o and -e<br>
> > > > > flags<br>
> > > > > have changed behavior (eg they now cant write to /home???), or<br>
> > > > > if<br>
> > > > > we<br>
> > > > > are using them incorrectly.<br>
> > > > ><br>
> > > > > But I think we need to go backwards and see why this is not<br>
> > > > > working<br>
> > > > > with the swift-generated qsub files.<br>
> > > > ><br>
> > > > > We should next add the two tags to the sites file to obtain a<br>
> > > > > log<br>
> > > > > from<br>
> > > > > the worker, on the (untested!) assumption that the worker is<br>
> > > > > really<br>
> > > > > starting in the automatic swift case:<br>
> > > > ><br>
> > > > > <profile namespace="globus"<br>
> > > > > key="workerLoggingLevel">DEBUG</profile><br>
> > > > > <profile namespace="globus"<br>
> > > > > key="workerLoggingDirectory">/lustre/beagle/wilde/beagle</profile><br>
> > > > ><br>
> > > > > - Mike<br>
> > > > ><br>
> > > > ><br>
> > > > > ----- Original Message -----<br>
> > > > > > From: "Ketan Maheshwari" <<a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a>><br>
> > > > > > To: "Michael Wilde" <<a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a>><br>
> > > > > > Cc: "Swift Devel" <<a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a>><br>
> > > > > > Sent: Sunday, November 13, 2011 7:35:24 AM<br>
> > > > > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > > > > On Sun, Nov 13, 2011 at 9:28 AM, Michael Wilde <<br>
> > > > > > <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > > > > > ><br>
> > > > > > wrote:<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > 2 thoughts here, Ketan:<br>
> > > > > ><br>
> > > > > > - when I tried my manual coaster test, I replaced the<br>
> > > > > > options<br>
> > > > > > "-n<br>
> > > > > > 3<br>
> > > > > > -N<br>
> > > > > > 1 -cc none -d 24 -F exclusive" on aprun with simply "-B"<br>
> > > > > > which<br>
> > > > > > says<br>
> > > > > > "use the options from qsub". I was going to go back and see<br>
> > > > > > if<br>
> > > > > > there<br>
> > > > > > was some subtle new mismatch between these qsub and aprun<br>
> > > > > > processor-layout options.<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > I tried the -B option:<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > #CoG This script generated by CoG<br>
> > > > > > #CoG by class: class<br>
> > > > > > org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor<br>
> > > > > > #CoG on date: 2011/11/13 02:16:54<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > #PBS -S /bin/bash<br>
> > > > > > #PBS -N Block-1113-1602<br>
> > > > > > #PBS -m n<br>
> > > > > > #PBS -A CI-DEB000002<br>
> > > > > > #PBS -l mppwidth=3,mppnppn=1,mppdepth=24<br>
> > > > > > #PBS -l walltime=00:10:00<br>
> > > > > > #PBS -o<br>
> > > > > > /home/ketan/.globus/scripts/PBS2583661693904024220.submit.stdout<br>
> > > > > > #PBS -e<br>
> > > > > > /home/ketan/.globus/scripts/PBS2583661693904024220.submit.stderr<br>
> > > > > > WORKER_LOGGING_LEVEL=NONE<br>
> > > > > > #PBS -v WORKER_LOGGING_LEVEL<br>
> > > > > > cd / && aprun -B /bin/sh -c /bin/date<br>
> > > > > > /bin/echo $?<br>
> > > > > > >/home/ketan/.globus/scripts/PBS2583661693904024220.submit.exitcode<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > And see the same behavior. The exitcode file is indeed<br>
> > > > > > updated<br>
> > > > > > each<br>
> > > > > > time with a code 0.<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > - I realized that manually testing the swift-generated<br>
> > > > > > submit<br>
> > > > > > file<br>
> > > > > > will give new errors because the swift service is no longer<br>
> > > > > > alive<br>
> > > > > > and<br>
> > > > > > listening on the port that the worker will try to connect<br>
> > > > > > to.<br>
> > > > > > Also,<br>
> > > > > > it<br>
> > > > > > seemed that the .pl file itself that automatic coaster<br>
> > > > > > bootstrap<br>
> > > > > > places in ~/.globus/coasters was not there. Im assuming that<br>
> > > > > > Swift<br>
> > > > > > removes these files when it exits, but need to verify that<br>
> > > > > > this<br>
> > > > > > is<br>
> > > > > > true and that the failure is not due to a missing .pl file.<br>
> > > > > > I<br>
> > > > > > suspect<br>
> > > > > > that this is normal and is not the problem, but again, we<br>
> > > > > > need<br>
> > > > > > to<br>
> > > > > > keep<br>
> > > > > > debugging until the root cause is found.<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > Mike<br>
> > > > > ><br>
> > > > > ><br>
> > > > > > ----- Original Message -----<br>
> > > > > ><br>
> > > > > > > From: "Ketan Maheshwari" < <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> ><br>
> > > > > > > To: "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > > > > > > Cc: "Swift Devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > > Sent: Sunday, November 13, 2011 7:20:25 AM<br>
> > > > > > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > > > > > I tried with a simple /bin/date command at the end of the<br>
> > > > > > > submit<br>
> > > > > > > script removing the call to <a href="http://worker.pl" target="_blank">worker.pl</a> :<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > #CoG This script generated by CoG<br>
> > > > > > > #CoG by class: class<br>
> > > > > > > org.globus.cog.abstraction.impl.scheduler.pbs.PBSExecutor<br>
> > > > > > > #CoG on date: 2011/11/13 02:16:54<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > #PBS -S /bin/bash<br>
> > > > > > > #PBS -N Block-1113-1602<br>
> > > > > > > #PBS -m n<br>
> > > > > > > #PBS -A CI-DEB000002<br>
> > > > > > > #PBS -l mppwidth=3,mppnppn=1,mppdepth=24<br>
> > > > > > > #PBS -l walltime=00:10:00<br>
> > > > > > > #PBS -o<br>
> > > > > > > /home/ketan/.globus/scripts/PBS2583661693904024220.submit.stdout<br>
> > > > > > > #PBS -e<br>
> > > > > > > /home/ketan/.globus/scripts/PBS2583661693904024220.submit.stderr<br>
> > > > > > > WORKER_LOGGING_LEVEL=NONE<br>
> > > > > > > #PBS -v WORKER_LOGGING_LEVEL<br>
> > > > > > > cd / && aprun -n 3 -N 1 -cc none -d 24 -F exclusive<br>
> > > > > > > /bin/sh<br>
> > > > > > > -c<br>
> > > > > > > /bin/date<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > =======<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > This fails too. The queue cancels the job as soon as it<br>
> > > > > > > starts<br>
> > > > > > > running, without writing anything to stdout or stderr.<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > On Sun, Nov 13, 2011 at 12:54 AM, Michael Wilde <<br>
> > > > > > > <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a><br>
> > > > > > > ><br>
> > > > > > > wrote:<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > OK, I dont need these; I can reproduce the problem as<br>
> > > > > > > well.<br>
> > > > > > ><br>
> > > > > > > For some reason, the coaster worker is exiting<br>
> > > > > > > immediately.<br>
> > > > > > ><br>
> > > > > > > I see a few possibilities:<br>
> > > > > > ><br>
> > > > > > > - Beagle networking may have changed, making it no longer<br>
> > > > > > > possible<br>
> > > > > > > to<br>
> > > > > > > reach the coaster service from the compute nodes using the<br>
> > > > > > > previous<br>
> > > > > > > IP<br>
> > > > > > > address ranges.<br>
> > > > > > ><br>
> > > > > > > - the <a href="http://worker.pl" target="_blank">worker.pl</a> script is not being created in<br>
> > > > > > > $HOME/.globus/coasters<br>
> > > > > > ><br>
> > > > > > > Mike<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > ----- Original Message -----<br>
> > > > > > > > From: "Michael Wilde" < <a href="mailto:wilde@mcs.anl.gov">wilde@mcs.anl.gov</a> ><br>
> > > > > > > > To: "Ketan Maheshwari" < <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a> ><br>
> > > > > > > > Cc: "Swift Devel" < <a href="mailto:swift-devel@ci.uchicago.edu">swift-devel@ci.uchicago.edu</a> ><br>
> > > > > > > > Sent: Saturday, November 12, 2011 8:39:36 PM<br>
> > > > > > > > Subject: Re: [Swift-devel] swift pbs/beagle broken<br>
> > > > > > > > Ketan, can you post the submit script and site file?<br>
> > > > > > > ><br>
> > > > > > > > On 11/12/11, Ketan Maheshwari <<br>
> > > > > > > > <a href="mailto:ketancmaheshwari@gmail.com">ketancmaheshwari@gmail.com</a><br>
> > > > > > > > ><br>
> > > > > > > > wrote:<br>
> > > > > > > > > Hi,<br>
> > > > > > > > ><br>
> > > > > > > > > It seems the pbs-coaster provider (local:pbs) is<br>
> > > > > > > > > broken<br>
> > > > > > > > > for<br>
> > > > > > > > > swift.<br>
> > > > > > > > > I<br>
> > > > > > > > > tried<br>
> > > > > > > > > swift trunk, 0.93 svn branch, 0.93RC3 and 0.93RC4 but<br>
> > > > > > > > > getting<br>
> > > > > > > > > the<br>
> > > > > > > > > same<br>
> > > > > > > > > response:<br>
> > > > > > > > ><br>
> > > > > > > > > Swift svn swift-r5205 cog-r3293<br>
> > > > > > > > ><br>
> > > > > > > > > RunID: 20111113-0216-1d35h7eb<br>
> > > > > > > > > Progress: time: Sun, 13 Nov 2011 02:16:54 +0000<br>
> > > > > > > > > site setting workersPerNode has been replaced with<br>
> > > > > > > > > jobsPerNode!<br>
> > > > > > > > > Progress: time: Sun, 13 Nov 2011 02:17:05 +0000<br>
> > > > > > > > > Active:1<br>
> > > > > > > > > Failed to transfer wrapper log for job cat-1hg8aoik<br>
> > > > > > > > > Exception in cat:<br>
> > > > > > > > > Arguments: [data.txt]<br>
> > > > > > > > > Host: pbs<br>
> > > > > > > > > Directory:<br>
> > > > > > > > > catsn-20111113-0216-1d35h7eb/jobs/1/cat-1hg8aoik<br>
> > > > > > > > > stderr.txt:<br>
> > > > > > > > ><br>
> > > > > > > > > stdout.txt:<br>
> > > > > > > > ><br>
> > > > > > > > > ----<br>
> > > > > > > > ><br>
> > > > > > > > > Caused by: Task failed: 1113-160254-000000 Block task<br>
> > > > > > > > > ended<br>
> > > > > > > > > prematurely<br>
> > > > > > > > ><br>
> > > > > > > > ><br>
> > > > > > > > > Final status: time: Sun, 13 Nov 2011 02:17:05 +0000<br>
> > > > > > > > > Failed:1<br>
> > > > > > > > > The following errors have occurred:<br>
> > > > > > > > > 1. Task failed: 1113-160254-000000 Block task ended<br>
> > > > > > > > > prematurely<br>
> > > > > > > > ><br>
> > > > > > > > ><br>
> > > > > > > > ><br>
> > > > > > > > > Trying the submit script outside of swift also does<br>
> > > > > > > > > not<br>
> > > > > > > > > seem<br>
> > > > > > > > > to<br>
> > > > > > > > > be<br>
> > > > > > > > > working.<br>
> > > > > > > > > The scripts get submitted to the queue and immediately<br>
> > > > > > > > > exits<br>
> > > > > > > > > without<br>
> > > > > > > > > writing anything to stdout or stderr.<br>
> > > > > > > > ><br>
> > > > > > > > > Were there any recent changes that could have affected<br>
> > > > > > > > > this?<br>
> > > > > > > > ><br>
> > > > > > > > > I remember to have tried this successfully in the last<br>
> > > > > > > > > week<br>
> > > > > > > > > of<br>
> > > > > > > > > last<br>
> > > > > > > > > month.<br>
> > > > > > > > ><br>
> > > > > > > > > Regards,<br>
> > > > > > > > > --<br>
> > > > > > > > > Ketan<br>
> > > > > > > > ><br>
> > > > > > > ><br>
> > > > > > > > --<br>
> > > > > > > > Sent from my mobile device<br>
> > > > > > > > _______________________________________________<br>
> > > > > > > > Swift-devel mailing list<br>
> > > > > > > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > > > > > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > > > > > ><br>
> > > > > > > --<br>
> > > > > > > Michael Wilde<br>
> > > > > > > Computation Institute, University of Chicago<br>
> > > > > > > Mathematics and Computer Science Division<br>
> > > > > > > Argonne National Laboratory<br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > ><br>
> > > > > > > --<br>
> > > > > > > Ketan<br>
> > > > > ><br>
> > > > > > --<br>
> > > > > > Michael Wilde<br>
> > > > > > Computation Institute, University of Chicago<br>
> > > > > > Mathematics and Computer Science Division<br>
> > > > > > Argonne National Laboratory<br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > ><br>
> > > > > > --<br>
> > > > > > Ketan<br>
> > > > ><br>
> > > > > --<br>
> > > > > Michael Wilde<br>
> > > > > Computation Institute, University of Chicago<br>
> > > > > Mathematics and Computer Science Division<br>
> > > > > Argonne National Laboratory<br>
> > > > ><br>
> > > > > _______________________________________________<br>
> > > > > Swift-devel mailing list<br>
> > > > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > > ><br>
> > > > --<br>
> > > > Michael Wilde<br>
> > > > Computation Institute, University of Chicago<br>
> > > > Mathematics and Computer Science Division<br>
> > > > Argonne National Laboratory<br>
> > > ><br>
> > > > _______________________________________________<br>
> > > > Swift-devel mailing list<br>
> > > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> > ><br>
> > > --<br>
> > > Michael Wilde<br>
> > > Computation Institute, University of Chicago<br>
> > > Mathematics and Computer Science Division<br>
> > > Argonne National Laboratory<br>
> > ><br>
> > > _______________________________________________<br>
> > > Swift-devel mailing list<br>
> > > <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> > > <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
> ><br>
> > --<br>
> > Michael Wilde<br>
> > Computation Institute, University of Chicago<br>
> > Mathematics and Computer Science Division<br>
> > Argonne National Laboratory<br>
><br>
> --<br>
> Michael Wilde<br>
> Computation Institute, University of Chicago<br>
> Mathematics and Computer Science Division<br>
> Argonne National Laboratory<br>
><br>
> _______________________________________________<br>
> Swift-devel mailing list<br>
> <a href="mailto:Swift-devel@ci.uchicago.edu">Swift-devel@ci.uchicago.edu</a><br>
> <a href="https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel" target="_blank">https://lists.ci.uchicago.edu/cgi-bin/mailman/listinfo/swift-devel</a><br>
<br>
--<br>
Michael Wilde<br>
Computation Institute, University of Chicago<br>
Mathematics and Computer Science Division<br>
Argonne National Laboratory<br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br>Ketan<br><br><br>