[Swift-user] Re: Copy of parallel scripting paper
William Emmanuel S. Yu
william.yu at novare.com.hk
Mon Nov 16 02:48:44 CST 2009
Hey. Thanks for the quick response. I do have to note that I haven't
gone through much of the documentation yet. But, these tidbits
definitely help.
Again thanks.
On Mon, 2009-11-16 at 02:35 -0600, Allan Espinosa wrote:
> Hi william,
>
> Yeah. typically i place a tc.data in my current directory and then
> specify it with the "-tc.file ./tc.data" option.
>
> I guess we can run on PBS Torque. Here's a sample sites.xml config:
>
> <config>
> <pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
> <execution provider="pbs" url="none" />
> <gridftp url="local://localhost" />
> <workdirectory >/home/aespinosa</workdirectory>
>
> <profile namespace="karajan" key="initialScore">2.02</profile>
> <profile namespace="karajan" key="jobThrottle">1.98</profile>
>
> <profile namespace="globus" key="queue">fast</profile>
> <profile namespace="globus" key="maxwalltime">01:00:00</profile>
> </pool>
> </config>
>
> My initialScore and jobThrottle score enables me to submit a maximum
> of 200 jobs at a time.
>
>
> Btw, I'm cc'ing you to the swift-user mailing list. Hope you enjoy
> tinkering around Swift!
>
> -Allan
>
> 2009/11/16 William Emmanuel S. Yu <william.yu at novare.com.hk>:
> >
> > I found the tc.data file. geez... its in a strange place! The etc folder
> > in the distribution directory.
> >
> > On Mon, 2009-11-16 at 16:22 +0800, William Emmanuel Yu wrote:
> >> On Mon, 2009-11-16 at 02:14 -0600, Allan Espinosa wrote:
> >> > I guess that works. But if you have a scheduler, you should use it.
> >> > we have support for pbs, cobalt and condor. any unsupported scheduler
> >> > can be interfaced with Globus GRAM2 or GRAM4. From my perspective,
> >> > the cluster sysads should make it easy for its users to submit jobs :)
> >> >
> >> Ok right now. That systems uses Torque and is really maintained at this
> >> point. So it was a real challenge using it. But, I managed to write a
> >> script with bash and mpirun that got the job done. However, I did
> >> encounter a lot of dependency issues. Haha. As expected.
> >>
> >> Do you have a document on torque integration? At least a quick one.
> >>
> >> Btw, I am using the default swift binary on my laptop with no installed
> >> scheduler. I ran into the following error:
> >>
> >> "Could not find any valid host for task "Task(type=UNKNOWN,
> >> identity=urn:cog-1258359742934)" with constraints {tr=convert,
> >> filenames=[Ljava.lang.String;@2de41d, trfqn=convert,
> >> filecache=org.griphyn.vdl.karajan.lib.cache.CacheMapAdapter at db4bcf}"
> >>
> >> Which is strange as I am not using any scheduler.
> >>
> >> Thanks!
> >>
> >>
> >>
> >> > 2009/11/15 William Emmanuel Yu <wyu at ateneo.edu>:
> >> > >
> >> > > After reading the article, I think I have a better appreciate of the
> >> > > problems you are trying to solve. Let me try to read about SWIFT more.
> >> > >
> >> > > What is the easier way to run SWIFT on a small cluster (say 64 cores on
> >> > > 16 nodes)? Can I just do an NFS and passwordless-SSH thing? I think I
> >> > > will try to help a buddy of mine with his thesis on aquatic migrations
> >> > > in UP as practice.
> >> > >
> >> > > Thanks.
> >> > >
> >> > > On Thu, 2009-11-12 at 20:11 -0600, Allan Espinosa wrote:
> >> > >> you can try the swift tutorial designed for localhost:
> >> > >>
> >> > >> http://www.ci.uchicago.edu/swift/guides/tutorial.php
> >> > >>
> >> > >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
> >> > >> > let me review first if this is easier to teach ... but, if you have a
> >> > >> > cookbook for a quick local install then that would also be cool.
> >> > >> >
> >> > >> > On Thu, 2009-11-12 at 19:08 -0600, Allan Espinosa wrote:
> >> > >> >> its user and setup easy when you use a local scheduler (run from
> >> > >> >> localhost). I can help you on the setup on local host or via ssh.
> >> > >> >> Some of my colleagues have also tried using this over Amazon EC2.
> >> > >> >>
> >> > >> >> -Allan
> >> > >> >>
> >> > >> >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
> >> > >> >> >
> >> > >> >> > interesting.. but this is setup heavy but user easy right?
> >> > >> >> >
> >> > >> >> > On Thu, 2009-11-12 at 18:58 -0600, Allan Espinosa wrote:
> >> > >> >> >> hi william
> >> > >> >> >>
> >> > >> >> >> see attached file.
> >> > >> >> >>
> >> > >> >> >> We are using Swift (http://www.ci.uchicago.edu/swift). It has
> >> > >> >> >> adapters to cluster schedulers like PBS, GRAM2, GRAM4, Cobalt and
> >> > >> >> >> Condor. But you can use ssh and local (fork) for a start.
> >> > >> >> >>
> >> > >> >> >> We couple it to Falkon (http://dev.globus.org/wiki/Incubator/Falkon)
> >> > >> >> >> for faster job throughput in short-time jobs and deployments on
> >> > >> >> >> supercomputers.
> >> > >> >> >>
> >> > >> >> >> -Allan
> >> > >> >> >>
> >> > >> >> >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
> >> > >> >> >> >
> >> > >> >> >> > Can I get a copy of this paper? What tools are you using now?
> >> > >> >> >> >
> >> > >> >> >> > --
> >> > >> >> >> > -------------------------------------------------------
> >> > >> >> >> > William Emmanuel S. Yu (杨怀义)
> >> > >> >> >> > Department of Information Systems and Computer Science
> >> > >> >> >> > Ateneo de Manila University
> >> > >> >> >> > email : wyu at ateneo dot edu
> >> > >> >> >> > blog : http://hip2b2.yutivo.org/
> >> > >> >> >> > web : http://CNG.ateneo.edu/cng/wyu/
> >> > >> >> >> > phone : +63(2)4266001 loc. 4186
> >> > >> >> >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
> >> > >> >> >> >
> >> > >> >> >> > Confidentiality Issue: This message is intended only for the use of the
> >> > >> >> >> > addressee and may contain information that is privileged and
> >> > >> >> >> > confidential. If you are not the intended recipient, you are hereby
> >> > >> >> >> > notified that any use or dissemination of this communication is strictly
> >> > >> >> >> > prohibited. If you have received this communication in error, please
> >> > >> >> >> > notify us immediately by reply and delete this message from your system.
> >> > >> >> >> >
> >> > >> >> >> >
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> >>
> >> > >> >> > --
> >> > >> >> > -------------------------------------------------------
> >> > >> >> > William Emmanuel S. Yu (杨怀义)
> >> > >> >> > Department of Information Systems and Computer Science
> >> > >> >> > Ateneo de Manila University
> >> > >> >> > email : wyu at ateneo dot edu
> >> > >> >> > blog : http://hip2b2.yutivo.org/
> >> > >> >> > web : http://CNG.ateneo.edu/cng/wyu/
> >> > >> >> > phone : +63(2)4266001 loc. 4186
> >> > >> >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
> >> > >> >> >
> >> > >> >> > Confidentiality Issue: This message is intended only for the use of the
> >> > >> >> > addressee and may contain information that is privileged and
> >> > >> >> > confidential. If you are not the intended recipient, you are hereby
> >> > >> >> > notified that any use or dissemination of this communication is strictly
> >> > >> >> > prohibited. If you have received this communication in error, please
> >> > >> >> > notify us immediately by reply and delete this message from your system.
> >> > >> >> >
> >> > >> >> >
> >> > >> >>
> >> > >> >>
> >> > >> >>
> >> > >> > --
> >> > >> > -------------------------------------------------------
> >> > >> > William Emmanuel S. Yu (杨怀义)
> >> > >> > Department of Information Systems and Computer Science
> >> > >> > Ateneo de Manila University
> >> > >> > email : wyu at ateneo dot edu
> >> > >> > blog : http://hip2b2.yutivo.org/
> >> > >> > web : http://CNG.ateneo.edu/cng/wyu/
> >> > >> > phone : +63(2)4266001 loc. 4186
> >> > >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
> >> > >> >
> >> > >> > Confidentiality Issue: This message is intended only for the use of the
> >> > >> > addressee and may contain information that is privileged and
> >> > >> > confidential. If you are not the intended recipient, you are hereby
> >> > >> > notified that any use or dissemination of this communication is strictly
> >> > >> > prohibited. If you have received this communication in error, please
> >> > >> > notify us immediately by reply and delete this message from your system.
> >> > >> >
> >> > >> >
> >> > >>
> >> > >>
> >> > >>
> >> > > --
> >> > > -------------------------------------------------------
> >> > > William Emmanuel S. Yu (杨怀义)
> >> > > Department of Information Systems and Computer Science
> >> > > Ateneo de Manila University
> >> > > email : wyu at ateneo dot edu
> >> > > blog : http://hip2b2.yutivo.org/
> >> > > web : http://CNG.ateneo.edu/cng/wyu/
> >> > > phone : +63(2)4266001 loc. 4186
> >> > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
> >> > >
> >> > > Confidentiality Issue: This message is intended only for the use of the
> >> > > addressee and may contain information that is privileged and
> >> > > confidential. If you are not the intended recipient, you are hereby
> >> > > notified that any use or dissemination of this communication is strictly
> >> > > prohibited. If you have received this communication in error, please
> >> > > notify us immediately by reply and delete this message from your system.
> >> > >
> >> > >
> >> >
> >> >
> >> >
> > --
> > -------------------------------------------------------
> > William Emmanuel S. Yu (杨怀义)
> > Novare Technologies Inc.
> > 6th Floor Peninsula Court Building,
> > Makati Avenue corner Paseo de Roxas Avenue,
> > Makati City, 1226 Philippines
> > email : william dot yu at novare dot com dot hk
> > web : www.novare.com.hk
> >
> > Confidentiality Issue: This message is intended only for the use of the
> > addressee and may contain information that is privileged and
> > confidential. If you are not the intended recipient, you are hereby
> > notified that any use or dissemination of this communication is strictly
> > prohibited. If you have received this communication in error, please
> > notify us immediately by reply and delete this message from your system.
> >
> >
>
>
>
--
-------------------------------------------------------
William Emmanuel S. Yu (杨怀义)
Novare Technologies Inc.
6th Floor Peninsula Court Building,
Makati Avenue corner Paseo de Roxas Avenue,
Makati City, 1226 Philippines
email : william dot yu at novare dot com dot hk
web : www.novare.com.hk
Confidentiality Issue: This message is intended only for the use of the
addressee and may contain information that is privileged and
confidential. If you are not the intended recipient, you are hereby
notified that any use or dissemination of this communication is strictly
prohibited. If you have received this communication in error, please
notify us immediately by reply and delete this message from your system.
More information about the Swift-user
mailing list