[Swift-user] Re: Copy of parallel scripting paper
Allan Espinosa
yecartes at gmail.com
Mon Nov 16 02:35:58 CST 2009
Hi william,
Yeah. typically i place a tc.data in my current directory and then
specify it with the "-tc.file ./tc.data" option.
I guess we can run on PBS Torque. Here's a sample sites.xml config:
<config>
<pool handle="TERAPORT" sysinfo="INTEL32::LINUX">
<execution provider="pbs" url="none" />
<gridftp url="local://localhost" />
<workdirectory >/home/aespinosa</workdirectory>
<profile namespace="karajan" key="initialScore">2.02</profile>
<profile namespace="karajan" key="jobThrottle">1.98</profile>
<profile namespace="globus" key="queue">fast</profile>
<profile namespace="globus" key="maxwalltime">01:00:00</profile>
</pool>
</config>
My initialScore and jobThrottle score enables me to submit a maximum
of 200 jobs at a time.
Btw, I'm cc'ing you to the swift-user mailing list. Hope you enjoy
tinkering around Swift!
-Allan
2009/11/16 William Emmanuel S. Yu <william.yu at novare.com.hk>:
>
> I found the tc.data file. geez... its in a strange place! The etc folder
> in the distribution directory.
>
> On Mon, 2009-11-16 at 16:22 +0800, William Emmanuel Yu wrote:
>> On Mon, 2009-11-16 at 02:14 -0600, Allan Espinosa wrote:
>> > I guess that works. But if you have a scheduler, you should use it.
>> > we have support for pbs, cobalt and condor. any unsupported scheduler
>> > can be interfaced with Globus GRAM2 or GRAM4. From my perspective,
>> > the cluster sysads should make it easy for its users to submit jobs :)
>> >
>> Ok right now. That systems uses Torque and is really maintained at this
>> point. So it was a real challenge using it. But, I managed to write a
>> script with bash and mpirun that got the job done. However, I did
>> encounter a lot of dependency issues. Haha. As expected.
>>
>> Do you have a document on torque integration? At least a quick one.
>>
>> Btw, I am using the default swift binary on my laptop with no installed
>> scheduler. I ran into the following error:
>>
>> "Could not find any valid host for task "Task(type=UNKNOWN,
>> identity=urn:cog-1258359742934)" with constraints {tr=convert,
>> filenames=[Ljava.lang.String;@2de41d, trfqn=convert,
>> filecache=org.griphyn.vdl.karajan.lib.cache.CacheMapAdapter at db4bcf}"
>>
>> Which is strange as I am not using any scheduler.
>>
>> Thanks!
>>
>>
>>
>> > 2009/11/15 William Emmanuel Yu <wyu at ateneo.edu>:
>> > >
>> > > After reading the article, I think I have a better appreciate of the
>> > > problems you are trying to solve. Let me try to read about SWIFT more.
>> > >
>> > > What is the easier way to run SWIFT on a small cluster (say 64 cores on
>> > > 16 nodes)? Can I just do an NFS and passwordless-SSH thing? I think I
>> > > will try to help a buddy of mine with his thesis on aquatic migrations
>> > > in UP as practice.
>> > >
>> > > Thanks.
>> > >
>> > > On Thu, 2009-11-12 at 20:11 -0600, Allan Espinosa wrote:
>> > >> you can try the swift tutorial designed for localhost:
>> > >>
>> > >> http://www.ci.uchicago.edu/swift/guides/tutorial.php
>> > >>
>> > >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
>> > >> > let me review first if this is easier to teach ... but, if you have a
>> > >> > cookbook for a quick local install then that would also be cool.
>> > >> >
>> > >> > On Thu, 2009-11-12 at 19:08 -0600, Allan Espinosa wrote:
>> > >> >> its user and setup easy when you use a local scheduler (run from
>> > >> >> localhost). I can help you on the setup on local host or via ssh.
>> > >> >> Some of my colleagues have also tried using this over Amazon EC2.
>> > >> >>
>> > >> >> -Allan
>> > >> >>
>> > >> >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
>> > >> >> >
>> > >> >> > interesting.. but this is setup heavy but user easy right?
>> > >> >> >
>> > >> >> > On Thu, 2009-11-12 at 18:58 -0600, Allan Espinosa wrote:
>> > >> >> >> hi william
>> > >> >> >>
>> > >> >> >> see attached file.
>> > >> >> >>
>> > >> >> >> We are using Swift (http://www.ci.uchicago.edu/swift). It has
>> > >> >> >> adapters to cluster schedulers like PBS, GRAM2, GRAM4, Cobalt and
>> > >> >> >> Condor. But you can use ssh and local (fork) for a start.
>> > >> >> >>
>> > >> >> >> We couple it to Falkon (http://dev.globus.org/wiki/Incubator/Falkon)
>> > >> >> >> for faster job throughput in short-time jobs and deployments on
>> > >> >> >> supercomputers.
>> > >> >> >>
>> > >> >> >> -Allan
>> > >> >> >>
>> > >> >> >> 2009/11/12 William Emmanuel Yu <wyu at ateneo.edu>:
>> > >> >> >> >
>> > >> >> >> > Can I get a copy of this paper? What tools are you using now?
>> > >> >> >> >
>> > >> >> >> > --
>> > >> >> >> > -------------------------------------------------------
>> > >> >> >> > William Emmanuel S. Yu (杨怀义)
>> > >> >> >> > Department of Information Systems and Computer Science
>> > >> >> >> > Ateneo de Manila University
>> > >> >> >> > email : wyu at ateneo dot edu
>> > >> >> >> > blog : http://hip2b2.yutivo.org/
>> > >> >> >> > web : http://CNG.ateneo.edu/cng/wyu/
>> > >> >> >> > phone : +63(2)4266001 loc. 4186
>> > >> >> >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
>> > >> >> >> >
>> > >> >> >> > Confidentiality Issue: This message is intended only for the use of the
>> > >> >> >> > addressee and may contain information that is privileged and
>> > >> >> >> > confidential. If you are not the intended recipient, you are hereby
>> > >> >> >> > notified that any use or dissemination of this communication is strictly
>> > >> >> >> > prohibited. If you have received this communication in error, please
>> > >> >> >> > notify us immediately by reply and delete this message from your system.
>> > >> >> >> >
>> > >> >> >> >
>> > >> >> >>
>> > >> >> >>
>> > >> >> >>
>> > >> >> > --
>> > >> >> > -------------------------------------------------------
>> > >> >> > William Emmanuel S. Yu (杨怀义)
>> > >> >> > Department of Information Systems and Computer Science
>> > >> >> > Ateneo de Manila University
>> > >> >> > email : wyu at ateneo dot edu
>> > >> >> > blog : http://hip2b2.yutivo.org/
>> > >> >> > web : http://CNG.ateneo.edu/cng/wyu/
>> > >> >> > phone : +63(2)4266001 loc. 4186
>> > >> >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
>> > >> >> >
>> > >> >> > Confidentiality Issue: This message is intended only for the use of the
>> > >> >> > addressee and may contain information that is privileged and
>> > >> >> > confidential. If you are not the intended recipient, you are hereby
>> > >> >> > notified that any use or dissemination of this communication is strictly
>> > >> >> > prohibited. If you have received this communication in error, please
>> > >> >> > notify us immediately by reply and delete this message from your system.
>> > >> >> >
>> > >> >> >
>> > >> >>
>> > >> >>
>> > >> >>
>> > >> > --
>> > >> > -------------------------------------------------------
>> > >> > William Emmanuel S. Yu (杨怀义)
>> > >> > Department of Information Systems and Computer Science
>> > >> > Ateneo de Manila University
>> > >> > email : wyu at ateneo dot edu
>> > >> > blog : http://hip2b2.yutivo.org/
>> > >> > web : http://CNG.ateneo.edu/cng/wyu/
>> > >> > phone : +63(2)4266001 loc. 4186
>> > >> > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
>> > >> >
>> > >> > Confidentiality Issue: This message is intended only for the use of the
>> > >> > addressee and may contain information that is privileged and
>> > >> > confidential. If you are not the intended recipient, you are hereby
>> > >> > notified that any use or dissemination of this communication is strictly
>> > >> > prohibited. If you have received this communication in error, please
>> > >> > notify us immediately by reply and delete this message from your system.
>> > >> >
>> > >> >
>> > >>
>> > >>
>> > >>
>> > > --
>> > > -------------------------------------------------------
>> > > William Emmanuel S. Yu (杨怀义)
>> > > Department of Information Systems and Computer Science
>> > > Ateneo de Manila University
>> > > email : wyu at ateneo dot edu
>> > > blog : http://hip2b2.yutivo.org/
>> > > web : http://CNG.ateneo.edu/cng/wyu/
>> > > phone : +63(2)4266001 loc. 4186
>> > > GPG : http://CNG.ateneo.net/cng/wyu/wyy.pgp
>> > >
>> > > Confidentiality Issue: This message is intended only for the use of the
>> > > addressee and may contain information that is privileged and
>> > > confidential. If you are not the intended recipient, you are hereby
>> > > notified that any use or dissemination of this communication is strictly
>> > > prohibited. If you have received this communication in error, please
>> > > notify us immediately by reply and delete this message from your system.
>> > >
>> > >
>> >
>> >
>> >
> --
> -------------------------------------------------------
> William Emmanuel S. Yu (杨怀义)
> Novare Technologies Inc.
> 6th Floor Peninsula Court Building,
> Makati Avenue corner Paseo de Roxas Avenue,
> Makati City, 1226 Philippines
> email : william dot yu at novare dot com dot hk
> web : www.novare.com.hk
>
> Confidentiality Issue: This message is intended only for the use of the
> addressee and may contain information that is privileged and
> confidential. If you are not the intended recipient, you are hereby
> notified that any use or dissemination of this communication is strictly
> prohibited. If you have received this communication in error, please
> notify us immediately by reply and delete this message from your system.
>
>
--
Allan M. Espinosa <http://allan.88-mph.net/blog>
PhD student, Computer Science
University of Chicago <http://people.cs.uchicago.edu/~aespinosa>
More information about the Swift-user
mailing list