[Swift-devel] Fwd: Re: globus jobs killed on NCSA's IA64 cluster (mercury)

[Mihael Hategan]

On Wed, 2007-03-21 at 10:27 -0500, Veronika V. Nefedova wrote:
> Hi, Mihael:
> 
> I have these properties modified in my scheduler.xml file:
> 
>                  <property name="jobThrottle" value="384"/>
> <property name="maxSimultaneousJobs" value="384"/>
> 
> Are you suggesting to add also this inside <scheduler>...</scheduler> :
> 
> <property name="submitThrottle" value="1"/> ?
> 
> Do these set parameters guarantee me that:
> 
> 1. I have no more then 384 jobs in a queue at any time
> and
> 2. Jobs are submitted to the queue with at least 1 sec delay

No. They don't. But they may get you closer to that.

> 
> (these are the requirements from TG NCSA).
> 
> Thanks!
> 
> Nika
> 
> At 09:18 AM 3/17/2007, Mihael Hategan wrote:
> >There is no direct rate limiter unfortunately. There is a submit
> >throttle which tells the number of concurrent submissions. Setting that
> >to 1 may work.
> >
> >On Fri, 2007-03-16 at 17:02 -0500, Veronika V. Nefedova wrote:
> > > Hi, Mihael:
> > >
> > > how do I set this throttling parameter ?
> > >
> > > Thanks,
> > >
> > > Nika
> > >
> > > >Date: Fri, 16 Mar 2007 15:53:57 -0600
> > > >Subject: Re: globus jobs killed on NCSA's IA64 cluster (mercury)
> > > >To: nefedova at mcs.anl.gov
> > > >From: consult at ncsa.uiuc.edu
> > > >Cc:
> > > >X-Mailer: Perl5 Mail::Internet v1.74
> > > >Sender: Nobody <nobody at ncsa.uiuc.edu>
> > > >X-Null-Tag: 2edd4a9833fa010df5441f1443ff58a9
> > > >X-NCSA-MailScanner-Information: Please contact help at ncsa.uiuc.edu for 
> > more
> > > >information, amantadine.ncsa.uiuc.edu
> > > >X-NCSA-MailScanner: Found to be clean
> > > >X-Virus-Scanned: by amavisd-new-20030616-p10 (Debian) at 
> > mailgw.mcs.anl.gov
> > > >
> > > >FROM: Arnold, Galen
> > > >(Concerning ticket No. 137212)
> > > >
> > > >Veronika,
> > > >
> > > >If you can throttle the job submission so that there's more than 1 second
> > > >between them, that would probably help us out.
> > > >
> > > >-Galen
> > > >
> > > >Veronika V. Nefedova <nefedova at mcs.anl.gov> writes:
> > > > >Hi, Galen:
> > > > >
> > > > >I was told that I could have 384 jobs in PBS at the same time. By my
> > > > >estimation I had no more then 136 jobs in the queue. What are other 
> > limits
> > > > >I should keep in mind ?
> > > > >
> > > > >Thanks,
> > > > >
> > > > >Veronika
> > > > >
> > > > >At 04:31 PM 3/16/2007, consult at ncsa.uiuc.edu wrote:
> > > > >>FROM: Arnold, Galen
> > > > >>(Concerning ticket No. 137212)
> > > > >>
> > > > >>Veronika,
> > > > >>
> > > > >>We see many sequences in our globus-gatekeeper.log on tg-login1 
> > like this
> > > > >>
> > > > >>Notice: 5: Authenticated globus user:
> > > > >>/DC=org/DC=doegrids/OU=People/CN=Veronika
> > > > >>Nefedova 137823
> > > > >>Notice: 0: GRID_SECURITY_HTTP_BODY_FD=6
> > > > >>Notice: 5: Requested service: jobmanager-pbs
> > > > >>Notice: 5: Authorized as local user: nefedova
> > > > >>Notice: 5: Authorized as local uid: 29202
> > > > >>Notice: 5:           and local gid: 11467
> > > > >>Notice: 0: executing
> > > > >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager
> > > > >>Notice: 0: GRID_SECURITY_CONTEXT_FD=9
> > > > >>Notice: 0: Child 16725 started
> > > > >>Notice: 5: Authenticated globus user:
> > > > >>/DC=org/DC=doegrids/OU=People/CN=Veronika
> > > > >>Nefedova 137823
> > > > >>Notice: 0: GRID_SECURITY_HTTP_BODY_FD=6
> > > > >>Notice: 5: Requested service: jobmanager-pbs
> > > > >>Notice: 5: Authorized as local user: nefedova
> > > > >>Notice: 5: Authorized as local uid: 29202
> > > > >>Notice: 5:           and local gid: 11467
> > > > >>Notice: 0: executing
> > > > >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager
> > > > >>Notice: 0: GRID_SECURITY_CONTEXT_FD=9
> > > > >>Notice: 0: Child 16726 started
> > > > >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16769
> > > > >>starting at Fri Mar 16 09:34:12 2007
> > > > >>
> > > > >>
> > > > >>...your DN is also popular in the globus-gatekeeper.log.
> > > > >>
> > > > >>tg-login1:/usr/local/globus-2.4.3-gcc-r5/var # grep 'CN=Veronika 
> > Nefedova'
> > > > >>globus-gatekeeper.log |wc -l
> > > > >>   15215
> > > > >>
> > > > >>
> > > > >>The large number of connects around the same time:
> > > > >>
> > > > >>  grep --after-context=10 Veronika globus-gatekeeper.log | grep 
> > starting |
> > > > >> grep
> > > > >>'Mar 16 09:3' | wc -l
> > > > >>      65
> > > > >>...
> > > > >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=15872
> > > > >>starting at Fri Mar 16 09:34:02 2007
> > > > >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16644
> > > > >>starting at Fri Mar 16 09:34:11 2007
> > > > >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16769
> > > > >>starting at Fri Mar 16 09:34:12 2007
> > > > >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16926
> > > > >>starting at Fri Mar 16 09:34:14 2007
> > > > >>...
> > > > >>
> > > > >>
> > > > >>...could be overwhelming the gatekeeper on tg-login1.
> > > > >>
> > > > >>-Galen
> > > > >>
> > > > >>
> > > > >>Veronika V. Nefedova <nefedova at mcs.anl.gov> writes:
> > > > >> >Hi, Dave:
> > > > >> >
> > > > >> >Could you please give me a bit more information on what has 
> > happened ?
> > > > >> >
> > > > >> >Thanks,
> > > > >> >
> > > > >> >Nika
> > > > >> >
> > > > >> >At 02:12 PM 3/16/2007, consult at ncsa.uiuc.edu wrote:
> > > > >> >>FROM: McWilliams, David G
> > > > >> >>(Concerning ticket No. 137212)
> > > > >> >>
> > > > >> >>Veronika,
> > > > >> >>
> > > > >> >>The system administrator had to kill several of your globus
> > > > processes today
> > > > >> >>because the load average on the node was over 25. Below is a list of
> > > > >> processes
> > > > >> >>that were killed. Please let us know if you need help 
> > identifying the
> > > > >> case of
> > > > >> >>the problem.
> > > > >> >>
> > > > >> >>Regards,
> > > > >> >>
> > > > >> >>Dave McWilliams            (217) 
> > 244-1144          consult at ncsa.uiuc.edu
> > > > >> >>NCSA Consulting
> > > > Services    http://www.ncsa.uiuc.edu/UserInfo/Consulting/
> > > > >> >>----------------------------------------------------------------- 
> > ----
> > > > -----
> > > > >> >>
> > > > >> >>nefedova 11084     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11165     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11166     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11277     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11295     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11379     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11402     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11480     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11522     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11602     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11668     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11715     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11785     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11813     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11892     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 11948     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12004     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12030     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12133     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12172     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12253     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12256     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12443     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12460     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12504     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12534     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12657     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12668     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12773     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12806     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12892     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 12946     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13023     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13072     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13142     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13233     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13245     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13352     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13379     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13474     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13488     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13618     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13663     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13743     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13820     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13887     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 13952     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14046     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14048     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14172     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14196     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14319     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14366     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14430     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14539     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14572     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14703     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14725     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14832     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 14849     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15009     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15017     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15164     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15165     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15322     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15332     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15526     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15544     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15671     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15672     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15787     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15842     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15981     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 15982     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16131     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16132     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16320     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16358     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16553     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16622     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16725     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16726     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16845     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16925     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 16989     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 17095     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 17262     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 17305     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 17375     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > > > >> pbs_gcc -rdn
> > > > >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > > > >> >>nefedova 31347 14172  4 13:35 ?        00:00:00 /usr/bin/perl
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script 
> > .pl -m
> > > > >> >>pbs_gcc
> > > > >> >>-f /tmp/gram_1Drc0X -c poll
> > > > >> >>nefedova 31349 11295  4 13:35 ?        00:00:00 /usr/bin/perl
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script 
> > .pl -m
> > > > >> >>pbs_gcc
> > > > >> >>-f /tmp/gram_fSWNvt -c poll
> > > > >> >>nefedova 31352 11522  4 13:35 ?        00:00:00 /usr/bin/perl
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script 
> > .pl -m
> > > > >> >>pbs_gcc
> > > > >> >>-f /tmp/gram_a3whWx -c poll
> > > > >> >>nefedova 31380 16845  4 13:35 ?        00:00:00 /usr/bin/perl
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script 
> > .pl -m
> > > > >> >>pbs_gcc
> > > > >> >>-f /tmp/gram_zz0JuC -c poll
> > > > >> >>nefedova 31396 14832  4 13:35 ?        00:00:00 /usr/bin/perl
> > > > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script 
> > .pl -m
> > > > >> >>pbs_gcc
> > > > >> >>-f /tmp/gram_5jGFI1 -c poll
> > > > >> >>nefedova 31512 31347  0 13:35 ?        00:00:00 sh -c
> > > > >> >>/usr/local/pbs/ia64/bin/qstat -f 905629.tg-master.ncsa.teragrid.org
> > > > >> >>2>/dev/null
> > > > >> >>nefedova 31546 31512  0 13:35 ?        00:00:00
> > > > >> >>/usr/local/pbs/ia64/bin/qstat -f
> > > > >> >>905629.tg-master.ncsa.teragrid.org
> > > > >> >>nefedova 31578 31555  0 13:35 ?        00:00:00
> > > > >> >>/usr/local/pbs/ia64/bin/qstat -f
> > > > >> >>905638.tg-master.ncsa.teragrid.org
> > > > >> >>nefedova 31629 31560  0 13:35 ?        00:00:00 sh -c
> > > > >> >>/usr/local/torque-2.1.7/ia64/sbin/pbs_iff 
> > tg-master.ncsa.teragrid.org
> > > > >> 15001 3
> > > > >> >>nefedova 31634 31578  0 13:35 ?        00:00:00 sh -c
> > > > >> >>/usr/local/torque-2.1.7/ia64/sbin/pbs_iff 
> > tg-master.ncsa.teragrid.org
> > > > >> 15001 3
> > >
> > >
> > > _______________________________________________
> > > Swift-devel mailing list
> > > Swift-devel at ci.uchicago.edu
> > > http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
> > >
> 
>