[Swift-devel] Fwd: Re: globus jobs killed on NCSA's IA64 cluster (mercury)

[Mihael Hategan]

On Wed, 2007-03-21 at 16:50 -0500, Veronika V. Nefedova wrote:
> Ok, NCSA wants to space job submissions to 5 seconds now... I am not sure 
> if we can do that ?

The only thing left is to add a submission rate limiter, but this won't
happen overnight.

> (like inserting 'sleep' before each submission)
> 
> Nika
> 
> 
> >Date: Wed, 21 Mar 2007 16:17:59 -0500
> >Subject: Re: globus jobs killed on NCSA's IA64 cluster (mercury)
> >To: nefedova at mcs.anl.gov
> >From: consult at ncsa.uiuc.edu
> >
> >FROM: Arnold, Galen
> >(Concerning ticket No. 137212)
> >
> >Veronika,
> >
> >Could you space these out a little more?
> >
> >Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper pid=12889
> >starting at Wed Mar 21 15:52:23 2007
> >Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper pid=12999
> >starting at Wed Mar 21 15:52:25 2007
> >Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper pid=13024
> >starting at Wed Mar 21 15:52:27 2007
> >Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper pid=13142
> >starting at Wed Mar 21 15:52:29 2007
> >
> >
> >...looks like they're at 2s intervals.  If many of them get queued in a few
> >minutes, it starts to get ahead of that gatekeeper and jobmanager.  Try 5s 
> >next.
> >
> >-Galen
> >
> >Veronika V. Nefedova <nefedova at mcs.anl.gov> writes:
> > >Yes, Thank you. I'll set this throttling parameter in my workflow.
> > >
> > >Nika
> > >
> > >At 04:53 PM 3/16/2007, consult at ncsa.uiuc.edu wrote:
> > >>FROM: Arnold, Galen
> > >>(Concerning ticket No. 137212)
> > >>
> > >>Veronika,
> > >>
> > >>If you can throttle the job submission so that there's more than 1 second
> > >>between them, that would probably help us out.
> > >>
> > >>-Galen
> > >>
> > >>Veronika V. Nefedova <nefedova at mcs.anl.gov> writes:
> > >> >Hi, Galen:
> > >> >
> > >> >I was told that I could have 384 jobs in PBS at the same time. By my
> > >> >estimation I had no more then 136 jobs in the queue. What are other 
> > limits
> > >> >I should keep in mind ?
> > >> >
> > >> >Thanks,
> > >> >
> > >> >Veronika
> > >> >
> > >> >At 04:31 PM 3/16/2007, consult at ncsa.uiuc.edu wrote:
> > >> >>FROM: Arnold, Galen
> > >> >>(Concerning ticket No. 137212)
> > >> >>
> > >> >>Veronika,
> > >> >>
> > >> >>We see many sequences in our globus-gatekeeper.log on tg-login1 like 
> > this
> > >> >>
> > >> >>Notice: 5: Authenticated globus user:
> > >> >>/DC=org/DC=doegrids/OU=People/CN=Veronika
> > >> >>Nefedova 137823
> > >> >>Notice: 0: GRID_SECURITY_HTTP_BODY_FD=6
> > >> >>Notice: 5: Requested service: jobmanager-pbs
> > >> >>Notice: 5: Authorized as local user: nefedova
> > >> >>Notice: 5: Authorized as local uid: 29202
> > >> >>Notice: 5:           and local gid: 11467
> > >> >>Notice: 0: executing
> > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager
> > >> >>Notice: 0: GRID_SECURITY_CONTEXT_FD=9
> > >> >>Notice: 0: Child 16725 started
> > >> >>Notice: 5: Authenticated globus user:
> > >> >>/DC=org/DC=doegrids/OU=People/CN=Veronika
> > >> >>Nefedova 137823
> > >> >>Notice: 0: GRID_SECURITY_HTTP_BODY_FD=6
> > >> >>Notice: 5: Requested service: jobmanager-pbs
> > >> >>Notice: 5: Authorized as local user: nefedova
> > >> >>Notice: 5: Authorized as local uid: 29202
> > >> >>Notice: 5:           and local gid: 11467
> > >> >>Notice: 0: executing
> > >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager
> > >> >>Notice: 0: GRID_SECURITY_CONTEXT_FD=9
> > >> >>Notice: 0: Child 16726 started
> > >> >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16769
> > >> >>starting at Fri Mar 16 09:34:12 2007
> > >> >>
> > >> >>
> > >> >>...your DN is also popular in the globus-gatekeeper.log.
> > >> >>
> > >> >>tg-login1:/usr/local/globus-2.4.3-gcc-r5/var # grep 'CN=Veronika 
> > Nefedova'
> > >> >>globus-gatekeeper.log |wc -l
> > >> >>   15215
> > >> >>
> > >> >>
> > >> >>The large number of connects around the same time:
> > >> >>
> > >> >>  grep --after-context=10 Veronika globus-gatekeeper.log | grep 
> > starting |
> > >> >> grep
> > >> >>'Mar 16 09:3' | wc -l
> > >> >>      65
> > >> >>...
> > >> >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=15872
> > >> >>starting at Fri Mar 16 09:34:02 2007
> > >> >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16644
> > >> >>starting at Fri Mar 16 09:34:11 2007
> > >> >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16769
> > >> >>starting at Fri Mar 16 09:34:12 2007
> > >> >>Notice: 6: /usr/local/globus-2.4.3-gcc-r5/sbin/globus-gatekeeper 
> > pid=16926
> > >> >>starting at Fri Mar 16 09:34:14 2007
> > >> >>...
> > >> >>
> > >> >>
> > >> >>...could be overwhelming the gatekeeper on tg-login1.
> > >> >>
> > >> >>-Galen
> > >> >>
> > >> >>
> > >> >>Veronika V. Nefedova <nefedova at mcs.anl.gov> writes:
> > >> >> >Hi, Dave:
> > >> >> >
> > >> >> >Could you please give me a bit more information on what has happened ?
> > >> >> >
> > >> >> >Thanks,
> > >> >> >
> > >> >> >Nika
> > >> >> >
> > >> >> >At 02:12 PM 3/16/2007, consult at ncsa.uiuc.edu wrote:
> > >> >> >>FROM: McWilliams, David G
> > >> >> >>(Concerning ticket No. 137212)
> > >> >> >>
> > >> >> >>Veronika,
> > >> >> >>
> > >> >> >>The system administrator had to kill several of your globus
> > >> processes today
> > >> >> >>because the load average on the node was over 25. Below is a list of
> > >> >> processes
> > >> >> >>that were killed. Please let us know if you need help identifying the
> > >> >> case of
> > >> >> >>the problem.
> > >> >> >>
> > >> >> >>Regards,
> > >> >> >>
> > >> >> >>Dave McWilliams            (217) 
> > 244-1144          consult at ncsa.uiuc.edu
> > >> >> >>NCSA Consulting
> > >> Services    http://www.ncsa.uiuc.edu/UserInfo/Consulting/
> > >> >> >>------------------------------------------------------------------ 
> > ---
> > >> -----
> > >> >> >>
> > >> >> >>nefedova 11084     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11165     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11166     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11277     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11295     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11379     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11402     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11480     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11522     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11602     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11668     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11715     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11785     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11813     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11892     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 11948     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12004     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12030     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12133     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12172     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12253     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12256     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12443     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12460     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12504     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12534     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12657     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12668     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12773     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12806     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12892     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 12946     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13023     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13072     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13142     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13233     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13245     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13352     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13379     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13474     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13488     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13618     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13663     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13743     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13820     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13887     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 13952     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14046     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14048     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14172     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14196     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14319     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14366     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14430     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14539     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14572     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14703     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14725     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14832     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 14849     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15009     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15017     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15164     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15165     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15322     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15332     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15526     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15544     1  0 09:33 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15671     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15672     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15787     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15842     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15981     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 15982     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16131     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16132     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16320     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16358     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16553     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16622     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16725     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16726     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16845     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16925     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 16989     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 17095     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 17262     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 17305     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 17375     1  0 09:34 ?        00:00:02 
> > globus-job-manager -conf
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/etc/globus-job-manager.conf -type
> > >> >> pbs_gcc -rdn
> > >> >> >>jobmanager-pbs -machine-type unknown -publish-jobs
> > >> >> >>nefedova 31347 14172  4 13:35 ?        00:00:00 /usr/bin/perl
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script. 
> > pl -m
> > >> >> >>pbs_gcc
> > >> >> >>-f /tmp/gram_1Drc0X -c poll
> > >> >> >>nefedova 31349 11295  4 13:35 ?        00:00:00 /usr/bin/perl
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script. 
> > pl -m
> > >> >> >>pbs_gcc
> > >> >> >>-f /tmp/gram_fSWNvt -c poll
> > >> >> >>nefedova 31352 11522  4 13:35 ?        00:00:00 /usr/bin/perl
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script. 
> > pl -m
> > >> >> >>pbs_gcc
> > >> >> >>-f /tmp/gram_a3whWx -c poll
> > >> >> >>nefedova 31380 16845  4 13:35 ?        00:00:00 /usr/bin/perl
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script. 
> > pl -m
> > >> >> >>pbs_gcc
> > >> >> >>-f /tmp/gram_zz0JuC -c poll
> > >> >> >>nefedova 31396 14832  4 13:35 ?        00:00:00 /usr/bin/perl
> > >> >> >>/usr/local//globus-2.4.3-gcc-r5/libexec/globus-job-manager-script. 
> > pl -m
> > >> >> >>pbs_gcc
> > >> >> >>-f /tmp/gram_5jGFI1 -c poll
> > >> >> >>nefedova 31512 31347  0 13:35 ?        00:00:00 sh -c
> > >> >> >>/usr/local/pbs/ia64/bin/qstat -f 905629.tg-master.ncsa.teragrid.org
> > >> >> >>2>/dev/null
> > >> >> >>nefedova 31546 31512  0 13:35 ?        00:00:00
> > >> >> >>/usr/local/pbs/ia64/bin/qstat -f
> > >> >> >>905629.tg-master.ncsa.teragrid.org
> > >> >> >>nefedova 31578 31555  0 13:35 ?        00:00:00
> > >> >> >>/usr/local/pbs/ia64/bin/qstat -f
> > >> >> >>905638.tg-master.ncsa.teragrid.org
> > >> >> >>nefedova 31629 31560  0 13:35 ?        00:00:00 sh -c
> > >> >> >>/usr/local/torque-2.1.7/ia64/sbin/pbs_iff tg-master.ncsa.teragrid.org
> > >> >> 15001 3
> > >> >> >>nefedova 31634 31578  0 13:35 ?        00:00:00 sh -c
> > >> >> >>/usr/local/torque-2.1.7/ia64/sbin/pbs_iff tg-master.ncsa.teragrid.org
> > >> >> 15001 3
> 
> 
> _______________________________________________
> Swift-devel mailing list
> Swift-devel at ci.uchicago.edu
> http://mail.ci.uchicago.edu/mailman/listinfo/swift-devel
>