[mpich-discuss] Hydra: Sorting node list by number of CPUs

[Reuti]

Hi,

Am 21.10.2010 um 18:51 schrieb Yauheni Zelenko:

> Hi, Reuti!
> 
> I need to reorder the list of granted slots in the cluster you got from resource manager.
> 
> It's not only related to LSF, but to Load Leveler, Sun Grid Engine, etc.
> 
> In case of LSF it's possible to rearrange environment variable. But it may be harder in case of Load Leveler and Sun Grid Engine, where slots are written in file, which may be not accessable for write.

in SGE you will get only one entry per host in the hostfile, followed by some information incuding the number of granted slots per machine ("$fiil_up" would be a good "allocation_rule" for your case then). AFAIK the rank0 is always where the jobscript and hence the `mpiexec` is executed. This you can't change even by changing the hostfile, SGE will schedule the jobscript to this node or did already so when you have the chance to rewrite the hostfile.

==

Anyway, in case of SGE the list is in the spool directory of the job and can be referenced by $PE_HOSTFILE in a queue or global prolog. As you are right that you can't change this depending on the permissions of the $SGE_JOB_SPOOL_DIR, you could execute the prolog under the account of the designated admin user like:

$ qconf -sq all.q
...
prolog    sgeadmin@/usr/sge/cluster/change_hostfile.sh
...

If you want to make a script w/o if-then-else also for the serial case, the file will also be created for serial jobs but $PE_HOSTFILE won't be set. So it can be referenced by $SGE_JOB_SPOOL_DIR/pe_hostfile in any case.

This way you can keep the tight integration of MPICH2 into SGE with Hydra without the need to supply a custom hostfile to `mpiexec`, which you could nevertheless create in the defined "start_proc_args" of the PE (parallel environment) definition.

-- Reuti


> 
> Eugene.
> ________________________________________
> From: mpich-discuss-bounces at mcs.anl.gov [mpich-discuss-bounces at mcs.anl.gov] On Behalf Of Reuti [reuti at staff.uni-marburg.de]
> Sent: Thursday, October 21, 2010 2:22 AM
> To: mpich-discuss at mcs.anl.gov
> Subject: Re: [mpich-discuss] Hydra: Sorting node list by number of CPUs
> 
> Hi,
> 
> Am 21.10.2010 um 01:32 schrieb Yauheni Zelenko:
> 
>> I think will be good idea to have command line option which will force Hydra to sort node list by number of CPUs before proxies creation.
>> 
>> Our application is working in master-slaves mode and master is always have rank 0. From performance point of view it's good idea to put as much slaves as possible to same host to use shared memory.
>> 
>> However managed environment like LSF could distribute CPUs across hosts randomly. For example we had 1 CPU in host1 and seven on host2.
>> 
>> So sorting node list should help. I think it may be implemented with qsort().
> 
> do you want a similar feature of LSF outside of LSF, or do you want to reorder the list of granted slots in the cluster you got from LSF?
> 
> -- Reuti
> 
> 
>> 
>> Eugene.
>> 
>> _______________________________________________
>> mpich-discuss mailing list
>> mpich-discuss at mcs.anl.gov
>> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> 
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
> _______________________________________________
> mpich-discuss mailing list
> mpich-discuss at mcs.anl.gov
> https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss