[mpich2-dev] [mvapich-discuss] MVAPICH2 does not work with specified PKEYs.

[Hari Subramoni]

Hi Mike,

I just verified with the latest trunk version of MVAPICH2 stack (should
be same for older revisions as well) and the MV2_DEFAULT_PKEY parameter
is getting passed correctly by mpirun_rsh to the spawned processes.

The way we give environment variable like 'MV2_DEFAULT_PKEY' for
mpirun_rsh framework and mpiexec is different. For mpirun_rsh we need to
give these along with the command itself as given below.

./bin/mpirun_rsh -np 2 amd1 amd2 MV2_DEFAULT_PKEY=0xffff ./a.out

For mpiexec, we generally export this to the environment. Could this be
the reason you observed that mpirun_rsh is not passing the environment
variable properly to the processes?

As to the error you see with mpiexec (which you should also see with
mpirun_rsh) when setting the MV2_DEFAULT_PKEY to some user defined value,
it is because of a small coding error. The following one line patch should
fix it. Could you please apply it to your trunk version of MVAPICH2 and
let us know if things works fine?

Index: src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c
===================================================================
--- src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c        (revision 3451)
+++ src/mpid/ch3/channels/mrail/src/gen2/rdma_iba_priv.c        (working copy)
@@ -1590,7 +1590,7 @@
         uint16_t curr_pkey;
         ibv_query_pkey(MPIDI_CH3I_RDMA_Process.nic_context[hca_num],
                 (uint8_t)port_num, (int)i ,&curr_pkey);
-        if (pkey == ntohs(curr_pkey) & PKEY_MASK) {
+        if (pkey == (ntohs(curr_pkey) & PKEY_MASK)) {
             *index = i;
             return 1;
         }

I am not very familiar with PKEY's . So, I will have to look it up
and get back to you on the reason for using the PKEY_MASK. Sorry about the
delay.

Thx,
Hari.

On Wed, 12 Aug 2009, Mike Heinz wrote:

> My testers are reporting further problems with mvapich2. On a fabric where the use of pkeys is required, mvapich2 is failing.
>
>
> 1)      The MV2_DEFAULT_PKEY parameter does not appear to be supported when using mpirun_rsh.
>
> 2)      When using mpd and mpiexec, the MV2_DEFAULT_PKEY parameter gets passed, but then fails. For example:
>
> [root at homer mpi_apps]#  export MV2_DEFAULT_PKEY=0xffff
> [root at homer mpi_apps]#  /usr/mpi/gcc/mvapich2-1.2p1/bin/mpiexec -machinefile /opt/iba/src/mpi_apps/mpi_hosts -n 2 osu2/osu_bw
>  [0] Abort: Can't find PKEY INDEX according to given PKEY
>  at line 1190 in file rdma_iba_priv.c
> rank 0 in job 6  homer.dev.silverstorm.com_33133   caused collective abort of all ranks
>   exit status of rank 0: killed by signal 9
>
> (Note that 0xffff is actually the default PKEY).
>
> A quick saquery reveals that the pkey is, in fact in the table:
>
> [root at homer mpi_apps]# iba_saquery -o pkey -l 1
> LID: 0x0001 PortNum:  1 BlockNum:  0
>       0-   7:  0x9001  0xffff  0x9002  0x0000  0x0000  0x0000  0x0000  0x0000
>       8-  15:  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000
>      16-  23:  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000
>      24-  31:  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000  0x0000
>
> When I examine ibv_param.c to see what was going on, here is what I found:
>
>     if ((value = getenv("MV2_DEFAULT_PKEY")) != NULL) {
>         rdma_default_pkey = (uint16_t)strtol(value, (char **) NULL,0) & PKEY_MASK;
>     }
> And...
>
>     #define PKEY_MASK 0x7fff /* the last bit is reserved */
>
> This makes it clear that mpiexec is doing bad things to the pkey - if nothing else, the high bit must be set in order for the connection to have full membership in an  Infiniband partition. Without setting this bit, a node will only have "limited membership", and limited nodes are not permitted to talk to each other.
>
> I'm going to try and see if I can quickly put together a patch for you that fixes the problems with mpiexec - but I'm not sure what the correct fix is for mpirun_rsh.
>
> --
> Michael Heinz
> Principal Engineer, Qlogic Corporation
> King of Prussia, Pennsylvania
>