[mpich2-dev] ROMIO: Persistent File Realms (Patch proposed and questions to use them)

[Pascal Deveze]

Hi all,

Thanks for all of you that provided me with documents about Persistent 
File Realms.
I really appreciated to learn more about this new optimized "2 phase IO 
method".

I tried to make it work with mpich2-1.3rc1 using an hint file containing :
romio_cb_pfr enable
striping_factor 1
striping_unit 1048576
romio_cb_fr_alignment 1048576

I exported the ROMIO_HINTS variable pointing to this file.

I tried then to run coll_test on 2 processes (on one node only):
mpirun -np 2 ./coll_test -fname /tmp/data

The processes were blocking in MPI_Alltoall.

So, I tried to disable the call to MPI_Alltoall by adding the following 
line to the hint file:
romio_cb_alltoall disable

The processes were still blocking in MPI_Alltoall.
In the file romio/adio/common/ad_hints.c, there seems to be a mistake
in setting fd->hints->cb_alltoall.
You will find hereafter the proposed patch.

After that correction, I ran coll_perf again, and I got blocked in 
MPI_Waitany.

I also tried coll_perf with 1, 4 and 12 processes (on one node) and got 
a problem in MPI_Type_free:

  Fatal error in PMPI_Type_free: Invalid datatype, error stack:
  PMPI_Type_free(156): MPI_Type_free(datatype_p=0x1791170) failed
  PMPI_Type_free(108): Cannot free permanent data type
  APPLICATION TERMINATED WITH THE EXIT STRING: Hangup (signal 1)

For 1, 4 and 12 processes, when I enable the use of MPI_Alltoall, I get 
the following error :
  coll_test: ad_coll_build_req_new.c:1018: ADIOI_Build_client_reqs: 
Assertion `cur_off_proc != -1' failed.

I am a bit disappointed not to be able to run that new very interesting 
"2 phase IO method".
So, if somebody could help me, I will really appreciate it.

Pascal

======== Proposed patch =========
--- ad_hints.c.OLD      2010-05-25 20:59:13.000000000 +0200
+++ ad_hints.c  2010-09-27 09:56:05.000000000 +0200
@@ -198,16 +198,16 @@
        if (flag) {
            if (!strcmp(value, "enable") || !strcmp(value, "ENABLE")) {
                ADIOI_Info_set(info, "romio_cb_alltoall", value);
-               fd->hints->cb_read = ADIOI_HINT_ENABLE;
+               fd->hints->cb_alltoall = ADIOI_HINT_ENABLE;
            }
            else if (!strcmp(value, "disable") || !strcmp(value, 
"DISABLE")) {
                ADIOI_Info_set(info, "romio_cb_alltoall", value);
-               fd->hints->cb_read = ADIOI_HINT_DISABLE;
+               fd->hints->cb_alltoall = ADIOI_HINT_DISABLE;
            }
            else if (!strcmp(value, "automatic") || !strcmp(value, 
"AUTOMATIC"))
            {
                ADIOI_Info_set(info, "romio_cb_alltoall", value);
-               fd->hints->cb_read = ADIOI_HINT_AUTO;
+               fd->hints->cb_alltoall = ADIOI_HINT_AUTO;
            }
 
            tmp_val = fd->hints->cb_alltoall;