[mpich-discuss] MPICH2 + Boost.MPI under Cygwin

[Murphy, John T.]

Hello all,

This is cross-posted with the Boost.MPI list; I'm not sure whether it is a
Boost problem or an MPICH problem, or even how to go about diagnosing
this. It seems more likely to be a boost-related problem, but perhaps
someone here has seen it or can help guide me to a solution.

Problem: Boost.MPI code that compiles, links, and runs successfully on
other platforms fails under Cygwin + MPICH2, giving the following runtime
error:

"Attempting to use an MPI routine before initializing MPICH"


The error seems related to calls to non-Boost MPI functions; it can occur
when the Boost.MPI library is linked even though no functions in the
boost::mpi namespace are used. Altering the order in which libraries are
given to the gcc linker can resolve the issue in simple cases (not shown
here), but does not for our full code.

Platform:
Windows Vista, Cygwin newly installed with gcc4, openssh, curl and z
libraries, MPICH2 built from source, Boost 1.48 and Boost 1.49 tested with
same results (the current release of our code is incompatible with Boost
1.50 due to changes in the Boost.Filesystem library).

Details:

Initial test code (test.cpp):
===================================

#include <iostream>
#include <mpi.h>

int main(int argc, char** argv){
  MPI_Init(&argc, &argv);
  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);

  std::cout << " RANK " << rank << " RUNNING " << std::endl;

  if(rank == 0){
    MPI_Status status;
    int mode = MPI_MODE_CREATE | MPI_MODE_WRONLY | MPI_MODE_APPEND;
    MPI_File out;
    std::string fileName("LOG.txt");
    MPI_File_open(MPI_COMM_SELF, (char*)fileName.c_str(), mode,
MPI_INFO_NULL, &out);
    MPI_File_close(&out);
  }

  MPI_Finalize();
  std::cout << " RANK " << rank << "       ENDING " << std::endl;
  return 0;
}
======================================


Compilation, linking, invocation and output without using Boost.MPI:

========================================
John at Vista2 ~/TEST
$ mpicxx -c -o test.o test.cpp

John at Vista2 ~/TEST
$ mpicxx -o test test.o

John at Vista2 ~/TEST
$ mpirun -n 2 ./test.exe
RANK 0 RUNNING
RANK 1 RUNNING
RANK 0       ENDING
RANK 1       ENDING

John at Vista2 ~/TEST
$ mpicxx -show
c++ -I/home/John/MPICH/include -L/home/John/MPICH/lib -lmpichcxx -lpmpich
-lmpich -lopa -lmpl -lpthread
=========================================


Same code, but add an unnecessary link to Boost.MPI:
=========================================
John at Vista2 ~/TEST
$ mpicxx -L/home/John/boost_1_48_0/stage/lib -o test test.o -lboost_mpi-mt

John at Vista2 ~/TEST
$ mpirun -n 2 ./test.exe
RANK 0 RUNNING
RANK 1 RUNNING
Attempting to use an MPI routine before initializing MPICH
=========================================
(Note: the LOG.txt file is not created.)


Code using Boost::MPI functions expclitly (test_boost.cpp):
=============================================
#include <boost/mpi.hpp>
#include <boost/mpi/environment.hpp>
#include <boost/mpi/communicator.hpp>
#include <iostream>

namespace mpi = boost::mpi;

int main(int argc, char** argv){
  mpi::environment env(argc, argv);
  mpi::communicator world;
  int rank = world.rank();
  std::cout << " RANK " << rank << " RUNNING " << std::endl;

  if(rank == 0){
    MPI_Status status;
    int mode = MPI_MODE_CREATE | MPI_MODE_WRONLY | MPI_MODE_APPEND;
    MPI_File out;
    std::string fileName("LOG.txt");
    MPI_File_open(MPI_COMM_SELF, (char*)fileName.c_str(), mode,
MPI_INFO_NULL, &out);
    MPI_File_close(&out);
  }

  std::cout << " RANK " << rank << "       ENDING " << std::endl;
  return 0;
}
==========================================

Compilation, linking, and running:

==========================================
John at Vista2 ~/TEST
$ mpicxx -c -I/home/John/boost_1_48_0/ -o test_boost.o test_boost.cpp

John at Vista2 ~/TEST
$ mpicxx -L/home/John/boost_1_48_0/stage/lib  -o test_boost test_boost.o
-lboost_mpi-mt

John at Vista2 ~/TEST
$ mpirun -n 2 ./test_boost.exe
RANK 0 RUNNING
RANK 1 RUNNING
RANK 1       ENDING
Attempting to use an MPI routine before initializing MPICH
===========================================

The other system I have easy access to, and on which this code runs
correctly, is a Mac/Unix-like using OpenMPI, but the need is for this code
(and, more importantly, for our larger project, where the error was first
observed) to run under Cygwin. I do not know how to diagnose whether
this is a Boost.MPI issue, an MPICH2 issue, or something related to how
the two libraries work together.

Any help appreciated- thanks,
John



-- 
John T. Murphy
Computational Postdoctoral Fellow
Decision and Information Sciences and
Argonne Leadership Computing Facility
Argonne National Laboratory
jtmurphy at anl.gov