[MOAB-dev] commit/MOAB: iulian07: example for multiple communicators

[Grindeanu, Iulian R.]

No, this would be wrong, I think.
Some more major changes are needed, as down deep, we use crystal router in ParallelComm;
they can only intra-communicate 

This is more complicated than I thought :( 

Maybe overlapping communicators

________________________________________
From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Grindeanu, Iulian R. [iulian at mcs.anl.gov]
Sent: Tuesday, June 10, 2014 12:05 PM
To: Jacob, Robert L.; moab-dev at mcs.anl.gov
Subject: Re: [MOAB-dev] commit/MOAB: iulian07: example for      multiple        communicators

In the coupler / test example, we have as many ParallelComm instances as number of files (usually 2, one for target, one for source), and one moab instance
but in the example, both use MPI_WORLD_COMM

I think we can split it in the coupler example;

some work might be required to show how that can be done.

maybe it makes sense to have also 2 different moab instances (one for target, one for source), at least in the example.

Iulian

________________________________________
From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on behalf of Robert Jacob [jacob at mcs.anl.gov]
Sent: Tuesday, June 10, 2014 11:47 AM
To: moab-dev at mcs.anl.gov
Subject: Re: [MOAB-dev] commit/MOAB: iulian07: example for multiple     communicators

On 6/10/14 11:41 AM, commits-noreply at bitbucket.org wrote:
>
> The original MPI_WORLD_COMM is split into nbComms (a second
> optional argument to ./HelloParMOAB, first is the file to load)
> each communicator will load a full mesh in parallel, which will be distributed
> among  its tasks
>
> the communicators are completely independent from MOAB's point of view, and they
> will never "inter-communicate" in moab code.
>
> so MOAB will handle only "intra-communication"

In MCT, we advocate that each model have its own communicator but there
is still a copy of MPI_WORLD_COMM that can be used for
inter-communication.  What is the solution for a coupled model with
MOAB/Coupe?

Rob

>
> Affected #:  1 file
>
> diff --git a/examples/HelloParMOAB.cpp b/examples/HelloParMOAB.cpp
> index 5bcc4ab..3611a8d 100644
> --- a/examples/HelloParMOAB.cpp
> +++ b/examples/HelloParMOAB.cpp
> @@ -2,6 +2,12 @@
>    * \brief Read mesh into MOAB and resolve/exchange/report shared and ghosted entities \n
>    * <b>To run</b>: mpiexec -np 4 HelloMoabPar [filename]\n
>    *
> + *  It shows how to load the mesh independently, on multiple
> + *  communicators (with second argument, the number of comms)
> + *
> + *
> + *
> + *  mpiexec -np 8 HelloMoabPar [filename] [nbComms]
>    */
>
>   #include "moab/ParallelComm.hpp"
> @@ -26,6 +32,10 @@ int main(int argc, char **argv)
>       test_file_name = argv[1];
>     }
>
> +  int nbComms = 1;
> +  if (argc > 2)
> +    nbComms = atoi(argv[2]);
> +
>     options = "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS";
>
>     // Get MOAB instance and read the file with the specified options
> @@ -33,15 +43,38 @@ int main(int argc, char **argv)
>     if (NULL == mb)
>       return 1;
>
> +  MPI_Comm comm ;
> +  int global_rank, global_size;
> +  MPI_Comm_rank( MPI_COMM_WORLD, &global_rank );
> +  MPI_Comm_rank( MPI_COMM_WORLD, &global_size );
> +
> +  int color = global_rank%nbComms; // for each angle group a different color
> +  if (nbComms>1)
> +  {
> +    // split the communicator, into ngroups = nbComms
> +    MPI_Comm_split( MPI_COMM_WORLD, color, global_rank, &comm );
> +  }
> +  else
> +  {
> +    comm = MPI_COMM_WORLD;
> +  }
>     // Get the ParallelComm instance
> -  ParallelComm* pcomm = new ParallelComm(mb, MPI_COMM_WORLD);
> +  ParallelComm* pcomm = new ParallelComm(mb, comm);
>     int nprocs = pcomm->proc_config().proc_size();
>     int rank = pcomm->proc_config().proc_rank();
> -  MPI_Comm comm = pcomm->proc_config().proc_comm();
> +  MPI_Comm rcomm = pcomm->proc_config().proc_comm();
> +  assert(rcomm==comm);
> +  if (global_rank == 0)
> +    cout<< " global rank:" <<global_rank << " color:" << color << " rank:" << rank << " of " << nprocs << " processors\n";
> +
> +  if (global_rank == 1)
> +    cout<< " global rank:" <<global_rank << " color:" << color << " rank:" << rank << " of " << nprocs << " processors\n";
>
> -  if (rank == 0)
> +  MPI_Barrier(MPI_COMM_WORLD);
> +
> +  if (global_rank == 0)
>       cout << "Reading file " << test_file_name << "\n  with options: " << options << endl
> -         << " on " << nprocs << " processors\n";
> +         << " on " << nprocs << " processors on " << nbComms << " communicator(s) \n";
>
>     ErrorCode rval = mb->load_file(test_file_name.c_str(), 0, options.c_str());
>     if (rval != MB_SUCCESS) {
> @@ -69,9 +102,9 @@ int main(int argc, char **argv)
>     for (int i = 0; i < 4; i++)
>       nums[i] = (int)owned_entities.num_of_dimension(i);
>     vector<int> rbuf(nprocs*4, 0);
> -  MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, MPI_COMM_WORLD);
> +  MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
>     // Print the stats gathered:
> -  if (rank == 0) {
> +  if (global_rank == 0) {
>       for (int i = 0; i < nprocs; i++)
>         cout << " Shared, owned entities on proc " << i << ": " << rbuf[4*i] << " verts, " <<
>             rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, " << rbuf[4*i + 3] << " elements" << endl;
> @@ -109,7 +142,7 @@ int main(int argc, char **argv)
>
>     // gather the statistics on processor 0
>     MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
> -  if (rank == 0) {
> +  if (global_rank == 0) {
>       cout << " \n\n After exchanging one ghost layer: \n";
>       for (int i = 0; i < nprocs; i++) {
>         cout << " Shared, owned entities on proc " << i << ": " << rbuf[4*i] << " verts, " <<
>
> Repository URL: https://bitbucket.org/fathomteam/moab/
>
> --
>
> This is a commit notification from bitbucket.org. You are receiving
> this because you have the service enabled, addressing the recipient of
> this email.
>