[MOAB-dev] commit/MOAB: iulian07: example for multiple communicators
Vijay S. Mahadevan
vijay.m at gmail.com
Wed Jun 11 09:55:16 CDT 2014
>> In MCT, we advocate that each model have its own communicator but there
>> is still a copy of MPI_WORLD_COMM that can be used for
>> inter-communication. What is the solution for a coupled model with
>> MOAB/Coupe?
Rob, with CouPE we haven't had an explicit use case so far that
requires using two different subgroup communicators for different
meshes but this is something that I'll be looking at soon. All base
objects take the communicator as an argument during creation in both
CouPE and MOAB (ParallelComm) and so any instances that use
MPI_COMM_WORLD explicitly will be replaced with the one provided by
the user.
> For disjoint or at least
> different partitions for source/target, the coupling (CouPE and Coupler)
> will need to operate on the union of both, while each works on its own for
> mesh sharing.
Tim, the Coupler has access to both the source and target ParallelComm
objects and can technically just query based on them. Currently the
tests use co-located mode and we are changing it to have the option to
also perform disjoint point location to do various timing related
studies. I do completely agree that the Coupler will have to operate
on the union but as it stands, I don't see any large changes needed in
the way the interpolated data gets transferred back to the querying
process. It might however require some changes in the driver/test
programs to correctly accumulate the queried data and this is what
should be in the example.
Iulian will revert the above commit to HelloParMOAB since this is
supposed to be an introductory example for the parallel functionality
and should not have any advanced usage. We will have a separate
example that explores handling communication between instances
distributed in both co-located and disjoint settings.
Vijay
On Tue, Jun 10, 2014 at 12:09 PM, Tim Tautges
<timothy.tautges at cd-adapco.com> wrote:
>
>
> On 06/10/2014 12:05 PM, Grindeanu, Iulian R. wrote:
>>
>>
>> In the coupler / test example, we have as many ParallelComm instances as
>> number of files (usually 2, one for target, one for source), and one moab
>> instance
>> but in the example, both use MPI_WORLD_COMM
>>
>> I think we can split it in the coupler example;
>>
>> some work might be required to show how that can be done.
>>
>> maybe it makes sense to have also 2 different moab instances (one for
>> target, one for source), at least in the example.
>>
>
> Well, that's definitely not how the Coupler stuff was designed to happen.
> In all that's been done with CouPE so far, the source and target meshes were
> both distributed on the world communicator. For disjoint or at least
> different partitions for source/target, the coupling (CouPE and Coupler)
> will need to operate on the union of both, while each works on its own for
> mesh sharing.
>
> - tim
>
>
>> Iulian
>>
>> ________________________________________
>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on
>> behalf of Robert Jacob [jacob at mcs.anl.gov]
>> Sent: Tuesday, June 10, 2014 11:47 AM
>> To: moab-dev at mcs.anl.gov
>> Subject: Re: [MOAB-dev] commit/MOAB: iulian07: example for multiple
>> communicators
>>
>> On 6/10/14 11:41 AM, commits-noreply at bitbucket.org wrote:
>>>
>>>
>>> The original MPI_WORLD_COMM is split into nbComms (a second
>>> optional argument to ./HelloParMOAB, first is the file to load)
>>> each communicator will load a full mesh in parallel, which will be
>>> distributed
>>> among its tasks
>>>
>>> the communicators are completely independent from MOAB's point of view,
>>> and they
>>> will never "inter-communicate" in moab code.
>>>
>>> so MOAB will handle only "intra-communication"
>>
>>
>> In MCT, we advocate that each model have its own communicator but there
>> is still a copy of MPI_WORLD_COMM that can be used for
>> inter-communication. What is the solution for a coupled model with
>> MOAB/Coupe?
>>
>> Rob
>>
>>>
>>> Affected #: 1 file
>>>
>>> diff --git a/examples/HelloParMOAB.cpp b/examples/HelloParMOAB.cpp
>>> index 5bcc4ab..3611a8d 100644
>>> --- a/examples/HelloParMOAB.cpp
>>> +++ b/examples/HelloParMOAB.cpp
>>> @@ -2,6 +2,12 @@
>>> * \brief Read mesh into MOAB and resolve/exchange/report shared and
>>> ghosted entities \n
>>> * <b>To run</b>: mpiexec -np 4 HelloMoabPar [filename]\n
>>> *
>>> + * It shows how to load the mesh independently, on multiple
>>> + * communicators (with second argument, the number of comms)
>>> + *
>>> + *
>>> + *
>>> + * mpiexec -np 8 HelloMoabPar [filename] [nbComms]
>>> */
>>>
>>> #include "moab/ParallelComm.hpp"
>>> @@ -26,6 +32,10 @@ int main(int argc, char **argv)
>>> test_file_name = argv[1];
>>> }
>>>
>>> + int nbComms = 1;
>>> + if (argc > 2)
>>> + nbComms = atoi(argv[2]);
>>> +
>>> options =
>>> "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS";
>>>
>>> // Get MOAB instance and read the file with the specified options
>>> @@ -33,15 +43,38 @@ int main(int argc, char **argv)
>>> if (NULL == mb)
>>> return 1;
>>>
>>> + MPI_Comm comm ;
>>> + int global_rank, global_size;
>>> + MPI_Comm_rank( MPI_COMM_WORLD, &global_rank );
>>> + MPI_Comm_rank( MPI_COMM_WORLD, &global_size );
>>> +
>>> + int color = global_rank%nbComms; // for each angle group a different
>>> color
>>> + if (nbComms>1)
>>> + {
>>> + // split the communicator, into ngroups = nbComms
>>> + MPI_Comm_split( MPI_COMM_WORLD, color, global_rank, &comm );
>>> + }
>>> + else
>>> + {
>>> + comm = MPI_COMM_WORLD;
>>> + }
>>> // Get the ParallelComm instance
>>> - ParallelComm* pcomm = new ParallelComm(mb, MPI_COMM_WORLD);
>>> + ParallelComm* pcomm = new ParallelComm(mb, comm);
>>> int nprocs = pcomm->proc_config().proc_size();
>>> int rank = pcomm->proc_config().proc_rank();
>>> - MPI_Comm comm = pcomm->proc_config().proc_comm();
>>> + MPI_Comm rcomm = pcomm->proc_config().proc_comm();
>>> + assert(rcomm==comm);
>>> + if (global_rank == 0)
>>> + cout<< " global rank:" <<global_rank << " color:" << color << "
>>> rank:" << rank << " of " << nprocs << " processors\n";
>>> +
>>> + if (global_rank == 1)
>>> + cout<< " global rank:" <<global_rank << " color:" << color << "
>>> rank:" << rank << " of " << nprocs << " processors\n";
>>>
>>> - if (rank == 0)
>>> + MPI_Barrier(MPI_COMM_WORLD);
>>> +
>>> + if (global_rank == 0)
>>> cout << "Reading file " << test_file_name << "\n with options: "
>>> << options << endl
>>> - << " on " << nprocs << " processors\n";
>>> + << " on " << nprocs << " processors on " << nbComms << "
>>> communicator(s) \n";
>>>
>>> ErrorCode rval = mb->load_file(test_file_name.c_str(), 0,
>>> options.c_str());
>>> if (rval != MB_SUCCESS) {
>>> @@ -69,9 +102,9 @@ int main(int argc, char **argv)
>>> for (int i = 0; i < 4; i++)
>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>> vector<int> rbuf(nprocs*4, 0);
>>> - MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, MPI_COMM_WORLD);
>>> + MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
>>> // Print the stats gathered:
>>> - if (rank == 0) {
>>> + if (global_rank == 0) {
>>> for (int i = 0; i < nprocs; i++)
>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>> rbuf[4*i] << " verts, " <<
>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, "
>>> << rbuf[4*i + 3] << " elements" << endl;
>>> @@ -109,7 +142,7 @@ int main(int argc, char **argv)
>>>
>>> // gather the statistics on processor 0
>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
>>> - if (rank == 0) {
>>> + if (global_rank == 0) {
>>> cout << " \n\n After exchanging one ghost layer: \n";
>>> for (int i = 0; i < nprocs; i++) {
>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>> rbuf[4*i] << " verts, " <<
>>>
>>> Repository URL: https://bitbucket.org/fathomteam/moab/
>>>
>>> --
>>>
>>> This is a commit notification from bitbucket.org. You are receiving
>>> this because you have the service enabled, addressing the recipient of
>>> this email.
>>>
>
> --
> Timothy J. Tautges
> Manager, Directed Meshing, CD-adapco
> Phone: 608-354-1459
> timothy.tautges at cd-adapco.com
More information about the moab-dev
mailing list