[MOAB-dev] commit/MOAB: iulian07: example for multiple communicators
Tim Tautges
timothy.tautges at cd-adapco.com
Wed Jun 11 10:06:44 CDT 2014
On 06/11/2014 09:55 AM, Vijay S. Mahadevan wrote:
>>> In MCT, we advocate that each model have its own communicator but there
>>> is still a copy of MPI_WORLD_COMM that can be used for
>>> inter-communication. What is the solution for a coupled model with
>>> MOAB/Coupe?
>
> Rob, with CouPE we haven't had an explicit use case so far that
> requires using two different subgroup communicators for different
> meshes but this is something that I'll be looking at soon. All base
> objects take the communicator as an argument during creation in both
> CouPE and MOAB (ParallelComm) and so any instances that use
> MPI_COMM_WORLD explicitly will be replaced with the one provided by
> the user.
>
>> For disjoint or at least
>> different partitions for source/target, the coupling (CouPE and Coupler)
>> will need to operate on the union of both, while each works on its own for
>> mesh sharing.
>
> Tim, the Coupler has access to both the source and target ParallelComm
> objects and can technically just query based on them. Currently the
> tests use co-located mode and we are changing it to have the option to
> also perform disjoint point location to do various timing related
> studies. I do completely agree that the Coupler will have to operate
> on the union but as it stands, I don't see any large changes needed in
> the way the interpolated data gets transferred back to the querying
> process. It might however require some changes in the driver/test
> programs to correctly accumulate the queried data and this is what
> should be in the example.
I agree it shouldn't require substantial changes, but it will require changes, probably just in CouPE, to initialize
MOAB with each mesh on a mesh-specific communicator, then the Coupler on a union communicator. Otherwise it should work.
- tim
>
> Iulian will revert the above commit to HelloParMOAB since this is
> supposed to be an introductory example for the parallel functionality
> and should not have any advanced usage. We will have a separate
> example that explores handling communication between instances
> distributed in both co-located and disjoint settings.
>
> Vijay
>
> On Tue, Jun 10, 2014 at 12:09 PM, Tim Tautges
> <timothy.tautges at cd-adapco.com> wrote:
>>
>>
>> On 06/10/2014 12:05 PM, Grindeanu, Iulian R. wrote:
>>>
>>>
>>> In the coupler / test example, we have as many ParallelComm instances as
>>> number of files (usually 2, one for target, one for source), and one moab
>>> instance
>>> but in the example, both use MPI_WORLD_COMM
>>>
>>> I think we can split it in the coupler example;
>>>
>>> some work might be required to show how that can be done.
>>>
>>> maybe it makes sense to have also 2 different moab instances (one for
>>> target, one for source), at least in the example.
>>>
>>
>> Well, that's definitely not how the Coupler stuff was designed to happen.
>> In all that's been done with CouPE so far, the source and target meshes were
>> both distributed on the world communicator. For disjoint or at least
>> different partitions for source/target, the coupling (CouPE and Coupler)
>> will need to operate on the union of both, while each works on its own for
>> mesh sharing.
>>
>> - tim
>>
>>
>>> Iulian
>>>
>>> ________________________________________
>>> From: moab-dev-bounces at mcs.anl.gov [moab-dev-bounces at mcs.anl.gov] on
>>> behalf of Robert Jacob [jacob at mcs.anl.gov]
>>> Sent: Tuesday, June 10, 2014 11:47 AM
>>> To: moab-dev at mcs.anl.gov
>>> Subject: Re: [MOAB-dev] commit/MOAB: iulian07: example for multiple
>>> communicators
>>>
>>> On 6/10/14 11:41 AM, commits-noreply at bitbucket.org wrote:
>>>>
>>>>
>>>> The original MPI_WORLD_COMM is split into nbComms (a second
>>>> optional argument to ./HelloParMOAB, first is the file to load)
>>>> each communicator will load a full mesh in parallel, which will be
>>>> distributed
>>>> among its tasks
>>>>
>>>> the communicators are completely independent from MOAB's point of view,
>>>> and they
>>>> will never "inter-communicate" in moab code.
>>>>
>>>> so MOAB will handle only "intra-communication"
>>>
>>>
>>> In MCT, we advocate that each model have its own communicator but there
>>> is still a copy of MPI_WORLD_COMM that can be used for
>>> inter-communication. What is the solution for a coupled model with
>>> MOAB/Coupe?
>>>
>>> Rob
>>>
>>>>
>>>> Affected #: 1 file
>>>>
>>>> diff --git a/examples/HelloParMOAB.cpp b/examples/HelloParMOAB.cpp
>>>> index 5bcc4ab..3611a8d 100644
>>>> --- a/examples/HelloParMOAB.cpp
>>>> +++ b/examples/HelloParMOAB.cpp
>>>> @@ -2,6 +2,12 @@
>>>> * \brief Read mesh into MOAB and resolve/exchange/report shared and
>>>> ghosted entities \n
>>>> * <b>To run</b>: mpiexec -np 4 HelloMoabPar [filename]\n
>>>> *
>>>> + * It shows how to load the mesh independently, on multiple
>>>> + * communicators (with second argument, the number of comms)
>>>> + *
>>>> + *
>>>> + *
>>>> + * mpiexec -np 8 HelloMoabPar [filename] [nbComms]
>>>> */
>>>>
>>>> #include "moab/ParallelComm.hpp"
>>>> @@ -26,6 +32,10 @@ int main(int argc, char **argv)
>>>> test_file_name = argv[1];
>>>> }
>>>>
>>>> + int nbComms = 1;
>>>> + if (argc > 2)
>>>> + nbComms = atoi(argv[2]);
>>>> +
>>>> options =
>>>> "PARALLEL=READ_PART;PARTITION=PARALLEL_PARTITION;PARALLEL_RESOLVE_SHARED_ENTS";
>>>>
>>>> // Get MOAB instance and read the file with the specified options
>>>> @@ -33,15 +43,38 @@ int main(int argc, char **argv)
>>>> if (NULL == mb)
>>>> return 1;
>>>>
>>>> + MPI_Comm comm ;
>>>> + int global_rank, global_size;
>>>> + MPI_Comm_rank( MPI_COMM_WORLD, &global_rank );
>>>> + MPI_Comm_rank( MPI_COMM_WORLD, &global_size );
>>>> +
>>>> + int color = global_rank%nbComms; // for each angle group a different
>>>> color
>>>> + if (nbComms>1)
>>>> + {
>>>> + // split the communicator, into ngroups = nbComms
>>>> + MPI_Comm_split( MPI_COMM_WORLD, color, global_rank, &comm );
>>>> + }
>>>> + else
>>>> + {
>>>> + comm = MPI_COMM_WORLD;
>>>> + }
>>>> // Get the ParallelComm instance
>>>> - ParallelComm* pcomm = new ParallelComm(mb, MPI_COMM_WORLD);
>>>> + ParallelComm* pcomm = new ParallelComm(mb, comm);
>>>> int nprocs = pcomm->proc_config().proc_size();
>>>> int rank = pcomm->proc_config().proc_rank();
>>>> - MPI_Comm comm = pcomm->proc_config().proc_comm();
>>>> + MPI_Comm rcomm = pcomm->proc_config().proc_comm();
>>>> + assert(rcomm==comm);
>>>> + if (global_rank == 0)
>>>> + cout<< " global rank:" <<global_rank << " color:" << color << "
>>>> rank:" << rank << " of " << nprocs << " processors\n";
>>>> +
>>>> + if (global_rank == 1)
>>>> + cout<< " global rank:" <<global_rank << " color:" << color << "
>>>> rank:" << rank << " of " << nprocs << " processors\n";
>>>>
>>>> - if (rank == 0)
>>>> + MPI_Barrier(MPI_COMM_WORLD);
>>>> +
>>>> + if (global_rank == 0)
>>>> cout << "Reading file " << test_file_name << "\n with options: "
>>>> << options << endl
>>>> - << " on " << nprocs << " processors\n";
>>>> + << " on " << nprocs << " processors on " << nbComms << "
>>>> communicator(s) \n";
>>>>
>>>> ErrorCode rval = mb->load_file(test_file_name.c_str(), 0,
>>>> options.c_str());
>>>> if (rval != MB_SUCCESS) {
>>>> @@ -69,9 +102,9 @@ int main(int argc, char **argv)
>>>> for (int i = 0; i < 4; i++)
>>>> nums[i] = (int)owned_entities.num_of_dimension(i);
>>>> vector<int> rbuf(nprocs*4, 0);
>>>> - MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, MPI_COMM_WORLD);
>>>> + MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
>>>> // Print the stats gathered:
>>>> - if (rank == 0) {
>>>> + if (global_rank == 0) {
>>>> for (int i = 0; i < nprocs; i++)
>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>> rbuf[4*i] << " verts, " <<
>>>> rbuf[4*i + 1] << " edges, " << rbuf[4*i + 2] << " faces, "
>>>> << rbuf[4*i + 3] << " elements" << endl;
>>>> @@ -109,7 +142,7 @@ int main(int argc, char **argv)
>>>>
>>>> // gather the statistics on processor 0
>>>> MPI_Gather(nums, 4, MPI_INT, &rbuf[0], 4, MPI_INT, 0, comm);
>>>> - if (rank == 0) {
>>>> + if (global_rank == 0) {
>>>> cout << " \n\n After exchanging one ghost layer: \n";
>>>> for (int i = 0; i < nprocs; i++) {
>>>> cout << " Shared, owned entities on proc " << i << ": " <<
>>>> rbuf[4*i] << " verts, " <<
>>>>
>>>> Repository URL: https://bitbucket.org/fathomteam/moab/
>>>>
>>>> --
>>>>
>>>> This is a commit notification from bitbucket.org. You are receiving
>>>> this because you have the service enabled, addressing the recipient of
>>>> this email.
>>>>
>>
>> --
>> Timothy J. Tautges
>> Manager, Directed Meshing, CD-adapco
>> Phone: 608-354-1459
>> timothy.tautges at cd-adapco.com
--
Timothy J. Tautges
Manager, Directed Meshing, CD-adapco
Phone: 608-354-1459
timothy.tautges at cd-adapco.com
More information about the moab-dev
mailing list