[petsc-users] A bad commit affects MOOSE

[Jed Brown]

Matthew Knepley <knepley at gmail.com> writes:

> On Tue, Apr 3, 2018 at 6:06 PM, Jed Brown <jed at jedbrown.org> wrote:
>
>> Derek Gaston <friedmud at gmail.com> writes:
>>
>> > Sounds great to me - what library do I download that we're all going to
>> use
>> > for managing the memory pool?  :-)
>> >
>> > Seriously though: why doesn't MPI give us an ability to get unique tag
>> IDs
>> > for a given communicator?
>>
>> It's called a dup'd communicator.
>>
>> > I like the way libMesh deals with this:
>> > https://github.com/libMesh/libmesh/blob/master/include/
>> parallel/parallel_implementation.h#L1343
>>
>> PETSc does something similar, but using attributes inside the MPI_Comm
>> instead of as a wrapper that goes around the communicator.
>>
>> https://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/Sys/
>> PetscCommGetNewTag.html
>>
>> > I would definitely sign on for all of us to use the same library for
>> > getting unique tag IDs... and then we would need a lot less
>> communicators...
>>
>> Communicators should be cheap.  One per library per "size" isn't a huge
>> number of communicators.
>>
>
> And this
> https://experts.illinois.edu/en/publications/mpi-on-millions-of-cores still
> got published? I guess
> the reviewers never wanted any more than 2K communicators ;)

These are unrelated concerns.  A million is only 2^20 and 20 is much
less thann 2000.  The issue is that communicator etiquette between
libraries isn't expressed in some visible statement of best practices,
perhaps even a recommendation in the MPI standard.  If hypre dup'd
communicators like PETSc, then we would all have less code and it would
be nowhere near 2000 even in the massive MOOSE systems.