[petsc-users] DMDA and local-to-global scatters

[Coon, Ethan]

Hi Barry, all,

I’m trying to understand some extremely old code (4 years now!) that people insist on using and therefore having me support, despite the fact that the DOE doesn’t think it is important enough to pay me to support.

Barry — can you explain to me the history and logic in your change:

https://bitbucket.org/petsc/petsc/commits/bd1fc5ae41626b6cf1674a6070035cfd93e0c1dd

that removed DMDA’s local-to-global map, in favor of doing the reverse scatter on global-to-local?  When I wrote the aforementioned LBM code, the local-to-global scatter had different semantics than the reverse global-to-local scatter.  The latter merged owned and ghosted values from the local Vec into the owned global Vec, while the former ignored ghost values and did a direct copy from owned local values to owned global values.  Under INSERT_VALUES, this was important as the latter was a race condition while the former was a well-posed operation.

Now I see that this L2G scatter has been removed (in 2014), which introduced a race condition.  It was tweaked a bit for 1 process by this commit:

https://bitbucket.org/petsc/petsc/commits/1eb28f2e8c580cb49316c983b5b6ec6c58d77ab8

which refers to a Lisandro email that I’m having trouble finding.  Fortunately this caused some errors that I did see, as opposed to the previous race conditions which I didn’t see.

Additionally there was documentation added to not do INSERT_VALUES with DMDAs at some point.  (Maybe because it causes race conditions!)  This documentation suggests to "simply compute the values directly into a global vector instead of a local one."  This isn’t a good choice in my application, where I do many time steps in local vectors, using repeated calls to “DMLocalToLocalBegin()”, and only go back to the global vector when I want to do i/o.  Computing into global vectors requires managing two Vecs throughout the entire code, when otherwise I only manage one (except in the i/o portion of the code).  I guess the answer is to create and use a L2G forward scatter myself?  Is there a better solution I’m not thinking of?

Thanks,

Ethan



------------------------------------------------------------------------
Ethan Coon
Research Scientist
Computational Earth Science -- EES-16
Los Alamos National Laboratory
505-665-8289

http://www.lanl.gov/expertise/profiles/view/ethan-coon
------------------------------------------------------------------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mcs.anl.gov/pipermail/petsc-users/attachments/20151029/7d11b1f2/attachment.html>