[petsc-dev] Integrating PFLOTRAN, PETSC & SAMRAI

[Barry Smith]

On Jun 6, 2011, at 3:21 PM, Boyce Griffith wrote:

> 
> 
> On 6/6/11 4:16 PM, Barry Smith wrote:
>> 
>> On Jun 6, 2011, at 3:14 PM, Boyce Griffith wrote:
>> 
>>> 
>>> 
>>> On 6/6/11 3:55 PM, Barry Smith wrote:
>>>> 
>>>> On Jun 6, 2011, at 2:50 PM, Boyce Griffith wrote:
>>>> 
>>>>> I maintain similar SAMRAI-PETSc stuff, although I'm only synched up with PETSc 3.1, not petsc-dev.  VecNorm caching caused me endless grief until I started making lots of calls to PetscObjectStateIncrease(), both in the Vec routines and in the Mat/nonlinear function wrappers.
>>>>> 
>>>>> If you are "building your own" Vec class, is there a better/safer way to ensure that cached Vec data is properly invalidated?
>>>> 
>>>>    Hmm, in theory the "base" vector class that calls your implementation is suppose to manage the cache/state stuff, for example VecAXPY() updates the state of the y vector. So this should not be problem except in nonlinear function evaluates where YOU are changing the vector entries directly and need to update the state.  Of course, there could be bugs in some of our base Vec classes, please report any you know about and we'll get them fixed.
>>> 
>>> All I can tell you is that, many years ago, at least some of these calls to PetscObjectStateIncrease() were needed in the Vec routines in order to get PETSc solvers to work with the SAMRAI PETSc Vec class.  To try to avoid the issues that Bobby is reporting, I went ahead and made any Vec routine that could conceivably result in an invalidation of cached Vec data call PetscObjectStateIncrease().  I probably went overboard and call it unnecessarily in some places.
>>> 
>>> I will work on systematically removing these calls and report any problems.
>>> 
>>> Besides nonlinear function evalutations, is it also necessary in implementations of MatVecMult/MatVecMultAdd/etc?  Or should these also be handled automatically by PETSc?
>> 
>> 
>>    In theory the MatMult/Add etc also update the state but it is possible that some obscure ones do not have it implemented properly.
> 
> Could calls to PetscObjectStateIncrease be added to SNES so that they are not needed anywhere in wrapper/application code?

   SNESComputeFunction() could crank up the state on the output function. I think that is the only officially "missing" place.  Putting an extra state increase there is probably fine.

   As Jed notes since the application code should always be calling MatAssemblyEnd() even on wrapper matrices that takes care of that state automatically.


   Barry


> 
> -- Boyce