[petsc-dev] Integrating PFLOTRAN, PETSC & SAMRAI

[Barry Smith]

On Jun 6, 2011, at 2:50 PM, Boyce Griffith wrote:

> I maintain similar SAMRAI-PETSc stuff, although I'm only synched up with PETSc 3.1, not petsc-dev.  VecNorm caching caused me endless grief until I started making lots of calls to PetscObjectStateIncrease(), both in the Vec routines and in the Mat/nonlinear function wrappers.
> 
> If you are "building your own" Vec class, is there a better/safer way to ensure that cached Vec data is properly invalidated?

   Hmm, in theory the "base" vector class that calls your implementation is suppose to manage the cache/state stuff, for example VecAXPY() updates the state of the y vector. So this should not be problem except in nonlinear function evaluates where YOU are changing the vector entries directly and need to update the state.  Of course, there could be bugs in some of our base Vec classes, please report any you know about and we'll get them fixed.

   Barry



> 
> -- Boyce
> 
> On 6/6/11 3:40 PM, Bobby Philip wrote:
>> What usually breaks for me is VecNorm caching, arguments for functions,
>> and introduction of new functions. Once in a while the PETScObject structure
>> changes and that causes trouble too. But this is my usual list.
>> 
>> Any time petsc-dev changes, I end up having to build:
>> 
>> 1. SAMRAI
>> 2. SAMRUtils
>> 3. SAMRSolvers
>> 4. PFLOTRAN
>> 
>> with stops along the way to fix anything that broke mainly in 1 and 4. Also, SAMRAI
>> continues to only want to support version 2.3.3 of PETSc so I end up keeping tabs
>> on what needs to be changed with each new SAMRAI version. A set of automated
>> tests I hope might be the right catalyst to move them to the present generation(s) of PETSc.
>> 
>> Bobby
>> 
>> On Jun 4, 2011, at 9:09 AM, Matthew Knepley wrote:
>> 
>>> On Sat, Jun 4, 2011 at 8:03 AM, Bobby Philip<philipb at ornl.gov>  wrote:
>>> 
>>>> Barry:
>>>> 
>>>> I heartily agree with all that you have said. Thanks for your suggestions.
>>>> I do believe what you have suggested is the best both in the shorter and
>>>> longer term. As a clarification on freezing the petsc-dev version, my
>>>> suggestion was to freeze
>>>> the version for a few months at a time as it reduces the burden on me on
>>>> fixing frequent breakages.
>>>> 
>>> 
>>> We don't have to bog down pflotran-dev with particulars, but can you
>>> characterize what breaks
>>> most often for you, or what you find yourself doing most often for SAMRAI?
>>> 
>>>  Thanks,
>>> 
>>>     Matt
>>> 
>>> 
>>>> Regards,
>>>> Bobby
>>>> 
>>>> 
>>>>> 
>>>>> Peter,
>>>>> 
>>>>>   In the mid to longer term we need to make sure that SAMRAI has more
>>>> resources than a one man operation, so this is something that needs coverage
>>>> in the next round of SciDAC.
>>>>> 
>>>>>   When dealing with tracking package changes I've found the best way is
>>>> to track them continuously so each new issue (due to a change) gets fixed
>>>> immediately rather than waiting until a bunch of stuff is broken before
>>>> fixing it.  This way determining what change broke what, is much easier.
>>>> This means we should set up nightly tests of PETSc+SAMRAI+PFLOTRAN and deal
>>>> with each issue the day after it breaks instead of weeks later. Probably
>>>> Richard could set up such a test and have it send email to say Richard,
>>>> Bobby and I each time a breakage occurs.
>>>>> 
>>>>> 
>>>>>  Barry
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Jun 3, 2011, at 5:49 PM, Lichtner, Peter C wrote:
>>>>> 
>>>>>> Folks: we have a problem trying to integrate PFLOTRAN with PETSc and
>>>> SAMRAI. The basic problem is that when PETSc is updated it often breaks
>>>> SAMRAI in nontrivial ways. For PFLOTRAN itself the changes needed are
>>>> usually minor to bring PFLOTRAN back in sync with PETSc. But this is not the
>>>> case for SAMRAI. Bobby Philip, the lead on SAMRAI, does not have the
>>>> resources to develop the PFLOTRAN implementation of SAMRAI and keep it up to
>>>> date with changes in PETSc. The question is how to resolve this difficulty.
>>>> We have expended a considerable effort in adding SAMRAI to PFLOTRAN and
>>>> several directions of PFLOTRAN development rely on SAMRAI including
>>>> applications to CO2 sequestration, the Hanford uranium plume, and a hybrid
>>>> model using the lattice Boltzmann method at the pore scale coupled to the
>>>> Darcy continuum scale with SAMRAI providing the bridge between the two
>>>> scales. Bobby has suggested freezing the PETSc developer version used in
>>>> PFLOTRAN. However, this would potentially limit the development of PFLOTRAN.
>>>> An alternative approach is to create a separate branch of PFLOTRAN for use
>>>> with SAMRAI. However, this would limit our ability to develop a hybrid
>>>> model, not to mention causing difficulties down the road when the two
>>>> branches are finally merged. Any thoughts, suggestions, comments, etc.
>>>> appreciated on how we can best resolve this issue.
>>>>>> Thanks, ...Peter
>>>>>> <><><><><><><><><><><><><><><><><><><><><>
>>>>>> Peter C. Lichtner (lichtner at lanl.gov)
>>>>>> LANL EES-16: MS D469
>>>>>> (505) 667-3420 (o), 665-3285 (fax), 795-2881 (cell)
>>>>>> Los Alamos National Laboratory
>>>>>> SM-30 Bikini Atoll Road
>>>>>> Los Alamos, NM 87545
>>>>>> blockedblockedhttp://www.ees.lanl.gov/pflotran/
>>>>>> <><><><><><><><><><><><><><><><><><><><><>
>>>>>> 
>>>>>> 
>>>>>> 
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>>> --
>>> What most experimenters take for granted before they begin their experiments
>>> is infinitely more interesting than any results to which their experiments
>>> lead.
>>> -- Norbert Wiener
>> 
>>