[petsc-dev] [petsc-users] new book introducing PETSc for PDEs

[Barry Smith]

> On Oct 31, 2020, at 10:59 PM, Jed Brown <jed at jedbrown.org> wrote:
> 
> Barry Smith <bsmith at petsc.dev> writes:
> 
>>> On Oct 31, 2020, at 5:21 PM, Jed Brown <jed at jedbrown.org> wrote:
>>> 
>>> Barry Smith <bsmith at petsc.dev> writes:
>>> 
>>>>> On Oct 31, 2020, at 9:35 AM, Jed Brown <jed at jedbrown.org> wrote:
>>>>> 
>>>>> Barry Smith <bsmith at petsc.dev> writes:
>>>>> 
>>>>>> I have no problem updating any of Ed's examples if we need to with each release, so the burdern doesn't fall on him. We simply make a fork of his 
>>>>>> repository with a new branch and update that and make a MR to Ed for each release and he can have a new branch or tag of his examples for each new 
>>>>>> PETSc release.
>>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>> We just make this part of our release process.
>>>>> 
>>>>> If we add it to CI, the workflow is
>>>>> 
>>>> 
>>>> I would keep the  fork always petsc/pkg-eds-repo   and have PETSc master always point to the fork. At releases we would have release point directly to eds. 
>>>> This would remove the need to fork constantly I use this model partially for adol-c
>>>> 
>>>> Anything wrong with this model?
>>> 
>>> This takes Ed out of the loop, but doesn't avoid the CI circus that I outlined. It's more reproducible to always pin to a commit hash in the repository, in which case there may be no need to circle back and update PETSc after merging in Ed's repo (or our fork).
>>> 
>>> An alternative would be to subtree the examples into PETSc and export upon release. Ed could be the CODEOWNER so he can weigh in on proposed changes or more idiomatic interfaces that may become available. His repo would always work with the latest release.
>> 
>>  What do you mean by release (six month release or all the time with master?)
>> 
>>  I have no problem with alternative approaches but they must truly eliminate part of the extra work required by my approach on everyone's part not increase the work. I did find some possible issues with subtree which may or may not apply 
>> 
>> https://www.atlassian.com/git/tutorials/git-subtree
>> 
>> Drawbacks (but in our opinion they're largely acceptable):
>> 
>> You must learn about a new merge strategy (i.e.git subtree).
>> Contributing code back upstream for the sub-projects is slightly more complicated.
>> The responsibility of not mixing super and sub-project code in commits lies with you.
>> 1 and 3 seem a bit concerning, I will certainly make this mistake constantly without an automatic system to prevent it.
>> 
>> ----
>> This is not a good sign, git subtree exists,  but
>> $ git help subtree
>> No manual entry for git-subtree
> 
> You're just using an old Git, for which you need to enable it explicitly.
> 
>> -----
>> 
>> Anyways back to my concern.
>> 
>> git subtree does require use of new commands every time you mess with it (say every three months) that we do not know and since each of 
>> us will do this infrequently it is likely we will not remember them (I won't)  while my approach does not require remembering new commands. 
>> My approach only requires branching, push, and pulling on the fork and updating EdsRepo.py which is things all the developers know about and 
>> do regularly since we update other external packages.

>> Circular dependencies (PETSc CI depends on Ed's p4pdes depends on PETSc) are significantly more labor-intensive and it would need to be done on each change, versus once per release cycle.

    I think we are not communicating on exactly the  same wave length. 

    I am not advocating we test on Ed's I am advocating we test on our fork of Ed's and how often our fork gets passed to Ed with MR is a choice all of us make together, now we could do it once at release time with all the fixes we made, every month or each time. Totally up to Ed how often he wants to get them, and he is free to ignore them for weeks (up to the next release if he likes) since our testing will continue because we use our fork. How often we pass them on to Ed is not related to how we maintain our CI.

    There is no circular dependency on my approach with Ed's p4pdes.

    I vote to fix things in our fork or gittree thing continuously since it makes it easier to fix things rather than wait to the release when we try to find and fix everything and it also helps tell us if we introduced a real bug into PETSc and fix PETSc immediately instead of waiting up to 6 months, just like we now we test immediately with Petsc4py and we should do with SLEPc.  How often we give the updates to Ed is a completely different issue. 

   So again back to my original statement ,it comes down to if the subtree or the fork approach is easier for all the PETSc developers who do not currently know gittree and would need to learn it with your approach. I don't know which is easier learning to use gittree which has its own gotcha's or using mine which we all know but may require an extra step (not involving Ed, just updating the p4pdes.py commit each time we change something in the fork.)

  I think using --download-p4pdes on a couple of systems in the CI is enough, I don't think we need to put it all CI pipelines (I would like slepc in all pipelines).but we could put in all pipelines if we want.

  For completeness I show the exact the work flow for my suggestion

   pipelines --download-p4pdes and runs its tests
   if it breaks the developer uses --download-p4pdes  on their system 
     they fix the problem either by fixing PETSc or what is downloaded from the p4pde fork
        if the fix is in the p4pdes fork they make a branch in the p4pdes fork, which they already have since they used --download-p4pdes and thus 
                              have the fork on their system
            they put the fix in new branch in the p4pdes fork and push it
            they edit p4pdes.py and put a new commit in it pointing to their branch in the fork
   run the pipeline again
       if fails with p4pdes they do the above again
       else the PETSc branch gets accepted and merged to master
          depending on Ed's choice we make an MR for p4pdes depending on the agreed upon cycle. If Ed puts the fix into his master then we just update
               our fork with his latest master with a simple merge of his.  This will then become the new one we test against. If Ed doesn't respond to the MR all is fine we 
              just continue on our fork. If he puts other things in his branch but not our MR we just merge that into our fork and so are still testing with his latest master.

  This happens to be the exact same thing I do now with slepc and any other git based package that I use now (except for some I need to make my own fork of the external package because we don't have one always available eventually likely we will likely put more forks into gitlab/petsc so each person doesn't constantly need to make fresh forks of external packages)

  If subtree requires fewer steps than this and has no new oddball git subtree commands then we should definitely use subtree, if it requires other steps involving subtree we need see those commands written in a workflow like I have written above and decide if it is still simpler or not.
  
  I am not rejecting subtree, we just need to explicitly see its complete work flow and hence the differences to decide, that is what I asking for.

  Barry

  It seems to me that with subtree we also need to maintain a fork of Ed's stuff or else that will have a circular dependency. But perhaps I do not understand it.