[petsc-dev] including PETSc4py (and Tao4py) inside PETSc repository

Barry Smith bsmith at mcs.anl.gov
Thu Oct 22 22:30:11 CDT 2015 

> On Oct 22, 2015, at 10:24 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> As I said - depending on the precondition for how tightly petsc &
> petsc4py should be synced - we can choose the appropriate model.
> 
> [for the metioned precondition - I was suggestion the model].
> 
> As you say - the initilal phrasing is wrong - so the model is not
> suitable.
> 
> A single repo does make it easy for 'all feature branches' to be in
> sync..
> 
> In that model with-petsc4py=1 would be the default. However anyone
> changing the petsc UI would also have to update petsc4py with the
> change.. [and Lisandro says - that change could be buggy - and he
> would like to review it -before its applied]

   Just like Jed was going to review everything anybody put into PETSc before it got into master. That is just unrealistic and won't last long. Better to just have a good test suite.

> 
> For 2 repos model - perhaps sync at 'master' might work.
> 
> Satish
> 
> On Thu, 22 Oct 2015, Barry Smith wrote:
> 
>> 
>>> On Oct 22, 2015, at 9:53 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>> 
>>> If dmplex with petsc4py is the primary use case [and assuming the
>>> usage is primarily --download-petsc4py] - we chould use the first
>>> model I mentioned - and that should not be too difficult.
>>> 
>>> The workflow would be something like:
>>> 
>>> - Matt would do the updates in petsc/dmplex feature branch & petsc4py
>>> dmplex branches simultaneosusly.
>> 
>>   That's nuts; Matt is a college professor he doesn't have time to keep two branches in sync for no good reason when they could be in the same repository.
>> 
>>   Plus other people will be making other changes in PETSc that break petsc4py
>> 
>>   This discussion has shown to me that keeping petsc and petsc4py in different repositories is just silly and unproductive.
>>> 
>>> - And would always use --download-petsc4py for all his build tests
>>> [we'll have to fix auto-update for --download-pkg for git version of
>>> pkg]
>>> 
>>> - Any isues that come up - Matt will fix in petsc4py/dmplex and update
>>> petsc4py [before this branch is ever merged to next]
>>> 
>>> So after next testing - a merge to master - Matt could issue a pull
>>> requst to Lisandro [or leave it for Lisandro to figureout when
>>> petsc4py-master gets updated with the fix].
>>> 
>>> The update would involve merging/updating petsc4py-master [and and
>>> update to petsc4py.py in petsc-master]
>>> 
>>> [even if this petsc4py-master update doesn't happen immediately] it
>>> won't affect petsc4py/dmplex users as they would be using
>>> --download-petsc4py [which will use the working petsc4py.commitid set
>>> my Matt anyway]
>>> 
>>> If the usage is not via --download-petsc4py - then there could be a
>>> more appropriate workflow for that.. [and a different testsuite to
>>> match that usage..]
>>> 
>>> Satish
>>> 
>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>> 
>>>> 
>>>> The model we adopt also depends on how cutting edge petsc4py users want to be with petsc. For example if people want to use dmplex with petsc4py then the two packages have to move very closely in sync.
>>>> 
>>>>> On Oct 22, 2015, at 9:11 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>> 
>>>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>>>> 
>>>>>> 
>>>>>>> On Oct 22, 2015, at 7:28 PM, Satish Balay <balay at mcs.anl.gov> wrote:
>>>>>>> 
>>>>>>> The current infrastruture tests --download-petsc4py in petsc 'next'
>>>>>>> and 'master' branches.
>>>>>>> 
>>>>>>> Wrt next - usually a change in a feature branch (when it gets merged
>>>>>>> to next) triggers this breakage.  And usual thing to do is to fix it
>>>>>>> in the feature branch (and merge to next again)
>>>>>>> 
>>>>>>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with
>>>>>>> the petsc4py repo gitcommit [optionally also tarball ] that
>>>>>>> corresponds to the fix. [one way to handle this on petsc4py repo is to
>>>>>>> create a feature branch for this fix]
>>>>>> 
>>>>>> Yeah this is nuts, having to maintaining matching branches between petsc and petsc4py and merge them at the same time into next and then master. Clearly no one will ever do this and hence petsc4py will always be out of step from PETSc. I do really believe that incorporating petsc4py into PETSc would just make all these difficulties go away. For example maintaining Tao is a breeze now since it is updated in the same branches as PETSc.
>>>>>> 
>>>>> 
>>>>> Perhaps I should rephrase the issue in the following way:
>>>>> 
>>>>> We should decide on what level petsc4py and petsc should be synced.
>>>>> [and the testsuite should be modified appropriately for that]
>>>>> 
>>>>> Currently its done at next, master branchs. For this to work - the
>>>>> above - previously mentioned workflow is required. [and for this
>>>>> workflow - a single repo does makes thing easier]
>>>>> 
>>>>> We could say sync should happen only between petsc4py-master &
>>>>> petsc-master. [so issues - if any in feature branches/next wont't be
>>>>> dealt with - and its ok for the master branches to be out of sync for
>>>>> a brief time for the fix to propergate].
>>>>> 
>>>>> For this - we would have a different workflow [wrt syncing petsc4py
>>>>> wrt petsc - and a slightly different test suite - testing only
>>>>> master. [We could continue having the curent test suite - but ignore
>>>>> errors in next].  And e-mails to Lisandro will trigger only on
>>>>> petsc4py errors on master.  Lisandro would then fix petsc4py (master)
>>>>> - and then update petsc4py.py in petsc-master.
>>>>> 
>>>>> Or perhaps some other mode is preferable..
>>>>> 
>>>>> Satish
>>>>> 
>>>>>> Barry
>>>>>> 
>>>>>>> 
>>>>>>> When this (petsc) feature branch is merged into master - the
>>>>>>> corresponding feature branch on the petsc4py side can be merged into
>>>>>>> petsc4py master [if petsc4py-master is to be in sync with
>>>>>>> petsc-master].
>>>>>>> 
>>>>>>> [if there are multiple feature branches on the petsc side requiring
>>>>>>> corresponding petsc4py changes - somehow that has to be handled]
>>>>>>> 
>>>>>>> Perhaps testing petsc4py via petsc infrastructure [using
>>>>>>> --download-petsc4py] is not the appropriate approach?
>>>>>>> 
>>>>>>> Or we should do this only on master - not next?  [or just have a
>>>>>>> single --download-petsc4py test - thats not mixed with other tests? -
>>>>>>> so its breakage is not critical for other things]
>>>>>>> 
>>>>>>> Also we don't currently run any petsc4py tests. [we just test if it
>>>>>>> compiles or not]
>>>>>>> 
>>>>>>> Satish
>>>>>>> 
>>>>>>> On Thu, 22 Oct 2015, Barry Smith wrote:
>>>>>>> 
>>>>>>>> 
>>>>>>>> Lisandro,
>>>>>>>> 
>>>>>>>> It is tested nightly but there is apparently currently no email sent on failure. See the bottom of, for example, http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log
>>>>>>>> 
>>>>>>>> I think what should happen is the petsc4py log file should be published to the web (like the other log files) and a script look through it and decide to send email if it detects a problem. Like we currently have for the other log files. Someone has to dig into the nightly test infrastructure to figure out where the new logic needs to go. Currently Satish and Karl understand that infrastructure the best.
>>>>>>>> 
>>>>>>>> Barry
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin <dalcinl at gmail.com> wrote:
>>>>>>>>> 
>>>>>>>>> On 22 October 2015 at 23:20, Barry Smith <bsmith at mcs.anl.gov> wrote:
>>>>>>>>>> 
>>>>>>>>>> Lisandro,
>>>>>>>>>> 
>>>>>>>>>> You never responded to this. I assume it was because you did not like the idea? This is not an attempt to take control of petsc4py or to take credit away from petsc4py from you. Petsc4py is your package and you will always be the one who deserves the credit.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Oh, sorry! I missed this email.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>> The problem is that mutual efficient development of very intertwined packages is difficult if they are in different respositories. For example petsc4py has been out of date with petsc master for many weeks now because it is too much effort to update the petsc4py repository each time we make some change in PETSc. Thus it becomes a burden on you to go in every once in a while and fix up petsc4py. With one repository changes would happen in both PETSc source and petsc4py source at the same time and tests would detect problems immediately.
>>>>>>>>>> 
>>>>>>>>>> Thoughts?
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> I general terms, I prefer to keep things separate as much a possible.
>>>>>>>>> IMHO, a much important step would be to incorporate petsc4py to the
>>>>>>>>> nightly tests, clearly separating the petsc and petsc4py tests runs. I
>>>>>>>>> would love to get an email every time petsc4py nightly breaks, this
>>>>>>>>> way I would be able to push fixes within a day or two, say.
>>>>>>>>> Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so
>>>>>>>>> ultimately I'm the one in best position to do such work, if you guys
>>>>>>>>> start adding commits to petsc4py withing the main petsc repo, I'm
>>>>>>>>> likely going to miss these changes. For example, today Michael Lange
>>>>>>>>> made some changes, I got an email from Bitbucket, then did a review
>>>>>>>>> and spotted a few issues.
>>>>>>>>> 
>>>>>>>>> My hesitation to incorporate petsc4py within petsc repo is, in
>>>>>>>>> Python's [and linear algebra :-)] Zen : "Sparse is better than dense".
>>>>>>>>> If we continue adding stuff to core PETSc, at same point we will endup
>>>>>>>>> like Trilinos ( DON'T MISS the sencond entry in the FAQ:
>>>>>>>>> https://trilinos.org/download/public-git-repository/).
>>>>>>>>> 
>>>>>>>>> All that being said, I don't have any problem at all adding petsc4py
>>>>>>>>> to the main PETSc repository. But I still think this is not a good
>>>>>>>>> idea, it will not solve all the problems. Nightly tests with automatic
>>>>>>>>> notifications to me about breakages is perhaps is all what we need to
>>>>>>>>> keep petsc4py in close sync with petsc/master. What about trying this
>>>>>>>>> first to see how it goes?
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> PS: Please understand my position has nothing to do with credits about
>>>>>>>>> the project or ego-related crap. Actually, your proposal would mean
>>>>>>>>> less responsibility and maintenance work for me. But I don't think it
>>>>>>>>> is a move in the right direction.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> -- 
>>>>>>>>> Lisandro Dalcin
>>>>>>>>> ============
>>>>>>>>> Research Scientist
>>>>>>>>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE)
>>>>>>>>> Numerical Porous Media Center (NumPor)
>>>>>>>>> King Abdullah University of Science and Technology (KAUST)
>>>>>>>>> http://numpor.kaust.edu.sa/
>>>>>>>>> 
>>>>>>>>> 4700 King Abdullah University of Science and Technology
>>>>>>>>> al-Khawarizmi Bldg (Bldg 1), Office # 4332
>>>>>>>>> Thuwal 23955-6900, Kingdom of Saudi Arabia
>>>>>>>>> http://www.kaust.edu.sa
>>>>>>>>> 
>>>>>>>>> Office Phone: +966 12 808-0459
>>>> 
>>>> 
>>> 
>> 
>> 
>

More information about the petsc-dev mailing list