[petsc-dev] broken nightlybuilds (next vs next-tmp)

[Satish Balay]

On Sat, 11 Nov 2017, Matthew Knepley wrote:

> On Sat, Nov 11, 2017 at 12:37 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> > On Sat, 11 Nov 2017, Matthew Knepley wrote:
> >
> > > > In the long term - Barry wants to get rid of next..
> > >
> > >
> > > 1) I think next really prevents master from getting screwed up (witness
> > > next)
> > >
> > > 2) I think we are actually finding interaction bugs there.
> > >
> > > Are those points wrong, or is there another way to do these things?
> >
> > Next is an intergration testing mechanism. The prerequisite for it [I
> > think] is - one should test the branch properly before merging to
> > next. However we are not doing proper testing before merge to next -
> > and relying on next to do this part aswell.
> >
> > So with current next - it one has to figure out which branches are
> > breaking the tests [takes time - which most of us are not doing] - and
> > then hope it gets fixed quickly. Otherwise next stay broken for a long
> > time [and other branches in next - which could be clean - don't
> > receive sufficient confidence to graduate to master]
> >
> > So Barry's thought wrt getting next is to have a better way of testing
> > feature branch one wants to test (i.e master+feature). Its not clear
> > to me how many integration issues we've addressed with our current
> > next model. [Its mostly been indvidual branch issues]
> >
> > Also if feature-1 and feature-2 are feature branches that are tested
> > in next [wrt integration]. The following should be equivalent to
> > testing 'master + feature1 + feature2' - aka current next model:
> >
> > 1. test master+feature1
> > 2. success => merge feature1 to master
> > 3. tets master+ feature2
> > 3. success => merge feature2 to master
> >
> > Note: my next-tmp is an attempt to get closer to 'master+feature1'
> > testing from 'master+feature1+feature2' testing [yeah - its more like
> > master+2/3 branches in next-tmp vs master+10/15 branches in next.]
> >
> > Also I'm restarting next-tmp from a clean master when merging new set
> > of branches to test. And throwing away branches with problems - and
> > retest only after it has fixes [This way a broken branch does not keep
> > next-tmp broken until it gets fixed.]
> 
> 
> I don't think we have the resources to run full tests on every branch one
> at a time. Do we?

We don't yet have this - but thats what I'm effectively doing manually
with next-tmp [instead of 1 - its 2-3]

Barry's thought here (I think) is - once we convert all the examples
to the new test suite - it whould run much quicker - so might be able
to a whole test suite on each branch.

Or mabe have a way to have a few (fast) tests that can uncover most of
the issues [aka travis-ci, pipelines] . Once the branch clears this -
it can go into longer test queue (aka next equivalnet - but not next).

None of this infrastructure currently exists.

BTW: Ultimlately if you want to improve current next model - everyone
has to do a 'make alltests DIFF=$PETSC_DIR/bin/petscdiff' for atleast
one build that has relavent feature options enabled - before merging
the feature branch to next.

Note - this doesn't have to be on ones laptop - it can be on es.mcs.

Satish