[petsc-dev] solicitation of ideas for managing transition to master under new test harness of new feature/bug fix branches.

[Franck Houssen]

As a end user of PETSc, I do not know much about the way you manage it internally (workflow, tests, ...).

If you don't use it already (?), the concept of "Test Impact Analysis" (= run only tests impacted by modified/pushed code) may help:
- if your tests are wrapped with python (?), this proof of concept (for python, based on nosetests) may illustrate the idea:
  https://paulhammant.com/2015/01/18/reducing-test-times-by-only-running-impacted-tests-python-edition/
- if you use/like windows (I don't), it seems you may found some stuffs in Visual Studio (I've never tested this):
  https://martinfowler.com/articles/rise-test-impact-analysis.html#Pre-calculatedGraphsOfSourceVsTests

The python proof of concept may give ideas/guidelines to write your own TIA (years ago, we did our own based on gcov).

Hope this helps,

Franck

----- Mail original -----
> De: "Barry F. Smith" <bsmith at mcs.anl.gov>
> À: "petsc-dev" <petsc-dev at mcs.anl.gov>
> Envoyé: Dimanche 14 Janvier 2018 05:48:13
> Objet: [petsc-dev] solicitation of ideas for managing transition to master under new test harness of new feature/bug
> fix branches.
> 
> 
>    With the new test harness I am soliciting ideas of how to manage the
>    transition of new feature and bug fix branches from developers/pull
>    requests into master to insure master remains clean and reliable but it
>    doesn't take days or weeks to get new features and bug fixes into master.
> 
>    I am working with the assumption that with our current hodge-podge of test
>    hardware it would take about 3 hours to run the equivalent of the current
>    nightly builds on a new feature/bug branch (this means many OS, many
>    compiles, many variations of external packages, many variations of
>    configure options (single, double, quad, complex, 64 bit indices). With
>    the purchase of new hardware it may be possible to get it down to 30
>    minutes. It will be impossible to get the full test suite with many
>    variants down to a couple minutes.
> 
>  There are issues with race conditions if multiple people want to test and
>  then move branches over at the same time. I'm looking for any ideas on how
>  to limit/eliminate them. We can't have the current situation where multiple
>  people put broken things into next and it is difficult to figure out what
>  branches caused the failures in next and get them fixed promptly, thus
>  resulting in a sluggish next (and hence sluggish PETSc development).
> 
>   Thanks
> 
>    Barry
> 
>    Let's not waste time with another war over the status of next. The issues
>    are largely the same with or without next. So if you like the next model,
>    think we are keeping it, if you don't like it, think we are getting rid
>    of it. But regardless, next (nor master) cannot be the dumping ground for
>    broken branches as it has been.
> 
>    Also a couple of simple mined automated tests on GitHub or Bitbucket are
>    not good enough to move branches into next/master. We need extensive
>    testing with many compilers/OS that these sites do not support before
>    something moves up to next/master. Neither next nor master is the place
>    to find bug issues, they (95+ % of bug issues) must be dealt with before
>    the branch ever hits next/master.
> 
> 
> 
> 
>