[petsc-dev] solicitation of ideas for managing transition to master under new test harness of new feature/bug fix branches.

[Smith, Barry F.]

  Franck,

     Thanks. Having a way to compute the dependencies of the tests on the chances could dramatically reduce the testing time. Thanks for the references.

   Barry


> On Jan 15, 2018, at 3:52 AM, Franck Houssen <franck.houssen at inria.fr> wrote:
> 
> As a end user of PETSc, I do not know much about the way you manage it internally (workflow, tests, ...).
> 
> If you don't use it already (?), the concept of "Test Impact Analysis" (= run only tests impacted by modified/pushed code) may help:
> - if your tests are wrapped with python (?), this proof of concept (for python, based on nosetests) may illustrate the idea:
>  https://paulhammant.com/2015/01/18/reducing-test-times-by-only-running-impacted-tests-python-edition/
> - if you use/like windows (I don't), it seems you may found some stuffs in Visual Studio (I've never tested this):
>  https://martinfowler.com/articles/rise-test-impact-analysis.html#Pre-calculatedGraphsOfSourceVsTests
> 
> The python proof of concept may give ideas/guidelines to write your own TIA (years ago, we did our own based on gcov).
> 
> Hope this helps,
> 
> Franck
> 
> ----- Mail original -----
>> De: "Barry F. Smith" <bsmith at mcs.anl.gov>
>> À: "petsc-dev" <petsc-dev at mcs.anl.gov>
>> Envoyé: Dimanche 14 Janvier 2018 05:48:13
>> Objet: [petsc-dev] solicitation of ideas for managing transition to master under new test harness of new feature/bug
>> fix branches.
>> 
>> 
>>   With the new test harness I am soliciting ideas of how to manage the
>>   transition of new feature and bug fix branches from developers/pull
>>   requests into master to insure master remains clean and reliable but it
>>   doesn't take days or weeks to get new features and bug fixes into master.
>> 
>>   I am working with the assumption that with our current hodge-podge of test
>>   hardware it would take about 3 hours to run the equivalent of the current
>>   nightly builds on a new feature/bug branch (this means many OS, many
>>   compiles, many variations of external packages, many variations of
>>   configure options (single, double, quad, complex, 64 bit indices). With
>>   the purchase of new hardware it may be possible to get it down to 30
>>   minutes. It will be impossible to get the full test suite with many
>>   variants down to a couple minutes.
>> 
>> There are issues with race conditions if multiple people want to test and
>> then move branches over at the same time. I'm looking for any ideas on how
>> to limit/eliminate them. We can't have the current situation where multiple
>> people put broken things into next and it is difficult to figure out what
>> branches caused the failures in next and get them fixed promptly, thus
>> resulting in a sluggish next (and hence sluggish PETSc development).
>> 
>>  Thanks
>> 
>>   Barry
>> 
>>   Let's not waste time with another war over the status of next. The issues
>>   are largely the same with or without next. So if you like the next model,
>>   think we are keeping it, if you don't like it, think we are getting rid
>>   of it. But regardless, next (nor master) cannot be the dumping ground for
>>   broken branches as it has been.
>> 
>>   Also a couple of simple mined automated tests on GitHub or Bitbucket are
>>   not good enough to move branches into next/master. We need extensive
>>   testing with many compilers/OS that these sites do not support before
>>   something moves up to next/master. Neither next nor master is the place
>>   to find bug issues, they (95+ % of bug issues) must be dealt with before
>>   the branch ever hits next/master.
>> 
>> 
>> 
>> 
>>