[petsc-dev] broken nightlybuilds (next vs next-tmp)

[Smith, Barry F.]

> On Nov 11, 2017, at 11:17 AM, Matthew Knepley <knepley at gmail.com> wrote:
> 
> On Fri, Nov 10, 2017 at 4:50 PM, Satish Balay <balay at mcs.anl.gov> wrote:
> 
> On Fri, 10 Nov 2017, Richard Tran Mills wrote:
> 
> > Hi Satish,
> >
> > Thanks for taking the initiative to switch to testing next-tmp to help
> > clear up the constipation with moving things into master. It looks like
> > there hasn't been any graduation of the branches you've been putting into
> > next-tmp in a few days, though. Is this just because you haven't had time
> > to do any more of these merges, or are the tests breaking with next-tmp now?
> >
> > If I go to the dashboard at
> > http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/10/next.html,
> > is this showing me the info for 'next' or 'next-tmp'? If it is showing me
> > 'next', how do I find the results for 'next-tmp'?
> 
> Sorry - I haven't been checking properly on this.
> 
> The last next-tmp build was on:
> 
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/07/next.html
> 
> There were some errors - which I didn't debug.
> 
> $ git fetch -p && comm -12 <(git branch -r --merged origin/next-tmp | sort) <(git branch -r --no-merged origin/master | sort)
>   origin/hongzh/add-tstraj-filename
>   origin/hongzh/copy_l2g_stencil
>   origin/jed/variadic-malloc
>   origin/rmills/feature-aijmkl-matmatmult
> 
> [Its probably one of the above branches. I'll look for it - and then
> merge the others to master.]
> 
> So I reverted the tests back to next stating 08
> 
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/08/next.html
> 
> Yeah - next vs next-tmp is not clearly visible here - but if you go to
> any of the logs - you'll see next or next-tmp on the filename.
> 
> For eg:
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/07/examples_full_next-tmp.log
> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2017/11/08/examples_full_next.log
> 
> > If we think a branch may be ready for 'master', should we still be merging
> > to 'next', given it's current broken state?
> 
> For now - I would say - if you now a bunch of branches that you think
> are ready (master) - I can schedule a next-tmp test on them to verify.
> 
> The tricky thing here is for all of us to be aware of this - and not
> assume next is tested - and merge then go ahead and merge the wrong
> branches to master.
> 
> > Lastly, a question for everyone: If someone knows that they have merged
> > something into 'next' that has broken the builds or tests, and it is going
> > to be a while before this is fixed, should they revert that changeset in
> > 'next'? I see some reverts in the 'next' logs, but not that many. Maybe
> > this is because it's not always easy to tell if one's particular changeset
> > broke things when there are all these other changes being merged.
> 
> Reverts are tricky - hence I avoid them. But if needed - we could do
> that [it requires extra care in the workflow - i.e when you have to
> merge back later - or revert again for the second time etc.]
> 
> Alternative is to delete/recreate next - if needed. [but it requires
> all next users to do this delete/recreation]
> 
> In the long term - Barry wants to get rid of next..
> 
> 1) I think next really prevents master from getting screwed up (witness next)

   If we run literally all the tests on the same machines as next currently does on a branch before it goes into master then we would be doing no less testing when we switch to the new model.
> 
> 2) I think we are actually finding interaction bugs there.

   Bullfucking shit! Maybe one in all the years and the hassles that next introduces are a million times worse then the one interaction bug found.

  The plan is that all the tests in next will take say 3 hours with conversion to the new test harness, hence this gives us enough time to do the complete test on each branch before master.

   BTW: anyone who has free time I urge to convert more tests from the old test harness to the new.

  Barry

> 
> Are those points wrong, or is there another way to do these things?
> 
>   Thanks,
> 
>      Matt
>  
> 
> Satish
> 
> 
> 
> -- 
> What most experimenters take for granted before they begin their experiments is infinitely more interesting than any results to which their experiments lead.
> -- Norbert Wiener
> 
> https://www.cse.buffalo.edu/~knepley/