[petsc-users] [petsc-dev] petsc git next branch *is* unwound!

[Satish Balay]

On Fri, 10 Oct 2014, Dave Nystrom wrote:

> Thanks for your detailed reply.  Suppose I start working with wdn_mods_master
> instead of wdn_mods_next where wdn_mods_master is master + my local changes
> to master.  Now, suppose I find a bug which I report.  With the current petsc
> workflow, it seems like that bug would get fixed on next and then merged to
> master after some amount of time.  Is that correct? 

Nope - the bug is fixed in a corresponding 'feature' branch [where the
bug was originally introduced] or a new 'bug-fix' branch [created off
latest master/maint] or directly in master/maint (if we think its a
trivial fix that wont' break stuff). And then this branch is merged to
next for further testing.

Note: We don't develop (or make direct commits) on 'next'. It is an
integration branch where feature/bugfix branches are merged into - and
from where nightly tests are done. We find bugs with this testing - or
we realize (very rarely) some of these feature branches won't graduate
and make it to master [or need to be redone]. Hence its discard after
every release.

You are correct in the sense that most branches [sometimes bugfixes]
have to stay in next for some time - before they get merged to
master. And this 'time' 1 day or more [due to the way some of this
testing is done - party the shedule - partly the amount of testing -
and partly how we manually interpret test results and match them with
these branches]

> If so, then I have to
> wait for that amount of time before I can have the fix.  Or I would need to
> figure out how to pull it into my wdn_mods_master from next.

You can temporarily switch to the 'bug-fix' branch and use it - or
merge it [but then be prepared to revert it - if this branch is
rebased or discarded]. Yeah - its not ideal. But one of the points of
git is to use/discard branches as you need. [trying to stick to one
branch to track multi-branch development can be difficult]

> And then most of the petsc features I am most interested in show up in next
> before master - like the stuff with cuda, threads and ViennaCL, etc.  So I'm
> not sure how to really dispense with my use of and reliance on next.  But I
> will give it more thought.

Yes - next is where the features show up first. But thats different
then 'use next for bugfixes' - as 'next' is likely to have more new
bugs [due to these new features] than bugfixes.


> And I will study the details of your reply below and see if I can upgrade my
> git skills.  But I've never had to use the rebase, reset or cherry picking
> features of git before.  So it will take take some study.

And based on the issue you are having with 'git reset --hard next'
(the 'next' part - not the 'git reset' part) - an understanding of git
branches is perhaps needed. Without it - its not easy to use git [or
communicate about it]

for eg:

- understaning origin/branchname vs branchname [and how fetch/pull
  (and other basic operations) differ on origin/branchname vs
  branchname

- use of commit-id vs branchname in various git commands

- branchname vs tag

- what is a 'new branch' - and what does 'delete branch' mean

etc..

Satish

> 
> Satish Balay writes:
>  > I believe the workflow used by petsc is same as what most projects
>  > with git do - so there is nothing special about it.
>  > 
>  > Try 'man gitworkflows'
> 
> I'll check this out.
> 
>  > I think your reason for using next is flawed (you get more bugs than
>  > bugfixes on next - as its an integration branch - and features are
>  > merged here for testing purposes). But if thats what you want to do -
>  > thats fine. We can benefit with the extra testing that you are now
>  > providing on next. And I believe git gives you tools to manage
>  > whatever workflow you choose.
>  > 
>  > Generally we don't branch off next. New branches are created from
>  > master (or maint). This works well for new features workflow (or bug
>  > fix workflow).
>  > 
>  > In your case - you'd like to maintain 'wdn_mods_next' branch - which
>  > is 'next' + local fixes that are specific to next (and not shared
>  > across other branches - like master). And these fixes are next
>  > specific - hence you want this branch to start off next. Presumably
>  > this is low over head stuff - and don't need permanant history for
>  > this (as we usually do with feature branches)
> 
> It is low over head and I don't need a permanent history of petsc changes.
> 
>  > For the above requirement - currently you do:
>  > 
>  > a. branch wdn_mods_next from next
>  > b. add local commits in this branch
>  > c. merge latest next into wdn_mods_next. (as needed)
> 
> Right.
> 
>  > Alternative is to use 'git rebase' to manage your local commits.  This
>  > is not critical - but it lets you keep track of them more sanely.
>  > 
>  > <rebase local commits over latest next>
>  > git checkout next
>  > git pull
>  > git checkout wdn_mods_next
>  > git rebase next
>  > 
>  > [also check 'git rebase -i']
> 
> OK, I don't really know what rebase does - so I will have to study the
> documentation and my git book.
> 
>  > In the present situation - when next is unwound - you have to unwind
>  > and recreate wdn_mods_next. For this you'd like to extract your
>  > current modifications from wdn_mods_next - and apply to a new
>  > wdn_mods_next branch.
>  > 
>  > <First step is to backup current wdn_mods_next. i.e:>
>  > git checkout wdn_mods_next
>  > git checkout -b wdn_mods_next-oct-2014
>  > 
>  > <Now reset wdn_mods_next to current next>
>  > git checkout wdn_mods_next
>  > git reset --hard next # assuming you've already reset 'next' to latest
> 
> At the moment, I'm not sure the above git reset command will work - based on
> trying it and getting a git error message.
> 
>  > <check previous local modifications>
>  > git log origin/next-oct-2014..wdn_mods_next-oct-2014
>  > git diff origin/next-oct-2014..wdn_mods_next-oct-2014
>  > 
>  > Now you can 'git cherrypick' the commits you need [from the above
>  > list] or grab the above diff - and apply to dn_mods_next(new) branch. for eg:
>  > 
>  > git diff origin/next-oct-2014..wdn_mods_next-oct-2014 > my_changes
>  > patch -Np1 < my_changes
>  > <resolve patch conflicts manually>
>  > git commit
> 
> This seems like a complicated procedure but maybe it will seem simpler and
> more intuitive if I study this more.
> 
> Thanks,
> 
> Dave
> 
>  > Satish
>  > 
>  > On Fri, 10 Oct 2014, Nystrom, William David wrote:
>  > 
>  > > Jed,
>  > > 
>  > > I don't think your reply below really does what I want to do.  My wdn_mods_next branch is
>  > > a branch that I maintain that was derived from the next branch.  I do not merge changes
>  > > from wdn_mods_next into the next branch.  The next branch is supposed to be a mirror of
>  > > the official petsc next branch and I only update the next branch in my petsc clones by first
>  > > performing a "git remote -v update origin" in my bare mirror repo which mirrors the petsc maint,
>  > > master and next branches as well as containing any additional branches that I create.
>  > > 
>  > > So, I update wdn_mods_next from the latest copy of next that I have by using the following
>  > > command:
>  > > 
>  > > git pull origin next
>  > > 
>  > > in the petsc directory where I have wdn_mods_next checked out.  Generally this is easy and
>  > > has no conflicts to resolve but on a rare occasion, I might have to resolve a conflict.  This is
>  > > because my wdn_mods_next branch does not differ much from next - just a few hacks I have
>  > > had to resort to.
>  > > 
>  > > So here is one way I think I could accomplish my goal.  In my bare, mirror repo execute
>  > > 
>  > > git remote -v update origin
>  > > 
>  > > At this point my bare mirror repo now has a copy of the new unwound next branch - if I am
>  > > understanding correctly.
>  > > 
>  > > Then in a clone of my mirror repo execute
>  > > 
>  > > git checkout wdn_mods_next
>  > > git reset --hard next
>  > > git push origin +wdn_mods_next
>  > > 
>  > > If I understand correctly, then my wdn_mods_next branch would be identical to the next
>  > > branch and I would not have the changes I made to next that resulted in wdn_mods_next.
>  > > 
>  > > Then I could untar a file of those changes on top of my clone with wdn_mods_next checked
>  > > out and recommit those changes.  This is about half a dozen files.  But this seems like a
>  > > hacky way to do business that is only reasonable because I have a small number of files.
>  > > 
>  > > Isn't there a better way to do this?
>  > > 
>  > > Perhaps you or Satish might ask why I am using next instead of master.  The reason is that
>  > > I am forever uncovering some bug in petsc that you guys fix on the next branch and I don't
>  > > want to wait for several days before it gets pulled into master and I can resume productive
>  > > work.
>  > > 
>  > > So maybe there is some flaw or weakness in my git work flow.  I have been using git for
>  > > several years now but am not a power user.  I like git but do not really want to have to
>  > > become a git guru.  However, this petsc policy of periodically unwinding the next branch
>  > > seems to always disrupt my workflow - so I have not found a robust way to deal with it
>  > > yet.
>  > > 
>  > > Dave
>  > > 
>  > > ________________________________________
>  > > From: Jed Brown [jed at jedbrown.org]
>  > > Sent: Thursday, October 02, 2014 4:39 PM
>  > > To: Satish Balay; Nystrom, William David
>  > > Cc: petsc-users; petsc-dev at mcs.anl.gov
>  > > Subject: Re: [petsc-dev] [petsc-users] petsc git next branch *is* unwound!
>  > > 
>  > > Satish Balay <balay at mcs.anl.gov> writes:
>  > > 
>  > > > You would have to delete both 'next' and 'wdn_mods_next' branches from
>  > > > all your clones - and recreate them.
>  > > 
>  > > Do your normal "git remote update" (or git fetch) in your "master" clone
>  > > (confusing to use the same name to refer to a repository and a branch).
>  > > I'll call the repository Mirror.  Anyway, you should see that all three
>  > > branches are updated.  'maint' and 'master' should fast-forward, while
>  > > 'next' will say "forced update".
>  > > 
>  > > After that, recreate 'next' on the clones just like in Satish's original
>  > > email.
>  > > 
>