www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - github release procedure

reply Walter Bright <newshound2 digitalmars.com> writes:
As always, when I try to do a release, problems crop up. For example,
the github procedure agreed upon and outlined here:

http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

Issues:

1. you cannot have a tag and a branch with the same name. At least, you cannot 
push them with:

     git push origin 2.N+1

because it's ambiguous. So I prepended a v to the tag name.

2. The:

    git checkout staging
    git merge master

It merges master into staging, wiping out my changes in staging, and does not 
delete staging. Now that the release is done, we're done with staging. What is 
needed is the ability to merge from staging to master all commits in staging 
that occurred after it branched off from master.

I did this by going through the git commit history and cherry-picking one by 
one. There's got to be better way.

3. There is no mention of where and when the:

    git push

and:

    git pull

get done. I also had to add staging to .git/config, can that be done from the 
push & pull command?

4. I think that staging should be deleted after the branch is done?

5. Since essentially the staging branch gets replaced by the 2.061 branch, why 
have a staging branch at all? Just make a 2.061 branch, then tag it when the 
release happens.
Jan 02 2013
next sibling parent reply "Jason House" <jason.james.house gmail.com> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.

It looks like branch should be 2.N while tags are 2.N.M. It looks like the base of branch 2.N will be tagged as 2.N.0. If minor changes are needed, they go into the 2.N branch, and when ready the next tag on that branch will be 2.N.1, then 2.N.2, etc...
 2. The:

    git checkout staging
    git merge master

 It merges master into staging, wiping out my changes in 
 staging, and does not delete staging. Now that the release is 
 done, we're done with staging. What is needed is the ability to 
 merge from staging to master all commits in staging that 
 occurred after it branched off from master.

I believe the intent of staging is to get stable snapshots of master. At a minimum, I believe you'd merge master into staging when you're ready to do a dmdbeta release. I would recommend merging master into staging more frequently, but definitely not as often as master.
Jan 02 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 10:11 AM, Johannes Pfau wrote:
 What I think is weird in the current workflow is that we branch the
 release branch if we actually ship the release (after beta testing,
 etc). Why not just start the release branch if we start to prepare the
 release (before beta testing).

Yeah, that's what I was thinking.
Jan 03 2013
prev sibling next sibling parent "David Nadlinger" <see klickverbot.at> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:
 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.

This is okay – the leading 'v' for version tags is common practice with Git, and if the Git workflow described on the wiki is intended to be official, it should be updated to reflect that (in a hurry right now, so I'll pass). David
Jan 03 2013
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-03 01:58, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D


 Issues:

 1. you cannot have a tag and a branch with the same name. At least, you
 cannot push them with:

      git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.

That is the convention used by git.
 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging, and
 does not delete staging. Now that the release is done, we're done with
 staging. What is needed is the ability to merge from staging to master
 all commits in staging that occurred after it branched off from master.

 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.

Can't you just merge staging into master?
 3. There is no mention of where and when the:

     git push

 and:

     git pull

Reading "Release a new version of D", I would say that after you run "git checkout staging" you need to make sure that you're local changes the the upstream changes are in sync. That can mean that you need to both run "git push" and "git pull". After that the tag needs to be pushed. If you then also merges master into staging that should be pushed as well.
 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?

 4. I think that staging should be deleted after the branch is done?

 5. Since essentially the staging branch gets replaced by the 2.061
 branch, why have a staging branch at all? Just make a 2.061 branch, then
 tag it when the release happens.

-- /Jacob Carlborg
Jan 03 2013
parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 3:59 AM, Jacob Carlborg wrote:
 On 2013-01-03 01:58, Walter Bright wrote:
 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging, and
 does not delete staging. Now that the release is done, we're done with
 staging. What is needed is the ability to merge from staging to master
 all commits in staging that occurred after it branched off from master.

 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.

Can't you just merge staging into master?

And won't that remove the commits that are in master but not in staging?
 3. There is no mention of where and when the:

     git push

 and:

     git pull

Reading "Release a new version of D", I would say that after you run "git checkout staging" you need to make sure that you're local changes the the upstream changes are in sync. That can mean that you need to both run "git push" and "git pull". After that the tag needs to be pushed. If you then also merges master into staging that should be pushed as well.

I know there need to be push's and pull's. The question is the sequence, and the actual commands, like push origin or just push. The trouble is the wiki lists a sequence of commands, but assumes the user realizes that there are a bunch of other commands that need to be interleaved. Imagine it's a script. Steps cannot be skipped :-)
Jan 03 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 02 Jan 2013 16:58:22 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 
 2. The:
 
     git checkout staging
     git merge master
 
 It merges master into staging, wiping out my changes in staging

What changes? All changes should be made in master, then applied to staging via cherry picking. So there should never be changes in staging that are not in master so there's nothing that could be overwritten?
, and does not delete staging. Now that the release is done, we're done
 with staging.

Why would you want to delete staging? This could be done, but I don't see the benefit. Actually staging is quite redundant afaik. Why not just create the release branch directly when we start to prepare a release, then cherry pick from master into that release branch (but never the other way round). Once the release is stable, give it a tag. Then leave the branch alone (or continue the branch by cherry picking further commits for a bugfix 2.061.1 release) What's the benefit of first cherry-picking into staging and the creating the release branch? If we use release branches directly we can avoid the merge from master to staging after every release as we'll just create a new branch for the new release. (Mixing cherry picking and merging isn't exactly a best practice)
 What is needed is the ability to merge from staging to
 master all commits in staging that occurred after it branched off
 from master.

Wait- The wiki page isn't too clear here ("Development takes place on the master branch"), but as far as I understand all changes should go to master first. Even if those are bugfixes which will be used in staging. The those will be cherry picked into staging if necessary. We should never merge staging into master.
 
 I did this by going through the git commit history and cherry-picking
 one by one. There's got to be better way.

We should cherry pick one by one from master into staging. But we should never have to merge anything from staging back into master. (Technically you can probably pick many commits at once, so it's not really one by one)
 
 3. There is no mention of where and when the:
 
     git push
 
 and:
 
     git pull
 
 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?

git fetch origin #update remote branch list git checkout staging #should automatically track origin/staging The wiki page could need some enhancements regarding pushing/pulling.
 
 4. I think that staging should be deleted after the branch is done?

I don't know why staging was proposed. As described above I don't know why it's necessary.
 5. Since essentially the staging branch gets replaced by the 2.061
 branch, why have a staging branch at all? Just make a 2.061 branch,
 then tag it when the release happens.

Ah right, exactly my question. I should have read the whole message first ;-)
Jan 03 2013
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 9:56 AM, Johannes Pfau wrote:
 Am Wed, 02 Jan 2013 16:58:22 -0800
 schrieb Walter Bright <newshound2 digitalmars.com>:

 2. The:

      git checkout staging
      git merge master

 It merges master into staging, wiping out my changes in staging

What changes? All changes should be made in master, then applied to staging via cherry picking.

I think that's a vast assumption that there would never be changes specific to the release.
 , and does not delete staging. Now that the release is done, we're done
 with staging.

Why would you want to delete staging? This could be done, but I don't see the benefit.

Turn that around - what's the benefit of keeping it? It's just clutter.
 What's the benefit of first cherry-picking into staging and the
 creating the release branch?

Beats me. That's why I asked.
 3. There is no mention of where and when the:

      git push

 and:

      git pull

 get done. I also had to add staging to .git/config, can that be done
 from the push & pull command?

git fetch origin #update remote branch list git checkout staging #should automatically track origin/staging The wiki page could need some enhancements regarding pushing/pulling.

As I suggested to Jacob, if the wiki lists git command sequences, it should be complete (like a script), and not full of assumptions about other commands that need to be inserted.
Jan 03 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 1:58 PM, Walter Bright wrote:
 As I suggested to Jacob, if the wiki lists git command sequences, it
 should be complete (like a script), and not full of assumptions about
 other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it? Thanks, Andrei
Jan 03 2013
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 3:17 PM, Rob T wrote:
 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu wrote:
 On 1/3/13 1:58 PM, Walter Bright wrote:
 As I suggested to Jacob, if the wiki lists git command sequences, it
 should be complete (like a script), and not full of assumptions about
 other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it? Thanks, Andrei

I'm rather concerned when I see comments that suggest that the purpose of the staging branch was not understood in the slightest. There's a lot of discussion on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com The people who are attempting to use the new process have to at least take some time to read through the process discussion threads, certainly that is not ideal, and the wiki should explain everything much better.

That is correct. The discussion was the crucible and the wiki page should be the result of what boiled down into it. There continues to be a bit a debate on the exact details of the procedure to be followed, which has naturally led to a scant wiki page. That all is natural and expected - no need for it to cause concern.
 The reason for a less than adequate process description, is that the
 process was not yet fully completed and was difficult to complete
 without a dry run to work out the bugs and oversights etc. Instead of a
 dry run, we're experiencing the real thing, started in the middle of the
 holidays to boot.

 We have to expect some problems of course!

There we go :o).
 It would be nice to contain everything under one discussion thread so
 that the same people who started this whole thing are more likely to
 notice the complaints and misunderstandings and so forth.

It's funny - that was my exact intent when I started the thread "Next focus: PROCESS"! Andrei
Jan 03 2013
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/3/13 6:40 PM, foobar wrote:
 I suggest to read about how git works and how best to utilize its
 features, and NOT apply common wisdom from CVS/SVN/RCS which are based
 on a _completely different_ conceptual model.

Better yet, since you are already so conversant with git, you could help us improve the process directly. Care to make a pass through the wiki page? Andrei
Jan 03 2013
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 11:19 AM, Andrei Alexandrescu wrote:
 I think this is a pertinent point - the process proposed at github is
incomplete
 and scantily motivated. Can the experts make one more pass through it?

One feature I really liked about github was when you created a new repository, it would give you an exact list of corresponding git commands to set up your local version.
Jan 03 2013
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 1/3/2013 12:17 PM, Rob T wrote:
 I'm rather concerned when I see comments that suggest that the purpose of the
 staging branch was not understood in the slightest. There's a lot of discussion
 on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

 The people who are attempting to use the new process have to at least take some
 time to read through the process discussion threads, certainly that is not
 ideal, and the wiki should explain everything much better.

I agree that the pertinent rationales must be in the wiki.
Jan 03 2013
prev sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
05-Jan-2013 00:01, Rob T пишет:
 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:
 The more I think about it, staging really seems to be useful to avoid
 some corner cases.

I haven't had time to study the newly proposed process in enough detail to properly comment, but it always seemed to me that you cannot get away without a staging branch. You can try very hard to do without, but it means something else will suffer for the lack of it, so it seems you are coming to the same conclusion which is good (unless I'm wrong). If we look at how the current 2.061.0 release went down, there was clearly no staging at all, it went straight out just as it did in the past, and we're already seeing complaints about broken code and bug fixes that should have gone into the release that were left out. We're also finding bugs discovered right after the release was made, this is not good and should not happen.

Staging branch was there, so did the betas. What could have helped is announcing beta at D.announce instead of separate list. Plus at holidays there was not much of beta test coverage. -- Dmitry Olshansky
Jan 04 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu 
wrote:
 On 1/3/13 1:58 PM, Walter Bright wrote:
 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it? Thanks, Andrei

I'm rather concerned when I see comments that suggest that the purpose of the staging branch was not understood in the slightest. There's a lot of discussion on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com The people who are attempting to use the new process have to at least take some time to read through the process discussion threads, certainly that is not ideal, and the wiki should explain everything much better. The reason for a less than adequate process description, is that the process was not yet fully completed and was difficult to complete without a dry run to work out the bugs and oversights etc. Instead of a dry run, we're experiencing the real thing, started in the middle of the holidays to boot. We have to expect some problems of course! It would be nice to contain everything under one discussion thread so that the same people who started this whole thing are more likely to notice the complaints and misunderstandings and so forth. --rt
Jan 03 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Thu, Jan 03, 2013 at 09:17:08PM +0100, Rob T wrote:
 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu
 wrote:
On 1/3/13 1:58 PM, Walter Bright wrote:
As I suggested to Jacob, if the wiki lists git command sequences, it
should be complete (like a script), and not full of assumptions
about other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it?


 I'm rather concerned when I see comments that suggest that the
 purpose of the staging branch was not understood in the slightest.
 There's a lot of discussion on the issue here
 http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

In my mind, 'staging' is just a placeholder name for the place where we can freeze new features and stabilize them before a release. The actual mechanics of doing this is independent of the concept itself. We could either: (1) Have an actual branch called 'staging', where changes made to master since the last merge are merged in immediately after release, and when things have sufficiently stabilized, we tag/branch that particular commit as release 2.X.Y; (2) Or, we can follow Walter's suggestion of branching master into 2.X.Y after a release and just tag it as a release when it's sufficiently stable. In this case, the latest 2.X.Y branch acts as 'staging'. Part of the draw of (1) is that betatesters and adventurous people have a fixed place to go to, to get the latest and greatest features, but without the instability of master. I think (2) is more commonly what is actually done by software companies and projects. In this case, 'staging' isn't so much as an actual physical branch, as the general concept of having a branch (2.X.Y, which changes per release) to stabilize things for a release, while development on master continues unhindered. The Linux kernel also uses a similar kind of process: X.Y.Z is a development release (can be thought of as 'master') when Y is odd, and a stable release (can be thought of as 'staging' or Walter's release branches) when Y is even. Basically, the idea is that master should be able to receive the latest and greatest features, completely unhindered, while stabilizing the code for a release is done in a separate branch which doesn't receive new features, but only bugfixes, prior to an actual release.
 The people who are attempting to use the new process have to at
 least take some time to read through the process discussion threads,
 certainly that is not ideal, and the wiki should explain everything
 much better.

I confess to having contributed some poorly-written material on the wiki page. Please feel free to revise it. T -- The computer is only a tool. Unfortunately, so is the user. -- Armaphine, K5
Jan 03 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 22:16:07 UTC, Andrei Alexandrescu 
wrote:
 It's funny - that was my exact intent when I started the thread 
 "Next focus: PROCESS"!


 Andrei

I know that was your intent and we tried hard to stay focused in that thread, and it was working reasonably well, but now we have another thread to deal with that is disconnected from the first thread, resulting in people repeating themselves all over again. I know it was rejected a thousand times before, but solving the disconnection problem was one reason for asking to move the forum into a web based version that could be managed (ie this type of forum cannot be managed at all it seems). The other alternative was to discuss everything in the wiki page, but people found that to be not so good as can be seen by looking at the wiki as well as the comments concerning its use for that purpose. It would be nice to have a better solution for the community to perform cooperative problem solving than what we have now, which also leads on to resolving other problems such as the endless repeating disconnected discussions on other subjects of importance. I don't want to stray off topic, so just keep in mind that there may be a more significant issue that should be resolved immediately after we get past the current one. --rt
Jan 03 2013
prev sibling next sibling parent "foobar" <foo bar.com> writes:
On Thursday, 3 January 2013 at 20:42:39 UTC, Johannes Pfau wrote:
 Am Thu, 03 Jan 2013 21:17:08 +0100
 schrieb "Rob T" <rob ucora.com>:

 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei 
 Alexandrescu wrote:
 On 1/3/13 1:58 PM, Walter Bright wrote:
 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it? Thanks, Andrei

I'm rather concerned when I see comments that suggest that the purpose of the staging branch was not understood in the slightest. There's a lot of discussion on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

Sorry, but I'm not sure if we have a consensus on some things discussed there: This is probably the best summary: http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com?page=16#post-woyboqigqbkqjxmshppn:40forum.dlang.org "4. Feature can than be further refined and _integration bugs_ can be fixed by the general dev team." What's the overhead for many small feature branches? DMD work is probably 90% small bug fixes and 10% bigger features. So most of the time there are no "integration bugs" and we just waste man power reviewing commits again and copying those into staging? "When the "dev" branch is considered stable enough by the team (exact criteria to be defined later), the changes are merged to the _2nd level of integration_ - the "staging" branch. This allows for a wider audience to test and provide real-world feedback." The exact criteria would be kinda important. But as we have many small bugfixes which are merged every day I don't see how the master branch would suddenly get more "stable". What I am most concerned about are the timespans discussed: "I propose to go for a yearly release of the stable branches with one year support (In the beginning)." The wiki discussion page even mentions "I don't think 4 months are a ridiculously long time for staging if the release is made every 3 years."

Let's clarify a bunch of things here: 1. There is no overhead with maintaining many feature brunches. 2. Feature branches can be safely removed after their contents have been merged. This is useful for avoiding clutter. 3. There is no need to re-review the _same_ code on each branch it is merged to. That's just plain stupid. 4. Chery picking is a HORRIBLE idea for managing releases and should only be used on development. Chery-picking COPIES commits and thus loses the connections in the history graph. 5. There is no problem developing bug_fixes on separate topic branches. 6. There is no problem merging back to master. 7. Let me say this again. Merges are cheap and easy in git. So are re-merges. Avoiding merges is an artifact of client-server designs such as SVN and is NOT compatible with Git's conceptual model. I suggest to read about how git works and how best to utilize its features, and NOT apply common wisdom from CVS/SVN/RCS which are based on a _completely different_ conceptual model.
Jan 03 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Thursday, 3 January 2013 at 22:11:59 UTC, H. S. Teoh wrote:
 We could either:

 (1) Have an actual branch called 'staging', where changes made 
 to master
 since the last merge are merged in immediately after release, 
 and when
 things have sufficiently stabilized, we tag/branch that 
 particular
 commit as release 2.X.Y;

 (2) Or, we can follow Walter's suggestion of branching master 
 into 2.X.Y
 after a release and just tag it as a release when it's 
 sufficiently
 stable. In this case, the latest 2.X.Y branch acts as 'staging'.

 Part of the draw of (1) is that betatesters and adventurous 
 people have
 a fixed place to go to, to get the latest and greatest 
 features, but
 without the instability of master.

(1) is what I was advocating precisely so that beta testers could use it shortly after a release and up to the final next release. There are a few benefits to this that may be under appreciated. There's plenty of benefits to having a lot of beta testers, so we want to draw them in and encourage them to use the latest beta version as soon as possible. Real world programmers do not want to make use of something that is too unstable, so they will tend to avoid the master development branch and will prefer the latest stable release instead. The only reason they will try out the latest beta version, is to gain access to a bug fix or new feature that they really need for their real world programming tasks, but they will only try it out if they think that the beta is stable enough and is not going to get horribly broken by anything radical being merged in without proper oversight and planning (that's what master and the related feature branches are for). If we immediately move master into the next staging after a new release, and freeze the introduction of new features, only to concentrate on stabilizing it for the next release, then we're more likely to attract beta testers from the real world, these are the guys who will do testing under the most realistically conditions possible, something that in invaluable in gaining feed back and routing our difficult to discover bugs.
 I think (2) is more commonly what is actually done by software 
 companies
 and projects. In this case, 'staging' isn't so much as an actual
 physical branch, as the general concept of having a branch 
 (2.X.Y, which
 changes per release) to stabilize things for a release, while
 development on master continues unhindered.

 The Linux kernel also uses a similar kind of process: X.Y.Z is a
 development release (can be thought of as 'master') when Y is 
 odd, and a
 stable release (can be thought of as 'staging' or Walter's 
 release
 branches) when Y is even.

 Basically, the idea is that master should be able to receive 
 the latest
 and greatest features, completely unhindered, while stabilizing 
 the code
 for a release is done in a separate branch which doesn't 
 receive new
 features, but only bugfixes, prior to an actual release.

I'm not certain what the major difference is between (1) and (2), because they both sound very nearly identical. In both cases, they seem to provide the same goals of stabilizing the next release, while allowing development to proceed unhindered, so if (2) can be made to achieve the goal I described for attracting real world beta testers, it may not matter which is used, however I would suggest that we always focus on serving the needs of the end user as a top priority as much as is possible. D requires plenty of significant end users working on significant projects. You did mention that (1) is more suitable for beta testers, so I ask how is (2) not as suitable? --rt
Jan 03 2013
prev sibling next sibling parent Russel Winder <russel winder.org.uk> writes:
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Fri, 2013-01-04 at 00:40 +0100, foobar wrote:
[=E2=80=A6]
 Let's clarify a bunch of things here:
 1. There is no overhead with maintaining many feature brunches.

Which is why Git is superior to Mercurial for the sort of process going on here.
 2. Feature branches can be safely removed after their contents=20
 have been merged. This is useful for avoiding clutter.

See above :-)
 3. There is no need to re-review the _same_ code on each branch=20
 it is merged to. That's just plain stupid.

I am not so sure on this. A changeset in a different context is a different changeset, which is why Mercurial is the way it is and why it has some benefits overf Git. The effect of a code change in one branch is not necessarily the same as the effect of the same change in a different branch. However review is not the right tool here, CI is. Each and every branch in the mainline must have a CI listener that runs all the code on every commit. Branches in the mainline are CVCS thinking, not wrong, but must be treated as an integral part of the project, hence the CI requirement. Branches held in forks of the mainline have no such requirement, but people should run all the tests after each commit as a matter of professionalism. Others can of course pull these feature branches into their own fork repository for working on. This is the DVCS way. Eventually of course commit to the mainline is required otherwise a project fork has been made.
 4. Chery picking is a HORRIBLE idea for managing releases and=20
 should only be used on development. Chery-picking COPIES commits=20
 and thus loses the connections in the history graph.

Agreed.
 5. There is no problem developing bug_fixes on separate topic=20
 branches.

See 1 above. Git makes this trivially easy.
 6. There is no problem merging back to master.

See 1 above. Git makes this trivially easy.
 7. Let me say this again. Merges are cheap and easy in git. So=20
 are re-merges. Avoiding merges is an artifact of client-server=20
 designs such as SVN and is NOT compatible with Git's conceptual=20
 model.

It is important to note that Mercurial has different properties again, so leads to different workflow. Given the choice has been Git then Git workflows apply. Every developer should be keeping master as a mirror of the mainline master and working on feature branches. When a changeset is accepted after review, the committer commits to the mainline and then everyone can pull and garbage collect, and in the process delete the feature branch they have. I tend to fork the mainline repository locally and also on GitHub and then manually link my two forks. This means "git pull" locally pulls all branches from the mainline repository. I work on feature branches pushed to my GitHub fork, people can then pull them as they wish. You have to remember to fetch from all the remote branches you are tracking. I can then make a pull request on my GitHub fork and still continue to track the mainline locally.
 I suggest to read about how git works and how best to utilize its=20
 features, and NOT apply common wisdom from CVS/SVN/RCS which are=20
 based on a _completely different_ conceptual model.

Indeed. Star model has a role in DVCS, the mainline repository is the mainline from which all else stems. But not all traffic has to be routed through it with Git. People can share feature branches between themselves to collaborate to create a pull request without ever having to use the mainline. That is the whole point. It means that the mainline committer role is not to do all the development work, but is to coordinate and mentor activity which they then review and when happy merge into mainline. No-one should ever be doing development on master. No-one should ever push to mainline without someone at least knowing they are about to do it and have previously checked the work. No mainline repository should ever be without a full CI. http://d.puremagic.com/test-results/ Looks like part of a solution. It should also provides artefacts for download and install as nightly releases. It assumes there is only master in the repository. If there are any other branches in the mainline repository then where are the CIs for those? In my clone of DMD I see master, staging, dmd-1.x branches. Given DMD is a single branch development, I don't see a need for staging. I shall have to go through all the email to find the rationale, unless someone has a short paragraph rationale. What I find really bizarre is that the tags v2.061 and v1.076 are on branches other than master. To me the whole Git history looks a mess, essentially broken. I suggest some serious rebasing and merging to make the history consistent with the version numbering. The questions to ask: If I want to create a feature branch, which branch do I use as a base? Is staging just the 2.x maintenance branch? When do changes on master make it to staging? Should staging and dmd-1.x branch HEADS be moved forwards on the branch? Sorry if this is all answered somewhere, just point me at the URL where the answers are explained. =20 --=20 Russel. =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D Dr Russel Winder t: +44 20 7585 2200 voip: sip:russel.winder ekiga.n= et 41 Buckmaster Road m: +44 7770 465 077 xmpp: russel winder.org.uk London SW11 1EN, UK w: www.russel.org.uk skype: russel_winder
Jan 04 2013
prev sibling next sibling parent "mist" <none none.none> writes:
Weird. While I was reading "Next focus: PROCESS" topic and 
http://wiki.dlang.org/Proposed_new_D_development_process entries, 
I was completely convinced that switch to faster release model is 
proposed, so that master replaces staging the very moment one 
version is released (and it is the same moment as testing of next 
release begins). Plus once a year or so release is marked LTS and 
cherry-pick bug-fixes until next LTS is out. Was my understanding 
wrong?

Anyway, now I am reading all those comments and it feels absurdly 
like no one has discussed this stuff few weeks ago and different 
topic and argument has started from beginning. This is madness.
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 3 January 2013 at 18:58:51 UTC, Walter Bright wrote:
 Turn that around - what's the benefit of keeping it? It's just 
 clutter.

It has been discussed here before, and many people agreed that stagging can go away, as changes made in the proposal made it quite redundant. That sounds the reasonable thing to do.
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 03:00:10PM +0100, Johannes Pfau wrote:
 Am Fri, 04 Jan 2013 07:59:05 +0100
 schrieb "Rob T" <rob ucora.com>:
 
 I'm not certain what the major difference is between (1) and (2), 
 because they both sound very nearly identical.


That's because they are essentially two ways of implementing the same concept.
 In both cases, they seem to provide the same goals of stabilizing
 the next release, while allowing development to proceed unhindered,
 so if (2) can be made to achieve the goal I described for attracting
 real world beta testers, it may not matter which is used, however I
 would suggest that we always focus on serving the needs of the end
 user as a top priority as much as is possible. D requires plenty of
 significant end users working on significant projects.
 
 You did mention that (1) is more suitable for beta testers, so I ask
 how is (2) not as suitable?


I don't think the difference is that serious. (1) is just more _convenient_ for beta testers, because there's an actual branch called 'staging', so they just have to grab that. (2) requires that they know which version branch is the latest, which again, isn't that big of a deal. [...]
 http://wiki.dlang.org/User:Jpf/Release_Process
 
 I updated the page with some more git commands, ascii art and color so
 it's hopefully easier to understand now. Please tell me if I got
 anything wrong. The only thing I (intended to change) changed is that I
 removed staging.

I like what you did with the page. I think it makes it a lot clearer. I read through "Release Schedule" and "Branching model", and I think it pretty much captures what I described as approach (2).
 Although I can see the benefits of staging as you describe it, we can
 get most of those by just creating a new release/version branch as
 soon as the current release is released. This removes some complexity
 from the workflow.

If this approach is more suitable for the core devs, then I think we should go with it.
 AFAIK it should be easy enough to introduce staging later on, when
 everyone is comfortable with the new workflow. (Please consider that
 many of us are not git experts - so if we can avoid some complexity
 now and add it later if needed, I'd prefer that)

Whether or not one is a git expert ought not matter, if we define the process sufficiently clearly that one can simply look up the wiki page and type in the commands as-is, like a script, as Walter puts it. The important point is to agree on a single implementation of the process, and supply complete and unambiguous series of git commands to carry out the process. T -- The fact that anyone still uses AOL shows that even the presence of options doesn't stop some people from picking the pessimal one. - Mike Ellis
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 04:14:00PM +0100, Johannes Pfau wrote:
[...]
 I just thought about something - maybe staging isn't 100% redundant.
 I assume my conclusions on
 http://wiki.dlang.org/User:Jpf/Release_Process are correct:
 
 If we want release stabilization time == time between 2 releases i.e.
 we start a new release/version branch after every major release: 
 
 Let's say 2.062 is in stabilization phase and we use the 2.062 branch.
 Then bugfixes are based on / merged into 2.062 branch. So someone
 opens a pull request targeting the 2.062 branch with a bugfix. For
 some reason (lack of time) this request isn't merged till 2.062 is
 released.  After the release only regression fixes should go into
 2.062.  But we now have a bugfix (not regression) pull request
 targeting the 2.062 branch!
 
 Staging completely avoids that issue. So is this reason enough to have
 staging?

I'm not sure, but in my experience at my day jobs, any bugfix that *can* be applied to master (i.e., the code hasn't diverged so much that the fix doesn't apply anymore), *should* be. For this reason, it's best to always base bugfixes on master, unless the code has diverged so much the fix only applies to a particular release branch. Otherwise you end up with the problem somebody mentioned, that the bug is fixed in a previous release, but is "unfixed" in a later release because we neglected to merge the fix into master. Of course, it's possible to base the fix on the release branch and merge it back to master, but the latter step is more prone to be forgotten. But then again, post-release regression fixes should probably be based on the release branch, so that we know that the fix works, as master may have diverged significantly since the original release branch was made. So either way, we have to make sure fixes are applied back to master. T -- May you live all the days of your life. -- Jonathan Swift
Jan 04 2013
prev sibling next sibling parent reply "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:
 The more I think about it, staging really seems to be useful to 
 avoid
 some corner cases.

I haven't had time to study the newly proposed process in enough detail to properly comment, but it always seemed to me that you cannot get away without a staging branch. You can try very hard to do without, but it means something else will suffer for the lack of it, so it seems you are coming to the same conclusion which is good (unless I'm wrong). If we look at how the current 2.061.0 release went down, there was clearly no staging at all, it went straight out just as it did in the past, and we're already seeing complaints about broken code and bug fixes that should have gone into the release that were left out. We're also finding bugs discovered right after the release was made, this is not good and should not happen. The whole point of having a process has been defeated in terms of the goal of creating highly stable releases. What has been released is in effect a beta release or the same thing as a staging release, it is *not* a stable release! If it was a stable release, we would not see so many immediate problems. I realize this was our fist pass through with an incomplete process, and almost no buy in from the devs who were to use it, so with that said we have clearly made some significant improvements, and I do not want to take away from that, however I hope we can all refocus on the goal of releasing only high quality software, not half baked betas. Thanks! --rt
Jan 04 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-01-04 21:11, Jonathan M Davis wrote:

 That's what the beta is for, which we did. Clearly, not enough people tried it
 out, or those regressions would have been caught. Creating a separate branch
 would not fix that.

It fixes other problems, like merging a pull request that should have been merge just because someone has missed a beta was out. -- /Jacob Carlborg
Jan 05 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-01-05 12:26, Jacob Carlborg wrote:
 On 2013-01-04 21:11, Jonathan M Davis wrote:

 That's what the beta is for, which we did. Clearly, not enough people
 tried it
 out, or those regressions would have been caught. Creating a separate
 branch
 would not fix that.

It fixes other problems, like merging a pull request that should have been merge just because someone has missed a beta was out.

should _not_. -- /Jacob Carlborg
Jan 05 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 20:41:47 UTC, Dmitry Olshansky wrote:
 Staging branch was there, so did the betas.
 What could have helped is announcing beta at D.announce instead 
 of separate list. Plus at holidays there was not much of beta 
 test coverage.

Yes, being the first pass through with an incomplete and poorly understood process didn't help, along with the timing, so the result is what the process produced in its current form. We're definitely making progress however, and hopefully the criticisms are taken constructively as they should be. We just want to see things improve, that's all. One improvement for next pass through, is to release installable snapshots of the next beta directly on the download page (with appropriate labeling) so that it is trivially easy for end users to download and install the latest beta for testing. Also, as I mentioned in my previous post, we ought to define what "stable" means so that we have a much better way to judge if the current beta is ready for release or not. As it sits, we have no definition, so it's not surprising that some people thought the current version was ready, while others are not so sure of that. --rt
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 05, 2013 at 12:02:39AM +0100, Rob T wrote:
[...]
 We're definitely making progress however, and hopefully the
 criticisms are taken constructively as they should be. We just want
 to see things improve, that's all.
 
 One improvement for next pass through, is to release installable
 snapshots of the next beta directly on the download page (with
 appropriate labeling) so that it is trivially easy for end users to
 download and install the latest beta for testing.

+1. More beta test coverage can only be a good thing. T -- Life is complex. It consists of real and imaginary parts. -- YHL
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Sat, Jan 05, 2013 at 11:42:23AM +0100, Johannes Pfau wrote:
 Am Fri, 04 Jan 2013 21:01:41 +0100
 schrieb "Rob T" <rob ucora.com>:
 
 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:
 The more I think about it, staging really seems to be useful to
 avoid some corner cases.

I haven't had time to study the newly proposed process in enough detail to properly comment, but it always seemed to me that you cannot get away without a staging branch. You can try very hard to do without, but it means something else will suffer for the lack of it, so it seems you are coming to the same conclusion which is good (unless I'm wrong).

Yep. As we have lost most attention in this thread(most people have probably set "ignore this thread") we should check if we all agree on the process as outlined on the updated wiki page (I've updated the original page now) and improve it, if necessary, do some proof reading / spell checking and then open a new thread, asking for comments from the main devs.

I like the new page!! One question, though: in the git workflow section, it says that a non-regression bugfix should go directly into staging, but there's no mention of merging it back to master. Is this deliberate? It seems to me to be kinda important that we merge all fixes back to master so that bugs don't get "unfixed". Or am I missing something here? T -- Why waste time learning, when ignorance is instantaneous? -- Hobbes, from Calvin & Hobbes
Jan 05 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
One quick note, is that we may want to change the title and 
description, since I think we're taking about a "Development and 
Release Process" rather than only a "Release Process".

The new wiki, and even the process page should be linked in on 
the main page to make it easy for the devs to get access to it.

--rt
Jan 05 2013
prev sibling next sibling parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Thursday, 3 January 2013 at 17:56:05 UTC, Johannes Pfau wrote:
 What changes? All changes should be made in master, then 
 applied to
 staging via cherry picking. So there should never be changes in 
 staging
 that are not in master so there's nothing that could be 
 overwritten?

I strongly disagree here. We want pull requests to be made against the oldest supported/relevant branch. This could be merged to both staging and master (which can simply be done by merging staging into master) there is no need to complicate things with cherry picking. As for history, I think that clearly shows what the history is, staging gets fixes and master gets staging that is what we want.
Jan 09 2013
prev sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Wed, Jan 09, 2013 at 09:40:59PM +0100, Johannes Pfau wrote:
[...]
 I've updated the http://wiki.dlang.org/Development_and_Release_Process
 page last week (I even added a "do not cherry pick from one official
 branch into another" rule), so you won't have to convince me ;-)
 
 Do you think the wiki page is good / correct? I've verified that all
 git commands work, so if everyone agrees with the workflow (should be
 the same as on the old page) we can probably start a new thread and
 ask for comments from the core devs.

The earlier we can get input from the core devs, the better. Otherwise we're just building castles in the air. T -- It only takes one twig to burn down a forest.
Jan 09 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 03:51:48 +0100
schrieb "Jason House" <jason.james.house gmail.com>:

 On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.

It looks like branch should be 2.N while tags are 2.N.M. It looks like the base of branch 2.N will be tagged as 2.N.0. If minor changes are needed, they go into the 2.N branch, and when ready the next tag on that branch will be 2.N.1, then 2.N.2, etc...

This would mean changing the dmd versioning. We currently don't increase the second number. To match our versioning, branch should be 2.0.N and tag should be 2.0.N.M. But I like Walter's proposal more, as 2.0.61 should be the same as 2.0.60.0 anyway so the .0 is redundant.
 I believe the intent of staging is to get stable snapshots of 
 master. At a minimum, I believe you'd merge master into staging 
 when you're ready to do a dmdbeta release. I would recommend 
 merging master into staging more frequently, but definitely not 
 as often as master.

This would be an explanation, but we'd have to define what "stable snapshot" means. If it's just "staging must always compile" that's ok. But everything more advanced will need someone to review every single commit. And when we actually want to prepare a release we need all commits / new features anyway? Imho staging isn't really necessary. We need a branch for every "big" release, 2.0.60, 2.0.61 and so on. Then we add tags, v2.0.60 v2.0.61 and if we wan't a stable dmd, we use one of the release branches (e.g. 2.0.61) cherry pick bugfixes from master and the add a v2.0.61.1 tag and release that. What I think is weird in the current workflow is that we branch the release branch if we actually ship the release (after beta testing, etc). Why not just start the release branch if we start to prepare the release (before beta testing).
Jan 03 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 10:54:25 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 1/3/2013 3:59 AM, Jacob Carlborg wrote:
 On 2013-01-03 01:58, Walter Bright wrote:
 2. The:

     git checkout staging
     git merge master

 It merges master into staging, wiping out my changes in staging,
 and does not delete staging. Now that the release is done, we're
 done with staging. What is needed is the ability to merge from
 staging to master all commits in staging that occurred after it
 branched off from master.

 I did this by going through the git commit history and
 cherry-picking one by one. There's got to be better way.

Can't you just merge staging into master?

And won't that remove the commits that are in master but not in staging?

No, merging will never remove commits. You aren't really supposed to delete commits in git as it can lead to all kinds of trouble and therefore deleting has to be done explicitly. But I still advise not to merge staging into master. As all commits should go to master first there should be no need to merge back. If we have commits in staging now which are not in master you can probably merge them into master. It won't look nice in the git history but I shouldn't be a real problem.
 
 I know there need to be push's and pull's. The question is the
 sequence, and the actual commands, like push origin or just push.

The 'remote' to which you push the changes actually depends on your setup. 'origin' is called a 'remote' and it's the repository where you've git cloned from. So in a simple setup, it's always push origin. Git can also 'track' branches. In that case your local branch tracks a 'remote' + branch. In that case you can use git push and it'll push to the tracked remote branch. If you didn't create the staging branch manually but just used git checkout staging it should be tracking origin automatically. Anyway, you can just try git push, as long as you don't have other remotes than origin, nothing can go wrong. (you can use git remote to show all remotes)
 
 The trouble is the wiki lists a sequence of commands, but assumes the
 user realizes that there are a bunch of other commands that need to
 be interleaved.
 
 Imagine it's a script. Steps cannot be skipped :-)

I'll update the wiki if nobody outpaces me. But I'd like to have some feedback/conclusion on the staging issue first.
Jan 03 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 10:58:46 -0800
schrieb Walter Bright <newshound2 digitalmars.com>:


 What changes? All changes should be made in master, then applied to
 staging via cherry picking.

I think that's a vast assumption that there would never be changes specific to the release.

If there's a change which is 100% specific to a release it can be pushed to staging directly, but then there's no need to merge it back into master. If the change makes sense in master, push it to master then cherry pick into staging.
Jan 03 2013
prev sibling next sibling parent "alex" <info alexanderbothe.com> writes:
Couldn't we just make an auto-publishing cron-job service or so 
which periodically (once a day or even just after a push/merge) 
compiles dmd and everything, packs it and makes this archive 
downloadable then?

Of course this literally rolling release should be marked as 
unstable/nightly build. Anyway, if a halfway stable state has 
been reached or if the community thinks that it's enough for a 
version bump,
the latest build is simply tagged and marked as latest stable 
version.
How easy is that?
Jan 03 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Thu, 03 Jan 2013 21:17:08 +0100
schrieb "Rob T" <rob ucora.com>:

 On Thursday, 3 January 2013 at 19:19:49 UTC, Andrei Alexandrescu 
 wrote:
 On 1/3/13 1:58 PM, Walter Bright wrote:
 As I suggested to Jacob, if the wiki lists git command 
 sequences, it
 should be complete (like a script), and not full of 
 assumptions about
 other commands that need to be inserted.

I think this is a pertinent point - the process proposed at github is incomplete and scantily motivated. Can the experts make one more pass through it? Thanks, Andrei

I'm rather concerned when I see comments that suggest that the purpose of the staging branch was not understood in the slightest. There's a lot of discussion on the issue here http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com

Sorry, but I'm not sure if we have a consensus on some things discussed there: This is probably the best summary: http://forum.dlang.org/thread/ka5rv5$2k60$1 digitalmars.com?page=16#post-woyboqigqbkqjxmshppn:40forum.dlang.org "4. Feature can than be further refined and _integration bugs_ can be fixed by the general dev team." What's the overhead for many small feature branches? DMD work is probably 90% small bug fixes and 10% bigger features. So most of the time there are no "integration bugs" and we just waste man power reviewing commits again and copying those into staging? "When the "dev" branch is considered stable enough by the team (exact criteria to be defined later), the changes are merged to the _2nd level of integration_ - the "staging" branch. This allows for a wider audience to test and provide real-world feedback." The exact criteria would be kinda important. But as we have many small bugfixes which are merged every day I don't see how the master branch would suddenly get more "stable". What I am most concerned about are the timespans discussed: "I propose to go for a yearly release of the stable branches with one year support (In the beginning)." The wiki discussion page even mentions "I don't think 4 months are a ridiculously long time for staging if the release is made every 3 years."
Jan 03 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months are a
 ridiculously long time for staging if the release is made every 3
 years."

That makes it sound like they want the current stuff to be marked as staging and then have some other release that sits around for a very long time being treated as somehow more stable than the staging branch. In general, there's nothing about bug fixing which is stable, and separating out bug fixes between a more stable branch and a less stable one doesn't make all that much sense. Separating out new features and not putting them in the "stable" branch makes sense, but that really doesn't make sense for bugs. Also, the name "staging" implies that it's purely for preparing a release, in which case keeping it around makes _no_ sense. Not to mention, as already mentioned, it would make more sense to simply create a new branch for each release to begin with than have a staging branch if that's what it's for. And if that's not what it's for, then it's a horrible name. - Jonathan M Davis
Jan 03 2013
prev sibling next sibling parent "foobar" <foo bar.com> writes:
On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis 
wrote:
 On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months 
 are a
 ridiculously long time for staging if the release is made 
 every 3
 years."

That makes it sound like they want the current stuff to be marked as staging and then have some other release that sits around for a very long time being treated as somehow more stable than the staging branch. In general, there's nothing about bug fixing which is stable, and separating out bug fixes between a more stable branch and a less stable one doesn't make all that much sense. Separating out new features and not putting them in the "stable" branch makes sense, but that really doesn't make sense for bugs. Also, the name "staging" implies that it's purely for preparing a release, in which case keeping it around makes _no_ sense. Not to mention, as already mentioned, it would make more sense to simply create a new branch for each release to begin with than have a staging branch if that's what it's for. And if that's not what it's for, then it's a horrible name. - Jonathan M Davis

As I explained several times before, this is a HORRIBLE idea. If each release has its own branch than a bug fix needs to be applied to _all_ such branches _manually_ and it is easy to get a situation where a fix on say version 3.1 was accidentally not merged to version 4.0 (forgotten) and suddenly the bug was "unfixed". Whatever the name of this branch is, the semantics/purpose of it is to provide integration.
Jan 03 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 00:47:08 foobar wrote:
 On Thursday, 3 January 2013 at 22:04:28 UTC, Jonathan M Davis
 
 wrote:
 On Thursday, January 03, 2013 21:42:37 Johannes Pfau wrote:
 What I am most concerned about are the timespans discussed:
 "I propose to go for a yearly release of the
 stable branches with one year support (In the beginning)."
 
 The wiki discussion page even mentions "I don't think 4 months
 are a
 ridiculously long time for staging if the release is made
 every 3
 years."

That makes it sound like they want the current stuff to be marked as staging and then have some other release that sits around for a very long time being treated as somehow more stable than the staging branch. In general, there's nothing about bug fixing which is stable, and separating out bug fixes between a more stable branch and a less stable one doesn't make all that much sense. Separating out new features and not putting them in the "stable" branch makes sense, but that really doesn't make sense for bugs. Also, the name "staging" implies that it's purely for preparing a release, in which case keeping it around makes _no_ sense. Not to mention, as already mentioned, it would make more sense to simply create a new branch for each release to begin with than have a staging branch if that's what it's for. And if that's not what it's for, then it's a horrible name. - Jonathan M Davis

As I explained several times before, this is a HORRIBLE idea. If each release has its own branch than a bug fix needs to be applied to _all_ such branches _manually_ and it is easy to get a situation where a fix on say version 3.1 was accidentally not merged to version 4.0 (forgotten) and suddenly the bug was "unfixed". Whatever the name of this branch is, the semantics/purpose of it is to provide integration.

??? Bug fixes go to master. New release branches are branched from master. So, you're not going to have future branches not getting previous bug fixes. The only way that you're going to get problems with bug fixes not making it into future releases is if you commit them to the branch first rather than directly to master (which is a bad idea IMHO) and then you forget to merge that back into master. - Jonathan M Davis
Jan 03 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 00:40:03 +0100
schrieb "foobar" <foo bar.com>:
 
 Let's clarify a bunch of things here:
 1. There is no overhead with maintaining many feature brunches.

of merging those first into master. Then later determining if master is stable (how?) and then merging into staging. The wiki says this is useful to allow fixing integration bugs. I think with mostly bugfixes there won't be many integration bugs. And merging into staging always requires someone to do it, which is imho overhead (regarding "human resources", not technical).
 2. Feature branches can be safely removed after their contents 
 have been merged. This is useful for avoiding clutter.

 3. There is no need to re-review the _same_ code on each branch 
 it is merged to. That's just plain stupid.

considered sufficiently stable. Who determines what is stable and how?
 4. Chery picking is a HORRIBLE idea for managing releases and 
 should only be used on development. Chery-picking COPIES commits 
 and thus loses the connections in the history graph.

Sure, this is why I said mixing cherry picking and merging is not a good idea. As long as you only cherry pick into a branch and never merge into that branch or merge it back there is no issue. I missed the detail that you suppose basing bugfix branches on the oldest applicable release branch. That should be made very clear in the wiki. I just think that this process is counter-productive for dmd as most changes are bugfixes and regressions are caused by bug fixes as well. Applying _every_ bugfix in older releases has a high risk of reintroducing regressions (and / or breaking user code. If the compiler didn't enforce const in some situations, fixing it is a bug fix. But fixing it will break user code - so you have a 2.061 that works for users and a 2.061.1 which breaks code). It makes more sense to me to only fix regressions that were introduced with the base release. So 2.061.1 would only fix regressions introduced in 2.061. (This can still be done by basing that fix on the latest applicable version and merging back into master / staging). Also the lowest applicable version for most dmd bugs is probably 2.01. So if we start this process now, we'll still be fixing bugs in 2.061 in 10 years, then merging into 2.062, 2.063.... Here "lowest _supported_ applicable version" makes more sense imho, where supported needs to be defined (e.g. 2 releases). This is not a big problem if we only talk about regressions, but I'd still prefer "lowest _supported_ applicable version"
 5. There is no problem developing bug_fixes on separate topic 
 branches.

 6. There is no problem merging back to master.

we have already cherry picked commits from master into staging. Then merging staging back into master is not good - It will probably work but as you said cherry picking duplicates commits and we'll therefore merge identical changes with different ids. We should probably delete staging and start again as soon as the workflow is clear to everyone.
 7. Let me say this again. Merges are cheap and easy in git. So 
 are re-merges. Avoiding merges is an artifact of client-server 
 designs such as SVN and is NOT compatible with Git's conceptual 
 model.
 
 I suggest to read about how git works and how best to utilize its 
 features, and NOT apply common wisdom from CVS/SVN/RCS which are 
 based on a _completely different_ conceptual model.

I don't have common wisdom from cvs/svn ;-) Merges are not that easy if you want to merge only a single commit or skip certain commits. Although you can tell merge to skip commits it's in no way easy. http://stackoverflow.com/a/729723/471401 It's probably best to not get into this situation. The workflow described on the wiki can indeed work well without cherry picking. But what was done till 2.061 was basing bug fixes on the master branch. If you want to get these into a release branch without features, etc cherry-picking is way easier than merging.
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob ucora.com>:

 
 You did mention that (1) is more suitable for beta testers, so I 
 ask how is (2) not as suitable?

With (1) you can always use staging, it'll always be the latest beta. With (2) you'll have to figure out the most recent branch (2.061, 2.062, 2.063). But I think we can start with (2) and change to (1) later if we think it's useful. Staging complicates the concept(on a cognitive level) and it needs additional "human resources" to do the merges into it. I think I understand the whole concept now. I seems to be a good solution but it requires some discipline (basing changes on the correct branches) so we should document it as clearly as possible. I'd strongly recommend to only base regression fixes on release branches and usual bug fixes on master (see my other message).
Jan 04 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-01-03 01:58, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For example,
 the github procedure agreed upon and outlined here:

Don't know if this is the exact model we want to follow but here are some commands: https://github.com/nvie/gitflow These are based on "A successful Git branching model" that has been talked about before: http://nvie.com/posts/a-successful-git-branching-model/ -- /Jacob Carlborg
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 07:59:05 +0100
schrieb "Rob T" <rob ucora.com>:

 
 I'm not certain what the major difference is between (1) and (2), 
 because they both sound very nearly identical. In both cases, 
 they seem to provide the same goals of stabilizing the next 
 release, while allowing development to proceed unhindered, so if 
 (2) can be made to achieve the goal I described for attracting 
 real world beta testers, it may not matter which is used, however 
 I would suggest that we always focus on serving the needs of the 
 end user as a top priority as much as is possible. D requires 
 plenty of significant end users working on significant projects.
 
 You did mention that (1) is more suitable for beta testers, so I 
 ask how is (2) not as suitable?
 
 --rt

http://wiki.dlang.org/User:Jpf/Release_Process I updated the page with some more git commands, ascii art and color so it's hopefully easier to understand now. Please tell me if I got anything wrong. The only thing I (intended to change) changed is that I removed staging. Although I can see the benefits of staging as you describe it, we can get most of those by just creating a new release/version branch as soon as the current release is released. This removes some complexity from the workflow. AFAIK it should be easy enough to introduce staging later on, when everyone is comfortable with the new workflow. (Please consider that many of us are not git experts - so if we can avoid some complexity now and add it later if needed, I'd prefer that)
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 12:02:44 +0100
schrieb "mist" <none none.none>:

 Weird. While I was reading "Next focus: PROCESS" topic and 
 http://wiki.dlang.org/Proposed_new_D_development_process entries, 
 I was completely convinced that switch to faster release model is 
 proposed, so that master replaces staging the very moment one 
 version is released (and it is the same moment as testing of next 
 release begins). Plus once a year or so release is marked LTS and 
 cherry-pick bug-fixes until next LTS is out. Was my understanding 
 wrong?
 
 Anyway, now I am reading all those comments and it feels absurdly 
 like no one has discussed this stuff few weeks ago and different 
 topic and argument has started from beginning. This is madness.

So someone who was part of that discussion: Please give us a short summary. You can't expect everyone to read through all those posts again, 95% is probably obsolete. And most important: Were the main developers part of that discussion / have they agreed? While I agree LTS would be nice it needs someone to do it. You also have to consider the available man power. But this is not very relevant for this discussion: It's always possible to increase support time later.
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For 
 example,
 the github procedure agreed upon and outlined here:

 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_version_of_D

 Issues:

 1. you cannot have a tag and a branch with the same name. At 
 least, you cannot push them with:

     git push origin 2.N+1

 because it's ambiguous. So I prepended a v to the tag name.

Let's get a very practical case here. 2.061 have been released, and Kenji already fixed a bug in it (unreachable statement issue). Some people (including me, but I'm not the only one) would be interested by a new revision of dmd with that fix. It is why the proposal include branches as 2.N and revision as 2.N.M . So the same version of D, with bugfixes can be published. The branch has a 2.N form, the tag has a 2.N.M form.
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 15:55:01 +0100
schrieb "deadalnix" <deadalnix gmail.com>:

 On Thursday, 3 January 2013 at 18:58:51 UTC, Walter Bright wrote:
 Turn that around - what's the benefit of keeping it? It's just 
 clutter.

It has been discussed here before, and many people agreed that stagging can go away, as changes made in the proposal made it quite redundant. That sounds the reasonable thing to do.

I just thought about something - maybe staging isn't 100% redundant. I assume my conclusions on http://wiki.dlang.org/User:Jpf/Release_Process are correct: If we want release stabilization time == time between 2 releases i.e. we start a new release/version branch after every major release: Let's say 2.062 is in stabilization phase and we use the 2.062 branch. Then bugfixes are based on / merged into 2.062 branch. So someone opens a pull request targeting the 2.062 branch with a bugfix. For some reason (lack of time) this request isn't merged till 2.062 is released. After the release only regression fixes should go into 2.062. But we now have a bugfix (not regression) pull request targeting the 2.062 branch! Staging completely avoids that issue. So is this reason enough to have staging?
Jan 04 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:
 And most important: Were the main developers part of that discussion /
 have they agreed? While I agree LTS would be nice it needs someone to
 do it. You also have to consider the available man power.

For the most part, I don't think that they were. I think that Andrei was really the only one of us involved at all, though I'd have to look at the thread again to be sure. - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 15:26:20 deadalnix wrote:
 On Thursday, 3 January 2013 at 00:59:04 UTC, Walter Bright wrote:
 As always, when I try to do a release, problems crop up. For
 example,
 the github procedure agreed upon and outlined here:
 
 http://wiki.dlang.org/Proposed_new_D_development_process#Release_a_new_ver
 sion_of_D
 
 Issues:
 
 1. you cannot have a tag and a branch with the same name. At
 
 least, you cannot push them with:
     git push origin 2.N+1
 
 because it's ambiguous. So I prepended a v to the tag name.

Let's get a very practical case here. 2.061 have been released, and Kenji already fixed a bug in it (unreachable statement issue). Some people (including me, but I'm not the only one) would be interested by a new revision of dmd with that fix. It is why the proposal include branches as 2.N and revision as 2.N.M . So the same version of D, with bugfixes can be published. The branch has a 2.N form, the tag has a 2.N.M form.

And why would that particular bug get a release as opposed to another? Why wouldit be special? Or are you proposing that we do a 2.N.M release for every bug fix? That seems insane. Why wouldn't you just use master if you want the latest? - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 08:15:34AM -0800, Jonathan M Davis wrote:
 On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:
 And most important: Were the main developers part of that discussion
 / have they agreed? While I agree LTS would be nice it needs someone
 to do it. You also have to consider the available man power.

For the most part, I don't think that they were. I think that Andrei was really the only one of us involved at all, though I'd have to look at the thread again to be sure.

We solicited for core dev involvement in that thread, but AFAIK, there was only limited participation. It felt like we were just talking about castles in the air, because the people who would actually be implementing the process didn't say very much at all. So I wasn't at all surprised at the misunderstandings in this thread. Still, I think we're finally converging on the same page here, so that's a good thing. T -- Amateurs built the Ark; professionals built the Titanic.
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 08:18:52AM -0800, Jonathan M Davis wrote:
 On Friday, January 04, 2013 15:26:20 deadalnix wrote:

 Let's get a very practical case here. 2.061 have been released, and
 Kenji already fixed a bug in it (unreachable statement issue).
 
 Some people (including me, but I'm not the only one) would be
 interested by a new revision of dmd with that fix.
 
 It is why the proposal include branches as 2.N and revision as
 2.N.M . So the same version of D, with bugfixes can be published.
 
 The branch has a 2.N form, the tag has a 2.N.M form.

And why would that particular bug get a release as opposed to another?

It doesn't. It just gets merged in to the 2.N branch post-release, and once enough fixes have accumulated, we make a 2.N.M release.
 Why wouldit be special? Or are you proposing that we do a 2.N.M
 release for every bug fix? That seems insane.

Some bugs are important enough to push out to the release branch, e.g. regressions or blocker bugs. Nobody is suggesting we make a new release for *every* bugfix pushed to the release branch -- that *would* be insane. The release manager, whoever that person is, makes the judgment call on when a release branch has enough bugfixes to warrant another 2.N.M release. That can be just a single bug, if it's a critical blocker bug, or it can be 20 bugs, if none of them are too important, but, taken together, constitute a significant improvement over the original release.
 Why wouldn't you just use master if you want the latest?

Because master potentially has breaking changes that one may not want to deal with. E.g. property enforcement, or breaking syntax changes, etc.. Enterprises who adopt D will want to stay with a stable release that won't randomly break code all over the place just because a particular bugfix was needed. That's why it's important to push regressions and blockers to release branches, without the new features in master. T -- Doubt is a self-fulfilling prophecy.
Jan 04 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 08:23:58 H. S. Teoh wrote:
 On Fri, Jan 04, 2013 at 08:15:34AM -0800, Jonathan M Davis wrote:
 On Friday, January 04, 2013 15:20:02 Johannes Pfau wrote:
 And most important: Were the main developers part of that discussion
 / have they agreed? While I agree LTS would be nice it needs someone
 to do it. You also have to consider the available man power.

For the most part, I don't think that they were. I think that Andrei was really the only one of us involved at all, though I'd have to look at the thread again to be sure.

[...] We solicited for core dev involvement in that thread, but AFAIK, there was only limited participation. It felt like we were just talking about castles in the air, because the people who would actually be implementing the process didn't say very much at all. So I wasn't at all surprised at the misunderstandings in this thread. Still, I think we're finally converging on the same page here, so that's a good thing.

The thread was way too big, and I had too little time to get into that conversation, which is why I wasn't involved. - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 07:46:55 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:

 
 I like what you did with the page. I think it makes it a lot clearer.
 I read through "Release Schedule" and "Branching model", and I think
 it pretty much captures what I described as approach (2).
 

similar, except that I removed staging.
 If this approach is more suitable for the core devs, then I think we
 should go with it.

I also created a version with staging: http://wiki.dlang.org/User:Jpf/Release_Process_Staging It occurred to me that there is one detil where staging is really useful: If we have bugfix pull requests which target a version branch which is stabilizing (e.g 2.062) that's fine. But if those aren't merged when 2.062 is released, all open pull requests will still target 2.062 but they should target 2.063 instead. So we have to cycle through all pull requests and ask the contributors to retarget 2.063. With staging bug fixes can just always target staging and this is not an issue. I think in the end the core devs should choose. But it seems like an extra staging is more difficult to understand but can avoid some additional work in long term.
 Whether or not one is a git expert ought not matter, if we define the
 process sufficiently clearly that one can simply look up the wiki page
 and type in the commands as-is, like a script, as Walter puts it. The
 important point is to agree on a single implementation of the process,
 and supply complete and unambiguous series of git commands to carry
 out the process.

guarantee that it's complete.
Jan 04 2013
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 08:20:00 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:

 
 I'm not sure, but in my experience at my day jobs, any bugfix that
 *can* be applied to master (i.e., the code hasn't diverged so much
 that the fix doesn't apply anymore), *should* be. For this reason,
 it's best to always base bugfixes on master, unless the code has
 diverged so much the fix only applies to a particular release branch.
 Otherwise you end up with the problem somebody mentioned, that the
 bug is fixed in a previous release, but is "unfixed" in a later
 release because we neglected to merge the fix into master.

But if you base bug fixes on master you have to merge master into the release branch to get the bug fixes. But master may contain newly added features which you don't want in the release branch. The more I think about it, staging really seems to be useful to avoid some corner cases.
Jan 04 2013
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 21:01:41 Rob T wrote:
 If we look at how the current 2.061.0 release went down, there
 was clearly no staging at all, it went straight out just as it
 did in the past, and we're already seeing complaints about broken
 code and bug fixes that should have gone into the release that
 were left out. We're also finding bugs discovered right after the
 release was made, this is not good and should not happen.

That's what the beta is for, which we did. Clearly, not enough people tried it out, or those regressions would have been caught. Creating a separate branch would not fix that. - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 4 Jan 2013 08:31:57 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:

 On Fri, Jan 04, 2013 at 08:18:52AM -0800, Jonathan M Davis wrote:
 On Friday, January 04, 2013 15:26:20 deadalnix wrote:
 It is why the proposal include branches as 2.N and revision as
 2.N.M . So the same version of D, with bugfixes can be published.
 
 The branch has a 2.N form, the tag has a 2.N.M form.

And why would that particular bug get a release as opposed to another?


I would propose to only merge regression fixes into minor releases. At least for dmd every bug fix also has potential to break user code. I think if 2.061 compiled some code, 2.061.1 should as well and the only way to ensure that is by only merging regression fixes (regressions which started to occur in 2.061). The decision what fixed bug / regression justifies an immediate release is up to the release manager. But to make things simple we can just have a fixed schedule for less urgent fixes: release a update every x weeks.
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 08:35:17 -0800
schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 The thread was way too big, and I had too little time to get into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis

I've restructured the wiki page here: http://wiki.dlang.org/User:Jpf/Release_Process_Staging Do you think that's clearer? (To my understanding it's the same process as proposed on the "official" wiki page)
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 16:19:45 UTC, Jonathan M Davis wrote:
 And why would that particular bug get a release as opposed to 
 another? Why
 wouldit be special? Or are you proposing that we do a 2.N.M 
 release for every
 bug fix? That seems insane. Why wouldn't you just use master if 
 you want the
 latest?

That is not latest, that is bug fixes. That is the whole point of changing process : being able to publish bug fixes without new features, so users don't need to wait mounth to get their bug fixed and get bunch of unrelated stuff with the fix that may introduce bugs as well. I took this precise example, but I'm not suggesting that a revision is made for EVERY bug fix. Only for severe bugs (regressions, corrupted codegen, . . .) or when the quantity is judged enough.
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:
 Am Fri, 04 Jan 2013 08:35:17 -0800
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 The thread was way too big, and I had too little time to get 
 into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis

I've restructured the wiki page here: http://wiki.dlang.org/User:Jpf/Release_Process_Staging Do you think that's clearer?

This is very good, however, some thing are still unclear. First, regression fix don't make any sense to me. You suggest to fix bug in master and fix regression in older branches. This should be the opposite IMO. A regression is something that used to work, but don't work anymore. So Correcting them in older version seems kind of contradictory, especially when other bugs aren't. I don't think separating those 2 type of bugs is really beneficial and I would keep only the process for regression fixes. Secondly, in every git commands block you have all the commands to set-up the repository. I think this belong to its own paragraph or even its own page. Then I'd replace them with git remote update in git command block. Finally, it is weird that we have v2.062 -> v2.062.1 . I'd prefers v2.062.0 for the first one. I like very much the ASCII art on top, which make thing really clear. Overall, it is better that the actual version on the wiki.
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 07:35:17PM +0100, deadalnix wrote:
 On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:

I've restructured the wiki page here:
http://wiki.dlang.org/User:Jpf/Release_Process_Staging

Do you think that's clearer?


 I like very much the ASCII art on top, which make thing really
 clear. Overall, it is better that the actual version on the wiki.

I think we should replace the actual version with this one. No point working on it as a user page when it's what's going to be implemented anyway. Wiki history will let us go back to review the old version if necessary. T -- Life is too short to run proprietary software. -- Bdale Garbee
Jan 04 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 18:35:18 UTC, deadalnix wrote:
 On Friday, 4 January 2013 at 17:26:02 UTC, Johannes Pfau wrote:
 Am Fri, 04 Jan 2013 08:35:17 -0800
 schrieb Jonathan M Davis <jmdavisProg gmx.com>:

 The thread was way too big, and I had too little time to get 
 into
 that conversation, which is why I wasn't involved.
 
 - Jonathan M Davis



OK, but it would have been a lot shorter had you and the other devs become involved ;)
 First, regression fix don't make any sense to me. You suggest 
 to fix bug in master and fix regression in older branches. This 
 should be the opposite IMO.

 A regression is something that used to work, but don't work 
 anymore. So Correcting them in older version seems kind of 
 contradictory, especially when other bugs aren't.

 I don't think separating those 2 type of bugs is really 
 beneficial and I would keep only the process for regression 
 fixes.

Terminology aside, we need to differentiate between bug fixes that do not introduce new bugs vs bug fixes that may introduce new bugs *or* break existing code for the current release version. We also *have* to prevent new features and structural changes (destabilizers) from leaking into a stable release.
 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.

 Finally, it is weird that we have v2.062 -> v2.062.1 . I'd 
 prefers v2.062.0 for the first one.

Absolutely! Otherwise someone is going to think v2.062 is greater than v2.062.1. Guaranteed. I already got semi-confused looking at the latest download page and I know what's going on far more than Joe Smith who walks in tomorrow checking out D for the first time.
 I like very much the ASCII art on top, which make thing really 
 clear. Overall, it is better that the actual version on the 
 wiki.

Looks good yes. --rt
Jan 04 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 20:30:28 Rob T wrote:
 Absolutely! Otherwise someone is going to think v2.062 is greater
 than v2.062.1. Guaranteed. I already got semi-confused looking at
 the latest download page and I know what's going on far more than
 Joe Smith who walks in tomorrow checking out D for the first time.

Really? Why on earth would you think that 2.062 was greater than 2.062.1? Also, I believe that it's very common with Linux packages (and probably the projects themselves) to do that sort of versioning where there's never a .0 and the last part only gets added when you actually get a .1. - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 19:30:29 UTC, Rob T wrote:
 Terminology aside, we need to differentiate between bug fixes 
 that do not introduce new bugs vs bug fixes that may introduce 
 new bugs *or* break existing code for the current release 
 version. We also *have* to prevent new features and structural 
 changes (destabilizers) from leaking into a stable release.

Understood. The things is that it is more dependent of the code than of the bug itself. Basically, it say that we should refuse risky bug fixes in released versions, and fix them into master. This probably should be specified at some point, but this is quite badly defined now, and I'm not sure this is really important to define this before actually using the process. I'd go for a use you judgment here, or some very generic guideline like « if the bug fix require important refactoring, or change the behavior of an existing feature, then it should only be fixed in master ».
Jan 04 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:
 Really? Why on earth would you think that 2.062 was greater 
 than 2.062.1?

I was asking for clarity so that no one can possibly get confused. If you look at the download page, the .0 is missing on some of the packages, but shows up as a -0 on some of the others, and that is simply confusing and totally unnecessary. If it is necessary for some reason, then it needs to be explained.
 Also, I believe that it's very common with Linux packages (and 
 probably the
 projects themselves) to do that sort of versioning where 
 there's never a .0
 and the last part only gets added when you actually get a .1.

There's no law that states that we must follow old conventions. --rt
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:
 On Friday, January 04, 2013 20:30:28 Rob T wrote:
 Absolutely! Otherwise someone is going to think v2.062 is 
 greater
 than v2.062.1. Guaranteed. I already got semi-confused looking 
 at
 the latest download page and I know what's going on far more 
 than
 Joe Smith who walks in tomorrow checking out D for the first 
 time.

Really? Why on earth would you think that 2.062 was greater than 2.062.1? Also, I believe that it's very common with Linux packages (and probably the projects themselves) to do that sort of versioning where there's never a .0 and the last part only gets added when you actually get a .1. - Jonathan M Davis

Both debian and ubuntu uses version.of.software-packageversion For instance : $ apt-cache policy audacious audacious: Installé : 3.2.4-1 Candidat : 3.2.4-1 Table de version : 3.3.3-2 0 10 http://ftp.fr.debian.org/debian/ experimental/main amd64 Packages *** 3.2.4-1 0 990 http://ftp.fr.debian.org/debian/ testing/main amd64 Packages 150 http://ftp.fr.debian.org/debian/ unstable/main amd64 Packages 100 /var/lib/dpkg/status 2.3-2 0 800 http://ftp.fr.debian.org/debian/ stable/main amd64 Packages 3.2.4 is the version of audacious. 1 is the version of the package. Distro rarely interfers with the version of the software itself and simply resuse what software devs choose, adding their own system on top of it.
Jan 04 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Friday, January 04, 2013 21:10:32 Rob T wrote:
 On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis wrote:
 Really? Why on earth would you think that 2.062 was greater
 than 2.062.1?

I was asking for clarity so that no one can possibly get confused. If you look at the download page, the .0 is missing on some of the packages, but shows up as a -0 on some of the others, and that is simply confusing and totally unnecessary. If it is necessary for some reason, then it needs to be explained.
 Also, I believe that it's very common with Linux packages (and
 probably the
 projects themselves) to do that sort of versioning where
 there's never a .0
 and the last part only gets added when you actually get a .1.

There's no law that states that we must follow old conventions.

True, but you also shouldn't do something different just to do something different. You need a good reason. I think that it's pretty ugly to have 2.062.0, and in my experience, that's a very abnormal thing to do. You have 2.062 followed by 2.062.1 if you ever have any point releases, but you don't start with .0. I don't recall ever seeing that before. And the fact that with have -0 on the latest release on the download page is downright bizarre too. I don't think that I've ever seen -0 before either. I'd expect you to add the -1 if you ever need one but not start with -0. And actually, looking at my Linux box now, it looks like Arch starts with -1, not -0. I don't know what other distros do. Regardless, I'm not at all in favor of having .0 on any release. Only add the minor versions to releases which are minor versions, not to the major ones. - Jonathan M Davis
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 20:48:34 UTC, Jonathan M Davis wrote:
 On Friday, January 04, 2013 21:10:32 Rob T wrote:
 On Friday, 4 January 2013 at 19:59:19 UTC, Jonathan M Davis 
 wrote:
 Really? Why on earth would you think that 2.062 was greater
 than 2.062.1?

I was asking for clarity so that no one can possibly get confused. If you look at the download page, the .0 is missing on some of the packages, but shows up as a -0 on some of the others, and that is simply confusing and totally unnecessary. If it is necessary for some reason, then it needs to be explained.
 Also, I believe that it's very common with Linux packages 
 (and
 probably the
 projects themselves) to do that sort of versioning where
 there's never a .0
 and the last part only gets added when you actually get a .1.

There's no law that states that we must follow old conventions.

True, but you also shouldn't do something different just to do something different. You need a good reason. I think that it's pretty ugly to have 2.062.0, and in my experience, that's a very abnormal thing to do. You have 2.062 followed by 2.062.1 if you ever have any point releases, but you don't start with .0. I don't recall ever seeing that before. And the fact that with have -0 on the latest release on the download page is downright bizarre too. I don't think that I've ever seen -0 before either. I'd expect you to add the -1 if you ever need one but not start with -0. And actually, looking at my Linux box now, it looks like Arch starts with -1, not -0. I don't know what other distros do. Regardless, I'm not at all in favor of having .0 on any release. Only add the minor versions to releases which are minor versions, not to the major ones. - Jonathan M Davis

I don't think anybody really care if this start with 0 or 1. What is weird is that you'll find 2 numbers versions and 3 numbers one, which is confusing (and I never saw that in any software).
Jan 04 2013
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 4 January 2013 at 21:02:45 UTC, deadalnix wrote:
 I don't think anybody really care if this start with 0 or 1. 
 What is weird is that you'll find 2 numbers versions and 3 
 numbers one, which is confusing (and I never saw that in any 
 software).

Looking at several software and what they do, it seems that starting with 0 and with 1 are both fairly common. WE ARE DEV, WE START COUNTING AT 0 !
Jan 04 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Friday, 4 January 2013 at 21:05:42 UTC, deadalnix wrote:
 On Friday, 4 January 2013 at 21:02:45 UTC, deadalnix wrote:
 I don't think anybody really care if this start with 0 or 1. 
 What is weird is that you'll find 2 numbers versions and 3 
 numbers one, which is confusing (and I never saw that in any 
 software).

Looking at several software and what they do, it seems that starting with 0 and with 1 are both fairly common. WE ARE DEV, WE START COUNTING AT 0 !

Even better is to also identify in the version sequence, what is a beta release and what is not. AFAIAC the current 2.061 release is in a beta stage because it is not yet "stable". The question though, is what does "stable" mean? For a definition, I propose something like: All known critical bugs have been resolved, and a certain percentage [to be determined] of all known non-critical bugs have been resolved, or some function thereof. We can settle on something I'm sure, but right now we have no definition of what stable means, so that's perhaps one reason why new releases are more buggy than I would expect them to be. But what does that mean? It means that I would *not* use the new release for anything that mattered in a production environment, not until it stabilized to a much higher standard than it currently is at. --rt
Jan 04 2013
prev sibling next sibling parent "H. S. Teoh" <hsteoh quickfur.ath.cx> writes:
On Fri, Jan 04, 2013 at 11:54:32PM +0100, Rob T wrote:
[...]
 Even better is to also identify in the version sequence, what is a
 beta release and what is not.
 
 AFAIAC the current 2.061 release is in a beta stage because it is
 not yet "stable". The question though, is what does "stable" mean?
 
 For a definition, I propose something like:
 
 All known critical bugs have been resolved, and a certain percentage
 [to be determined] of all known non-critical bugs have been
 resolved, or some function thereof. We can settle on something I'm
 sure, but right now we have no definition of what stable means, so
 that's perhaps one reason why new releases are more buggy than I
 would expect them to be. But what does that mean? It means that I
 would *not* use the new release for anything that mattered in a
 production environment, not until it stabilized to a much higher
 standard than it currently is at.

The problem is, what constitutes "stable" is a judgment call, because which bugs are critical (or, in this case, release-critical) are also a judgment call. So we would need a delegated Release Manager who decides when a particular release branch is ready to be released, and who holds high standards of what constitutes "release-ready". Or have some kind of voting system such that some fixed percentage of the top-voted bugs must be fixed before something can be released. The bug tracker already has a voting system, but (1) people don't pay attention to it, so the number of votes a bug gets doesn't seem to correlate with its likelihood of getting fixed; and (2) the number of votes you can cast is arbitrarily limited to 10, which discourages people from using the system. T -- Don't get stuck in a closet---wear yourself out.
Jan 04 2013
prev sibling next sibling parent "Rob T" <rob ucora.com> writes:
On Saturday, 5 January 2013 at 01:00:28 UTC, H. S. Teoh wrote:
 The problem is, what constitutes "stable" is a judgment call, 
 because
 which bugs are critical (or, in this case, release-critical) 
 are also a
 judgment call.

 So we would need a delegated Release Manager who decides when a
 particular release branch is ready to be released, and who 
 holds high
 standards of what constitutes "release-ready".

 [...]

Good points. As it stands right now, someone *has* to make a decision when to release, so we in effect have a "Release Manager" already, although who is making the decision and how the decision is being made is not clear at all. Personally, I would not want to be in that position because I would have nothing to guide me when deciding to make a release (or not), and if I made a mistake and released too soon or too late, then all the abuse for the mistake will be directed at me personally. I'd much rather have angry people place blame on a less than adequate process instead. Processes are much easier to improve on and no one but the process itself is too blame when a process fails to deliver. The point is, any definitions we come up with will be better than absolutely no definitions at all. For example the process as it is being defined, is making a positive impact even in its embryonic state. It allows us to look back at past less-than-perfect results, and move ahead with further incremental improvements. No one is to blame for the results, so we focus on the process improvements instead of pointing fingers at people. --rt
Jan 04 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 21:01:41 +0100
schrieb "Rob T" <rob ucora.com>:

 On Friday, 4 January 2013 at 17:11:54 UTC, Johannes Pfau wrote:
 The more I think about it, staging really seems to be useful to 
 avoid
 some corner cases.

I haven't had time to study the newly proposed process in enough detail to properly comment, but it always seemed to me that you cannot get away without a staging branch. You can try very hard to do without, but it means something else will suffer for the lack of it, so it seems you are coming to the same conclusion which is good (unless I'm wrong).

Yep. As we have lost most attention in this thread(most people have probably set "ignore this thread") we should check if we all agree on the process as outlined on the updated wiki page (I've updated the original page now) and improve it, if necessary, do some proof reading / spell checking and then open a new thread, asking for comments from the main devs.
Jan 05 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Fri, 04 Jan 2013 19:35:17 +0100
schrieb "deadalnix" <deadalnix gmail.com>:


 
 First, regression fix don't make any sense to me. You suggest to 
 fix bug in master and fix regression in older branches. This 
 should be the opposite IMO.
 
 A regression is something that used to work, but don't work 
 anymore. So Correcting them in older version seems kind of 
 contradictory, especially when other bugs aren't.

I added a "Definitions" section to explain what's meant by regression in that context. I also added a rationale, but already explained it well. This was written assuming dmd though, were bugfixes can easily cause regressions. For phobos/druntime we could use less strict rules, I added a not about that on the wiki. Now why we fix regressions in the release branches: Let's say we have dmd 2.061, 2.062, 2.062.1. A regression was introduced in 2.062 and my code that compiled fine in 2.061 doesn't work in 2.062 anymore. But then the regression fix is pushed to the 2.062 release branch, a minor release is made and 2.062.1 successfully compiles that code again. If the regression already occured in 2.061 should it be fixed in 2.062? This is a different question, but that situation can't happen as long as we only support 1 release and only fix newly introduced regressions.
 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.

Yep, will fix that.
Jan 05 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 5 Jan 2013 11:51:43 +0100
schrieb Johannes Pfau <nospam example.com>:

 Am Fri, 04 Jan 2013 19:35:17 +0100
 schrieb "deadalnix" <deadalnix gmail.com>:
 Secondly, in every git commands block you have all the commands 
 to set-up the repository. I think this belong to its own 
 paragraph or even its own page. Then I'd replace them with git 
 remote update in git command block.

Yep, will fix that.

Fixed, there's a "Local repository setup" section now. The instructions should be useable like a script now, although it would be good if someone proof-read them. BTW: especially http://wiki.dlang.org/Release_Process#Local_repository_setup should be proof-read. I'm not sure if it's the best solution, but I wanted something which works fine if the staging branch already exists on origin(new forks) and if it doesn't already exist.
Jan 05 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sat, 5 Jan 2013 10:21:30 -0800
schrieb "H. S. Teoh" <hsteoh quickfur.ath.cx>:

 
 One question, though: in the git workflow section, it says that a
 non-regression bugfix should go directly into staging, but there's no
 mention of merging it back to master. Is this deliberate? It seems to
 me to be kinda important that we merge all fixes back to master so
 that bugs don't get "unfixed". Or am I missing something here?
 

It's mentioned in the official branches section and in the summary "Regression fixes and bug fixes will be merged from staging into master." and "Merge staging branch into master regularly", but it should be added to the workflow as well. I don't think it's urgent to immediately merge it into master though and bugs can't really get 'unfixed' anyway: Releases are made from staging so the fix will be part of releases either way. It's enough to sporadically merge staging to master so that the feature branches and master also get the bug fixes.
Jan 05 2013
prev sibling next sibling parent Johannes Pfau <nospam example.com> writes:
Am Sun, 06 Jan 2013 00:18:16 +0100
schrieb "Rob T" <rob ucora.com>:

 One quick note, is that we may want to change the title and 
 description,

Done. http://wiki.dlang.org/Development_and_Release_Process I also verified and corrected the git commands. They should now all work as expected and can be used like a script. Do you agree with the process as described on that page now? Would be great to also get some feedback from other contributors of the original wiki page. I wonder whether a section should be added to the wiki explaining the transition from current workflow to the new workflow. For the first release we should do an additional merge from master into staging in ~4weeks as otherwise there won't be any new features in 2.062. And I'd recommend merging staging into master, then deleting staging and creating it again as we don't know whether staging already has cherry picked commits etc? If we want to maintain the current release we might have to use cherry-picking this one time as I think there are already regression fixes only pushed to master / staging which should be in the version branch.
Jan 06 2013
prev sibling next sibling parent "Jesse Phillips" <Jessekphillips+D gmail.com> writes:
On Friday, 4 January 2013 at 20:10:33 UTC, Rob T wrote:
 If you look at the download page, the .0 is missing on some of 
 the packages, but shows up as a -0 on some of the others, and 
 that is simply confusing and totally unnecessary. If it is 
 necessary for some reason, then it needs to be explained.

The -0 is a packaging version, if you repackage 2.63 (fixing dependencies) then the package is 2.63-1.
Jan 09 2013
prev sibling parent Johannes Pfau <nospam example.com> writes:
Am Wed, 09 Jan 2013 18:37:15 +0100
schrieb "Jesse Phillips" <Jessekphillips+D gmail.com>:

 On Thursday, 3 January 2013 at 17:56:05 UTC, Johannes Pfau wrote:
 What changes? All changes should be made in master, then 
 applied to
 staging via cherry picking. So there should never be changes in 
 staging
 that are not in master so there's nothing that could be 
 overwritten?

I strongly disagree here. We want pull requests to be made against the oldest supported/relevant branch. This could be merged to both staging and master (which can simply be done by merging staging into master) there is no need to complicate things with cherry picking. As for history, I think that clearly shows what the history is, staging gets fixes and master gets staging that is what we want.

You replied to one of my oldest posts in this discussion ;-) I didn't understand the staging concept when I wrote that - I think I've understood most of it now, though. I think the old release process wiki page wasn't clear enough. I've updated the http://wiki.dlang.org/Development_and_Release_Process page last week (I even added a "do not cherry pick from one official branch into another" rule), so you won't have to convince me ;-) Do you think the wiki page is good / correct? I've verified that all git commands work, so if everyone agrees with the workflow (should be the same as on the old page) we can probably start a new thread and ask for comments from the core devs.
Jan 09 2013