www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Incubated modules for Phobos

reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Hi all,

Normal Phobos submission procedure is usually like that:

1. write entire module from scratch by oneself
2. submit for voting
3. rewrite wrong parts, if there are none then add it to Phobos
4. otherwise goto 2

It is hard for one person to write entire module in such way it 
satisfies everyone, especially for very new or complex additions, such 
as database handling or cryptography.

Here, I propose that we add experimental "exp" hierarchy to Phobos for 
such projects. I know etc hierarchy may me used for that but IMHO 
separate "exp" would be more appropriate. In this namespace "beta" 
modules will slowly evolve into official std modules.

Some (obvious) advantages are:

* developers may receive feedback very early in the process, saving them 
from mass rewrites when in the opinion of community they made some wrong 
coding decission.
* developers may receive coding help, i.e. many of them may collaborate 
on one big module.
* users may test experimental modules early. This is important, because 
usability issues may be catched earlier, not after submission when API 
usually becomes frozen and it is too late.

"Exp" code may be shipped with each release just like "etc" code. Users 
using experimental code should be aware of breaking changes that may be 
introduced with each release or even with each commit.

Thoughts?
Dec 18 2011
next sibling parent Bane <branimir.milosavljevic gmail.com> writes:
Piotr Szturmaj Wrote:

 Hi all,
 
 Normal Phobos submission procedure is usually like that:
 
 1. write entire module from scratch by oneself
 2. submit for voting
 3. rewrite wrong parts, if there are none then add it to Phobos
 4. otherwise goto 2
 
 It is hard for one person to write entire module in such way it 
 satisfies everyone, especially for very new or complex additions, such 
 as database handling or cryptography.
 
 Here, I propose that we add experimental "exp" hierarchy to Phobos for 
 such projects. I know etc hierarchy may me used for that but IMHO 
 separate "exp" would be more appropriate. In this namespace "beta" 
 modules will slowly evolve into official std modules.
 
 Some (obvious) advantages are:
 
 * developers may receive feedback very early in the process, saving them 
 from mass rewrites when in the opinion of community they made some wrong 
 coding decission.
 * developers may receive coding help, i.e. many of them may collaborate 
 on one big module.
 * users may test experimental modules early. This is important, because 
 usability issues may be catched earlier, not after submission when API 
 usually becomes frozen and it is too late.
 
 "Exp" code may be shipped with each release just like "etc" code. Users 
 using experimental code should be aware of breaking changes that may be 
 introduced with each release or even with each commit.
 
 Thoughts?
I like it. More people will keep an eye on new versions for new modules that are coming in. And if one wants to write new module for his needs, it can easily check exp to see is he reinventing wheel. (exmple. my suggestion here few days back to upgrade std.process only to find there is upgraded module - if I didn't ask on this NG I would never know there is that module). So exp sounds good.
Dec 18 2011
prev sibling next sibling parent reply Peter Alexander <peter.alexander.au gmail.com> writes:
On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 Hi all,

 Normal Phobos submission procedure is usually like that:

 1. write entire module from scratch by oneself
 2. submit for voting
 3. rewrite wrong parts, if there are none then add it to Phobos
 4. otherwise goto 2

 It is hard for one person to write entire module in such way it
 satisfies everyone, especially for very new or complex additions, such
 as database handling or cryptography.

 Here, I propose that we add experimental "exp" hierarchy to Phobos for
 such projects. I know etc hierarchy may me used for that but IMHO
 separate "exp" would be more appropriate. In this namespace "beta"
 modules will slowly evolve into official std modules.

 Some (obvious) advantages are:

 * developers may receive feedback very early in the process, saving them
 from mass rewrites when in the opinion of community they made some wrong
 coding decission.
 * developers may receive coding help, i.e. many of them may collaborate
 on one big module.
 * users may test experimental modules early. This is important, because
 usability issues may be catched earlier, not after submission when API
 usually becomes frozen and it is too late.

 "Exp" code may be shipped with each release just like "etc" code. Users
 using experimental code should be aware of breaking changes that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Dec 18 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Peter Alexander wrote:
 On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 "Exp" code may be shipped with each release just like "etc" code. Users
 using experimental code should be aware of breaking changes that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.
Dec 18 2011
next sibling parent Chad J <chadjoan __spam.is.bad__gmail.com> writes:
On 12/18/2011 10:00 AM, Piotr Szturmaj wrote:
 Peter Alexander wrote:
 On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 "Exp" code may be shipped with each release just like "etc" code. Users
 using experimental code should be aware of breaking changes that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.
That would be because there are no branches for it. cd $DPATH/src/phobos git branch <crickets> This proposal seems good, but I would do it in the form of an "exp" branch rather than an "exp" directory. If this is desired for releases, then perhaps the release scripts could clone both the master branch and the exp branch so that it's all out in the open.
Dec 18 2011
prev sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj <bncrbme jadamspam.pl>  
wrote:

 Peter Alexander wrote:
 On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 "Exp" code may be shipped with each release just like "etc" code. Users
 using experimental code should be aware of breaking changes that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.
I think it would be enough to post something on D.announce and then whoever is interested can watch your branch. Maybe even a wiki page could categorize the branches. I agree with others that github's forking and branching mechanism works quite well for developing multi-user projects. For example, both Lars Kyllingstad and I are working on a new std.process module, he's doing the main design and unixen implementations, I'm doing the windows implementation. -Steve
Dec 19 2011
parent reply Piotr Szturmaj <bncrbme jadamspam.pl> writes:
Steven Schveighoffer wrote:
 On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
 <bncrbme jadamspam.pl> wrote:

 Peter Alexander wrote:
 On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 "Exp" code may be shipped with each release just like "etc" code. Users
 using experimental code should be aware of breaking changes that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.
I think it would be enough to post something on D.announce and then whoever is interested can watch your branch. Maybe even a wiki page could categorize the branches. I agree with others that github's forking and branching mechanism works quite well for developing multi-user projects. For example, both Lars Kyllingstad and I are working on a new std.process module, he's doing the main design and unixen implementations, I'm doing the windows implementation.
Okay, I agree too. The core problem is that modules after inclusion become official/public instantly and API must be frozen. Eventual API issues discovered after the inclusion are often impossible to fix because nobody "likes" deprecations and breaking changes. So, instead of making modules public from the start, I would rather give some additional "beta" time for them. Interested users may check out these modules in the "real world" during that time. Also, beta code may introduce breaking changes without worrying. I think review is a requirement of course but it is often not enough. Some things may not be catched during review. Code must be extensively _used_ to prove its quality. If you agree with me, there is only one additional rule to add to Phobos contribution process. After a review and voting process, add module to etc (or exp, whatever) for one month. Then move it to std if no/little changes were made during that trial. If changes are big the trial period should be extended.
Dec 19 2011
parent "Paul D. Anderson" <paul.d.removethis.anderson comcast.andthis.net> writes:
I think this idea needs further consideration.

Summarizing the earlier discussion, there were four schools of 
thought:

1. This is a good idea.
2. This is a good idea, but let's use github branches.
3. This is a good idea, but let's use newsgroup postings.
4. This is a good idea, but let's use the review queue page on 
the wiki.

I'd hate to see this good idea not get implemented because we 
can't agree on the mechanism. The suggestions above are not 
mutually exclusive.

It seems to me that there are two primary requirements for this 
to be effective:

1. There should be ONE place where people can go to see what's 
under development.

2. That ONE place needs to be easy to find, as in easy for 
someone visiting the D programming language website for the very 
first time.

Here's my proposal: (Pay attention, there's a quiz at the end.)

Add another menu item under the "Community" menu to the left 
here. Call it "Under Development" or "Work in Progress" or 
"Experimental Library modules" or redefine the "Projects & 
Libraries" link. Whatever. The link would lead to a list of 
modules under development. These could be github branches, 
dsource projects, review queue items, etc. It doesn't matter how 
the project is sourced or maintained.

Anyone who has a module under development could add a URL to 
their project page at that ONE place.
  These could be in any stage of development -- just a proposal or 
maybe just a good idea, a formal specification, a beta version, a 
complete library, or even a wish list...)

There may be other, better ways to implement this, but I would 
find it very useful to have that ONE place.

For example, I spent quite a bit of time, spread over a couple of 
weeks, looking for a D language statistics library. It turns out 
there is at least one.

Quiz: 1) Where is it? 2) Why isn't it being considered for Phobos?


On Monday, 19 December 2011 at 16:46:46 UTC, Piotr Szturmaj wrote:
 Steven Schveighoffer wrote:
 On Sun, 18 Dec 2011 10:00:35 -0500, Piotr Szturmaj
 <bncrbme jadamspam.pl> wrote:

 Peter Alexander wrote:
 On 18/12/11 2:18 PM, Piotr Szturmaj wrote:
 "Exp" code may be shipped with each release just like "etc" 
 code. Users
 using experimental code should be aware of breaking changes 
 that may be
 introduced with each release or even with each commit.

 Thoughts?
Isn't this just reinventing git branches? If people have their work-in-progress branches on GitHub then people can already try them out, submit pull requests etc. That's the whole point of a branch.
Yes, but what I propose is the centralized repository for that branches (eventual candidates to std). Currently no one knows all modules that are being worked on.
I think it would be enough to post something on D.announce and then whoever is interested can watch your branch. Maybe even a wiki page could categorize the branches. I agree with others that github's forking and branching mechanism works quite well for developing multi-user projects. For example, both Lars Kyllingstad and I are working on a new std.process module, he's doing the main design and unixen implementations, I'm doing the windows implementation.
Okay, I agree too. The core problem is that modules after inclusion become official/public instantly and API must be frozen. Eventual API issues discovered after the inclusion are often impossible to fix because nobody "likes" deprecations and breaking changes. So, instead of making modules public from the start, I would rather give some additional "beta" time for them. Interested users may check out these modules in the "real world" during that time. Also, beta code may introduce breaking changes without worrying. I think review is a requirement of course but it is often not enough. Some things may not be catched during review. Code must be extensively _used_ to prove its quality. If you agree with me, there is only one additional rule to add to Phobos contribution process. After a review and voting process, add module to etc (or exp, whatever) for one month. Then move it to std if no/little changes were made during that trial. If changes are big the trial period should be extended.
Jan 03 2012
prev sibling next sibling parent "jdrewsen" <jdrewsen nospam.com> writes:
On Sunday, 18 December 2011 at 14:18:34 UTC, Piotr Szturmaj wrote:
 Hi all,

 Normal Phobos submission procedure is usually like that:

 1. write entire module from scratch by oneself
 2. submit for voting
 3. rewrite wrong parts, if there are none then add it to Phobos
 4. otherwise goto 2

 It is hard for one person to write entire module in such way it 
 satisfies everyone, especially for very new or complex 
 additions, such as database handling or cryptography.

 Here, I propose that we add experimental "exp" hierarchy to 
 Phobos for such projects. I know etc hierarchy may me used for 
 that but IMHO separate "exp" would be more appropriate. In this 
 namespace "beta" modules will slowly evolve into official std 
 modules.

 Some (obvious) advantages are:

 * developers may receive feedback very early in the process, 
 saving them from mass rewrites when in the opinion of community 
 they made some wrong coding decission.
 * developers may receive coding help, i.e. many of them may 
 collaborate on one big module.
 * users may test experimental modules early. This is important, 
 because usability issues may be catched earlier, not after 
 submission when API usually becomes frozen and it is too late.

 "Exp" code may be shipped with each release just like "etc" 
 code. Users using experimental code should be aware of breaking 
 changes that may be introduced with each release or even with 
 each commit.

 Thoughts?
I think that people should just their WIP + links to github/docs on the "review queue" wiki page linked from the dlang.org left menu. /Jonas
Dec 18 2011
prev sibling parent "Marco Leise" <Marco.Leise gmx.de> writes:
The idea reminds me of how extensions are managed in OpenGL:  =

http://www.opengl.org/resources/features/OGLextensions/
Often hardware vendors like S3, nVidia or ATi invented cool stuff, like =
 =

texture compression and were free to add a prefixed function name to the=
ir  =

drivers (S3_=E2=80=A6, NV_=E2=80=A6, ATI_=E2=80=A6). Sometimes other ven=
dors would implement those  =

functions as well and they are eventually renamed to EXT_=E2=80=A6. Then=
 there is  =

the architecture review board (ARB) that picks up these extensions and  =

evaluates how they could fit into OpenGL. Often changes are necessary fo=
r  =

the function to become ARB_=E2=80=A6. I think this is all still independ=
ent of the  =

specification release cycle (OpenGL is just a specification). When a new=
  =

specification is planned, ARB_=E2=80=A6 functions make it into GL_=E2=80=
=A6 core  =

functionality.
I believe this concept helps innovation. But who would decide which modu=
le  =

is a good fit for Phobos and which one is better fetched through an  =

external package manager? For example, what if I came up a complete  =

containers package that has different approaches from std.containers or =
a  =

2D drawing/image API or a D compiler front-end?
In OpenGL terminology, I think it would be an ARB_=E2=80=A6 already, if =
it is in a  =

distribution of DMD. It would have a specification that some review boar=
d  =

has accepted for inclusion into Phobos. Anything with a clear  =

specification on paper or as an implementation could be accepted, but on=
  =

the other hand, just saying that "I start writing an image handling  =

module" should not be enough to be in .ext in Phobos. Otherwise some .ex=
t  =

modules will disappear suddenly or get stuck there forever, because as  =

they mature the review board realizes, that the API is not going to be  =

what they like to see in Phobos. That's frustrating for both the develop=
er  =

and users.

So in short: yes, but review what ends up in .ext
Dec 18 2011