www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Mars Drafting Library - Official community driven library

reply "Piotrek" <no_data no_data.pl> writes:
Hi,

The history of std.(experimental.)logger and the latest thread 
about gui functionality inclusion into Phobos made me think about 
how to solve the problem of adding new modules.

I came up with the idea (maybe not new) to create a additional 
library(along druntime and Phobos) delivered with dmd package and 
named "Mars" (Deimos is unfortunately already taken ).

The library itself would be driven by community (not individual 
library developer) in order to be... the standard.

The process would be something similar to that other committees 
use
(e.g http://www.iec.ch/standardsdev/how/processes/development/) 
where before the standard is approved it goes through a draft 
stage. Still a draft can be used to create a working product 
(e.g. some Wi-Fi solutions based on IEEE 802.11 drafts)

Possible initial prerequisites:
- User awareness about the usage consequences
- Library placed at https://github.com/D-Programming-Language/
- Only well recognized community members have pull rights
- design decision made on the best known sw engineering patterns 
used in D
- New module should be functional with D/Phobos standards applied
- API and implementation allowed to change any time in order to 
make a progress
- no external dependencies beside OS services
- "draft" as the root module name e.g. "module draft.gui"

Advantages:
- community driven process which ensures the lowest level of 
controversy
- fast path for modules like GUI to be standardized

Disadvantages:
- additional effort for the sw release process

Ready to be destroyed ;)

Piotrek
Jan 31 2015
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
 and named "Mars" (Deimos is unfortunately already taken ).
BTW, Mars is the name of the language, before people started calling it D (which stuck).
Jan 31 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 14:33:35 UTC, Vladimir Panteleev 
wrote:
 On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
 and named "Mars" (Deimos is unfortunately already taken ).
BTW, Mars is the name of the language, before people started calling it D (which stuck).
I know that, but the naming is the least important aspect of the proposal. We can refarm the unused "Mars" name or choose another one, maybe "Curiosity", or anything suitable ;) Piotrek
Jan 31 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 1/31/2015 7:10 AM, Piotrek wrote:
 I know that, but the naming is the least important aspect of the proposal.
I agree. Let's not have endless posts bikeshedding the name, no matter how tempting.
Jan 31 2015
parent "Craig Dillabaugh" <craig.dillabaugh gmail.com> writes:
On Saturday, 31 January 2015 at 20:48:01 UTC, Walter Bright wrote:
 On 1/31/2015 7:10 AM, Piotrek wrote:
 I know that, but the naming is the least important aspect of 
 the proposal.
I agree. Let's not have endless posts bikeshedding the name, no matter how tempting.
Yeah, lets just call it Tango and get it over with.
Jan 31 2015
prev sibling next sibling parent reply "Laeeth Isharc" <Laeethnospam nospam.laeeth.com> writes:
This has the advantage over existing situation where you have the 
official library where things need to go through exacting and 
time consuming process and then dub.  Within dub every project is 
at the same level and it is not obvious which projects are the 
ones to use, and there is not necessarily coherence in the 
different ways things are done, and the only one motivated to 
work on supplementary features like documentation is the project 
author.  So you have very useful, well written libraries that 
have no docs.
Jan 31 2015
parent reply "Israel" <tl12000 live.com> writes:
On Saturday, 31 January 2015 at 18:11:48 UTC, Laeeth Isharc wrote:
 This has the advantage over existing situation where you have 
 the official library where things need to go through exacting 
 and time consuming process and then dub.  Within dub every 
 project is at the same level and it is not obvious which 
 projects are the ones to use, and there is not necessarily 
 coherence in the different ways things are done, and the only 
 one motivated to work on supplementary features like 
 documentation is the project author.  So you have very useful, 
 well written libraries that have no docs.
What i hate about dub is that no every library is ready to use. Some require additional steps/resources in order to get them to work. Dub does not have any indication of that because the library authors dont usually include that information. What is good about this "extra community driven standard library" is that everything will be included in one place and will be at the ready when it is needed.
Jan 31 2015
next sibling parent reply "Laeeth Isharc" <Laeethnospam nospam.laeeth.com> writes:
On Saturday, 31 January 2015 at 18:45:13 UTC, Israel wrote:
 On Saturday, 31 January 2015 at 18:11:48 UTC, Laeeth Isharc 
 wrote:
 This has the advantage over existing situation where you have 
 the official library where things need to go through exacting 
 and time consuming process and then dub.  Within dub every 
 project is at the same level and it is not obvious which 
 projects are the ones to use, and there is not necessarily 
 coherence in the different ways things are done, and the only 
 one motivated to work on supplementary features like 
 documentation is the project author.  So you have very useful, 
 well written libraries that have no docs.
What i hate about dub is that no every library is ready to use. Some require additional steps/resources in order to get them to work. Dub does not have any indication of that because the library authors dont usually include that information. What is good about this "extra community driven standard library" is that everything will be included in one place and will be at the ready when it is needed.
Well, I certainly do appreciate that all these libraries exist and, as D matures, better documentation a and a general tidying up will surely be part of that process, as has already begun with phobos. There is something of a hacker disdain for docs - they are not fun to write (for most people), and one thinks that a few pointers should be enough for someone to figure it out. And that's true, for someone like the author of the code. But it is hard to imagine what it is like not to have the skillset and experience one has. And so it is a very different experience for the new user coming from python than what he experienced when he learnt python, even though if you have the basic ability D need not be intrinsically harder. The division of labour makes sense - the people who like and are good at writing docs are not necessarily those who like and are good at writing code. But its harder to recruit helpers for a personal project than a shared library.
Jan 31 2015
parent "Israel" <tl12000 live.com> writes:
On Saturday, 31 January 2015 at 19:03:55 UTC, Laeeth Isharc wrote:
 On Saturday, 31 January 2015 at 18:45:13 UTC, Israel wrote:

 the people who like and are good at writing docs are not 
 necessarily those who like and are good at writing code.
This is exactly how i feel.
Jan 31 2015
prev sibling parent "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 18:45:13 UTC, Israel wrote:
 On Saturday, 31 January 2015 at 18:11:48 UTC, Laeeth Isharc 
 wrote:
 This has the advantage over existing situation where you have 
 the official library where things need to go through exacting 
 and time consuming process and then dub.  Within dub every 
 project is at the same level and it is not obvious which 
 projects are the ones to use, and there is not necessarily 
 coherence in the different ways things are done, and the only 
 one motivated to work on supplementary features like 
 documentation is the project author.  So you have very useful, 
 well written libraries that have no docs.
What i hate about dub is that no every library is ready to use. Some require additional steps/resources in order to get them to work. Dub does not have any indication of that because the library authors don't usually include that information. What is good about this "extra community driven standard library" is that everything will be included in one place and will be at the ready when it is needed.
Thanks. I wanted to mention the DUB packages but the post would be too long. DUB can be seen as a repository of special purpose packages without the burden of standardization process and out of the box properties. In general, this proposal assumes DUB repository will serve the same role as it does now. Piotrek
Jan 31 2015
prev sibling next sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
 Hi,

 The history of std.(experimental.)logger and the latest thread 
 about gui functionality inclusion into Phobos made me think 
 about how to solve the problem of adding new modules.

 I came up with the idea (maybe not new) to create a additional 
 library(along druntime and Phobos) delivered with dmd package 
 and named "Mars" (Deimos is unfortunately already taken ).

 The library itself would be driven by community (not individual 
 library developer) in order to be... the standard.

 The process would be something similar to that other committees 
 use
 (e.g http://www.iec.ch/standardsdev/how/processes/development/) 
 where before the standard is approved it goes through a draft 
 stage. Still a draft can be used to create a working product 
 (e.g. some Wi-Fi solutions based on IEEE 802.11 drafts)

 Possible initial prerequisites:
 - User awareness about the usage consequences
 - Library placed at https://github.com/D-Programming-Language/
 - Only well recognized community members have pull rights
 - design decision made on the best known sw engineering 
 patterns used in D
 - New module should be functional with D/Phobos standards 
 applied
 - API and implementation allowed to change any time in order to 
 make a progress
 - no external dependencies beside OS services
 - "draft" as the root module name e.g. "module draft.gui"

 Advantages:
 - community driven process which ensures the lowest level of 
 controversy
 - fast path for modules like GUI to be standardized

 Disadvantages:
 - additional effort for the sw release process

 Ready to be destroyed ;)

 Piotrek
The most important thing about a standard library is decisiveness in the leadership about what *kinds* of things should be in it. When it's been made clear that a given task is worthy of inclusion in the standard library, there will be enough incentive for contributors to put in the time and effort required to meet its high standards. But there are so many tasks that aren't quite up to that bar, either because they serve too specific a niche, or because there are two or more good, but conflicting ways to implement solutions, and no good way to decide between them. For any such task that doesn't meet those high standards, the library's development is best left to the community. But there is a place for the official recognition of existing libraries which have reached a level of quality that can be safely recommended. This is what is currently missing. There needs to be a page for this. The existing Wiki page: http://wiki.dlang.org/Libraries_and_Frameworks ... suffers from exactly the same problem as code.dlang.org. It contains everything in existence. This is more of a "second-party" library system, for which idea I am indebted to this article: http://www.jfbillingsley.com/blog/?p=206 Suggestions for implementation?
Jan 31 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 20:34:32 UTC, Zach the Mystic 
wrote:

 The most important thing about a standard library is 
 decisiveness in the leadership about what *kinds* of things 
 should be in it. When it's been made clear that a given task is 
 worthy of inclusion in the standard library, there will be 
 enough incentive for contributors to put in the time and effort 
 required to meet its high standards.
The intention of creating draft modules would be the inclusion in Phobos. In simplistic way, the following stages of development will be applied: 1. Proposal (DIP, NG discussion, DUB package showcase, local meet-up events etc) 2. Draft module creation and development 3 Approval for Phobos merge, i.e. "draft" -> "std"
 But there are so many tasks that aren't quite up to that bar, 
 either because they serve too specific a niche, or because 
 there are two or more good, but conflicting ways to implement 
 solutions, and no good way to decide between them.

 For any such task that doesn't meet those high standards, the 
 library's development is best left to the community. But there 
 is a place for the official recognition of existing libraries 
 which have reached a level of quality that can be safely 
 recommended. This is what is currently missing.
To be clear. I treat current DMD and Phobos library development as community driven as well. The reason, the title includes "community driven" is that modules can't be made by independent developer, i.e. without community approval. As an example DUB packages don't fulfill this requirement.
 This is more of a "second-party" library system, for which idea 
 I am indebted to this article: 
 http://www.jfbillingsley.com/blog/?p=206

 Suggestions for implementation?
I briefly read the article and some parts are similar. However the difference is that Curiosity/Mars would form some kind of trinity with Druntime and Phobos. See also my answer to weaselcat's post (http://forum.dlang.org/post/mtqjtavxzjucixuycnlf forum.dlang.org). Piotrek
Jan 31 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Saturday, 31 January 2015 at 23:03:52 UTC, Piotrek wrote:
 On Saturday, 31 January 2015 at 20:34:32 UTC, Zach the Mystic 
 wrote:

 The most important thing about a standard library is 
 decisiveness in the leadership about what *kinds* of things 
 should be in it. When it's been made clear that a given task 
 is worthy of inclusion in the standard library, there will be 
 enough incentive for contributors to put in the time and 
 effort required to meet its high standards.
The intention of creating draft modules would be the inclusion in Phobos. In simplistic way, the following stages of development will be applied: 1. Proposal (DIP, NG discussion, DUB package showcase, local meet-up events etc) 2. Draft module creation and development 3 Approval for Phobos merge, i.e. "draft" -> "std"
I really can't see the difference between `std.experimental` and this. If `std.experimental` doesn't get used for this, `std.experimental` will end up a marginalized "experiment" itself. I think `std.experimental` runs the huge risk of not being recognized as what it is - i.e. a shop for building things (from scratch if necessary, IMO). If you're not worried about the name "Mars", why are you worried about `std.experimental`?
 But there are so many tasks that aren't quite up to that bar, 
 either because they serve too specific a niche, or because 
 there are two or more good, but conflicting ways to implement 
 solutions, and no good way to decide between them.

 For any such task that doesn't meet those high standards, the 
 library's development is best left to the community. But there 
 is a place for the official recognition of existing libraries 
 which have reached a level of quality that can be safely 
 recommended. This is what is currently missing.
To be clear. I treat current DMD and Phobos library development as community driven as well. The reason, the title includes "community driven" is that modules can't be made by independent developer, i.e. without community approval. As an example DUB packages don't fulfill this requirement.
 This is more of a "second-party" library system, for which 
 idea I am indebted to this article: 
 http://www.jfbillingsley.com/blog/?p=206

 Suggestions for implementation?
I briefly read the article and some parts are similar. However the difference is that Curiosity/Mars would form some kind of trinity with Druntime and Phobos. See also my answer to weaselcat's post (http://forum.dlang.org/post/mtqjtavxzjucixuycnlf forum.dlang.org). Piotrek
Yes, we're basically talking about the two categories I mentioned to begin with. You're focusing on those libraries which can be pre-approved as worthy of phobos. The way I figure it, only Andrei and Walter can ultimately give pre-approval for such libraries. But I think the second kind I mentioned -- high-quality libraries which aren't suited for phobos -- also need official, or at least prominent, recognition. It's really important for people not to have to investigate every program listed on code.lang.org in order to find high-quality existing code. I would even argue that such recognition is more important than the library you're proposing here (which already seems to exist with `std.experimental`). Meeting the pre-approval standards for phobos is a very high bar, and it certainly won't be the majority of high-quality libraries.
Feb 01 2015
next sibling parent "Laeeth Isharc" <Laeeth.nospam nospam-laeeth.com> writes:
Things are never quite the same, but it is still interesting to 
remind oneself of experience in other languages.  History of 
numpy /scipy here:

http://wiki.scipy.org/History_of_SciPy

Would it have made the cut if it had had to meet an 
std.experimental level of quality in its earlier days ?  And the 
fact that it didn't allowed also for greater flexibility of 
development and for exploring multiple approaches (eventually 
recombined).

New growth often comes from the fringes and so won't be 
recognised until so many people are using it it can no longer be 
ignored.  It is important then to have less regulated interstices 
where such things can prove themselves.  So it might be worth 
asking directly the leading library authors what they think, 
since if people have no interest in filling out DIPs then it is 
good to know what the trade off might be (rigour versus creative 
work).
Feb 01 2015
prev sibling next sibling parent reply "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 14:40:17 UTC, Zach the Mystic wrote:
 The intention of creating draft modules would be the inclusion 
 in Phobos.
 In simplistic way, the following stages of development will be 
 applied:

 1. Proposal (DIP, NG discussion, DUB package showcase, local 
 meet-up events etc)
 2. Draft module creation and development
 3  Approval for Phobos merge, i.e. "draft" -> "std"
I really can't see the difference between `std.experimental` and this. If `std.experimental` doesn't get used for this, `std.experimental` will end up a marginalized "experiment" itself. I think `std.experimental` runs the huge risk of not being recognized as what it is - i.e. a shop for building things (from scratch if necessary, IMO). If you're not worried about the name "Mars", why are you worried about `std.experimental`?
I initially thought about the "std.experimental", but came up to the conclusion that when modules are in drafting stage they shouldn't pollute the Phobos. Basically because the final standard is not defined. A simple distinction can be seen as follows: draft - drafting std.experimental - piloting The Drafting library can be omitted during DMD installation without any harm for Druntime and Phobos.
 I briefly read the article and some parts are similar. However 
 the difference is that Curiosity/Mars would form some kind of 
 trinity with Druntime and Phobos. See also my answer to 
 weaselcat's post 
 (http://forum.dlang.org/post/mtqjtavxzjucixuycnlf forum.dlang.org).

 Piotrek
Yes, we're basically talking about the two categories I mentioned to begin with. You're focusing on those libraries which can be pre-approved as worthy of phobos. The way I figure it, only Andrei and Walter can ultimately give pre-approval for such libraries.
I don't treat Walter and Andrei as a blocking point. I think they will do anything is good for the language. Many time the D community initiated successful campaigns seconded by the key designers.
 But I think the second kind I mentioned -- high-quality 
 libraries which aren't suited for phobos -- also need official, 
 or at least prominent, recognition. It's really important for 
 people not to have to investigate every program listed on 
 code.lang.org in order to find high-quality existing code. I 
 would even argue that such recognition is more important than 
 the library you're proposing here (which already seems to exist 
 with `std.experimental`).
I truly agree that there are many valuable DUB packages needing the advertisement. However this is out of the scope of the proposal. Piotrek
Feb 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
 came up to the conclusion that when modules are in drafting 
 stage they
shouldn't pollute the Phobos This is does not seem a strong enough point to warrant extra maintenance effort and extra newbie confusion.
Feb 01 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 22:45:32 UTC, Dicebot wrote:
 came up to the conclusion that when modules are in drafting 
 stage they
shouldn't pollute the Phobos This is does not seem a strong enough point to warrant extra maintenance effort and extra newbie confusion.
Newbie confusion? In what way? Piotrek
Feb 01 2015
parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 1 February 2015 at 22:57:07 UTC, Piotrek wrote:
 On Sunday, 1 February 2015 at 22:45:32 UTC, Dicebot wrote:
 came up to the conclusion that when modules are in drafting 
 stage they
shouldn't pollute the Phobos This is does not seem a strong enough point to warrant extra maintenance effort and extra newbie confusion.
Newbie confusion? In what way?
What library to use between Phobos and Mars, why those are separate, why those have different stability guranatees, where to submit new contributions, why bother - it all would need to be written down and explained over and over again. And this alone is huge "-".
Feb 01 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 23:22:55 UTC, Dicebot wrote:
 Newbie confusion? In what way?
What library to use between Phobos and Mars, why those are separate, why those have different stability guranatees, where to submit new contributions, why bother - it all would need to be written down and explained over and over again. And this alone is huge "-".
IMO, there is no more confusion than using std.experimental vs other modules. Drafting modules are for developing functionality not present in Phobos, but build upon Phobos foundation. I admit I don't understand where confusion may come from. I will pay more attention to it as you described it as a huge problem. Piotrek
Feb 02 2015
prev sibling parent reply "Tofu Ninja" <emmons0 purdue.edu> writes:
On Sunday, 1 February 2015 at 14:40:17 UTC, Zach the Mystic wrote:
 Yes, we're basically talking about the two categories I 
 mentioned to begin with. You're focusing on those libraries 
 which can be pre-approved as worthy of phobos. The way I figure 
 it, only Andrei and Walter can ultimately give pre-approval for 
 such libraries. But I think the second kind I mentioned -- 
 high-quality libraries which aren't suited for phobos -- also 
 need official, or at least prominent, recognition. It's really 
 important for people not to have to investigate every program 
 listed on code.lang.org in order to find high-quality existing 
 code. I would even argue that such recognition is more 
 important than the library you're proposing here (which already 
 seems to exist with `std.experimental`). Meeting the 
 pre-approval standards for phobos is a very high bar, and it 
 certainly won't be the majority of high-quality libraries.
I tend to agree with this. I am not sure how another library with phobos level requirements would actually help. To me is seems like std.experimental, just with a different name. The problems with code.dlang.org seem to be a more important thing to focus on. One thing that is a downside of code.dlang.org is that it is hard to tell which libraries are good, which are bad, and it is hard to find what does what. One thing that would definitely help with the situation is a better code.dlang.org site with ratings, tags, categories, and comments. Also it would be great to advertise some of the better packages by putting them on a more prominent display in something like a "featured" section. I really wish I knew website design, then I could help with some of this stuff rather than just talking about it.
Feb 01 2015
parent "Dicebot" <public dicebot.lv> writes:
On Monday, 2 February 2015 at 03:58:44 UTC, Tofu Ninja wrote:
 One thing that would definitely help with the situation is a 
 better code.dlang.org site with ratings, tags, categories, and 
 comments. Also it would be great to advertise some of the 
 better packages by putting them on a more prominent display in 
 something like a "featured" section.
I think this is much better approach in the long-term. CI result integration fits the same purpose. This is one task where getting some paid help from professinal web devs may be important.
Feb 01 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
...
+1, basically boost for D. I heavily agree that the libraries should complement Phobos and never be orthogonal to phobos lest we have another tangos/phobos situation.
Jan 31 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 20:44:38 UTC, weaselcat wrote:
 On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
...
+1, basically boost for D. I heavily agree that the libraries should complement Phobos and never be orthogonal to phobos lest we have another tangos/phobos situation.
Several points from prerequisites was intensionally chosen to prevent Phobos/Tango fiasco. The general rule is that the draft module should be treated as a potential Phobos module. Piotrek
Jan 31 2015
prev sibling next sibling parent reply "ZombineDev" <valid_email he.re> writes:
I like the idea of having an additional library that we would 
ship alongside Phobos with every release. There of course some 
obvious pros and cons for having 'Mars' (or whatever is called) 
as a DUB packages vs included in the standard library:

Pros for being a DUB package:
+ faster release cycles (push a new version any time).
+ semantic versioning (instead of dmd 2.0xx).
+ can depend on other DUB packages (doesn't need to include 
everything it uses beside Phobos, ofc).
+ can have different configurations / build types, be split in 
DUB subpackages and all the other nice things that the DUB 
package format provides.
+ can include other binaries (eg. C libs).
+ etc...
Cons:
- Probably not so well tested, because it won't get as wildly 
used as Phobos.
- Third-party with zero guarantees. (You have to trust the 
author(s) that it is not malicious or anything like that).
- Not everybody is comfortable with having DUB as a dependency on 
their project.
- Can be a bit of an inconvenience if you only want to write a 
small script.
- DUB doesn't have first class support in the major IDEs, AFAIK 
(for people that prefer them over the terminal). Actually I thing 
it's extremely simple to use on the command-ling and I have no 
problem with Mono-D or "dub generate visuald", but it's not like 
the NuGet support in VS for .NET projects (AFAIR you can now have 
language plugins/analyzers based on Roslyn (linting, DSLs, etc.) 
which are NuGet packages that you download from their gallery).
- etc...

Pros being included alongside Phobos:
+ Better testing because more people can/will use it
+ Potentially better code, because of code review during pull 
requests and generally high standards for new stuff (like with 
std.experimental.logger).
+ More stable, because people may care more for backwards 
comparability (though the points is that this will not be 
guaranteed).
+ People new to the language will feel more comfortable using 
'standard' libraries.
+ etc...
Cons:
- Extremely slow release cycle.
- Hard to get new stuff (controversial like GUI) in.
- Not being able to have external dependencies than druntime and 
Phobos (like bindings for C libraries)
- etc...

With all that said I still think that it is a good idea to have 
another library shipping alongside Phobos with every DMD release 
(or GDC, LDC, etc.).
I think a good compromise would be the following:

1. Include DUB with DMD. We don't need a stable DUB as a library 
API to just use it to get other packages.
2. Make 'Mars' a DUB package and use semantic versioning to tag 
new releases.
3. Move it to github.com/D-Programming-Language/.
4. Include last known 'well working' with every DMD release. (Of 
course other implementations are free to decide whether to 
include it). Or we can have some post-installation script which 
would call DUB.
5. Afterwards if a new version of 'Mars' is released users can 
just do a 'dub upgrade' to update the one that's already 
included, or wait for a new official release.

I really do thing that this would be a step in the right 
direction. For example Microsoft has been been using their 
package manager NuGet "as a .NET Framework release vehicle" quite 
successfully for the last couple of years. Here[1] you can see 
that they are providing quite large and heavily used parts of 
.NET.

Another good idea is to separate Phobos from DMD and also put it 
on DUB. As you can see[2] many of the integral parts of.NET are 
provided as packages and people have no problem using them as 
such (you can see by the large download numbers).


[1]: http://blogs.msdn.com/b/dotnet/p/nugetpackages.aspx
[2]: https://www.nuget.org/packages
Jan 31 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 20:47:43 UTC, ZombineDev wrote:
 I like the idea of having an additional library that we would 
 ship alongside Phobos with every release. There of course some 
 obvious pros and cons for having 'Mars' (or whatever is called) 
 as a DUB packages vs included in the standard library:
Thanks for comments:) I'm going to address most of them.
 Pros being included alongside Phobos:
 + Better testing because more people can/will use it
 + Potentially better code, because of code review during pull 
 requests and generally high standards for new stuff (like with 
 std.experimental.logger).
 + More stable, because people may care more for backwards 
 comparability (though the points is that this will not be 
 guaranteed).
 + People new to the language will feel more comfortable using 
 'standard' libraries.
 + etc...
 Cons:
 - Extremely slow release cycle.
There can be nightly/weekly builds if really needed. But in general, yes, release cycle would be the same as DMD package.
 - Hard to get new stuff (controversial like GUI) in.
The key aspect of the proposal it to have that kind of functionality go through a processes of eliminating controversy as much as possible during developing stage.
 - Not being able to have external dependencies than druntime 
 and Phobos (like bindings for C libraries)
 - etc...
Being a part of official packet it is actually advantage. As an opposite example I couldn't get curl module working under Ubuntu in sensible time.
 I think a good compromise would be the following:

 1. Include DUB with DMD. We don't need a stable DUB as a 
 library API to just use it to get other packages.
 2. Make 'Mars' a DUB package and use semantic versioning to tag 
 new releases.
 3. Move it to github.com/D-Programming-Language/.
 4. Include last known 'well working' with every DMD release. 
 (Of course other implementations are free to decide whether to 
 include it). Or we can have some post-installation script which 
 would call DUB.
 5. Afterwards if a new version of 'Mars' is released users can 
 just do a 'dub upgrade' to update the one that's already 
 included, or wait for a new official release.


 Another good idea is to separate Phobos from DMD and also put 
 it on DUB. As you can see[2] many of the integral parts of.NET 
 are provided as packages and people have no problem using them 
 as such (you can see by the large download numbers).


 [1]: http://blogs.msdn.com/b/dotnet/p/nugetpackages.aspx
 [2]: https://www.nuget.org/packages
This proposal's aim is to be the least intrusive so that kind of changes are out of the scope. DMD and Phobos are often coupled in terms of changes (bug fixes etc). I'm against moving standard library to DUB. Piotrek
Feb 01 2015
prev sibling next sibling parent reply "Jonathan Marler" <johnnymarler gmail.com> writes:
On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
 Hi,

 The history of std.(experimental.)logger and the latest thread 
 about gui functionality inclusion into Phobos made me think 
 about how to solve the problem of adding new modules.

 I came up with the idea (maybe not new) to create a additional 
 library(along druntime and Phobos) delivered with dmd package 
 and named "Mars" (Deimos is unfortunately already taken ).

 The library itself would be driven by community (not individual 
 library developer) in order to be... the standard.

 The process would be something similar to that other committees 
 use
 (e.g http://www.iec.ch/standardsdev/how/processes/development/) 
 where before the standard is approved it goes through a draft 
 stage. Still a draft can be used to create a working product 
 (e.g. some Wi-Fi solutions based on IEEE 802.11 drafts)

 Possible initial prerequisites:
 - User awareness about the usage consequences
 - Library placed at https://github.com/D-Programming-Language/
 - Only well recognized community members have pull rights
 - design decision made on the best known sw engineering 
 patterns used in D
 - New module should be functional with D/Phobos standards 
 applied
 - API and implementation allowed to change any time in order to 
 make a progress
 - no external dependencies beside OS services
 - "draft" as the root module name e.g. "module draft.gui"

 Advantages:
 - community driven process which ensures the lowest level of 
 controversy
 - fast path for modules like GUI to be standardized

 Disadvantages:
 - additional effort for the sw release process

 Ready to be destroyed ;)

 Piotrek
I approve :) +1
Jan 31 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Saturday, 31 January 2015 at 20:55:11 UTC, Jonathan Marler 
wrote:
 On Saturday, 31 January 2015 at 14:06:20 UTC, Piotrek wrote:
 Hi,

 The history of std.(experimental.)logger and the latest thread 
 about gui functionality inclusion into Phobos made me think 
 about how to solve the problem of adding new modules.

 I came up with the idea (maybe not new) to create a additional 
 library(along druntime and Phobos) delivered with dmd package 
 and named "Mars" (Deimos is unfortunately already taken ).

 The library itself would be driven by community (not 
 individual library developer) in order to be... the standard.

 The process would be something similar to that other 
 committees use
 (e.g 
 http://www.iec.ch/standardsdev/how/processes/development/) 
 where before the standard is approved it goes through a draft 
 stage. Still a draft can be used to create a working product 
 (e.g. some Wi-Fi solutions based on IEEE 802.11 drafts)

 Possible initial prerequisites:
 - User awareness about the usage consequences
 - Library placed at https://github.com/D-Programming-Language/
 - Only well recognized community members have pull rights
 - design decision made on the best known sw engineering 
 patterns used in D
 - New module should be functional with D/Phobos standards 
 applied
 - API and implementation allowed to change any time in order 
 to make a progress
 - no external dependencies beside OS services
 - "draft" as the root module name e.g. "module draft.gui"

 Advantages:
 - community driven process which ensures the lowest level of 
 controversy
 - fast path for modules like GUI to be standardized

 Disadvantages:
 - additional effort for the sw release process

 Ready to be destroyed ;)

 Piotrek
I approve :) +1
Thanks for your vote. If there is no official veto to the proposal, I will create a DIP in a day or two. Meanwhile, we can still discuss unclear or problematic issues. Piotrek
Feb 01 2015
prev sibling next sibling parent reply "HaraldZealot" <harald_zealot tut.by> writes:
 Possible initial prerequisites:
 - User awareness about the usage consequences
 - Library placed at https://github.com/D-Programming-Language/
 - Only well recognized community members have pull rights
 - design decision made on the best known sw engineering 
 patterns used in D
 - New module should be functional with D/Phobos standards 
 applied
 - API and implementation allowed to change any time in order to 
 make a progress
 - no external dependencies beside OS services
 - "draft" as the root module name e.g. "module draft.gui"

 Advantages:
 - community driven process which ensures the lowest level of 
 controversy
 - fast path for modules like GUI to be standardized

 Disadvantages:
 - additional effort for the sw release process

 Ready to be destroyed ;)

 Piotrek
Approximately a half year ago I have similar idea and suggestion. This is my idea: * make new feature in dub, that it can place some libraries in common namespace. For example CyberShadow's ae will be placed in something like advancex.image, or logger (lucky it is in std.experimental already, but as alternative history) is placed in stdx.logger. But they are not part of phobos in that time but usual dub package on dub registry. * create namespaces "advance" (or any other) for useful but not so common components (e.g. proposal windowing, image processing an so on), "bind" for Deimos. Phobos is "std" already :). And also create their experimental counterpart like "advancex", "bindx" and "stdx". (It can be other name but I prefer one worded "stdx" than two worded "std.experimental" in other level of hierarchy"). * make new feature in dub and dub register that counts download, likes and bugs. When some package receives essential feedback it can be started pull request process. * all packages in special namespaces can't be orphan. Experimental version must have at least one active maintainer. More approved namespace require two responsible maintainer one of which at least someone from trusted user. If package became orphan it will be dropped out. In this manner we can create stair like development. To place simple package in dub registry you have the same condition as for now. If you want place it in experimental namespace you must complete some easy condition (more complicated in case of "stdx" and perhaps "bindx"), and you package must go through full review process in case of phobos. You can see on archlinux as example of such package management. For now I drive two interesting project, but I also try to find forces for this one.
Feb 01 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 09:28:42 UTC, HaraldZealot wrote:
 Approximately a half year ago I have similar idea and 
 suggestion.
Thanks for your input. Yes, there are similarities, but there are also some differences. See some of my comments below:
 This is my idea:
 * make new feature in dub, that it can place some libraries in 
 common namespace.

 For example CyberShadow's ae will be placed in something like 
 advancex.image, or logger (lucky it is in std.experimental 
 already, but as alternative history) is placed in stdx.logger. 
 But they are not part of phobos in that time but usual dub 
 package on dub registry.

 * create namespaces "advance" (or any other) for useful but not 
 so common components (e.g. proposal windowing, image processing 
 an so on), "bind" for Deimos. Phobos is "std" already :). And 
 also create their experimental counterpart like "advancex", 
 "bindx" and "stdx". (It can be other name but I prefer one 
 worded "stdx" than two worded "std.experimental" in other level 
 of hierarchy").
This differ from the "Drafting Library" proposal in the following points: - modules/packages are owned/driven by one developer - dub packages are not inclined to work out of the box with dmd release package
 * make new feature in dub and dub register that counts 
 download, likes and bugs. When some package receives essential 
 feedback it can be started pull request process.
By the "pull request process" you mean inclusion in Phobos? Then, in result, wouldn't it just mean: make the most popular dub package in some category a standard?
 For now I drive two interesting project, but I also try to find 
 forces for this one.
The important thing about the proposal is to provide a process so one can be a part in creating the standard. You could contribute to drafts for the modules related to areas of your expertise. Piotrek
Feb 01 2015
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
Just few quick questions:

1) what would it give over std.experimental ?
2) what would it give over code.dlang.org ?
3) what problems are you trying to solve and why do you think 
this is suitable solution?
Feb 01 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 21:54:13 UTC, Dicebot wrote:
 Just few quick questions:
Hi
 1) what would it give over std.experimental ?
- draft modules will be more flexible for changes than in the ones in standard library - new drafting modules won't disturb usual users of the standard library IMO, std.experimental is not for the drafting stage of the SW development.
 2) what would it give over code.dlang.org ?
- community driven as opposed to individual driven - out of the box readiness - minimal fragmentation and controversy
 3) what problems are you trying to solve and why do you think 
 this is suitable solution?
Adding new modules (replacing the deprecated ones) in more robust and quicker manner. IMO, this is a change in the direction how standards are made. Piotrek
Feb 01 2015
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Sunday, 1 February 2015 at 22:54:51 UTC, Piotrek wrote:
 On Sunday, 1 February 2015 at 21:54:13 UTC, Dicebot wrote:
 Just few quick questions:
Hi
 1) what would it give over std.experimental ?
- draft modules will be more flexible for changes than in the ones in standard library
As per latest agreement everything in std.experimental is considered subject to any change so is perfectly flexible.
 - new drafting modules won't disturb usual users of the 
 standard library
That statements needs some hard data that current situation is disturbing to be considered as a rationale.
 IMO, std.experimental is not for the drafting stage of the SW 
 development.
Depends on your definition of "draft". Anything that is good enough to be actually used in real app is good enough for std.experimental - and anything less is of no use to end user anyway.
 2) what would it give over code.dlang.org ?
- community driven as opposed to individual driven - out of the box readiness - minimal fragmentation and controversy
code.dlang.org is actually much more community driven because it is naturally decentralized. Controversy is inevitable anyway (hello std.json). Fragmentation is a thing though - but I yet to be convinced that is a bad thing that needs to be fixed.
 3) what problems are you trying to solve and why do you think 
 this is suitable solution?
Adding new modules (replacing the deprecated ones) in more robust and quicker manner.
It is as quick as it can be for standard library - and code.dlang.org takes care of everything else. Any library that risks quick removal of deprecated modules / API is not acceptable for "standard" stamp. So far this does not seem a proposal that pulls own weight to me.
Feb 01 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Sunday, 1 February 2015 at 23:21:36 UTC, Dicebot wrote:
 As per latest agreement everything in std.experimental is 
 considered subject to any change so is perfectly flexible.
 - new drafting modules won't disturb usual users of the 
 standard library
That statements needs some hard data that current situation is disturbing to be considered as a rationale.
If you put to many uncompleted modules in the std.experimental it can influence stable part of the Phobos. - binary size - pull request with more experimental stuff etc.
 IMO, std.experimental is not for the drafting stage of the SW 
 development.
Depends on your definition of "draft". Anything that is good enough to be actually used in real app is good enough for std.experimental - and anything less is of no use to end user anyway.
I agree that all is a matter of definition. I still doubt that fast evolving drafts would be possible in std namespace.
 2) what would it give over code.dlang.org ?
- community driven as opposed to individual driven - out of the box readiness - minimal fragmentation and controversy
code.dlang.org is actually much more community driven because it is naturally decentralized. Controversy is inevitable anyway (hello std.json).
I used the "community driven" in contrast to "individually driven". The key property of the proposal is to minimize the controversy by community (common) drafting process.
 Fragmentation is a thing though - but I yet to be convinced 
 that is a bad thing that needs to be fixed.
I consider fragmentation a major problem for standards.
 3) what problems are you trying to solve and why do you think 
 this is suitable solution?
Adding new modules (replacing the deprecated ones) in more robust and quicker manner.
It is as quick as it can be for standard library - and code.dlang.org takes care of everything else.
I have non-trivial modules in mind like gui, json, database, etc.
 Any library that risks quick removal of deprecated modules / 
 API is not acceptable for "standard" stamp.
The drafting lib is a proposition of standard not a standard itself. And I didn't said anything about removal. I was referring to: "Warning: This module is considered out-dated and not up to Phobos' current standards. It will remain until we have a suitable replacement, but be aware that it will not remain long term."
 So far this does not seem a proposal that pulls own weight to 
 me.
I can put the burden on my shoulders. I'm in the middle of DIP creation. Unfortunately I have a regular job and other duties. Still, I hope to finish it as soon as promised. Piotrek
Feb 02 2015
prev sibling parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Sunday, 1 February 2015 at 22:54:51 UTC, Piotrek wrote:
 On Sunday, 1 February 2015 at 21:54:13 UTC, Dicebot wrote:
 1) what would it give over std.experimental ?
- draft modules will be more flexible for changes than in the ones in standard library - new drafting modules won't disturb usual users of the standard library IMO, std.experimental is not for the drafting stage of the SW development.
I don't see the return on investment for making a big distinction between a draft and a more polished, but still not guaranteed library. The big line that needs to be drawn is whether any given task is or is not an appropriate target for the standard library. Once something passes that stage, it needs to be decided whether the API is in flux (e.g. std.experimental) or stable (e.g. std proper). An additional distinction between "more in flux" (your idea) and "less in flux" (std.experimental) is both less useful and quite possibly dangerous: The big temptation for software developers is to *promise* stability in order to attract the users they need in order to get the feedback they need in order to create the best possible design, and then break stability with the new design. By dividing libraries into the hazy "more in flux" and "less in flux" categories, one is simply deferring the hard choice between "stable" and "not stable", and likely to start making promises one cannot -- or worse, does not even intend to -- keep as a result. Therefore, I think std.experimental is for all "in flux" APIs, from the drafting stage to the later "less in flux" stages. The danger is that the phobos management will want to have their cake and eat it too, as the saying goes. You simply can't promise users both stability and a perfectly designed API at the same time, tempting as it is.
Feb 01 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Monday, 2 February 2015 at 06:07:29 UTC, Zach the Mystic wrote:
 Therefore, I think std.experimental is for all "in flux" APIs, 
 from the drafting stage to the later "less in flux" stages.
Definitely this is what I thought initially. But, IMO, it can be really hard or impossible to carry out, as you pointed one of the issues in the following part:
 The danger is that the phobos management will want to have 
 their cake and eat it too, as the saying goes. You simply can't 
 promise users both stability and a perfectly designed API at 
 the same time, tempting as it is.
Hence the *proposal* of the drafting library. Piotrek
Feb 02 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 2 February 2015 at 20:09:42 UTC, Piotrek wrote:
 On Monday, 2 February 2015 at 06:07:29 UTC, Zach the Mystic 
 wrote:
 Therefore, I think std.experimental is for all "in flux" APIs, 
 from the drafting stage to the later "less in flux" stages.
Definitely this is what I thought initially. But, IMO, it can be really hard or impossible to carry out, as you pointed one of the issues in the following part:
 The danger is that the phobos management will want to have 
 their cake and eat it too, as the saying goes. You simply 
 can't promise users both stability and a perfectly designed 
 API at the same time, tempting as it is.
Hence the *proposal* of the drafting library. Piotrek
I think std.experimental should essentially be its own library, with its own slot next to phobos on the github repo. I'm open to changing the name or something... but if we have both std.experimental *and* your new mars, what do you think is the real difference? How can std.experimental be "sort of" experimental, but really not, whereas mars is "really actually" experimental? Where is the cutoff line? How would anyone know whether they reached it or not? My attitude is that any given module in std.experimental should simply indicate its current level stability at the top of its documentation - full disclosure about where it is from, say, 0 to 95%, (with anything above that already assumed qualified for std proper). I'd even make one more point, that all current phobos modules known to be in need of revamping be copied wholesale in their current forms to std.experimental, with the top documentation saying, "This is the experimental version of the old std.xxx, which needs *your* help with its redesign and implementation. Feel free to break the current API by improving it. This is *not* currently considered a stable module."
Feb 02 2015
parent reply "Piotrek" <no_data no_data.pl> writes:
On Monday, 2 February 2015 at 21:19:01 UTC, Zach the Mystic wrote:

 I think std.experimental should essentially be its own library, 
 with its own slot next to phobos on the github repo. I'm open 
 to changing the name or something... but if we have both 
 std.experimental *and* your new mars, what do you think is the 
 real difference? How can std.experimental be "sort of" 
 experimental, but really not, whereas mars is "really actually" 
 experimental? Where is the cutoff line? How would anyone know 
 whether they reached it or not?
Hard to admit but I thought about removal of std.experimental ;), then I tried to find a better solution (not sure I did). After all I don't want to make more enemies than I have. So I got an idea that std.experimental can be adopted for the piloting new modules into Phobos when approved to be a standard. After Wikipedia: "A pilot project refers to an initial roll-out of a system into production"
 My attitude is that any given module in std.experimental should 
 simply indicate its current level stability at the top of its 
 documentation - full disclosure about where it is from, say, 0 
 to 95%, (with anything above that already assumed qualified for 
 std proper).
Yes, the drafting stage can have many sub-stages, but I didn't want to complicate the initial proposal too much and risk both low chance of the approval and later, inability of proper implementation. I learned that there can be taken only limited steps at once.
 I'd even make one more point, that all current phobos modules 
 known to be in need of revamping be copied wholesale in their 
 current forms to std.experimental, with the top documentation 
 saying, "This is the experimental version of the old std.xxx, 
 which needs *your* help with its redesign and implementation. 
 Feel free to break the current API by improving it. This is 
 *not* currently considered a stable module."
Haha, let's not lose our nerves :) It's not that bad. Moreover. I can barely find the equivalent level of code awesomeness as it is in D libraries. We just need to speed up the progress and reduce work fragmentation. Piotrek
Feb 02 2015
parent reply "Zach the Mystic" <reachzach gggmail.com> writes:
On Monday, 2 February 2015 at 22:18:59 UTC, Piotrek wrote:
 On Monday, 2 February 2015 at 21:19:01 UTC, Zach the Mystic 
 wrote:

 I think std.experimental should essentially be its own 
 library, with its own slot next to phobos on the github repo. 
 I'm open to changing the name or something... but if we have 
 both std.experimental *and* your new mars, what do you think 
 is the real difference? How can std.experimental be "sort of" 
 experimental, but really not, whereas mars is "really 
 actually" experimental? Where is the cutoff line? How would 
 anyone know whether they reached it or not?
Hard to admit but I thought about removal of std.experimental ;), then I tried to find a better solution (not sure I did). After all I don't want to make more enemies than I have. So I got an idea that std.experimental can be adopted for the piloting new modules into Phobos when approved to be a standard.
I'm arguing from the perspective that the approval must come first, and the piloting second. Who would want to develop a module in the "pre-pilot" phase, without having any idea of whether they were developing it finally for phobos or just 3rd-party use, then wait months or years to find out if the leadership is even *interested* in what they're doing? Why would people try to meet high standards without knowing if those standards are actually going to be required or not? It's like an unpaid internship: "Work for us for free, and we'll let you know at the end whether we want your work or not."
 I'd even make one more point, that all current phobos modules 
 known to be in need of revamping be copied wholesale in their 
 current forms to std.experimental, with the top documentation 
 saying, "This is the experimental version of the old std.xxx, 
 which needs *your* help with its redesign and implementation. 
 Feel free to break the current API by improving it. This is 
 *not* currently considered a stable module."
Haha, let's not lose our nerves :) It's not that bad. Moreover. I can barely find the equivalent level of code awesomeness as it is in D libraries. We just need to speed up the progress and reduce work fragmentation.
It occurred to me that when copying a module to std.experimental, the documentation at the top of the original std module should say, "This module has been feature frozen while work on the new one continues at std.experimental.xxxx. The API for this module will not break, nor will it be improved. Please consider using std.experimental.xxxx and giving feedback for its design, since eventually this module will be replaced with that one." I do realize that this is kind of what happened with D1 and D2, of course. Now you're going to have competing modules, and some people won't want to give up on the old one. So you will have to deprecate every change that breaks the old API first before finally replacing it with the new one. Well, it's a hard problem, and yet, "Most D code has yet to be written." But then again, I'm not the one who has to take fire for breaking people's old code, and I don't know if my attitude would change if I were.
Feb 02 2015
parent "Piotrek" <no_data no_data.pl> writes:
On Tuesday, 3 February 2015 at 02:38:57 UTC, Zach the Mystic 
wrote:
 I'm arguing from the perspective that the approval must come 
 first, and the piloting second. Who would want to develop a 
 module in the "pre-pilot" phase, without having any idea of 
 whether they were developing it finally for phobos or just 
 3rd-party use, then wait months or years to find out if the 
 leadership is even *interested* in what they're doing? Why 
 would people try to meet high standards without knowing if 
 those standards are actually going to be required or not? It's 
 like an unpaid internship: "Work for us for free, and we'll let 
 you know at the end whether we want your work or not."
Al least the module will be available in draft namespace until controversy is resolved. I don't think there would be any blocking campaign without providing sane requirements to move on. BTW. I tried to publish the DIP but got into the following issue: http://forum.dlang.org/post/yoqwuqhtlpifgwudeevm forum.dlang.org It looks the DIP will be delayed at least by day. Piotrek
Feb 03 2015