www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Expanding Phobos from a flat hierachy

reply "Don" <don nospam.com> writes:
In the "Implementing Half Floats in D" thread, we seemed to have 
reached a consensus on two important points:
(a) Phobos should have a broad scope (rather than being small 
like the C standard library).
(b) The current flat structure of Phobos (every module in the 
root) does not scale to hundreds of modules.

It's not quite unanimous on (a), but seems to be close enough.

Together, (a) and (b) mean we need a plan. The sooner we can do 
it, the less painful it will be. But, on the other hand, it's 
something that we really don't want to get wrong.

Personal taste plays a huge role in this (Practically any 
structure can work, but we're looking for an arrangement which is 
intuitive and aesthetically pleasing to as many people as 
possible). It'll be most productive to stick to uncontroversial 
facts as long as possible.

It would be great if people with extensive experience in other 
languages could give a brief description of the structure of the 
corresponding libraries. (Also include pseudo-standard libraries, 
for example boost in C++).

1. How many total modules are there? How old is the library? What 
is the recent growth rate of the library (Percentage increase in 
size per year, over whatever time interval is reasonable for that 
library)?
2. How deep is the hierarchy? How many top-level branches are 
there?
3. How has the library dealt with obsolete functions and modules? 
Eg, are there entire top-level branches which are obsolete? How 
many stupid names are there (eg, "std.algorithm2") which were 
forced by the original design becoming obsolete?
4. How much of the library is successful/convincing? To make it a 
bit less subjective: how much of the library is commonly ignored, 
in favour of using third-party libraries instead? Which areas are 
failures in this way (eg, GUI, database, ...)? Which areas are 
the most often praised?

Maybe such a survey already exists, but I haven't been able to 
find an existing one.
Feb 05 2013
next sibling parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, February 06, 2013 08:56:26 Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to have
 reached a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small
 like the C standard library).
 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.
 
 It's not quite unanimous on (a), but seems to be close enough.
 
 Together, (a) and (b) mean we need a plan. The sooner we can do
 it, the less painful it will be. But, on the other hand, it's
 something that we really don't want to get wrong.
 
 Personal taste plays a huge role in this (Practically any
 structure can work, but we're looking for an arrangement which is
 intuitive and aesthetically pleasing to as many people as
 possible). It'll be most productive to stick to uncontroversial
 facts as long as possible.


I agree that we should look at minimizing how many top-level modules we add in 
the future, but I don't think that it's generally worth trying to rearrange 
the modules that we already have. So, adding std.halffloat would be bad, but
I'm 
not sure that it's worth doing something like moving std.uri into std.net. 
Even if we did, we'd be forced to leave std.uri around for quite some time (if 
not permanently, given Walter's attitude about such things), which negates the 
gain to some exent. You can also argue endlessly about what should be where in 
the hierarchy, which is needless bikeshedding. So, if we do any moving, we 
need to be sure that that's what we want to do and be done with it, and we 
shouldn't be moving new modules around. They need to go into the right spot 
when they're added, not be rearranged later.


 It would be great if people with extensive experience in other
 languages could give a brief description of the structure of the
 corresponding libraries. (Also include pseudo-standard libraries,
 for example boost in C++).



either of them recently. It's easy enough to look at the docs though. They're 
well-organized. However, it should be kept in mind that both of those 
languages have the downside of being forced to have one public class per file 

which can be good but is often overkill, which means that we wouldn't 
necessarily want to lay stuff out in quite the same way as they do, even if we 
agreed that they do a good job of it.

- Jonathan M Davis
Feb 06 2013
next sibling parent reply "Don" <don nospam.com> writes:
On Wednesday, 6 February 2013 at 08:16:38 UTC, Jonathan M Davis 
wrote:

 On Wednesday, February 06, 2013 08:56:26 Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to 
 have
 reached a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small
 like the C standard library).
 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.
 
 It's not quite unanimous on (a), but seems to be close enough.
 
 Together, (a) and (b) mean we need a plan. The sooner we can do
 it, the less painful it will be. But, on the other hand, it's
 something that we really don't want to get wrong.
 
 Personal taste plays a huge role in this (Practically any
 structure can work, but we're looking for an arrangement which 
 is
 intuitive and aesthetically pleasing to as many people as
 possible). It'll be most productive to stick to uncontroversial
 facts as long as possible.





 You can also argue endlessly about what should be where in
 the hierarchy, which is needless bikeshedding.


Yes, exactly. That's why I want to defer passing judgement on 
anything until we have a reasonable set of data. Just because a 
design works well for a particular language doesn't mean it would 
also be good for D, but it's still good to look at.

I think it's more constructive to say "that design was tried in 
Ruby, but they abandoned it" than to say "I personally don't like 
it". OTOH, if three different languages have totally different 
structures, and all have been widely praised, then it's clear 
it's a personal preference issue.

And if we can say, "all nine libraries we've looked at did it 
this way", then the argument to do the same thing is very strong.
Feb 06 2013
parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 6 February 2013 at 10:15:28 UTC, Don wrote:

 On Wednesday, 6 February 2013 at 08:16:38 UTC, Jonathan M Davis 
 wrote:

 On Wednesday, February 06, 2013 08:56:26 Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to 
 have
 reached a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small
 like the C standard library).
 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.
 
 It's not quite unanimous on (a), but seems to be close enough.
 
 Together, (a) and (b) mean we need a plan. The sooner we can 
 do
 it, the less painful it will be. But, on the other hand, it's
 something that we really don't want to get wrong.
 
 Personal taste plays a huge role in this (Practically any
 structure can work, but we're looking for an arrangement 
 which is
 intuitive and aesthetically pleasing to as many people as
 possible). It'll be most productive to stick to 
 uncontroversial
 facts as long as possible.





 You can also argue endlessly about what should be where in
 the hierarchy, which is needless bikeshedding.


 Yes, exactly. That's why I want to defer passing judgement on 
 anything until we have a reasonable set of data. Just because a 
 design works well for a particular language doesn't mean it 
 would also be good for D, but it's still good to look at.

 I think it's more constructive to say "that design was tried in 
 Ruby, but they abandoned it" than to say "I personally don't 
 like it". OTOH, if three different languages have totally 
 different structures, and all have been widely praised, then 
 it's clear it's a personal preference issue.

 And if we can say, "all nine libraries we've looked at did it 
 this way", then the argument to do the same thing is very 
 strong.


I don't know if this is related or not (I'd think it is), but if 
we were to re-think phobos' module organization, we may also want 
to think about how we want to deal with changing entire modules 
at once.

I'm mostly thinking about things like "random2.d", "xmk2.d" or 
whatnot.

Or it could be un-related. Just saying we might want to take this 
into account.
Feb 06 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, February 06, 2013 11:28:25 monarch_dodra wrote:

 I don't know if this is related or not (I'd think it is), but if
 we were to re-think phobos' module organization, we may also want
 to think about how we want to deal with changing entire modules
 at once.
 
 I'm mostly thinking about things like "random2.d", "xmk2.d" or
 whatnot.


I believe that that's what we decided on previously. It just hasn't actually 
been done yet, because no one has implemented such a module since the decision 
was made.

- Jonathan M Davis
Feb 06 2013
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 6 February 2013 at 10:51:19 UTC, Jonathan M Davis 
wrote:

 On Wednesday, February 06, 2013 11:28:25 monarch_dodra wrote:

 I don't know if this is related or not (I'd think it is), but 
 if
 we were to re-think phobos' module organization, we may also 
 want
 to think about how we want to deal with changing entire modules
 at once.
 
 I'm mostly thinking about things like "random2.d", "xmk2.d" or
 whatnot.


 I believe that that's what we decided on previously. It just 
 hasn't actually
 been done yet, because no one has implemented such a module 
 since the decision
 was made.


Andrei made a proposal to allow transforming a module into a 
package. It seems to me like the way to go for this.

This is even possible to manage several versions of the lib using 
version that way. And different module can even import their own. 
Sound like a better idea than messing up with names.
Feb 06 2013
parent reply Jonathan M Davis <jmdavisProg gmx.com> writes:
On Wednesday, February 06, 2013 12:06:18 deadalnix wrote:

 Andrei made a proposal to allow transforming a module into a
 package. It seems to me like the way to go for this.


Yes, but that's a different issue. It's one thing to take std.algorithm or 
std.datetime and turn them into packages while still allowing you to import 
std.algorithm or std.datetime as before. It's quite another to completely 
replace a module with another module. To do that, you need a new module. 
Anything else will break code. The question then is what to name the new 
module.


 This is even possible to manage several versions of the lib using
 version that way. And different module can even import their own.
 Sound like a better idea than messing up with names.


You could replace the entire module in-place, but doing so will break code, 
and Walter in particular is very much against that in general. Whatever we did 
would require allowing people to transition from one to the other and we might 
even have to leave the old one around permanently (which I don't like, but I'm 
sure that Walter would favor). Simply changing it with a new version of Phobos 
wouldn't cut it.

- Jonathan M Davis
Feb 06 2013
next sibling parent reply "Dicebot" <m.strashun gmail.com> writes:
AFAIR there was proposal of a "future" meta package for 
introducing new modules with time to adapt, similar to how it is 
done in few other languages.
Feb 06 2013
parent reply "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 6 February 2013 at 11:38:31 UTC, Dicebot wrote:

 AFAIR there was proposal of a "future" meta package for 
 introducing new modules with time to adapt, similar to how it 
 is done in few other languages.


I really like the idea of an experimental section of phobos. I've 
been thinking lately that the phobos review might be improved by 
doing something more like this:

1. Proposal
2. Review and vote of overall usefulness and whether it meets a 
certain quality requirement.
3. Module is placed in experimental section of phobos with strong 
warnings that it is a trial module and could be removed in a 
future update.
4. Following a set period of time after the module has been in a 
public release another vote is held for actual inclusion.
5. If yea, move to its final resting place, if nay remove it for 
the experimental section.

People are much more likely to actually test something if it's 
right at their fingertips and they don't have to download it and 
set up module paths and all of that. This isn't particularly hard 
with D but I think the easier it is for people to do, the better 
the reviews and module quality will be. Also, giving a hard time 
period within which the module can be modified freely because 
nobody expects it to be set in stone would help a lot.

BA
Feb 06 2013
next sibling parent Robert <jfanatiker gmx.at> writes:
+1 from me. As long as we don't have a central repository, where we
could have a separate phobos-staging package or something. At the moment
I don't think that additions to phobos were tested much in real world
before being adopted. Such an experimental section could really help and
we should try to get things in there quickly, to boost the whole process
a bit.

Best regards,

Robert

On Wed, 2013-02-06 at 19:47 +0100, Brad Anderson wrote:

 I really like the idea of an experimental section of phobos. I've 
 been thinking lately that the phobos review might be improved by 
 doing something more like this:
 
 1. Proposal
 2. Review and vote of overall usefulness and whether it meets a 
 certain quality requirement.
 3. Module is placed in experimental section of phobos with strong 
 warnings that it is a trial module and could be removed in a 
 future update.
 4. Following a set period of time after the module has been in a 
 public release another vote is held for actual inclusion.
 5. If yea, move to its final resting place, if nay remove it for 
 the experimental section.
Feb 06 2013
prev sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 6 February 2013 at 18:47:35 UTC, Brad Anderson 
wrote:

 On Wednesday, 6 February 2013 at 11:38:31 UTC, Dicebot wrote:

 AFAIR there was proposal of a "future" meta package for 
 introducing new modules with time to adapt, similar to how it 
 is done in few other languages.


 I really like the idea of an experimental section of phobos. 
 I've been thinking lately that the phobos review might be 
 improved by doing something more like this:

 1. Proposal
 2. Review and vote of overall usefulness and whether it meets a 
 certain quality requirement.
 3. Module is placed in experimental section of phobos with 
 strong warnings that it is a trial module and could be removed 
 in a future update.
 4. Following a set period of time after the module has been in 
 a public release another vote is held for actual inclusion.
 5. If yea, move to its final resting place, if nay remove it 
 for the experimental section.

 People are much more likely to actually test something if it's 
 right at their fingertips and they don't have to download it 
 and set up module paths and all of that. This isn't 
 particularly hard with D but I think the easier it is for 
 people to do, the better the reviews and module quality will 
 be. Also, giving a hard time period within which the module can 
 be modified freely because nobody expects it to be set in stone 
 would help a lot.

 BA


Just one last idea I had I'd like to share before I forget about 
it.  I think module reviews should, in part, come in the form of 
pull requests. High level reviews need discussion so they would 
still be done here on the forums but simple documentation fixes 
and other tweaks would be better handled by pull requests on the 
module's home repo.  Seems silly that people will type up all of 
their suggestions and then the module author will slowly go 
through the review responses looking for changes he should be 
making when actually making the changes and sending a pull 
request would take less time and as well as some of the huge 
burden off of the author of the module.

BA
Feb 07 2013
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 6 February 2013 at 11:33:20 UTC, Jonathan M Davis 
wrote:

 On Wednesday, February 06, 2013 12:06:18 deadalnix wrote:

 Andrei made a proposal to allow transforming a module into a
 package. It seems to me like the way to go for this.


 Yes, but that's a different issue. It's one thing to take 
 std.algorithm or
 std.datetime and turn them into packages while still allowing 
 you to import
 std.algorithm or std.datetime as before. It's quite another to 
 completely
 replace a module with another module. To do that, you need a 
 new module.
 Anything else will break code. The question then is what to 
 name the new
 module.


If we use the package trick as proposed, a possible solution is :

std/datetime/v1.d :
// Datetime version 1

std/datetime/v2.d :
// Datetime version 2

std/datetime.d or std/datetime/package.d (or whatever) :
version(DATETIME_V1) {
     public import std.datetime.v1;
} else {
     public import std.datetime.v2;
}


 You could replace the entire module in-place, but doing so will 
 break code,
 and Walter in particular is very much against that in general. 
 Whatever we did
 would require allowing people to transition from one to the 
 other and we might
 even have to leave the old one around permanently (which I 
 don't like, but I'm
 sure that Walter would favor). Simply changing it with a new 
 version of Phobos
 wouldn't cut it.


Well Walter seems very worried about breaking code, but seems 
unable to adopt any practice the reduce its effect, and in the 
facts, my cod break at any new dmd release (and right now compile 
with none released one).
Feb 06 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2013 6:58 PM, deadalnix wrote:

 Well Walter seems very worried about breaking code, but seems unable to adopt
 any practice the reduce its effect, and in the facts, my cod break at any new
 dmd release (and right now compile with none released one).


Here's the current list of regressions:

http://d.puremagic.com/issues/buglist.cgi?query_format=advanced&bug_severity=regression&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED

If what's causing your code to break isn't there, it is unlikely to get fixed.
Feb 07 2013
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02/07/2013 03:58 AM, deadalnix wrote:

 ... my cod break at any new dmd release (and right now compile with none
released one).


+1. (I keep reporting regressions though, dustmite works quite well.)
Feb 07 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2013 12:15 AM, Jonathan M Davis wrote:

 I don't think that it's generally worth trying to rearrange
 the modules that we already have. So, adding std.halffloat would be bad, but
I'm
 not sure that it's worth doing something like moving std.uri into std.net.
 Even if we did, we'd be forced to leave std.uri around for quite some time


At least the contents of std.uri can be nothing more than:

   module std.uri;
   public import std.net.uri;
Feb 06 2013
parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 6 February 2013 at 21:07:24 UTC, Walter Bright 
wrote:

 On 2/6/2013 12:15 AM, Jonathan M Davis wrote:

 I don't think that it's generally worth trying to rearrange
 the modules that we already have. So, adding std.halffloat


 would be bad, but I'm

 not sure that it's worth doing something like moving std.uri


 into std.net.

 Even if we did, we'd be forced to leave std.uri around for


 quite some time

 At least the contents of std.uri can be nothing more than:

   module std.uri;
   public import std.net.uri;


Just throwing in an idea, in case no one mentioned it yet:

To avoid cluttering the Phobos directory structure, 
documentation, makefiles, etc. with redirect modules, the 
redirect modules can be placed in a completely separate directory 
(which would also have a "std" subdirectory), which would be 
added to sc.ini/dmd.conf's include path.
Feb 06 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 07 Feb 2013 05:19:56 +0100
schrieb "Vladimir Panteleev" <vladimir thecybershadow.net>:


 On Wednesday, 6 February 2013 at 21:07:24 UTC, Walter Bright 
 wrote:

 On 2/6/2013 12:15 AM, Jonathan M Davis wrote:

 I don't think that it's generally worth trying to rearrange
 the modules that we already have. So, adding std.halffloat


 would be bad, but I'm

 not sure that it's worth doing something like moving std.uri


 into std.net.

 Even if we did, we'd be forced to leave std.uri around for


 quite some time

 At least the contents of std.uri can be nothing more than:

   module std.uri;
   public import std.net.uri;


 
 Just throwing in an idea, in case no one mentioned it yet:
 
 To avoid cluttering the Phobos directory structure, 
 documentation, makefiles, etc. with redirect modules, the 
 redirect modules can be placed in a completely separate directory 
 (which would also have a "std" subdirectory), which would be 
 added to sc.ini/dmd.conf's include path.


That's cool. Together with the other proposals I think we have
enough infrastructure ready to solve all problems with module
updates and moves. You can also remove that line from dmd.conf
and add it only to project configurations with 'legacy' code.
Since IDEs like Mono-D already scan available modules in the
include paths for imports, this would reduce the noise a bit.

-- 
Marco
Feb 07 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 08:56, Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to have reached
 a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small like the C
 standard library).
 (b) The current flat structure of Phobos (every module in the root) does
 not scale to hundreds of modules.

 It's not quite unanimous on (a), but seems to be close enough.

 Together, (a) and (b) mean we need a plan. The sooner we can do it, the
 less painful it will be. But, on the other hand, it's something that we
 really don't want to get wrong.


+1

This is long overdue.

-- 
/Jacob Carlborg
Feb 06 2013
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 08:56, Don wrote:


 It would be great if people with extensive experience in other languages
 could give a brief description of the structure of the corresponding
 libraries. (Also include pseudo-standard libraries, for example boost in
 C++).

 1. How many total modules are there? How old is the library? What is the
 recent growth rate of the library (Percentage increase in size per year,
 over whatever time interval is reasonable for that library)?


Tango has 365 modules according to the documentation for D1 (including 
the object module).


 2. How deep is the hierarchy? How many top-level branches are there?


These are spread over 9 packages in the top level "tango" package.

It seems to have at most four levels of packages, including the top 
level "tango" package.


 3. How has the library dealt with obsolete functions and modules? Eg,
 are there entire top-level branches which are obsolete? How many stupid
 names are there (eg, "std.algorithm2") which were forced by the original
 design becoming obsolete?


There are no obsolete top level packages. As far as I know there are no 
stupid names. At one time they did remove the "collection" package and 
replaced it with a "container" package. It's not really a stupid name. 
There can be other cases like this where the optimal name is not used 
but there are no names like "std.algorithm2".

-- 
/Jacob Carlborg
Feb 06 2013
prev sibling next sibling parent "rumbu" <rumbu rumbu.ro> writes:
On Wednesday, 6 February 2013 at 07:56:26 UTC, Don wrote:

 It would be great if people with extensive experience in other 
 languages could give a brief description of the structure of 
 the corresponding libraries. (Also include pseudo-standard 
 libraries, for example boost in C++).



map existing D modules to the same hierarchy, because D is 
lacking the ideea of namespace. More than that, the way things 
are organized in BCL doesn't match the D phobos rules, only the 
System namespace in BCL contains more than 300 different types 
with thousands of functions. I don't think anybody will like a 
system.d file containing that number of elements.

The class hierarchy in main BCL assembly (mscorlib.dll) is 

is a class) from D point of view:
- system - basic types, general purpose functions;
- system.collections - object ranges;
- system.collections.generic - generic ranges;
- system.diagnostics - debug helpers;
- system.globalization - locale support;
- system.io - streams, files, readers, writers
- system.reflection - runtime traits;
- system.resources - Windows resource management;
- system.runtime.interopservices - WinAPI;
- system.runtime.serialization - runtime serialization;
- system.security - general purpose security functions;
- system.security.cryptography - encryption algorithms;
- system.text - text encoding/decoding support
- system.threading - threading support
- system.threading.tasks - task based threading, asynchrony

Another important assembly (lib will be the immediate equivalent 
in D) is the system.dll assembly containing following important 
namespaces:
- system - general purpose types;
- system.codedom - compiler tools. somehow equivalent to ctfe in 
D;
- system.collections.concurrent - shared collections and 
algorithms
- system.componentmodel - particular cases of interfaces, events, 
classes
- system.configuration - user settings, reading registry, ini 
files or various forms of persiostent data;
- system.diagnostics - profiling, performance counters, unit 
testing;
- system.compression - well known compression algorithms and 
associated streams;
- system.io.ports - general purpose port interaction functions;
- system.net - various communication protocols
- system.security - various authentication systems
- system.text.regularexpressions - regex;

This is not a comprehensive list, there are more assemblies in 
BCL(eg: system.core.dll assembly contains also some important 
classes like linq extensions or system.xml.dll which contains all 
xml related stuff), and the namespaces above were shortened (eg: 

namespace for mime encoding).


file system, there are no files or packages, you can code any 
number of classes in the same file or you can even code a class 
in multiple files (partial classes).
Feb 06 2013
prev sibling next sibling parent reply Robert <jfanatiker gmx.at> writes:
On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? How 
 many stupid names are there (eg, "std.algorithm2") which were 


Well in order to avoid such "stupid" names, I think it would be a good
idea to keep the standard library small. Include only the things that
are important to be "standard" in order to ensure interoperability
between libraries.

Instead we should offer a central place for D libraries, like CPAN for
perl or http://pypi.python.org/pypi for python.

The benefits:
- The base installation stays small, which might be important for
everything that is not a PC.
- Versioning is easier, because every library can exist in multiple
versions, and every project can choose to which version it wants to
link. (For backwards compatiblity) -> No stupid names, if the libraries
are small, simply deprecate the complete library.



-> In fact such a central place is already on my todo list. I have the
idea of creating an infrastructure, where you can easily write D
scripts, which may import arbitrary libraries from the repository. A
tool like rdmd would then notice that the libraries are not installed
locally, so it does install them before running the script. (I think of
integration with http://openbuildservice.org/ for Linux based systems
and some custom installer for Windows.)

-> This way we would have a really convenient way for scripting in D and
it would be easy to toy around with an idea, before creating an actual
project.
-> So instead of creating a single monster standard library, just offer
a central place for D libraries and automatic dependency management.

I hope that I find the time soon to implement such a thing, but it will
be a huge amount of work and maybe this is the right occasion to ask,
whether anyone else thinks this would be a good idea.

Just some of my recent thoughts.

Best regards,

Robert
Feb 06 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-06 13:02, Robert wrote:

 On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? How
 many stupid names are there (eg, "std.algorithm2") which were


 Well in order to avoid such "stupid" names, I think it would be a good
 idea to keep the standard library small. Include only the things that
 are important to be "standard" in order to ensure interoperability
 between libraries.

 Instead we should offer a central place for D libraries, like CPAN for
 perl or http://pypi.python.org/pypi for python.


Here you go:

https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

I will replace the Ruby code with D.

-- 
/Jacob Carlborg
Feb 06 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/13 8:42 AM, Jacob Carlborg wrote:

 On 2013-02-06 13:02, Robert wrote:

 On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? How
 many stupid names are there (eg, "std.algorithm2") which were


 Well in order to avoid such "stupid" names, I think it would be a good
 idea to keep the standard library small. Include only the things that
 are important to be "standard" in order to ensure interoperability
 between libraries.

 Instead we should offer a central place for D libraries, like CPAN for
 perl or http://pypi.python.org/pypi for python.


 Here you go:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 I will replace the Ruby code with D.


That's awesome. I skimmed the docs and the capabilities are pretty much 
what I'd expect. We should push for integrating Orbit in the standard 
distro. That entails a formal proposal, community review and voting etc. 
etc.

One question - since you mention replacing Ruby with D, I'm thinking it 
may be a good opportunity to just use D syntax everywhere. The arguments 
have been hashed already - a D user is supposed to know of all syntaxes 
that of D itself etc.

As an example I was looking at 
https://github.com/jacob-carlborg/orbit/wiki/orbfile which has:




basically any comparison operator is allowed here

"0.x.y", i.e. won't use any "1.x.y"     version

Nice enough but there's no shame about

orb("dwt", "0.5.3"); // specifies an exact version
orb("sqlite"); // uses the latest version
orb("orange", "> 0.0.1"); // uses any version greater than "0.0.1", 
basically any comparison operator is allowed here
orb("derelict", "~> 0.3.4"); // uses any version above "0.3.4" that is 
"0.x.y", i.e. won't use any "1.x.y"     version

The alternative orbspec 
https://github.com/jacob-carlborg/orbit/wiki/orbspec%20specification 
also has its own syntax:



version "1.0.0"
author "Jacob Carlborg"
type :library


This is in a way worse because there's a tidbit of syntax for each 
element. I'm thinking:

spec = "orange";
version = "1.0.0";
author = "Jacob Carlborg";
type = "library";
files = ["a.d", "b.di"]; // an array of source files

This code can be imported into an environment that has the appropriate 
definitions.

Hash table literals may also be also very useful.

Thoughts?


Andrei
Feb 06 2013
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-06 15:03, Andrei Alexandrescu wrote:


 That's awesome. I skimmed the docs and the capabilities are pretty much
 what I'd expect. We should push for integrating Orbit in the standard
 distro. That entails a formal proposal, community review and voting etc.
 etc.


This tool has a couple of dependencies on my own libraries (available of 
course) and Tango. I'm not feeling so enthusiastic to remove these 
dependencies.

https://github.com/jacob-carlborg/dstack
https://github.com/jacob-carlborg/mambo


 One question - since you mention replacing Ruby with D, I'm thinking it
 may be a good opportunity to just use D syntax everywhere. The arguments
 have been hashed already - a D user is supposed to know of all syntaxes
 that of D itself etc.


That was what I was referring to.


 Thoughts?


I was thinking having basically the same syntax. One syntax I'm not sure 
what to do about is the hash literal syntax Ruby uses.

orb "dwt", git: "git://github.com/jacob-carlborg/dwt.git"

https://github.com/jacob-carlborg/orbit/wiki/Integration

The above could be translated to:

orb("dwt", ["git" : "git://github.com/jacob-carlborg/dwt.git");

That doesn't look very nice.

Also I might need to be able pass values of different types in the same 
hash. Sure, Variant can be used in that case but that would look even 
more uglier.

-- 
/Jacob Carlborg
Feb 06 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/13 11:09 AM, Jacob Carlborg wrote:

 On 2013-02-06 15:03, Andrei Alexandrescu wrote:


 That's awesome. I skimmed the docs and the capabilities are pretty much
 what I'd expect. We should push for integrating Orbit in the standard
 distro. That entails a formal proposal, community review and voting etc.
 etc.


 This tool has a couple of dependencies on my own libraries (available of
 course) and Tango. I'm not feeling so enthusiastic to remove these
 dependencies.

 https://github.com/jacob-carlborg/dstack
 https://github.com/jacob-carlborg/mambo


I understand. The way I see this is as an motivation and opportunity to 
add the necessary functionality to Phobos. I may be uninformed, but the 
way I see it basic package management doesn't have to be very demanding 
on library functionality.


 One question - since you mention replacing Ruby with D, I'm thinking it
 may be a good opportunity to just use D syntax everywhere. The arguments
 have been hashed already - a D user is supposed to know of all syntaxes
 that of D itself etc.


 That was what I was referring to.


 Thoughts?


 I was thinking having basically the same syntax. One syntax I'm not sure
 what to do about is the hash literal syntax Ruby uses.

 orb "dwt", git: "git://github.com/jacob-carlborg/dwt.git"

 https://github.com/jacob-carlborg/orbit/wiki/Integration

 The above could be translated to:

 orb("dwt", ["git" : "git://github.com/jacob-carlborg/dwt.git");

 That doesn't look very nice.

 Also I might need to be able pass values of different types in the same
 hash. Sure, Variant can be used in that case but that would look even
 more uglier.


I'd say go with strings throughout. For the package manager 
"compile-time" is pretty much same as "run-time" so no need for early 
typechecking.


Andrei
Feb 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-06 18:22, Andrei Alexandrescu wrote:


 I understand. The way I see this is as an motivation and opportunity to
 add the necessary functionality to Phobos. I may be uninformed, but the
 way I see it basic package management doesn't have to be very demanding
 on library functionality.


No it doesn't need to have a demanding library functionality. But I have 
creating these libraries because they contain common code I use in 
several of my tools and libraries. I think duplicating that code just to 
avoid having any library dependencies would be quite foolish.

The reason why I use Tango is that it contains functionality that is 
either missing or not good enough in Phobos. Also, remember that I 
started this project several years ago.

-- 
/Jacob Carlborg
Feb 07 2013
parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Thursday, February 07, 2013 13:17:20 Jacob Carlborg wrote:

 On 2013-02-06 18:22, Andrei Alexandrescu wrote:

 I understand. The way I see this is as an motivation and opportunity to
 add the necessary functionality to Phobos. I may be uninformed, but the
 way I see it basic package management doesn't have to be very demanding
 on library functionality.


 
 No it doesn't need to have a demanding library functionality. But I have
 creating these libraries because they contain common code I use in
 several of my tools and libraries. I think duplicating that code just to
 avoid having any library dependencies would be quite foolish.
 
 The reason why I use Tango is that it contains functionality that is
 either missing or not good enough in Phobos. Also, remember that I
 started this project several years ago.


All of which is fine as long as it's a 3rd party project. I think that Andrei's 
point is that if you have any interest in it being D's official package
manager, 
it needs to be changed to use Phobos only, even if that requires adding or 
improving functionality in Phobos. An official package manager can't rely on
3rd 
party libraries or tools.

So, either Orbit needs to be updated to not depend on anything beyond D's 
standard library, or it should stop being presented as a possible official 
package manager. That's not saying anything bad about the project or the work 
that you've done. It's just saying that it does not currently fit the 
requirements for an official project.

- Jonathan M Davis
Feb 07 2013
prev sibling parent reply Jens Mueller <jens.k.mueller gmx.de> writes:
Jacob Carlborg wrote:

 On 2013-02-06 15:03, Andrei Alexandrescu wrote:

Thoughts?


 
 I was thinking having basically the same syntax. One syntax I'm not
 sure what to do about is the hash literal syntax Ruby uses.
 
 orb "dwt", git: "git://github.com/jacob-carlborg/dwt.git"
 
 https://github.com/jacob-carlborg/orbit/wiki/Integration
 
 The above could be translated to:
 
 orb("dwt", ["git" : "git://github.com/jacob-carlborg/dwt.git");
 
 That doesn't look very nice.


How about using YAML/JSON?
name: dwt
source: git://github.com/jacob-carlborg/dwt.git

Jens
Feb 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-06 18:35, Jens Mueller wrote:


 How about using YAML/JSON?
 name: dwt
 source: git://github.com/jacob-carlborg/dwt.git


The reason is that it will come a need for having conditions, variables 
and similar in the YAML/JSON file. So instead of creating a new format, 
or extending YAML/JSON with conditions and similar it's just easier to 
use a fully fledged language instead.

DSSS is a perfect example of that it's a bad idea to invent a new 
format. DSSS build scripts are basically an extended variant INI files. 
It do support some kind of conditions:

[main.d]
buildflags += -L-L.
version (Windows) {

}

It just happens that if you but the first brace on its own line, or 
skips the space between the closing parenthesis and the opening brace it 
can't parse the file. That's that things I like to avoid by using a 
fully fledged language.

That's way I originally chose Ruby, because it can look just like JSON:

foo = {
   a: 3,
   b: 4
}

With Ruby you can get a syntax that is declarative, just like JSON/YAML.

-- 
/Jacob Carlborg
Feb 07 2013
parent FG <home fgda.pl> writes:
On 2013-02-07 09:19, Jacob Carlborg wrote:

 That's way I originally chose Ruby, because it can look just like JSON:

 foo = {
    a: 3,
    b: 4
 }


This looks like JSON but doesn't look like Ruby. :)
Feb 07 2013
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2013 6:03 AM, Andrei Alexandrescu wrote:

 This is in a way worse because there's a tidbit of syntax for each element. I'm
 thinking:

 spec = "orange";
 version = "1.0.0";
 author = "Jacob Carlborg";
 type = "library";
 files = ["a.d", "b.di"]; // an array of source files


That's close enough to JSON to suggest: why not use JSON syntax?
Feb 06 2013
next sibling parent reply "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Wednesday, 6 February 2013 at 21:10:06 UTC, Walter Bright 
wrote:

 On 2/6/2013 6:03 AM, Andrei Alexandrescu wrote:

 This is in a way worse because there's a tidbit of syntax for 
 each element. I'm
 thinking:

 spec = "orange";
 version = "1.0.0";
 author = "Jacob Carlborg";
 type = "library";
 files = ["a.d", "b.di"]; // an array of source files


 That's close enough to JSON to suggest: why not use JSON syntax?


There is a competing project doing just that:

https://github.com/rejectedsoftware/dub

(doesn't look like it's been posted yet?)
Feb 06 2013
next sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 6 February 2013 at 21:16:01 UTC, Jakob Ovrum wrote:

 On Wednesday, 6 February 2013 at 21:10:06 UTC, Walter Bright 
 wrote:

 On 2/6/2013 6:03 AM, Andrei Alexandrescu wrote:

 This is in a way worse because there's a tidbit of syntax for 
 each element. I'm
 thinking:

 spec = "orange";
 version = "1.0.0";
 author = "Jacob Carlborg";
 type = "library";
 files = ["a.d", "b.di"]; // an array of source files


 That's close enough to JSON to suggest: why not use JSON 
 syntax?


 There is a competing project doing just that:

 https://github.com/rejectedsoftware/dub

 (doesn't look like it's been posted yet?)


I asked Sönke about possibly proposing this as the official D 
package manager awhile back when Dub was called VPM. You can find 
his answer here:

http://news.rejectedsoftware.com/groups/rejectedsoftware.vibed/thread/7/

BA
Feb 06 2013
parent reply "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 6 February 2013 at 21:55:53 UTC, Brad Anderson 
wrote:

 On Wednesday, 6 February 2013 at 21:16:01 UTC, Jakob Ovrum 
 wrote:

 On Wednesday, 6 February 2013 at 21:10:06 UTC, Walter Bright 
 wrote:

 On 2/6/2013 6:03 AM, Andrei Alexandrescu wrote:

 This is in a way worse because there's a tidbit of syntax 
 for each element. I'm
 thinking:

 spec = "orange";
 version = "1.0.0";
 author = "Jacob Carlborg";
 type = "library";
 files = ["a.d", "b.di"]; // an array of source files


 That's close enough to JSON to suggest: why not use JSON 
 syntax?


 There is a competing project doing just that:

 https://github.com/rejectedsoftware/dub

 (doesn't look like it's been posted yet?)


 I asked Sönke about possibly proposing this as the official D 
 package manager awhile back when Dub was called VPM. You can 
 find his answer here:

 http://news.rejectedsoftware.com/groups/rejectedsoftware.vibed/thread/7/

 BA


Reading more, it looks like Dub is his attempt to make VPM less 
Vibe.d specific and more for general D use.  It even generates 
VisualD project files.

Cool stuff.

BA
Feb 06 2013
parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Wednesday, 6 February 2013 at 22:25:03 UTC, Brad Anderson 
wrote:

 Reading more, it looks like Dub is his attempt to make VPM less 
 Vibe.d specific and more for general D use.  It even generates 
 VisualD project files.

 Cool stuff.

 BA


Yeah, I'm leaning towards DUB for a few reasons.

The implementation is written in idiomatic D2; it's much more 
likely to attract contributors this way.

The online package registry is already up and is entirely powered 
by vibe.d and its ecosystem: we're eating our own dog food 
throughout.

Also, using a data format (JSON, in this case) over a scripting 
language has its advantages. With DUB, there is only one piece of 
software you need to trust (which is DUB itself), as the package 
description file is not software. It's possible to do things like 
automatically building packages without having to vet a build 
script's quality. I'm not sure how sound this point really is, 
but it feels right (for what little it's worth...).
Feb 07 2013
prev sibling parent Robert <jfanatiker gmx.at> writes:
Ha! Great, I missed that. I thought it was vibe.d specific.

Thanks!
On Wed, 2013-02-06 at 22:15 +0100, Jakob Ovrum wrote:

 There is a competing project doing just that:
 
 https://github.com/rejectedsoftware/dub
Feb 06 2013
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 6 February 2013 at 21:10:06 UTC, Walter Bright 
wrote:

 On 2/6/2013 6:03 AM, Andrei Alexandrescu wrote:

 This is in a way worse because there's a tidbit of syntax for 
 each element. I'm
 thinking:

 spec = "orange";
 version = "1.0.0";
 author = "Jacob Carlborg";
 type = "library";
 files = ["a.d", "b.di"]; // an array of source files


 That's close enough to JSON to suggest: why not use JSON syntax?


Well if JSON syntax is better than D's why D don' use JSON syntax 
in the first place ?
Feb 06 2013
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Thu, 07 Feb 2013 04:02:29 +0100
schrieb "deadalnix" <deadalnix gmail.com>:


 Well if JSON syntax is better than D's why D don' use JSON syntax 
 in the first place ?


JSON is a well known format. The benefit is, parsers exist for
every practical language and developers know by heart the
syntax and can start editing. And mixing a data definition
language with a programming language doesn't work here. Maybe
if this was the LISP news group... :)

-- 
Marco
Feb 07 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 22:09, Walter Bright wrote:


 That's close enough to JSON to suggest: why not use JSON syntax?


See: 
http://forum.dlang.org/thread/ugmacrokqghrrwpfovam forum.dlang.org?page=4#post-kevo37:241gcn:241:40digitalmars.com

-- 
/Jacob Carlborg
Feb 07 2013
prev sibling next sibling parent reply Robert <jfanatiker gmx.at> writes:
On Wed, 2013-02-06 at 14:42 +0100, Jacob Carlborg wrote:

 Here you go:
 
 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D


:-) That is pretty much what I had in mind! Awesome! I would love to
help with this. I would start with some writing down of my ideas and
concepts I have in mind, if you are interested? If you are, where should
we discuss such things? On this mailing list?

 
 I will replace the Ruby code with D.


As the ultimate goal of my idea was to establish D as a compiled
language with all/most of the benefits of interpreted languages, it
would seem strange if the tool that made this possible was not written
in D. It would suggest that this was not possible in the first place. So
I think this is a good idea.

Concise definitions could be made possible, either by importing stuff in
an appropriate environment as Andrei suggested or if needed even via a
DSL, with string mixin's. So I think D already offers all we need, for
good concise configuration files?


Best regards,

Robert
Feb 06 2013
parent reply Jacob Carlborg <doob me.com> writes:
On 2013-02-06 16:41, Robert wrote:


 :-) That is pretty much what I had in mind! Awesome! I would love to
 help with this. I would start with some writing down of my ideas and
 concepts I have in mind, if you are interested? If you are, where should
 we discuss such things? On this mailing list?


Yes, I would be interested. I don't know what would be the best 
communication channel.


 I will replace the Ruby code with D.


 As the ultimate goal of my idea was to establish D as a compiled
 language with all/most of the benefits of interpreted languages, it
 would seem strange if the tool that made this possible was not written
 in D. It would suggest that this was not possible in the first place. So
 I think this is a good idea.

 Concise definitions could be made possible, either by importing stuff in
 an appropriate environment as Andrei suggested or if needed even via a
 DSL, with string mixin's. So I think D already offers all we need, for
 good concise configuration files?


As I said, I will replace Ruby with D, that includes the DSL. It's the 
only place where Ruby is used. The rest is D.

-- 
/Jacob Carlborg
Feb 07 2013
next sibling parent Robert <jfanatiker gmx.at> writes:
As Sönke Ludwig seems to work on something very similar, it might make
sense to join forces with him too. I will look into both implementations
next week, to see where we stand.


 
 As I said, I will replace Ruby with D, that includes the DSL. It's the 
 only place where Ruby is used. The rest is D.
Feb 07 2013
prev sibling parent Robert <jfanatiker gmx.at> writes:
On Thu, 2013-02-07 at 09:45 +0100, Jacob Carlborg wrote:

 Yes, I would be interested. I don't know what would be the best 
 communication channel.


Great! :-) I think for now, maybe this mailing list is not the worst
place. (At least I don't know a better one)

Best regards,

Robert
Feb 07 2013
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
06-Feb-2013 17:42, Jacob Carlborg пишет:

 On 2013-02-06 13:02, Robert wrote:

 On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? How
 many stupid names are there (eg, "std.algorithm2") which were


 Well in order to avoid such "stupid" names, I think it would be a good
 idea to keep the standard library small. Include only the things that
 are important to be "standard" in order to ensure interoperability
 between libraries.

 Instead we should offer a central place for D libraries, like CPAN for
 perl or http://pypi.python.org/pypi for python.


 Here you go:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D


I've been wondering what happened to it, looks fine I think.


 I will replace the Ruby code with D.


Great!

-- 
Dmitry Olshansky
Feb 06 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 18:15, Dmitry Olshansky wrote:


 I've been wondering what happened to it, looks fine I think.


I've been working on other projects for a while but now I've started to 
work on it again.

-- 
/Jacob Carlborg
Feb 07 2013
prev sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 6 February 2013 at 13:42:36 UTC, Jacob Carlborg 
wrote:

 On 2013-02-06 13:02, Robert wrote:

 On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? 
 How
 many stupid names are there (eg, "std.algorithm2") which were


 Well in order to avoid such "stupid" names, I think it would 
 be a good
 idea to keep the standard library small. Include only the 
 things that
 are important to be "standard" in order to ensure 
 interoperability
 between libraries.

 Instead we should offer a central place for D libraries, like 
 CPAN for
 perl or http://pypi.python.org/pypi for python.


 Here you go:

 https://github.com/jacob-carlborg/orbit/wiki/Orbit-Package-Manager-for-D

 I will replace the Ruby code with D.


I have a dream that someday phobos will just be a metapackage in 
a package manager that installs a set of core modules.

What is needed to get Orbit off the ground (Kepler's law, I 
guess, would be the joke answer)?

BA
Feb 06 2013
next sibling parent reply "David Nadlinger" <see klickverbot.at> writes:
On Wednesday, 6 February 2013 at 18:56:49 UTC, Brad Anderson 
wrote:

 What is needed to get Orbit off the ground (Kepler's law, I 
 guess, would be the joke answer)?


Somebody actually working on it.

No offense, Jacob, but to me it seems like you have quite a few 
interesting projects (DVM, DStep, Orbit, …), and although you are 
usually quick to advertise them here, it seems that none of them 
is quite ready for prime time.

Maybe concentrating your efforts on one of them would be a good 
idea? Otherwise, you'll inevitably leave behind a trail of broken 
expectations as you cannot realistically expect to support all of 
them at the same time…

David
Feb 06 2013
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/13 4:24 PM, David Nadlinger wrote:

 On Wednesday, 6 February 2013 at 18:56:49 UTC, Brad Anderson wrote:

 What is needed to get Orbit off the ground (Kepler's law, I guess,
 would be the joke answer)?


 Somebody actually working on it.

 No offense, Jacob, but to me it seems like you have quite a few
 interesting projects (DVM, DStep, Orbit, …), and although you are
 usually quick to advertise them here, it seems that none of them is
 quite ready for prime time.

 Maybe concentrating your efforts on one of them would be a good idea?
 Otherwise, you'll inevitably leave behind a trail of broken expectations
 as you cannot realistically expect to support all of them at the same time…

 David


I think one demotivating factor for both Jacob and the loosely-defined 
core D team is that we quickly reach irreducible positions on 
fundamental matter. It took literally two years for Jacob to decide to 
do away with Ruby. (Two years for real: 
https://github.com/jacob-carlborg/orbit/issues/1) Now we have 
dependencies on https://github.com/jacob-carlborg/dstack and 
https://github.com/jacob-carlborg/mambo. I took a look. This codebase is 
entirely neat and I'm glad a combination of author's talent and D makes 
the code so easy on the eyes, but there's no way we can ever make this 
part of the official distribution. It's parallel to the canon and some 
of it dedicates hundreds of lines to duplicating Phobos functionality 
with minor distinctions in functionality.

I'm looking at e.g. 
https://github.com/jacob-carlborg/mambo/blob/master/mambo/arguments/Arguments.d 
which I bet money does some cool things with command-line arguments, but 
I see no reason for using it over std.getopt. Similarly 
https://github.com/jacob-carlborg/mambo/blob/master/mambo/arguments/Formatter.d 
is yet another std.format. Then, 
https://github.com/jacob-carlborg/mambo/blob/master/mambo/core/Array.d 
seems to be in the midst of a retrofitting to use std.algorithm as much 
of what it does now is do algorithm-y work with slightly changed names.

Again, this is all fine and I'm not criticizing. If the purpose is to 
build a package manager for one's own use and to amass a community of 
users around it, great. But if the purpose is to make Orbit _the_ 
official package manager of the D programming language, there's no 
equivalent but distinct libraries for command-line arguments, no 
beginsWith that does what startsWith does, etc. etc. etc.

There are three ways I see out of this: (a) Jacob goes full bore and 
ports Orbit to Phobos, and proposes the generally useful parts of his 
library for inclusion in Phobos; (b) Jacob agrees for someone else to do 
a clean room implementation of his package manager design; (c) we agree 
with the status quo with the understanding there's no more dangling of 
Orbit as a possible official package manager.


Andrei
Feb 06 2013
parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 23:00, Andrei Alexandrescu wrote:

I'll start with some top posting and summarize. I use what works. If 
that means using some third party library I have no problem with that. 
Why should I reimplement something and hope for inclusion into Phobos 
when there already exist a perfectly good and working implementation.


 I think one demotivating factor for both Jacob and the loosely-defined
 core D team is that we quickly reach irreducible positions on
 fundamental matter. It took literally two years for Jacob to decide to
 do away with Ruby. (Two years for real:
 https://github.com/jacob-carlborg/orbit/issues/1)


I don't understand the hate about Ruby. People have just suggested to 
use JSON in this thread. Ruby can use the same syntax:

foo = {
   a: 3,
   b: 4
}

Now we have

 dependencies on https://github.com/jacob-carlborg/dstack and
 https://github.com/jacob-carlborg/mambo. I took a look. This codebase is
 entirely neat and I'm glad a combination of author's talent and D makes
 the code so easy on the eyes, but there's no way we can ever make this
 part of the official distribution. It's parallel to the canon and some
 of it dedicates hundreds of lines to duplicating Phobos functionality
 with minor distinctions in functionality.


Some of these modules go back to 2007, back when I was using D1. Some 
are just different names for functions in Phobos which I understand what 
you're not thrilled about including. Some are/were missing functions 
that I included in my own library instead of contributing to Phobos. I 
don't have time to wait five months for hoping something is included 
into Phobos.


 I'm looking at e.g.
 https://github.com/jacob-carlborg/mambo/blob/master/mambo/arguments/Arguments.d
 which I bet money does some cool things with command-line arguments, but
 I see no reason for using it over std.getopt.


This is a wrapper around the arguments parser in Tango which and some 
syntax sugar. It also helps integrating Formatter, mentioned below.

Similarly

 https://github.com/jacob-carlborg/mambo/blob/master/mambo/arguments/Formatter.d
 is yet another std.format. Then,
 https://github.com/jacob-carlborg/mambo/blob/master/mambo/core/Array.d
 seems to be in the midst of a retrofitting to use std.algorithm as much
 of what it does now is do algorithm-y work with slightly changed names.


Since you wrote that comment I assume you haven't even looked at it, 
just the name. This module is specifically designed to automatically 
generate a usage/help text from the Arguments module above. I can see no 
such functionality in std.format, am I wrong?


 Again, this is all fine and I'm not criticizing. If the purpose is to
 build a package manager for one's own use and to amass a community of
 users around it, great. But if the purpose is to make Orbit _the_
 official package manager of the D programming language, there's no
 equivalent but distinct libraries for command-line arguments, no
 beginsWith that does what startsWith does, etc. etc. etc.


Yes, this contains new names for functions already available in 
std.algorithm. It also contains some tweaks here and there, convenient 
functions and new functionality that is missing from Phobos.


 There are three ways I see out of this: (a) Jacob goes full bore and
 ports Orbit to Phobos, and proposes the generally useful parts of his
 library for inclusion in Phobos; (b) Jacob agrees for someone else to do
 a clean room implementation of his package manager design; (c) we agree
 with the status quo with the understanding there's no more dangling of
 Orbit as a possible official package manager.


Doing some renames and similar is one thing but I have no intention of 
reimplementing a lot of code that is working perfectly fine and hoping 
for a inclusion into Phobos that most likely won't happen.

I've tried to get a review of my serialization library for several 
years, sure it was a form or pre-review with the promise of removing any 
Tango and D1 related code if it was accepted. Nobody was interested. 
Sure I got the usual comments, I got real feedback from a single person 
here, but no formal review. Thank god I didn't remove the Tango and D1 
stuff, that would be been a lot of unnecessary work and wasted time. I 
tried to get a review several times, but I've lost interested now, since 
no one else was interested.

I'm also using a high level testing framework called Cucumber (uses 
Ruby). I'm 99% sure that you have no interest in this. It's a perfect 
framework to do this kind of testing I'm using it for. It's already 
available, working good and are supported by editors. Running tests, 
syntax highlighting and so on. Why should I give up that?

I'm also not a fan of putting unit tests inline. I prefer to have a 
completely separate directory structure for my tests. My unit tests also 
uses a simple unit testing library that allows to continue after a 
assertion failure and gives summary at the end if any tests failed. I'm 
pretty sure you don't want this either. It's already there, it's working 
fine. Why should I give up that?

I've compiled a list below of things used in Orbit (directly or 
indirectly) that Phobos is lacking in. Please tell me if I'm wrong:

* DStack - Is a library for helping structuring applications. It includes:
   * Handling of command line arguments
   * Components
   * Commands
   * Starting and initializing of the application
   * Configuration
I plan to add more stuff here when needed

* Serialization - Does not exist
* ConfigMap -  Does not exist. An associative array with opDispatch 
added to it. Allows to do things like:

auto config = ConfigMap();
config.foo = "asd";
config.bar.foo.baz = "foobar";

assert(config.foo == "asd");

* XML - The XML module is slow and has a cumbersome API

* Zip - I don't exactly remember what was wrong here. It either didn't 
work or I had to reimplement a lot of functionality already available in 
Tango.

* Net - std.curl is probably a good module but it wasn't available when 
I started. I also adds another dependency.

* std.process - I have not used it myself but I've heard it's not optimal

* std.getopt - Doesn't support the following:
   * Required arguments
   * Restricting the values of a given argument
   * No way to automatically create a help/usage list out of the arguments
   * Handling positional arguments
   * No support for commands/action. That is "git commit", "commit" 
would be the command/action
   * Handle multiple command lines
   * Validation of the arguments

Various convince functions:

* any - Opposite of empty
* last - Returns the last element of an array
* map, find and any for associative arrays
* Utility functions for handling UDA's

* isBlank - Check if a value if blank. If it's a string it's blank if 
it's only contains whitespace. If it you can call .empty return that. If 
it's a value type it's not empty. If it's a reference type and it's null 
it's empty

* isPresent - The opposite of isBlank

* pluralize - Takes a string and a count. If the count is greater than 1 
it converts the word in the string to plural

CTFE:

* format - A simple formatting function
* indexOf
* contains
* hasField - Returns true if the given type has the given field
* fieldsOf - Returns all the fields of the given type
* TypeOfField - Returns the type of a field
* nameOfFieldAt - Returns the name of the field at the given position
* set/getValueOfField - Sets/gets the value of a field
* newInstance - Returns a new instance based on the class name or 
ClassInfo. Can handle any class regardless of constructors

* A couple of traits

In the end it's all about what's working.

-- 
/Jacob Carlborg
Feb 07 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 22:24, David Nadlinger wrote:


 Somebody actually working on it.

 No offense, Jacob, but to me it seems like you have quite a few
 interesting projects (DVM, DStep, Orbit, …), and although you are
 usually quick to advertise them here, it seems that none of them is
 quite ready for prime time.


That's very true, that I'm working on several projects. Saying that none 
of them are ready for prime time is not really fair.

DVM - Has seen several releases and does what I intended it to do.
DStep - Has seen one release. It could use more work but it basically 
does what it should.
Orange - Has only seen one proper release. Could be faster and support 
more archives but this does what it's intended to do
Orbit - This is not finished yet. But many things do work.


 Maybe concentrating your efforts on one of them would be a good idea?
 Otherwise, you'll inevitably leave behind a trail of broken expectations
 as you cannot realistically expect to support all of them at the same time…


The problem is that I need all these tools and libraries. If one wants 
something done it's best to do it yourself.

Also it can be quite tiresome to work on the same project all the time. 
Especially if you get stuck, then it can be good to have something else 
to work on for a while.

-- 
/Jacob Carlborg
Feb 07 2013
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2013-02-06 19:56, Brad Anderson wrote:


 I have a dream that someday phobos will just be a metapackage in a
 package manager that installs a set of core modules.

 What is needed to get Orbit off the ground (Kepler's law, I guess, would
 be the joke answer)?


As with most things, it's time.

I have basically an hour per day to work on D. On weekends I can work 
more on D.

Quite a lot of that time goes to reading these newsgroups. I'm also not 
religiously working every one of these hours or being 100% focused.

-- 
/Jacob Carlborg
Feb 07 2013
prev sibling parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 06 Feb 2013 13:02:29 +0100
schrieb Robert <jfanatiker gmx.at>:


 On Wed, 2013-02-06 at 08:56 +0100, Don wrote:

 Eg, are there entire top-level branches which are obsolete? How 
 many stupid names are there (eg, "std.algorithm2") which were 


 
 Well in order to avoid such "stupid" names, I think it would be a good
 idea to keep the standard library small. Include only the things that
 are important to be "standard" in order to ensure interoperability
 between libraries.


I think the opposite. When we think about more concepts in
Phobos, like gui, serialization, cryptography and so on, we
will be able to create the right amount of hierarchy with
distinguishable names. I'm in favor of a big batteries
included library.


 The benefits:
 - The base installation stays small, which might be important for
   everything that is not a PC.


That's why we have modules and minimize dependencies between
them. Years ago when Java was introduced to mobile phones the
big library was no problem, because non-essential parts could
be removed.

-- 
Marco
Feb 06 2013
parent Robert <jfanatiker gmx.at> writes:
 I think the opposite. When we think about more concepts in
 Phobos, like gui, serialization, cryptography and so on, we
 will be able to create the right amount of hierarchy with
 distinguishable names. I'm in favor of a big batteries
 included library.


Yeah, but I still think that breaking independent parts up in
independent projects, will make the whole thing more manageable.

If some of these projects happen to be within the std package and be
installed by default, that is fine.

Best regards,

Robert
Feb 07 2013
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On 02/06/2013 08:56 AM, Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to have reached a
 consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small like the C
 standard library).
 (b) The current flat structure of Phobos (every module in the root) does not
 scale to hundreds of modules.


Possibly naive question -- assuming that one has a package of multiple modules, 
how does one go about importing the whole package?  Seems relevant if Phobos is 
to become nested.
Feb 06 2013
parent "bearophile" <bearophileHUGS lycos.com> writes:
Joseph Rushton Wakeling:


 Possibly naive question -- assuming that one has a package of 
 multiple modules, how does one go about importing the whole 
 package?  Seems relevant if Phobos is to become nested.


I think implementing this:
http://d.puremagic.com/issues/show_bug.cgi?id=3603

Bye,
bearophile
Feb 06 2013
prev sibling next sibling parent "Jonathan M Davis" <jmdavisProg gmx.com> writes:
On Wednesday, February 06, 2013 15:12:59 Joseph Rushton Wakeling wrote:

 On 02/06/2013 08:56 AM, Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to have reached a
 consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small like the C
 standard library).
 (b) The current flat structure of Phobos (every module in the root) does
 not scale to hundreds of modules.


 
 Possibly naive question -- assuming that one has a package of multiple
 modules, how does one go about importing the whole package? Seems relevant
 if Phobos is to become nested.


There are a couple of DIPS which relate to that (15 and 16 IIRC), but I would 
point out that in languages which _do_ permit that (e.g. Java), it's fairly 
common to consider it bad practice and insist on importing every module 
explicitly.

- Jonathan M Davis
Feb 06 2013
prev sibling next sibling parent reply Chad Joan <chadjoan gmail.com> writes:
On 02/06/2013 02:56 AM, Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to have reached
 a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small like the C
 standard library).
 (b) The current flat structure of Phobos (every module in the root) does
 not scale to hundreds of modules.

 It's not quite unanimous on (a), but seems to be close enough.

 Together, (a) and (b) mean we need a plan. The sooner we can do it, the
 less painful it will be. But, on the other hand, it's something that we
 really don't want to get wrong.

 ...


You ask for a lot of thorough research that I don't have time to provide 
at the moment.

I just want to say "PLEASE YES".  Phobos' module hierarchy is currently 
confusing at best, especially with various important elements being 
scattered across std.range, std.algorithm, std.array, and maybe another 
I'm forgetting.  Although I can sort of see the logic behind this 
convention, it is very hard to think of a function that operates on 
ranges and automatically know which module to look in.  I'm rambling, sorry.

PLEASE YES!
Feb 06 2013
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/6/2013 8:06 PM, Chad Joan wrote:

 Although I can sort of see the logic behind this convention, it is very hard to
 think of a function that operates on ranges and automatically know which module
 to look in.


Unfortunately, I don't think any reorg will help with that, as all functions 
that deal with input and output should use ranges.
Feb 07 2013
prev sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 06 Feb 2013 23:06:06 -0500
schrieb Chad Joan <chadjoan gmail.com>:


 I just want to say "PLEASE YES".  Phobos' module hierarchy is currently 
 confusing at best, especially with various important elements being 
 scattered across std.range, std.algorithm, std.array, and maybe another 
 I'm forgetting.  Although I can sort of see the logic behind this 
 convention, it is very hard to think of a function that operates on 
 ranges and automatically know which module to look in.  I'm rambling, sorry.
 
 PLEASE YES!


It is true.
I've had the same discussion on IRC a while ago. After
letting off some steam, I had the idea to make a map of
Phobos by functionality instead of module.

Sometimes simple stuff confuses me: is .empty in std.array or
std.range? And why is it not a standard property of arrays?
The module borders seem washy in std.rangorithray.

String formatting is another example with functions spread
over conv/string/format for: 'xformat' (std.string),
'text' (std.conv) or 'formattedWrite' (std.format).
Add manipulation to the list and the function you look for may
even be generalized in std.algorithm, like 'startsWith'.

String slicing/concatenation: 'stripRight' (std.string),
'startsWith' (std.algorithm), 'join' (std.array)

Text I/O: 'readText' (std.file), 'byLine' (std.stdio)

Temporary files: 'tmpfile' (std.stdio), 'tempDir' (std.file)

Charsets: 'toMBSz' (std.windows.charset),
'transcode' (std.encoding), 'toUTF16' (std.utf)

Memory streams: 'MemoryStream' (std.stream),
'OutBuffer' (std.outbuffer)

... and so on ...

-- 
Marco
Feb 07 2013
prev sibling next sibling parent "Paul D. Anderson" <paul.d.removethis.anderson comcast.andthis.net> writes:
On Wednesday, 6 February 2013 at 07:56:26 UTC, Don wrote:

 In the "Implementing Half Floats in D" thread, we seemed to 
 have reached a consensus on two important points:
 (a) Phobos should have a broad scope (rather than being small 
 like the C standard library).
 (b) The current flat structure of Phobos (every module in the 
 root) does not scale to hundreds of modules.

 It's not quite unanimous on (a), but seems to be close enough.

 Together, (a) and (b) mean we need a plan. The sooner we can do 
 it, the less painful it will be. But, on the other hand, it's 
 something that we really don't want to get wrong.

 Personal taste plays a huge role in this (Practically any 
 structure can work, but we're looking for an arrangement which 
 is intuitive and aesthetically pleasing to as many people as 
 possible). It'll be most productive to stick to uncontroversial 
 facts as long as possible.

 It would be great if people with extensive experience in other 
 languages could give a brief description of the structure of 
 the corresponding libraries. (Also include pseudo-standard 
 libraries, for example boost in C++).

 1. How many total modules are there? How old is the library? 
 What is the recent growth rate of the library (Percentage 
 increase in size per year, over whatever time interval is 
 reasonable for that library)?
 2. How deep is the hierarchy? How many top-level branches are 
 there?
 3. How has the library dealt with obsolete functions and 
 modules? Eg, are there entire top-level branches which are 
 obsolete? How many stupid names are there (eg, 
 "std.algorithm2") which were forced by the original design 
 becoming obsolete?
 4. How much of the library is successful/convincing? To make it 
 a bit less subjective: how much of the library is commonly 
 ignored, in favour of using third-party libraries instead? 
 Which areas are failures in this way (eg, GUI, database, ...)? 
 Which areas are the most often praised?

 Maybe such a survey already exists, but I haven't been able to 
 find an existing one.


Java Standard Library:

Total of 209 packages.

3 top-level packages:
   java  ( 56 packages, 13 top level within java)
   javax (117 packages, 20 top level within javax)
   org   ( 36 packages,  4 top-level within org)

Max depth of package tree:

   5

Duplicated Names:

   java.io and java.nio (new io)
   java.net and javax.net
   java.lang and javax.lang

Note that javax was initially intended to be an experimental 
library where its packages would get moved to the java library 
eventually. But the first time they tried to move a package from 
javax to java there were a lot of complaints. Too much broken 
code. So the experimental library become an addition to the 
standard library.

Paul
Feb 07 2013
prev sibling next sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/6/13, Don <don nospam.com> wrote:

 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.


I just want to add that we're still handicapped by D's access
attributes preventing us from moving private functionality into a
separate module.
Feb 07 2013
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 7 February 2013 at 22:56:39 UTC, Andrej Mitrovic 
wrote:

 On 2/6/13, Don <don nospam.com> wrote:

 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.


 I just want to add that we're still handicapped by D's access
 attributes preventing us from moving private functionality into 
 a
 separate module.


That is the whole point of private. If something is imported from 
elsewhere and used, by definition, it isn't private.
Feb 07 2013
prev sibling next sibling parent Jonathan M Davis <jmdavisProg gmx.com> writes:
On Thursday, February 07, 2013 23:56:30 Andrej Mitrovic wrote:

 On 2/6/13, Don <don nospam.com> wrote:

 (b) The current flat structure of Phobos (every module in the
 root) does not scale to hundreds of modules.


 
 I just want to add that we're still handicapped by D's access
 attributes preventing us from moving private functionality into a
 separate module.


And why is this a handicap? They're private, not part of the API, so they 
should totally irrelevant to how modules are laid out.

- Jonathan M Davis
Feb 07 2013
prev sibling parent reply Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/8/13, deadalnix <deadalnix gmail.com> wrote:

 That is the whole point of private. If something is imported from
 elsewhere and used, by definition, it isn't private.


On 2/8/13, Jonathan M Davis <jmdavisProg gmx.com> wrote:

 And why is this a handicap? They're private, not part of the API, so they
 should totally irrelevant to how modules are laid out.


Have you ever tried to used a project layout where you implement
private (e.g. OS-specific) functionality into their own subpackages,
which should only be accessible to the library and not the user?
Because this is impossible to do right now. IOW if you have:

./std/algorithm.d
./std/range.d

Say that both modules require some private internal template called
"MagicTemplate".

Personally I find it ideal to use internal packages for this, e.g.:

./std/algorithm.d
./std/range.d
./std/internal/templates.d  -- contains MagicTemplate, imported by
algorithm and range

But this forces you to make MagicTemplate public, as package access
won't work in this case.

Maybe "handicapped" was the wrong word for this. You could put the
template in algorithm.d or range.d and have one module import the
other (which has its own problems, e.g. with module constructors), or
you could put the internal.d module in the std package and set package
access to MagicTemplate:

./std/algorithm.d
./std/range.d
./std/internal.d  -- contains MagicTemplate with package access

But I prefer to have internal modules in subpackages, which is
impossible to do right now in D due to how access specifiers work
(i.e. everything ends up being public). Anyway it's not a blocker but
it's worth thinking about.
Feb 08 2013
next sibling parent "Dicebot" <m.strashun gmail.com> writes:
On Friday, 8 February 2013 at 13:14:09 UTC, Andrej Mitrovic wrote:

 ...


As far as I can see it, this will be automatically settled once 
package import DIP is done. Because then you can have hierarchy 
like this:

std/algorithm/_.d
std/algorithm/implementation.d

..and use "package" protection attribute.
Feb 08 2013
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Friday, 8 February 2013 at 13:14:09 UTC, Andrej Mitrovic wrote:

 But I prefer to have internal modules in subpackages, which is
 impossible to do right now in D due to how access specifiers 
 work
 (i.e. everything ends up being public). Anyway it's not a 
 blocker but
 it's worth thinking about.


Considering imports can be private, I don't see it as a big 
problem.

Surely the user won't have any expectations if he chooses to 
explicitly import a module that is clearly labeled internal.
Feb 08 2013