www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Segregating the standard library

reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Something I've been thinking about.

Whilst I think D is making great progress to a fantastic 1.0 d├ębut, the
standard library is somewhat lacking.  Not to knock Walter's efforts in
the slightest, but D's standard library is somewhat lean compared to,
say, Python's standard library.  Let's not even try comparing it to
Java's (arguably the only reason Java even succeeded in the first place.)

As such, I think there's something we need to do before D 1.0: we need
to split the compiler and the standard library.  Specifically:

* We need to make a clear distinction between the absolute bare-minimum
for the compiler and produced programs to function (let's call that the
core lib) and everything else (the standard lib.)

  The core lib would probably contain things like intrinsics, and
anything else necessary (I think that also includes Object, but I'm not
sure *why*...)  The core lib would be compiler-specific, and maintained
under Walter's control.

* The standard library should be split off as a separate project under
community control, with Walter periodically taking stable snapshots to
bundle with the D compiler.

The reason for this is that I'm worried people will look at D 1.0 The
Language and say "great steaming gouts of magma on a beeline for the
orphanage; forget C!", but then look at D 1.0 The Standard Library and
say "well, that's somewhat of an anticlimax."

Doing this, we can say: "here is the shiny new D 1.0 compiler.  Yeah,
the standard library is a little on the skinny side, but it's being
actively worked on, and you can update any time you like."

A few more random thoughts on this:

* I know Walter has said before that if anyone wants to contribute to
Phobos, that he's more than happy to take patches.  The problem is that
the website doesn't say this, and it doesn't have that air of "all
welcome to join in" feel that, say, dsource projects do.  Having a
public svn server and development forums/mailing list dedicated to the
standard library would probably help.

* Do we use Ares?  I suppose that's up to Walter; his choices are to
either bless Ares as what he will be distributing as "standard" with D,
or have a separate Phobos project.

* Irrespective, I think it's important that we structure the split so
that projects like Ares can peacefully coexist with whatever Walter
decides to ship with D.

* As such, D really, *really* needs a switch that says "don't link the
standard library."  At the very minimum, we should be able to specify
which standard library to use on either the command-line, or a
configuration file.  Also keep in mind some crazy people who might need
to link against either depending on the project at hand :)

* The standard library should maintain both a "1.0" version and a
"current" version, using whatever 1.0 version identifier Walter has the
compiler emit.  This means people can continue upgrading the standard
library whilst using the same compiler, or upgrade the compiler and use
the same standard library (so long as they use 1.0 as the middle ground).

* Incidentally, this brings up an interesting problem: we can't select
which symbols to link based on version identifiers.  Perhaps future D
compilers should link dstdlib.lib by default, and d1_0stdlib.lib when
the -D1.0 switch is thrown...

* The GC should be considered part of the standard lib, not core.  The
reason is that there are several people over the years who have
expressed interest in replacing or modifying the garbage collector D
uses (myself included.)  Currently this is very difficult simply because
you also need to rebuild the rest of the standard library, and then
completely replace phobos.lib.

* Perhaps we should even go so far as to put the standard library and GC
into separate .lib files...

Thoughts?  If this is way off base and unnecessary, just say so; I just
thought I'd throw this out there :)

	-- Daniel

P.S.  I said earlier today that there should be a document outlining
string usage in D.  If no one else wants to do that, I have a few ideas,
but they'll have to wait a bit... driving test in an hour or so :P

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even
make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/
Nov 17 2006
parent reply "Chris Miller" <chris dprogramming.com> writes:
On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep  =

<daniel.keep.lists gmail.com> wrote:

 * We need to make a clear distinction between the absolute bare-minimu=

 for the compiler and produced programs to function (let's call that th=

 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object

Agreed. The core should allow D to be used for OS development with little effort= , = and should allow D to be used as a C replacement, only using D's core = features and only C's library. This would make it a real systems programming language. Right now to = achieve this you have to do a bit of implementation-specific hacking. There's also a gray area where D itself depends on the GC, such as when = = using length properties, ~=3D and ~ operators, and array literals. Perha= ps = these should just segfault if used only with the D core and no replaceme= nt = GC. - Chris
Nov 17 2006
next sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Chris Miller wrote:
 On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep 
 <daniel.keep.lists gmail.com> wrote:
 
 * We need to make a clear distinction between the absolute bare-minimum
 for the compiler and produced programs to function (let's call that the
 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object

Agreed.

What should the overall goal be? A tight unified hierarchical set of packages like those found in Java/C#.NET? Or a more loose and ragtag library collection with a flat namespace, more like Python/Perl? While I think the former would be great to have, I think the latter is more realistic. Java and C# have nice libraries, but I think that kind of global consistency and hierarchical organization is not so feasible with the open source model. The "loose collection of useful things" model seem to work better in the open source world. Much of what's in the standard libraries of Python and Perl were once separate libraries written by 3rd parties. They just got to be good enough and popular enough that they were folded into the standard distributions. --bb
Nov 17 2006
parent reply Benji Smith <dlanguage benjismith.net> writes:
Bill Baxter wrote:
 What should the overall goal be?  A tight unified hierarchical set of 
 packages like those found in Java/C#.NET?  Or a more loose and ragtag 
 library collection with a flat namespace, more like Python/Perl?

Also, I wonder whether it's possible to offer a really standardized standard library to such a diverse collection of developers. Those of us from a Java/C# background have been trained to expect DescriptiveNamingConventions rather than terse names like "atoi" and "szptr". We also tend to use exceptions rather than error codes. And we prefer interfaces to operator overloads. We don't mind using templates for collections, but template metaprogramming doesn't interest us much. Personally, I'd rather have a String class than a char[]. I'd rather have a great HashMap implementation than built-in associative arrays. And I'd rather that foreach expected an Iterable object than a delegate, provided through an operator-overload. I use pointers occasionally in C# (for packing primitive data into arrays at runtime), but I'm generally more comfortable with references. Anyhow... If the D standard library looks like a re-vamped C library, I'll be disappointed. But if it looks like Java, the C++ crowd will be annoyed. At the same time, the standard library can't be a weird Frankenstein amalgamation of Java-style code and C-style code. Then no one would be happy. I really think the D standard library will be very small and minimal, providing some humble set of necessary core functionality. When people go looking for libraries (or when they author libraries themselves), they'll lean toward the programming paradigm they're most comfortable with. Some developers will prefer the C-style of coding, while others will tend to use Java/C# style libraries. I don't think there's anything wrong with that, but I think it's inevitable. --benji
Nov 17 2006
next sibling parent =?UTF-8?B?QW5kZXJzIEYgQmrDtnJrbHVuZA==?= <afb algonet.se> writes:
Benji Smith wrote:

 Anyhow...
 
 If the D standard library looks like a re-vamped C library, I'll be 
 disappointed.

Me too, considering that it already *includes* the standard C library... i.e. the functions in "std" needs to offer something extra over those in "std.c", or one could just as well stick with the C library stuff ?
 But if it looks like Java, the C++ crowd will be annoyed.

If the stdlib looks like Java with a mandatory String class and such, I think both the C and D crowds will be annoyed too ? We need *both* a low-level standard library and a high-level standard library for D...
 At the same time, the standard library can't be a weird Frankenstein 
 amalgamation of Java-style code and C-style code. Then no one would be 
 happy.
 
 I really think the D standard library will be very small and minimal, 
 providing some humble set of necessary core functionality.

Me too, I think it should offer a good base and then get out of the way. Decoupling the compiler runtime and the std library is a good thing too. And I would rather see the "Java-style" library done outside of DMD/GDC. I think it would be great if you could use D all these different ways; as a better and improved C with assembler, as a simpler alternative to C++, as a native alternative to Java/C#. All contained in one language ? --anders
Nov 18 2006
prev sibling parent reply Samuel MV <samuel jxdesigner.com> writes:
 Those of us from a Java/C# background have been trained to expect 
 DescriptiveNamingConventions rather than terse names like "atoi" and 
 "szptr". We also tend to use exceptions rather than error codes. And we 
 prefer interfaces to operator overloads. We don't mind using templates 
 for collections, but template metaprogramming doesn't interest us much.
 

I have both backgrounds: programmed 50% in C/asm/pascal and 50% in java/c#. And I find both extremes annoying: I liked the original java library structure, it was not too deep nor too flat ... C, PHP, etc. libraries seem just a bunch of functions thrown randomly ;-) But at the same time, I hope D's standard library doesn't become a new Java or C#, fill with 'AbstractFactoryFactoryNewlineChar' classes and 10-levels-deep qualifiers ... just to open a socket or to write in a file. Just look the problems the Java community has with EJB, JSP, servlets, struts, ... every year they have to solve the over-complexity they build the year before.
 Personally, I'd rather have a String class than a char[]. I'd rather 
 have a great HashMap implementation than built-in associative arrays. 
 And I'd rather that foreach expected an Iterable object than a delegate, 
 provided through an operator-overload. I use pointers occasionally in C# 
 (for packing primitive data into arrays at runtime), but I'm generally 
 more comfortable with references.
 

I want both: char[] for everyday use, and a String class for i18n and l10n, easy associative arrays, foreach D-style, Iterators java-style, operator-overload, ... I don't want to mess with classes and templates if the language can solve my most usual problems more easily. Complex solutions should be only for complex problems ...
 If the D standard library looks like a re-vamped C library, I'll be 
 disappointed.

I agree, for that we have std.c
 But if it looks like Java, the C++ crowd will be annoyed.

I can look D style, neither Java nor C++ :)
 At the same time, the standard library can't be a weird Frankenstein 
 amalgamation of Java-style code and C-style code. Then no one would be 
 happy.
 

It's the same in Java, there are classes (like Math), that are just a container for lots of static methods, i.e., a kind of C library. For me there are functionalities that are more easily solved with OOP, and other that are with classic procedural programming or functional programming. A 'pure' programming paradigm it's not practical, although it's nice to sell books ;-) It'll be ugly if it's done without order. I think we should have something like Python's PEP or Java's JSR. One for the language and another for the standard library, more community and more order at the same time.
 I really think the D standard library will be very small and minimal, 
 providing some humble set of necessary core functionality.
 

I totally agree.
 When people go looking for libraries (or when they author libraries 
 themselves), they'll lean toward the programming paradigm they're most 
 comfortable with. Some developers will prefer the C-style of coding, 
 while others will tend to use Java/C# style libraries.
 
 I don't think there's anything wrong with that, but I think it's 
 inevitable.
 

I think D community is too small to be able to afford that. It'll be better to have: a) one core library only with compiler and language stuff b) a standard library for basic functionality D-style (console, io, net, file, simple math, ... kind of phobos but revised and updated) c) and over this an OOP library for high-level/complex functionality (http, database, xml, windows toolkit, multimedia, ...) Best regards, Samuel.
Nov 18 2006
parent reply Jeff <jeffrparsons optusnet.com.au> writes:
Agreed on all points.

At very least I'd love to see a new newsgroup dedicated to standard 
library discussion, and a CVS/SVN server so, for example, people could 
easily make patches against the current development version for Walter's 
consideration.

Some kind of indication would be nice: are we likely to see anything 
like this soon, or is a different path intended?
Nov 20 2006
parent Lars Ivar Igesund <larsivar igesund.net> writes:
Jeff wrote:

 Agreed on all points.
 
 At very least I'd love to see a new newsgroup dedicated to standard
 library discussion, and a CVS/SVN server so, for example, people could
 easily make patches against the current development version for Walter's
 consideration.

Walter has indicated that this won't happen before D 1.0.
 Some kind of indication would be nice: are we likely to see anything
 like this soon, or is a different path intended?

I'm inclined to think that a different path might be needed in the long run. -- Lars Ivar Igesund blog at http://larsivi.net DSource & #D: larsivi
Nov 20 2006
prev sibling parent reply Alexander Panek <a.panek brainsware.org> writes:
The main problems are printf(), malloc() and similar. Everything else is 
held pretty platform independent, so it can either be just left out or 
be integrated without any worries. Apart from that, a compiler flag like 
GCC comes with one, "-ffreestanding", would be really great.


"-ffreestanding
       Assert that compilation takes place in a freestanding environment.
       This implies -fno-builtin.  A freestanding environment is one in
       which the standard library may not exist, and program startup may
       not necessarily be at "main".  The most obvious example is an OS
       kernel.  This is equivalent to -fno-hosted."

Also, possibility to link ELF files that can be loaded by Grub or 
similar would be great...don't know if that's possible atm with /link/.

Alex


Chris Miller wrote:
 On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep 
 <daniel.keep.lists gmail.com> wrote:
 
 * We need to make a clear distinction between the absolute bare-minimum
 for the compiler and produced programs to function (let's call that the
 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object

Agreed. The core should allow D to be used for OS development with little effort, and should allow D to be used as a C replacement, only using D's core features and only C's library. This would make it a real systems programming language. Right now to achieve this you have to do a bit of implementation-specific hacking. There's also a gray area where D itself depends on the GC, such as when using length properties, ~= and ~ operators, and array literals. Perhaps these should just segfault if used only with the D core and no replacement GC. - Chris

Nov 18 2006
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Alexander Panek wrote:
 Also, possibility to link ELF files that can be loaded by Grub or 
 similar would be great...don't know if that's possible atm with /link/.

It can certainly be done on Linux. AFAIK you'll need to invoke ld directly instead of going through dmd, though. Been there, done that, have the makefile in SVN :). I'm not aware of a way to make grub-compatible files on Windows (though there certainly may be one). The main problem there is that ld doesn't handle OMF, so maybe there are some unknown-to-me options to get optlink to do this, or there's another linker available somewhere that can manage it.
Nov 18 2006
parent reply Alexander Panek <a.panek brainsware.org> writes:
Of course, with GNU binutils you can do that without any problems. Been 
there, done that ;) . It'd just be really great to have a 'native' 
Digitalmars toolchain that can do the same out of the box - great point 
when it comes to marketing in system developing sector.

According to the ease of development on application development level 
(mango, DFL, DWT, DSSS, DDL, ... -- god damnit, D is already so great, 
btw!) and how different, still integrated, solutions are developed 
already, it shouldn't be such big of a problem to create a 
standard-way-to-go for lower level applications (such as kernels, real 
time applications, and whatnot).

I'll definitely give kernel development a go again as soon as D becomes 
1.0 and my schedule is somewhat free.

Alex

Frits van Bommel wrote:
 Alexander Panek wrote:
 Also, possibility to link ELF files that can be loaded by Grub or 
 similar would be great...don't know if that's possible atm with /link/.

It can certainly be done on Linux. AFAIK you'll need to invoke ld directly instead of going through dmd, though. Been there, done that, have the makefile in SVN :). I'm not aware of a way to make grub-compatible files on Windows (though there certainly may be one). The main problem there is that ld doesn't handle OMF, so maybe there are some unknown-to-me options to get optlink to do this, or there's another linker available somewhere that can manage it.

Nov 18 2006
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Alexander Panek wrote:
 Of course, with GNU binutils you can do that without any problems. Been 
 there, done that ;) . It'd just be really great to have a 'native' 
 Digitalmars toolchain that can do the same out of the box - great point 
 when it comes to marketing in system developing sector.

Well, GNU binutils is what DMD uses anyway on Linux, there is no independent 'native' DM toolchain. The dmd executable just invokes ld. It shouldn't be too hard to change dmd to allow this, though. It just needs a command-line switch to suppress -lphobos (and maybe a couple of other arguments to ld), the rest can be handled through -L.
Nov 18 2006