www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Segregating the standard library

reply Daniel Keep <daniel.keep.lists gmail.com> writes:
Something I've been thinking about.

Whilst I think D is making great progress to a fantastic 1.0 début, the
standard library is somewhat lacking.  Not to knock Walter's efforts in
the slightest, but D's standard library is somewhat lean compared to,
say, Python's standard library.  Let's not even try comparing it to
Java's (arguably the only reason Java even succeeded in the first place.)

As such, I think there's something we need to do before D 1.0: we need
to split the compiler and the standard library.  Specifically:

* We need to make a clear distinction between the absolute bare-minimum
for the compiler and produced programs to function (let's call that the
core lib) and everything else (the standard lib.)

  The core lib would probably contain things like intrinsics, and
anything else necessary (I think that also includes Object, but I'm not
sure *why*...)  The core lib would be compiler-specific, and maintained
under Walter's control.

* The standard library should be split off as a separate project under
community control, with Walter periodically taking stable snapshots to
bundle with the D compiler.

The reason for this is that I'm worried people will look at D 1.0 The
Language and say "great steaming gouts of magma on a beeline for the
orphanage; forget C!", but then look at D 1.0 The Standard Library and
say "well, that's somewhat of an anticlimax."

Doing this, we can say: "here is the shiny new D 1.0 compiler.  Yeah,
the standard library is a little on the skinny side, but it's being
actively worked on, and you can update any time you like."

A few more random thoughts on this:

* I know Walter has said before that if anyone wants to contribute to
Phobos, that he's more than happy to take patches.  The problem is that
the website doesn't say this, and it doesn't have that air of "all
welcome to join in" feel that, say, dsource projects do.  Having a
public svn server and development forums/mailing list dedicated to the
standard library would probably help.

* Do we use Ares?  I suppose that's up to Walter; his choices are to
either bless Ares as what he will be distributing as "standard" with D,
or have a separate Phobos project.

* Irrespective, I think it's important that we structure the split so
that projects like Ares can peacefully coexist with whatever Walter
decides to ship with D.

* As such, D really, *really* needs a switch that says "don't link the
standard library."  At the very minimum, we should be able to specify
which standard library to use on either the command-line, or a
configuration file.  Also keep in mind some crazy people who might need
to link against either depending on the project at hand :)

* The standard library should maintain both a "1.0" version and a
"current" version, using whatever 1.0 version identifier Walter has the
compiler emit.  This means people can continue upgrading the standard
library whilst using the same compiler, or upgrade the compiler and use
the same standard library (so long as they use 1.0 as the middle ground).

* Incidentally, this brings up an interesting problem: we can't select
which symbols to link based on version identifiers.  Perhaps future D
compilers should link dstdlib.lib by default, and d1_0stdlib.lib when
the -D1.0 switch is thrown...

* The GC should be considered part of the standard lib, not core.  The
reason is that there are several people over the years who have
expressed interest in replacing or modifying the garbage collector D
uses (myself included.)  Currently this is very difficult simply because
you also need to rebuild the rest of the standard library, and then
completely replace phobos.lib.

* Perhaps we should even go so far as to put the standard library and GC
into separate .lib files...

Thoughts?  If this is way off base and unnecessary, just say so; I just
thought I'd throw this out there :)

	-- Daniel

P.S.  I said earlier today that there should be a document outlining
string usage in D.  If no one else wants to do that, I have a few ideas,
but they'll have to wait a bit... driving test in an hour or so :P

-- 
Unlike Knuth, I have neither proven or tried the above; it may not even
make sense.

v2sw5+8Yhw5ln4+5pr6OFPma8u6+7Lw4Tm6+7l6+7D
i28a2Xs3MSr2e4/6+7t4TNSMb6HTOp5en5g6RAHCP  http://hackerkey.com/
Nov 17 2006
parent reply "Chris Miller" <chris dprogramming.com> writes:
On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep  =

<daniel.keep.lists gmail.com> wrote:


 * We need to make a clear distinction between the absolute bare-minimu=


m

 for the compiler and produced programs to function (let's call that th=


e

 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object


Agreed.

The core should allow D to be used for OS development with little effort=
,  =

and should allow D to be used as a C replacement, only using D's core  =

features and only C's library.

This would make it a real systems programming language. Right now to  =

achieve this you have to do a bit of implementation-specific hacking.

There's also a gray area where D itself depends on the GC, such as when =
 =

using length properties, ~=3D and ~ operators, and array literals. Perha=
ps  =

these should just segfault if used only with the D core and no replaceme=
nt  =

GC.

- Chris
Nov 17 2006
next sibling parent reply Bill Baxter <dnewsgroup billbaxter.com> writes:
Chris Miller wrote:

 On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep 
 <daniel.keep.lists gmail.com> wrote:
 

 * We need to make a clear distinction between the absolute bare-minimum
 for the compiler and produced programs to function (let's call that the
 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object


 
 Agreed.


What should the overall goal be?  A tight unified hierarchical set of 

library collection with a flat namespace, more like Python/Perl?

While I think the former would be great to have, I think the latter is 

of global consistency and hierarchical organization is not so feasible 
with the open source model.  The "loose collection of useful things" 
model seem to work better in the open source world.  Much of what's in 
the standard libraries of Python and Perl were once separate libraries 
written by 3rd parties.  They just got to be good enough and popular 
enough that they were folded into the standard distributions.

--bb
Nov 17 2006
parent reply Benji Smith <dlanguage benjismith.net> writes:
Bill Baxter wrote:

 What should the overall goal be?  A tight unified hierarchical set of 

 library collection with a flat namespace, more like Python/Perl?


Also, I wonder whether it's possible to offer a really standardized 
standard library to such a diverse collection of developers.


DescriptiveNamingConventions rather than terse names like "atoi" and 
"szptr". We also tend to use exceptions rather than error codes. And we 
prefer interfaces to operator overloads. We don't mind using templates 
for collections, but template metaprogramming doesn't interest us much.

Personally, I'd rather have a String class than a char[]. I'd rather 
have a great HashMap implementation than built-in associative arrays. 
And I'd rather that foreach expected an Iterable object than a delegate, 

(for packing primitive data into arrays at runtime), but I'm generally 
more comfortable with references.

Anyhow...

If the D standard library looks like a re-vamped C library, I'll be 
disappointed.

But if it looks like Java, the C++ crowd will be annoyed.

At the same time, the standard library can't be a weird Frankenstein 
amalgamation of Java-style code and C-style code. Then no one would be 
happy.

I really think the D standard library will be very small and minimal, 
providing some humble set of necessary core functionality.

When people go looking for libraries (or when they author libraries 
themselves), they'll lean toward the programming paradigm they're most 
comfortable with. Some developers will prefer the C-style of coding, 


I don't think there's anything wrong with that, but I think it's inevitable.

--benji
Nov 17 2006
next sibling parent =?UTF-8?B?QW5kZXJzIEYgQmrDtnJrbHVuZA==?= <afb algonet.se> writes:
Benji Smith wrote:


 Anyhow...
 
 If the D standard library looks like a re-vamped C library, I'll be 
 disappointed.


Me too, considering that it already *includes* the standard C library...
i.e. the functions in "std" needs to offer something extra over those
in "std.c", or one could just as well stick with the C library stuff ?


 But if it looks like Java, the C++ crowd will be annoyed.


If the stdlib looks like Java with a mandatory String class and such,
I think both the C and D crowds will be annoyed too ? We need *both*
a low-level standard library and a high-level standard library for D...


 At the same time, the standard library can't be a weird Frankenstein 
 amalgamation of Java-style code and C-style code. Then no one would be 
 happy.
 
 I really think the D standard library will be very small and minimal, 
 providing some humble set of necessary core functionality.


Me too, I think it should offer a good base and then get out of the way.
Decoupling the compiler runtime and the std library is a good thing too.
And I would rather see the "Java-style" library done outside of DMD/GDC.

I think it would be great if you could use D all these different ways;
as a better and improved C with assembler, as a simpler alternative to 


--anders
Nov 18 2006
prev sibling parent reply Samuel MV <samuel jxdesigner.com> writes:

 DescriptiveNamingConventions rather than terse names like "atoi" and 
 "szptr". We also tend to use exceptions rather than error codes. And we 
 prefer interfaces to operator overloads. We don't mind using templates 
 for collections, but template metaprogramming doesn't interest us much.
 


I have both backgrounds: programmed 50% in C/asm/pascal and 50% in 


I liked the original java library structure, it was not too deep nor too 
flat ... C, PHP, etc. libraries seem just a bunch of functions thrown 
randomly ;-)

But at the same time, I hope D's standard library doesn't become a new 

10-levels-deep qualifiers ... just to open a socket or to write in a file.

Just look the problems the Java community has with EJB, JSP, servlets, 
struts, ... every year they have to solve the over-complexity they build 
the year before.


 Personally, I'd rather have a String class than a char[]. I'd rather 
 have a great HashMap implementation than built-in associative arrays. 
 And I'd rather that foreach expected an Iterable object than a delegate, 

 (for packing primitive data into arrays at runtime), but I'm generally 
 more comfortable with references.
 


I want both: char[] for everyday use, and a String class for i18n and 
l10n, easy associative arrays, foreach D-style, Iterators java-style, 
operator-overload, ... I don't want to mess with classes and templates 
if the language can solve my most usual problems more easily. Complex 
solutions should be only for complex problems ...


 If the D standard library looks like a re-vamped C library, I'll be 
 disappointed.


I agree, for that we have std.c


 But if it looks like Java, the C++ crowd will be annoyed.


I can look D style, neither Java nor C++ :)


 At the same time, the standard library can't be a weird Frankenstein 
 amalgamation of Java-style code and C-style code. Then no one would be 
 happy.
 


It's the same in Java, there are classes (like Math), that are just a 
container for lots of static methods, i.e., a kind of C library.

For me there are functionalities that are more easily solved with OOP, 
and other that are with classic procedural programming or functional 
programming. A 'pure' programming paradigm it's not practical, although 
it's nice to sell books ;-)

It'll be ugly if it's done without order. I think we should have 
something like Python's PEP or Java's JSR. One for the language and 
another for the standard library, more community and more order at the 
same time.


 I really think the D standard library will be very small and minimal, 
 providing some humble set of necessary core functionality.
 


I totally agree.


 When people go looking for libraries (or when they author libraries 
 themselves), they'll lean toward the programming paradigm they're most 
 comfortable with. Some developers will prefer the C-style of coding, 

 
 I don't think there's anything wrong with that, but I think it's 
 inevitable.
 


I think D community is too small to be able to afford that. It'll be 
better to have:

a) one core library only with compiler and language stuff

b) a standard library for basic functionality D-style (console, io, net, 
file, simple math, ... kind of phobos but revised and updated)

c) and over this an OOP library for high-level/complex functionality 
(http, database, xml, windows toolkit, multimedia, ...)

Best regards,

            Samuel.
Nov 18 2006
parent reply Jeff <jeffrparsons optusnet.com.au> writes:
Agreed on all points.

At very least I'd love to see a new newsgroup dedicated to standard 
library discussion, and a CVS/SVN server so, for example, people could 
easily make patches against the current development version for Walter's 
consideration.

Some kind of indication would be nice: are we likely to see anything 
like this soon, or is a different path intended?
Nov 20 2006
parent Lars Ivar Igesund <larsivar igesund.net> writes:
Jeff wrote:


 Agreed on all points.
 
 At very least I'd love to see a new newsgroup dedicated to standard
 library discussion, and a CVS/SVN server so, for example, people could
 easily make patches against the current development version for Walter's
 consideration.


Walter has indicated that this won't happen before D 1.0.


 Some kind of indication would be nice: are we likely to see anything
 like this soon, or is a different path intended?


I'm inclined to think that a different path might be needed in the long run.

-- 
Lars Ivar Igesund
blog at http://larsivi.net
DSource & #D: larsivi
Nov 20 2006
prev sibling parent reply Alexander Panek <a.panek brainsware.org> writes:
The main problems are printf(), malloc() and similar. Everything else is 
held pretty platform independent, so it can either be just left out or 
be integrated without any worries. Apart from that, a compiler flag like 
GCC comes with one, "-ffreestanding", would be really great.


"-ffreestanding
       Assert that compilation takes place in a freestanding environment.
       This implies -fno-builtin.  A freestanding environment is one in
       which the standard library may not exist, and program startup may
       not necessarily be at "main".  The most obvious example is an OS
       kernel.  This is equivalent to -fno-hosted."

Also, possibility to link ELF files that can be loaded by Grub or 
similar would be great...don't know if that's possible atm with /link/.

Alex


Chris Miller wrote:

 On Fri, 17 Nov 2006 17:47:19 -0500, Daniel Keep 
 <daniel.keep.lists gmail.com> wrote:
 

 * We need to make a clear distinction between the absolute bare-minimum
 for the compiler and produced programs to function (let's call that the
 core lib) and everything else (the standard lib.)

   The core lib would probably contain things like intrinsics, and
 anything else necessary (I think that also includes Object


 
 Agreed.
 
 The core should allow D to be used for OS development with little 
 effort, and should allow D to be used as a C replacement, only using D's 
 core features and only C's library.
 
 This would make it a real systems programming language. Right now to 
 achieve this you have to do a bit of implementation-specific hacking.
 
 There's also a gray area where D itself depends on the GC, such as when 
 using length properties, ~= and ~ operators, and array literals. Perhaps 
 these should just segfault if used only with the D core and no 
 replacement GC.
 
 - Chris
Nov 18 2006
parent reply Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Alexander Panek wrote:

 Also, possibility to link ELF files that can be loaded by Grub or 
 similar would be great...don't know if that's possible atm with /link/.


It can certainly be done on Linux. AFAIK you'll need to invoke ld 
directly instead of going through dmd, though. Been there, done that, 
have the makefile in SVN :).

I'm not aware of a way to make grub-compatible files on Windows (though 
there certainly may be one). The main problem there is that ld doesn't 
handle OMF, so maybe there are some unknown-to-me options to get optlink 
to do this, or there's another linker available somewhere that can 
manage it.
Nov 18 2006
parent reply Alexander Panek <a.panek brainsware.org> writes:
Of course, with GNU binutils you can do that without any problems. Been 
there, done that ;) . It'd just be really great to have a 'native' 
Digitalmars toolchain that can do the same out of the box - great point 
when it comes to marketing in system developing sector.

According to the ease of development on application development level 
(mango, DFL, DWT, DSSS, DDL, ... -- god damnit, D is already so great, 
btw!) and how different, still integrated, solutions are developed 
already, it shouldn't be such big of a problem to create a 
standard-way-to-go for lower level applications (such as kernels, real 
time applications, and whatnot).

I'll definitely give kernel development a go again as soon as D becomes 
1.0 and my schedule is somewhat free.

Alex

Frits van Bommel wrote:

 Alexander Panek wrote:

 Also, possibility to link ELF files that can be loaded by Grub or 
 similar would be great...don't know if that's possible atm with /link/.


 
 It can certainly be done on Linux. AFAIK you'll need to invoke ld 
 directly instead of going through dmd, though. Been there, done that, 
 have the makefile in SVN :).
 
 I'm not aware of a way to make grub-compatible files on Windows (though 
 there certainly may be one). The main problem there is that ld doesn't 
 handle OMF, so maybe there are some unknown-to-me options to get optlink 
 to do this, or there's another linker available somewhere that can 
 manage it.
Nov 18 2006
parent Frits van Bommel <fvbommel REMwOVExCAPSs.nl> writes:
Alexander Panek wrote:

 Of course, with GNU binutils you can do that without any problems. Been 
 there, done that ;) . It'd just be really great to have a 'native' 
 Digitalmars toolchain that can do the same out of the box - great point 
 when it comes to marketing in system developing sector.


Well, GNU binutils is what DMD uses anyway on Linux, there is no 
independent 'native' DM toolchain. The dmd executable just invokes ld.
It shouldn't be too hard to change dmd to allow this, though. It just 
needs a command-line switch to suppress -lphobos (and maybe a couple of 
other arguments to ld), the rest can be handled through -L.
Nov 18 2006