www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Current state of "D as a better C" (Windows)?

reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
Hi,

What's the current situation of using D without Phobos/Druntime?

Sometimes, it's quite useful to be able to use D to create tiny 
EXEs/DLLs. For example, create simple DLL plugins that will be 
loaded into non-D programs, or even not have a 50-line program 
take half a meg. Additionally, some tasks require injecting a DLL 
systemwide (into every running process), so it is critical to 
minimize the impact by being as unintrusive as possible and 
having a minimal overhead.

I've started a "framework" for this purpose a while ago[1], but I 
heard of people starting similar projects since then and I think 
they've gotten further than me. Would there be any interest in 
collaborating and turning this into a community project?

I think some goals would be:
- A "runtime" (object.d) which provides either stubs for the 
current compiler hooks, or implementations with minimal 
dependencies
- A "standard library" which does not require runtime support, 
but can also be used when using Druntime (as a GC-free 
alternative to Phobos)
- Using MSVCRT on 32-bit as well
- Options to enable/disable various runtime features? For 
example, a project may want to use exceptions (which requires 
ClassInfo) but not the GC.
- Support for the Unilink linker? I think it is better at 
generating smaller binaries that OPTLINK.

There is also some ongoing work on the -betterC switch[2], which 
I'm quite excited about, but it seems to be currently blocked on 
waiting for the next DMD release as the changes[3] might break a 
lot of code.

  [1]: https://github.com/CyberShadow/SlimD
  [2]: https://d.puremagic.com/issues/show_bug.cgi?id=11881
  [3]: https://github.com/D-Programming-Language/dmd/pull/2561
Jan 21 2014
next sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev 
wrote:
 Hi,

 What's the current situation of using D without Phobos/Druntime?

I created a minimal, no runtime, "hello world" program [1], and I think Iain Buclaw answered your question when he commented on my example as follows: "that's not D! That's some D, a bit of some extended inline assembly, and a custom linker script." To paraphrase, if you really want the best of what D has to offer, you'll need to implement a good part of the runtime. But I argue, not all of it. I'm working on a bare-metal ARM Cortex-M _very minimal_ port of the D runtime with an emphasis on "minimal". I just created my repository and started documenting my progress [2]. It's in the very early stages, and I am quite new to D, so I have quite a ways to go, but my goals align with your stated intent. I'm able to do structs with just a few line of stubbed TypeInfo stuff in my object.d. I've reduced it to 21 lines [3]. I've also implemented a simple malloc/free and am able to do classes, but I haven't tested any D feature thoroughly yet.
 Sometimes, it's quite useful to be able to use D to create tiny 
 EXEs/DLLs. For example, create simple DLL plugins that will be 
 loaded into non-D programs, or even not have a 50-line program 
 take half a meg. Additionally, some tasks require injecting a 
 DLL systemwide (into every running process), so it is critical 
 to minimize the impact by being as unintrusive as possible and 
 having a minimal overhead.

 I've started a "framework" for this purpose a while ago[1], but 
 I heard of people starting similar projects since then and I 
 think they've gotten further than me. Would there be any 
 interest in collaborating and turning this into a community 
 project?

Didn't know about SlimD (catchy name by the way). I'll definitely be taking a look as this aligns with my goals. As the Cortex-M microcontrollers have very little FLASH memory, it is also important for me to reduce the executable size, and I'm focusing on that in my project. I've been able to get my "Hello, World!" program down to 68 bytes, with 15 of those bytes being the "Hello, World!\r\n" string constant.
 I think some goals would be:
 - A "runtime" (object.d) which provides either stubs for the 
 current compiler hooks, or implementations with minimal 
 dependencies

I would really like to see a more modular D runtime in which features of the D language can be selectively chosen. This is one of my goals. I'd also like to have the D runtime "reorganzied" to make hardware/OS/C-library abstraction more obvious. I submitted an enhancement to facilitate this [4]: I thin, a modular, well-organized, well-abstracted, D runtime will facilitate both your stated intention and my goals to bring D to the microcontroller world, and will also facilitate bringing D to a variety of other platforms as well. But, I really think the D runtime needs some refactoring to achieve this. In my project, I'm also trying to do everything in D ("Look, Walter!, No C!"). I'd like to see all the references to C's baggage left separate from the rest of the D runtime, included "size_t". Let's not track C's doo-doo into D's house, if it can be avoided.
 - A "standard library" which does not require runtime support, 
 but can also be used when using Druntime (as a GC-free 
 alternative to Phobos)

Phobos will not be useful to me unless I can find some way to make it lighter and GC-Free. I'll probably just end up taking pieces of it. I would _really_ like to have the whole concept of a GC decoupled from D. There are many suitable memory models to use with D, and the memory model seems like it should be a platform feature, not a language feature. But, I don't know how one could accomplish that without some changes to the language itself, and it seems the core language folks have their hands full.
 - Options to enable/disable various runtime features? For 
 example, a project may want to use exceptions (which requires 
 ClassInfo) but not the GC.

Yes, this definitely aligns with my goals, see my comment above about a modular D runtime. But I wouldn't necessarily be in favor of adding switches to the compiler, as I don't like to be at the mercy of the compiler implementers. I'd prefer some way where I can pass a user-defined variable to my compiler flags in concert with the version() feature. And if something is needed, but not implemented, the compiler/linker will throw an error saying so.
 - Support for the Unilink linker? I think it is better at 
 generating smaller binaries that OPTLINK.

Does D have to support Unilink, or does Unilink have to support D?
 There is also some ongoing work on the -betterC switch[2], 
 which I'm quite excited about, but it seems to be currently 
 blocked on waiting for the next DMD release as the changes[3] 
 might break a lot of code.

  [1]: https://github.com/CyberShadow/SlimD
  [2]: https://d.puremagic.com/issues/show_bug.cgi?id=11881
  [3]: https://github.com/D-Programming-Language/dmd/pull/2561

There was an interesting discussion, recently, that might relevant to this. I encourage you to check it out and maybe post your thoughts [5]. It's more about the required TypeInfo stuff, but nevertheless related to the -betterC idea. LCD has some pragmas that seem to align with the general idea of what you're looking for [6], and GDC has the -fno-emit-moduleinfo which is essential for what I'm doing. I'd love to contribute to something like this in anyway I can, but I'd probably be asking more questions than providing help. At the moment, my contribution is in the form of my repository [2]. I hope it will have some influence in the long run, and maybe when I come up to speed on the language I'll be able to contribute more. Mike [1] http://wiki.dlang.org/Minimal_semihosted_ARM_Cortex-M_%22Hello_World%22 [2] https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study [3] https://github.com/JinShil/D_Runtime_ARM_Cortex-M_study/blob/structs/source/object.d [4] https://d.puremagic.com/issues/show_bug.cgi?id=11666 [5] http://forum.dlang.org/post/jynxfglpulguvqbivrms forum.dlang.org [6] http://wiki.dlang.org/LDC-specific_language_changes
Jan 21 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-24 06:17, Mike wrote:

 That sounds very undesirable.  I still don't even understand what
 purpose modules and ModuleInfo really serve.  Right now, I'm just using
 modules for namespace scope and encapsulation.  If you know some
 documentation that helps demystify ModuleInfo and what its purpose is
 (besides the source code) please point me to it.

ModuleInfo contains, for example, module constructors: module foo; static this () { // run once before every thread is started } shared static this () { // run once before main is called } It also provides a way to iterate all modules, at runtime. From each module it's possible to iterate all local classes and all imported modules. Object.create which will return a new object of a class given by its fully qualified name, as a string. Object.create is implemented using ModuleInfo. -- /Jacob Carlborg
Jan 24 2014
prev sibling next sibling parent "Dominikus Dittes Scherkl" writes:
On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
 Didn't know about SlimD (catchy name by the way).

Jan 22 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev 
wrote:
 What's the current situation of using D without Phobos/Druntime?

With LDC I had to figure out how to make the compiler happy, but eventually got the job done.
 Sometimes, it's quite useful to be able to use D to create tiny 
 EXEs/DLLs. For example, create simple DLL plugins that will be 
 loaded into non-D programs, or even not have a 50-line program 
 take half a meg. Additionally, some tasks require injecting a 
 DLL systemwide (into every running process), so it is critical 
 to minimize the impact by being as unintrusive as possible and 
 having a minimal overhead.

 I've started a "framework" for this purpose a while ago[1], but 
 I heard of people starting similar projects since then and I 
 think they've gotten further than me. Would there be any 
 interest in collaborating and turning this into a community 
 project?

A framework doesn't seem like a good idea. You can have either big framework or small executable, not both. My code works with stock object.d just fine. In order to make injectable dll you only have to disable GC (and probably tls), and write otherwise normal D code.
Jan 22 2014
prev sibling next sibling parent "Atila Neves" <atila.neves gmail.com> writes:
On Wednesday, 22 January 2014 at 09:37:00 UTC, Dominikus Dittes 
Scherkl wrote:
 On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
 Didn't know about SlimD (catchy name by the way).


I still think that the original name "Mars" was far better, never mind the fact that it would've been far easier to Google. Ah well. Atila
Jan 22 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
Hi Mike, thanks for the extensive reply.

On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
 To paraphrase, if you really want the best of what D has to 
 offer, you'll need to implement a good part of the runtime.  
 But I argue, not all of it.

Well, even C with D's metaprogramming would be a big win. But picking-and-choosing runtime-supported features would be even better.
 I'm working on a bare-metal ARM Cortex-M _very minimal_ port of 
 the D runtime with an emphasis on "minimal". I just created my 
 repository and started documenting my progress [2].  It's in 
 the very early stages, and I am quite new to D, so I have quite 
 a ways to go, but my goals align with your stated intent.

 I'm able to do structs with just a few line of stubbed TypeInfo 
 stuff in my object.d.  I've reduced it to 21 lines [3].

 I've also implemented a simple malloc/free and am able to do 
 classes, but I haven't tested any D feature thoroughly yet.

 As the Cortex-M microcontrollers have very little FLASH memory, 
 it is also important for me to reduce the executable size, and 
 I'm focusing on that in my project.  I've been able to get my 
 "Hello, World!" program down to 68 bytes, with 15 of those 
 bytes being the "Hello, World!\r\n" string constant.

This is quite cool. I noticed you're using GNU make. On Windows, D uses Digital Mars make, which is quite minimalistic. I've considered writing a custom build tool for SlimD, but now that I think about it, I think GNU make will suffice.
 I would really like to see a more modular D runtime in which 
 features of the D language can be selectively chosen.  This is 
 one of my goals.

 I thin, a modular, well-organized, well-abstracted, D runtime 
 will facilitate both your stated intention and my goals to 
 bring D to the microcontroller world, and will also facilitate 
 bringing D to a variety of other platforms as well.  But, I 
 really think the D runtime needs some refactoring to achieve 
 this.

So, you think it would be better to make the standard D runtime more customizable in this regard, instead of putting together a new one? I believe the biggest factor for why someone would want to use a lighter runtime is the garbage collector. I think a lot of the code in Druntime still assumes that a GC is present... and then I'm not sure the D maintainers would want to maintain support for such rarely-needed features such as a custom entry point. (Case in point: pragma(entrypoint, symbol) has been broken since DMD 2.061.)
 In my project, I'm also trying to do everything in D ("Look, 
 Walter!,  No C!").  I'd like to see all the references to C's 
 baggage left separate from the rest of the D runtime, included 
 "size_t".

What's wrong with size_t? On Windows, it is quite necessary if you want to target both 32-bit and 64-bit (and if you want to do system-wide DLL injection, that's an absolute requirement).
 Phobos will not be useful to me unless I can find some way to 
 make it lighter and GC-Free.  I'll probably just end up taking 
 pieces of it.  I would _really_ like to have the whole concept 
 of a GC decoupled from D.  There are many suitable memory 
 models to use with D, and the memory model seems like it should 
 be a platform feature, not a language feature.  But, I don't 
 know how one could accomplish that without some changes to the 
 language itself, and it seems the core language folks have 
 their hands full.

There have been discussions of a compiler switch or attribute to make implicit allocations (such as array literals, or heap closures) generate compiler errors.
 Yes, this definitely aligns with my goals, see my comment above 
 about a modular D runtime.  But I wouldn't necessarily be in 
 favor of adding switches to the compiler, as I don't like to be 
 at the mercy of the compiler implementers.

I agree that hacking or forking the compiler should be avoided and left as a last resort. By "switches" I meant per-project (e.g. makefile) options.
 Does D have to support Unilink, or does Unilink have to support 
 D?

Well, neither. The "framework" would need to support linking your project with Unilink if you indicate that preference in your project's build configuration.
 There was an interesting discussion, recently, that might 
 relevant to this.  I encourage you to check it out and maybe 
 post your thoughts [5]. It's more about the required TypeInfo 
 stuff, but nevertheless related to the -betterC idea.

Well, as long as the linker throws away all compiler-emitted TypeInfo objects if your code doesn't use them, there doesn't seem to be any practical problem with them, is there?
 LCD has some pragmas that seem to align with the general idea 
 of what you're looking for [6], and GDC has the 
 -fno-emit-moduleinfo which is essential for what I'm doing.

ModuleInfo IS a problem, though. It is like a parasite of interdependence: it points towards all class information (for Object.factory), which itself pulls in all virtual methods of all classes in your program (through the Vtable). That means that anything that any virtual method in your program uses, regardless if you use that class or not, WILL end up in the executable. Ugh!
 I'd love to contribute to something like this in anyway I can, 
 but I'd probably be asking more questions than providing help.  
 At the moment, my contribution is in the form of my repository 
 [2].  I hope it will have some influence in the long run, and 
 maybe when I come up to speed on the language I'll be able to 
 contribute more.

Maybe we could join efforts, even though our goals target different platforms :) What do you say to this repository structure (inspired by the Windows DDK): Users start a new SlimD (or whatever name) project by adding a Makefile to their project directory. The makefile configures the project's requirements (target platform, used D features, linker of choice) and source files (*.d by default), and includes the main SlimD makefile, which does all the work based on the given configuration.
Jan 23 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 22 January 2014 at 11:13:53 UTC, Kagamin wrote:
 On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir 
 Panteleev wrote:
 I've started a "framework" for this purpose a while ago[1], 
 [...]

A framework doesn't seem like a good idea. You can have either big framework or small executable, not both.

I don't think you and I mean the same things ;) That's why I put "framework" in quotes. It was the most suitable word that came to mind - I basically meant "standard library and build system for your program". My goal is definitely not to replace one kitchen-sink library with another.
 My code works with stock object.d just fine. In order to make 
 injectable dll you only have to disable GC (and probably tls), 
 and write otherwise normal D code.

"Normal D code" involves plenty of explicit and implicit allocations. The practical and aesthetic considerations of hundreds of KB of useless baggage remain.
Jan 23 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Thursday, 23 January 2014 at 22:39:38 UTC, Vladimir Panteleev 
wrote:
 Well, even C with D's metaprogramming would be a big win. But 
 picking-and-choosing runtime-supported features would be even 
 better.

Quite right! I have actually delayed implementing classes a little bit because I found I could do so much with just structs, mixins, and templates. In fact, if there weren't so many types in the runtime implemented as classes, I could delay it even further.
 I noticed you're using GNU make. On Windows, D uses Digital 
 Mars make, which is quite minimalistic. I've considered writing 
 a custom build tool for SlimD, but now that I think about it, I 
 think GNU make will suffice.

Using GNU make was really an arbitrary decision for me. I really don't like GNU make and hate my own makefiles even more. Personally, I would prefer to write scripts in D, so I could automate whatever I wanted including builds. My project, however, is just an exercise. I expect to go through this exercise a couple times and fine tune what I want. Then I'll make a real repository with just the good stuff :-)
 So, you think it would be better to make the standard D runtime 
 more customizable in this regard, instead of putting together a 
 new one?

At the moment, I think throwing it all away and starting from scratch is the best approach, especially with the "minimal/modular" mindset, but I would like to get to a point where the two could be integrated (either I join D, or D joins me). I would prefer not to fragment D into "D", "D for embedded", "D for smartphones", "D for tablets", etc.. I think if the D runtime were modular and designed with platform ubiquity in mind, it would simply require a customized makefile like you've already alluded to, or something other approach (see my comments below)
 What's wrong with size_t?

There's nothing wrong with the concept of size_t. The problem is the "_t" suffix. That is C's baggage, and is inconsistent with other types in D. D doesn't use string_t, uint_t, etc..., so I don't see why this convention is being employed in D.
 There was an interesting discussion, recently, that might 
 relevant to this.  I encourage you to check it out and maybe 
 post your thoughts [5]. It's more about the required TypeInfo 
 stuff, but nevertheless related to the -betterC idea.

Well, as long as the linker throws away all compiler-emitted TypeInfo objects if your code doesn't use them, there doesn't seem to be any practical problem with them, is there?

Yeah, that's the conclusion I eventually came to. It was just a major inconvenience for me to have to spend hours studying and implementing all of this TypeInfo stuff, and all the stuff the TypeInfo uses, only to have it thrown away at link time. It was a huge waste of effort, but all that TypeInfo stuff may be needed in the end anyway, so I'm holding back too much harsh judgement at the moment.
 ModuleInfo IS a problem, though. It is like a parasite of 
 interdependence: it points towards all class information (for 
 Object.factory), which itself pulls in all virtual methods of 
 all classes in your program (through the Vtable). That means 
 that anything that any virtual method in your program uses, 
 regardless if you use that class or not, WILL end up in the 
 executable. Ugh!

That sounds very undesirable. I still don't even understand what purpose modules and ModuleInfo really serve. Right now, I'm just using modules for namespace scope and encapsulation. If you know some documentation that helps demystify ModuleInfo and what its purpose is (besides the source code) please point me to it.
 Maybe we could join efforts, even though our goals target 
 different platforms :)

 What do you say to this repository structure (inspired by the 
 Windows DDK):

 Users start a new SlimD (or whatever name) project by adding a 
 Makefile to their project directory. The makefile configures 
 the project's requirements (target platform, used D features, 
 linker of choice) and source files (*.d by default), and 
 includes the main SlimD makefile, which does all the work based 
 on the given configuration.

This is very much like what I had in mind, and I like it. However, I'm considering other methods, too. 1) Features/platforms can simply be provide at link time by adding a certain *.a or *.o files to the linker command. And if a needed feature is missing, linker errors will result. For example, exception handling could be provide with seh.o, eh.o, sjlj.o, or no_eh.o depending on which model the user wanted to use. (This example may not make any sense, as I know nothing about how to implement exception handling. I just couldn't think of better illustration of my thoughts). 2) Templates can be used to provide modularity/specialization in one's source code at compile time. There's a C++ graphics library [1] that uses this approach to create platform independent rendering pipelines. For example: //pseudo code typedef color_t typedef pixel_t typedef path_t typedef surface<pixel_t> surface_t typedef rasterizer<path_t> rasterizer_t typedef renderer<rasterizer_t, surface_t> renderer_t. //Then draw like this renderer_t r; r.DrawLine() r.DrawCircle() ... So, surface_t could be defined as... typedef surface<DirectX11> surface_t typedef surface<OpenGL> surface_t typedef surface<X11> surface_t ... to provide some platform specialization at compile time, and the user of the code doesn't know the difference. This is a pretty cool way to build a graphics library, and something along these lines could be used to provide some specialization to the D runtime too. User's would simply have to include a certain *.di file containing the required templated aliases/typedefs for a given platform. If user's want to port to a different platform, or choose a different feature subset, they just create a different set of templated aliases/typedefs. I'm brainstorming as I write this, so sorry for being so verbose. As you can see I don't have any opinion yet. Perhaps you could provide your opinion. Nevertheless, your proposed solution does basically what I'm looking for, and would be happy to help in any way I can. Mike [1] - http://www.antigrain.com/
Jan 23 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Thursday, 23 January 2014 at 22:39:38 UTC, Vladimir Panteleev 
wrote:
 I believe the biggest factor for why someone would want to use 
 a lighter runtime is the garbage collector. I think a lot of 
 the code in Druntime still assumes that a GC is present... and 
 then I'm not sure the D maintainers would want to maintain 
 support for such rarely-needed features such as a custom entry 
 point. (Case in point: pragma(entrypoint, symbol) has been 
 broken since DMD 2.061.)

The GC seems to be the biggest issue here, so I wanted to comment separately. Each target/platform/application will have different requirements and those requirements will dictate the type of memory management implementation to use. Ideally, I don't see why D has to be a garbage collected language. Can't it be agnostic to the memory management implementation? D seems to offer several options to the programmer, Scoped, RefCounted, new/destroy, and can even create their own garbage collection with the gcstub (or so I've heard). This is great!, but it's not idea as it forces the programmer to change their idioms. It appears that D is very tightly coupled to the memory management implementation, but does it need to be? Couldn't the runtime provide some hooks that can be used by the runtime implementation to know when a class is instantiated, when it goes out of scope, when it's assigned to another reference, etc... and use this to provide a variety of different memory management implementations without the programmer having to change their idioms? I'm hypothesizing that if these runtime hooks existed, one could create a transparent reference-counted implementation that would be mostly transparent to the programmer, for example. I still think the programmer will have to write code with their memory management implementation in mind, but they could at least stick to the same idioms. I'm not very optimistic, but I'm hoping something like this can be achieved, and that's what I'm shooting for. Absent this possibility, I'm not sure what to do. I believe there is probably a GC implementation that would be suitable for a my target platform (ARM Cortex-M microcontrollers). I can think of at least one: Some firmware for microcontrollers simply need to allocate all of their objects at the start of the program, but have no need to free memory: free is the power switch. For applications like this only gc.allocate() needs to be implemented, and the GC's background thread and gc.free() do not. I think this would allow the programmer to stick with traditional D idioms, but they have to keep in mind gc.free() will never be and should never be called. Some compile-time guarantee here would be nice. I could go on and on about some of my ideas, but I think I've said enough for now. I've got a lot to learn first, anyway. Mike
Jan 24 2014
prev sibling next sibling parent reply "Mike" <none none.com> writes:
On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev 
wrote:
 Hi,

 What's the current situation of using D without Phobos/Druntime?

 Sometimes, it's quite useful to be able to use D to create tiny 
 EXEs/DLLs. For example, create simple DLL plugins that will be 
 loaded into non-D programs, or even not have a 50-line program 
 take half a meg. Additionally, some tasks require injecting a 
 DLL systemwide (into every running process), so it is critical 
 to minimize the impact by being as unintrusive as possible and 
 having a minimal overhead.

 I've started a "framework" for this purpose a while ago[1], but 
 I heard of people starting similar projects since then and I 
 think they've gotten further than me. Would there be any 
 interest in collaborating and turning this into a community 
 project?

I'm curious, what is your motivation for pursuing this? What's your ultimate objective?
Jan 24 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 2:15 PM, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

 If your language does not have the C / C++ *semantics* of new /
 delete, or, better yet, owned pointers in Rust that automatically
 free their memory when they go out of scope, but instead forces a
 GC down your throat for some of the most basic array and standard
 library functionality, you will never, ever write kernel level
 code that stands a chance against C, C++ or Rust.

 Other than that, a "D with owned pointers" and an opt-in GC
 relegated to a library implementation (maybe with some minimal
 language hooks) would be my language of choice. I love D's
 templates.

 I understand that this would be a complete redesign of the
 language and Phobos, and would break most of the D code out
 there. That's why it won't happen.

 Everybody seems to realize this, finally. Just have a look at the
 last (long) GC thread. Comparisons of D with Java and C# all over.

Wait, what? Andrei
Jan 25 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 25.01.2014 23:33, schrieb Andrei Alexandrescu:
 On 1/25/14 2:15 PM, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

 If your language does not have the C / C++ *semantics* of new /
 delete, or, better yet, owned pointers in Rust that automatically
 free their memory when they go out of scope, but instead forces a
 GC down your throat for some of the most basic array and standard
 library functionality, you will never, ever write kernel level
 code that stands a chance against C, C++ or Rust.


The guys at Xerox PARC, Swiss Federal Institute of Technology in Zurich, Cisco, Olivetti seem to think otherwise. Sadly the industry never cared for their research in programming languages and safe OS systems. -- Paulo
Jan 25 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 1/25/14 6:32 PM, Frank Bauer wrote:
 There aren't that many language features that depend on a GC, and the
 ones that do can be disabled with a custom runtime.

I don't want to miss out on a single language feature of D. Where is that GC-free runtime?

GC is a language feature of D. As funny as that may seem (as a comeback timed almost too well) this has interesting truth to it. Some convenience features of D require GC to work, and that's as cut and dried as it gets. Without them, D would need to sacrifice some features or complicate the language or concentrate focus on that aspect alone (and I suspect Rust is guilty of at least some of the above). You can't have all of D without GC for the simple reason that GC is convenient and simplifies things, and often such things are desirable. Of course, reducing within reason, modularizing, and compartmentalizing the need/use of GC is a goal worth working toward. But all that requires reason and measure.
 druntime/Phobos are designed for a mixed allocation strategy, where
 tracing GC is one of the strategies used. Many Phobos types and
 functions do not rely on a GC.

In theory (which gives me hope), not in practice. Where is that GC-free Phobos?
 Java and C# do not function well without GC at all (disregarding the
 effort you'd have to go through to get a Java or C# implementation
 without GC).

 D definitely *does*.

Again, this gives me hope. Please Walter, Andrei: search all the forums for "I want D with opt-in GC" (fuzzy search, that is) and count matches.

We believe the clamor is overblown. The patterns in which D produces garbage (litter, for the most part) are very different from those in languages that don't offer a choice. Focusing on library functions that produce a few garbage-collected strings when the bulk of the load is elsewhere - that's penny wise and pound foolish. That said, it's good to have control and rein in over how and where litter is produced. We see this as an ongoing focus. Andrei
Jan 25 2014
prev sibling next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 26.01.2014 03:03, schrieb Frank Bauer:
 On Saturday, 25 January 2014 at 23:02:44 UTC, anonymous wrote:
 C doesn't have new/delete.

Please read-before-write (non atomic operation): C has the *semantics* of new / delete with malloc and free. D has too, if you constrain yourself to memory allocation via std.c.stdlib.malloc and free, which means no object allocation with *new* anymore. We don't want to constrain ourselves to that, do we? What we want is malloc/free *semantics* throughout if we choose so.

Whose semantics are those? Are you aware, that standard malloc/free vary a lot from compiler to compiler? Even generating cascading allocations or deallocations, as they tend to use more low level APIs on from the operating system? Anyone doing high performance computing or games with C ends up creating their own allocators anyway. So I wonder what is this magic performance semantics from malloc/free. -- Paulo
Jan 25 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-01-26 03:32, Frank Bauer wrote:

 I don't want to miss out on a single language feature of D. Where is
 that GC-free runtime?

Here it is: https://github.com/D-Programming-Language/druntime/blob/master/src/gcstub/gc.d It implements the GC API using malloc. One needs to manually delete the memory with GC.free. -- /Jacob Carlborg
Jan 26 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-01-26 16:18, Frank Bauer wrote:
 As you heard Andrei, too many *language features*, NOT *user code* rely
 on objects staying around till the GC collects them. It's not a matter
 of malloc/free everything in your user code. D as a language would not
 work with this little gcstub.

Sure it would, it just wouldn't free everything as it should. -- /Jacob Carlborg
Jan 27 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
D is NOT a systems language. So no point in application level
comparison with C. Set aside syntactic similarities.

If your language does not have the C / C++ *semantics* of new /
delete, or, better yet, owned pointers in Rust that automatically
free their memory when they go out of scope, but instead forces a
GC down your throat for some of the most basic array and standard
library functionality, you will never, ever write kernel level
code that stands a chance against C, C++ or Rust.

Other than that, a "D with owned pointers" and an opt-in GC
relegated to a library implementation (maybe with some minimal
language hooks) would be my language of choice. I love D's
templates.

I understand that this would be a complete redesign of the
language and Phobos, and would break most of the D code out
there. That's why it won't happen.

Everybody seems to realize this, finally. Just have a look at the
last (long) GC thread. Comparisons of D with Java and C# all over.
Jan 25 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Saturday, 25 January 2014 at 22:15:29 UTC, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

 If your language does not have the C / C++ *semantics* of new /
 delete, or, better yet, owned pointers in Rust that 
 automatically
 free their memory when they go out of scope, but instead forces 
 a
 GC down your throat for some of the most basic array and 
 standard
 library functionality, you will never, ever write kernel level
 code that stands a chance against C, C++ or Rust.

 Other than that, a "D with owned pointers" and an opt-in GC
 relegated to a library implementation (maybe with some minimal
 language hooks) would be my language of choice. I love D's
 templates.

 I understand that this would be a complete redesign of the
 language and Phobos, and would break most of the D code out
 there. That's why it won't happen.

 Everybody seems to realize this, finally. Just have a look at 
 the
 last (long) GC thread. Comparisons of D with Java and C# all 
 over.

Unfortunately, D is not similar to Java or C# (from the point of view of the garbage collector). :/
Jan 25 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 25 January 2014 at 22:15:29 UTC, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

Urgh...
 If your language does not have the C / C++ *semantics* of new /
 delete,

Setting aside syntax, it's extremely trivial to implement C++-style new/delete as function templates.
 or, better yet, owned pointers in Rust that automatically
 free their memory when they go out of scope, but instead forces 
 a
 GC down your throat for some of the most basic array and 
 standard
 library functionality,

Array functionality *that C and C++ do not have*. D slices are plenty useful without the primitives that allocate, such as concatenation. This very thread is an example of how you're not forced to use a runtime that implements these primitives.
 Other than that, a "D with owned pointers" and an opt-in GC
 relegated to a library implementation (maybe with some minimal
 language hooks) would be my language of choice. I love D's
 templates.

The GC is a library implementation with language hooks.
 I understand that this would be a complete redesign of the
 language and Phobos, and would break most of the D code out
 there. That's why it won't happen.

There aren't that many language features that depend on a GC, and the ones that do can be disabled with a custom runtime. druntime/Phobos are designed for a mixed allocation strategy, where tracing GC is one of the strategies used. Many Phobos types and functions do not rely on a GC.
 Everybody seems to realize this, finally. Just have a look at 
 the
 last (long) GC thread. Comparisons of D with Java and C# all 
 over.

Java and C# do not function well without GC at all (disregarding the effort you'd have to go through to get a Java or C# implementation without GC). D definitely *does*.
Jan 25 2014
prev sibling next sibling parent "anonymous" <anonymous example.com> writes:
On Saturday, 25 January 2014 at 22:15:29 UTC, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

 If your language does not have the C / C++ *semantics* of new /
 delete,

C doesn't have new/delete.
 or, better yet, owned pointers in Rust that automatically
 free their memory when they go out of scope, but instead forces 
 a
 GC down your throat for some of the most basic array

You can do C style array/pointer fiddling in D without the GC. You can't new or append, but you can't do that in C either. Seems to me, you're comparing more to C++/Rust than to C.
 and standard library functionality,

You have access to C's standard library from D.
 you will never, ever write kernel level
 code that stands a chance against C, C++ or Rust.

Your points don't apply to D vs C.
Jan 25 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Saturday, 25 January 2014 at 23:01:12 UTC, Jakob Ovrum wrote:
 Setting aside syntax, it's extremely trivial to implement 
 C++-style new/delete as function templates.

I am interested in this. I am currently using new/destroy. If you know of a way to use templates for this (aside from RefCounted and Scoped) please elaborate. It would be quite helpful for me.
 The GC is a library implementation with language hooks.

I am implementing a custom runtime and am very interested in this as well. This was the heart of my previous post. I see that the runtime hooks can be used to implement an alternate GC, but do you think these hooks could be used to implement a reference counted, owned pointer, etc... implementation as an alternative to a GC? Are there any currently missing runtime hooks preventing such an implementation? Mike
Jan 25 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
On Saturday, 25 January 2014 at 23:02:44 UTC, anonymous wrote:
 C doesn't have new/delete.

Please read-before-write (non atomic operation): C has the *semantics* of new / delete with malloc and free. D has too, if you constrain yourself to memory allocation via std.c.stdlib.malloc and free, which means no object allocation with *new* anymore. We don't want to constrain ourselves to that, do we? What we want is malloc/free *semantics* throughout if we choose so.
 You can do C style array/pointer fiddling in D without the GC. 
 You can't new or append, but you can't do that in C either. 
 Seems to me, you're comparing more to C++/Rust than to C.

 You have access to C's standard library from D.

I'm aware that I can use a D subset that is roughly equivalent to C and C's standard library. This subset is precisely what I am *not* talking about. I am talking about all the great modern architectural features of D and Phobos.
 Your points don't apply to D vs C.

It does if I want to use D in its entirety not just its subset equivalent to C. So if I want C++ new/delete or Rust's owned pointer semantics I should constrain myself to D's subset that is equivalent to C and forget about Phobos altogether? Welcome to the *new* D.
Jan 25 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
On Saturday, 25 January 2014 at 23:01:12 UTC, Jakob Ovrum wrote:
 Setting aside syntax, it's extremely trivial to implement 
 C++-style new/delete as function templates.

Fine. Can I use it with Phobos?
 Array functionality *that C and C++ do not have*. D slices are 
 plenty useful without the primitives that allocate, such as 
 concatenation.

I love D's slices. Super elegant. Can't imagine how one could ever invent half slices, i.e. array pointers without length ;)
 This very thread is an example of how you're not forced to use 
 a runtime that implements these primitives.

We must be in two different threads: On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev wrote:
 - A "standard library" which does not require runtime support, 
 but can also be used when using Druntime (as a GC-free 
 alternative to Phobos)

On Wednesday, 22 January 2014 at 04:13:41 UTC, Mike wrote:
 Phobos will not be useful to me unless I can find some way to 
 make it lighter and GC-Free. ... I would _really_ like to have 
 the
 whole concept of a GC decoupled from D. There are many suitable 
 memory models to use with D, and the memory model seems like it 
 should be a platform feature, not a language feature.

I understand from this thread that we are at a GC-free Hello World at this point. Well ...
 The GC is a library implementation with language hooks.

No it is not. It is part of the language. Every allocation *inside* the language (i.e. with new) goes through the GC.
 There aren't that many language features that depend on a GC, 
 and the ones that do can be disabled with a custom runtime.

I don't want to miss out on a single language feature of D. Where is that GC-free runtime?
 druntime/Phobos are designed for a mixed allocation strategy, 
 where tracing GC is one of the strategies used. Many Phobos 
 types and functions do not rely on a GC.

In theory (which gives me hope), not in practice. Where is that GC-free Phobos?
 Java and C# do not function well without GC at all 
 (disregarding the effort you'd have to go through to get a Java 
 or C# implementation without GC).

 D definitely *does*.

Again, this gives me hope. Please Walter, Andrei: search all the forums for "I want D with opt-in GC" (fuzzy search, that is) and count matches.
Jan 25 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 26 January 2014 at 02:32:29 UTC, Frank Bauer wrote:
 Fine. Can I use it with Phobos?

Quite a bit of it, actually. There's significant parts of phobos which don't do allocation at all: std.algorithm and std.digest are two big parts that are pretty good about it.
 No it is not. It is part of the language. Every allocation 
 *inside* the language (i.e. with new) goes through the GC.

Not necessarily, you can change the allocation strategy if you want by providing your own function. _d_newclass is a major example. The language sometimes makes tracking the reference really hard, for example, getting tho pointer to free with a closure is tricky, and tracking temporaries made with the binary concat operator. (You could perhaps do it but it would be really easy to lose a reference somewhere and leak it, I think binary concat should just be left unimplemented if you aren't using the gc (and avoided even if you are in any performance critical section)). So yeah, it is possible to go without the GC with the D language by changing runtime functions but the language doesn't help you get it *right*.
Jan 25 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Sunday, 26 January 2014 at 02:03:29 UTC, Frank Bauer wrote:
 On Saturday, 25 January 2014 at 23:02:44 UTC, anonymous wrote:
 You have access to C's standard library from D.

I'm aware that I can use a D subset that is roughly equivalent to C and C's standard library. This subset is precisely what I am *not* talking about. I am talking about all the great modern architectural features of D and Phobos.

You're asking for something to exist right now, which doesn't. You want a GC-free minimum runtime which allows use of the majority of the language (actually you sound like you want all, but that can't happen), and a standard library which can be used GC-free. This does not exist and everyone is well aware of it, hence the question of this thread "Current state of "D as a better C" (Windows)?" However your initial post claims that the language won't let us achieve that state, and people here are trying to explain that simply isn't true, we just don't have the libraries built up yet. Someone needs to take the approach of using D as C, that means give up all the nice feature of D and use it as if it were C. That person will then need to build out and submit patches to bring more of the D language into this "C" world.
Jan 25 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
 Adam: thanks for pointing out _d_newclass for me. But for 
classes I would just simply use custom allocators and 
deallocators and use new and delete as in C++.

Leaves dynamic arrays and the rest of Phobos (which parts?) which 
I have to forego if I don't like GC (but I do like slices so 
much, not much left for me to slice then ...). And no, anonymous, 
it's not just reallocating or appending to arrays, every dynamic 
array is allocated on the GC'd heap from the start.

I would really like to have something like Rust's owned pointer 
and refernce semantics in D by default:

Allocation with new gives an *owning pointer* similar to 
std.typecons.Unique, that frees its associated memory 
automatically when it goes out of scope.

Creating a *reference* from an owning pointer gives a 
'non-owning' pointer with no effect on automatic memory 
management whatsoever, but with the compiler-enforced requirement 
that its associated owning pointer can not be mutated as long as 
there are outstanding references in scope. Also the lifetime of a 
reference can not exceed the lifetime of the owning pointer it 
stems from.

This is why even a working 
(http://forum.dlang.org/thread/l96otj$tsj$1 digitalmars.com) 
std.typecons.Unique is not enough. The relationship between 
owning pointers and references needs compiler enforcement.

 Andrei: Short of dumping all GC-dependent language concepts, GC 
collection cycle times are key, no matter how little litter is 
produced. Hope the GC tracing algorithm you are implementing for 
the new custom allocators does not have to chase references too 
deep into the heap. Maybe implementing Rust-style owning pointers 
and references into the core language could turn out to be less 
time-consuming? We could still have GC around as an option a la 
GCollect!MyClass.
Jan 25 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
On Sunday, 26 January 2014 at 04:18:47 UTC, Jesse Phillips wrote:
 However your initial post claims that the language won't let us 
 achieve that state, and people here are trying to explain that 
 simply isn't true, we just don't have the libraries built up 
 yet.

Wrong. Read one post above: On Sunday, 26 January 2014 at 03:29:00 UTC, Andrei Alexandrescu wrote:
 GC is a language feature of D.

 As funny as that may seem (as a comeback timed almost too well) 
 this has interesting truth to it. Some convenience features of 
 D require GC to work, and that's as cut and dried as it gets.

But I agree with Andrei: it requires reason and measure. I just think that D could keep all of the features that make it great (modules, templates, slices are without rival, Phobos is well laid out), even after a complete overhaul of the memory system. Or watch part of the (real) systems programmers move from C++ to Rust (should it reach a stable 1.0, it already compiles itself) and the Java / C# crowd stay where they are.
Jan 25 2014
prev sibling next sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Sunday, 26 January 2014 at 05:17:35 UTC, Frank Bauer wrote:
  Adam: thanks for pointing out _d_newclass for me. But for 
 classes I would just simply use custom allocators and 
 deallocators and use new and delete as in C++.

Aye, I don't recommend overloading `new` to mean something else. Currently `new` = GC allocation, and code can rely on that. Better to have such code break (by not defining _d_newclass) than silently leak or worse.
 Leaves dynamic arrays and the rest of Phobos (which parts?) 
 which I have to forego if I don't like GC (but I do like slices 
 so much, not much left for me to slice then ...). And no, 
 anonymous, it's not just reallocating or appending to arrays, 
 every dynamic array is allocated on the GC'd heap from the 
 start.

Huh? You can slice any pointer: --- T[] allocArray(T)(size_t num) { if (auto chunk = cast(T*)calloc(num, T.sizeof)) return chunk[0 .. num]; else onOutOfMemoryError(); } --- As well as fixed-length arrays: --- char[128] buffer = ...; auto myStackString = buffer[0 .. len]; --- Slices are plenty useful without GC.
Jan 25 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Sunday, 26 January 2014 at 03:29:00 UTC, Andrei Alexandrescu 
wrote:

 GC is a language feature of D.

I hope what you mean is "automatic memory management is a feature of D". I want to clarify my previous statements by saying automatic memory management is something that I find attractive in D, and something I hope to build in my runtime. But as a rude janitor that barges into the party, orders the band to stop playing and the dancers to stop moving so he can sweep the floor (i.e. GC) ... There must be a better way...and I hope some of you will help me find it. Mike
Jan 26 2014
prev sibling next sibling parent "anonymous" <anonymous example.com> writes:
On Sunday, 26 January 2014 at 02:03:29 UTC, Frank Bauer wrote:
 C has the *semantics* of new / delete with malloc and free. D 
 has too, if you constrain yourself to memory allocation via 
 std.c.stdlib.malloc and free, which means no object allocation 
 with *new* anymore. We don't want to constrain ourselves to 
 that, do we? What we want is malloc/free *semantics* throughout 
 if we choose so.

 I'm aware that I can use a D subset that is roughly equivalent 
 to C and C's standard library. This subset is precisely what I 
 am *not* talking about. I am talking about all the great modern 
 architectural features of D and Phobos.

 Your points don't apply to D vs C.

It does if I want to use D in its entirety not just its subset equivalent to C. So if I want C++ new/delete or Rust's owned pointer semantics I should constrain myself to D's subset that is equivalent to C and forget about Phobos altogether? Welcome to the *new* D.

Welcome to "D as a better C", not "D as a better 'better C' than C++ and Rust". You might shed a tear for all the goodness you're missing out on, but there's still templates, slicing, methods, etc left.
Jan 26 2014
prev sibling next sibling parent "Frank Bauer" <y z.com> writes:
As you heard Andrei, too many *language features*, NOT *user 
code* rely on objects staying around till the GC collects them. 
It's not a matter of malloc/free everything in your user code. D 
as a language would not work with this little gcstub.
Jan 26 2014
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Wednesday, 22 January 2014 at 02:18:43 UTC, Vladimir Panteleev 
wrote:
 Hi,

 What's the current situation of using D without Phobos/Druntime?

 Sometimes, it's quite useful to be able to use D to create tiny 
 EXEs/DLLs. For example, create simple DLL plugins that will be 
 loaded into non-D programs, or even not have a 50-line program 
 take half a meg. Additionally, some tasks require injecting a 
 DLL systemwide (into every running process), so it is critical 
 to minimize the impact by being as unintrusive as possible and 
 having a minimal overhead.

 I've started a "framework" for this purpose a while ago[1], but 
 I heard of people starting similar projects since then and I 
 think they've gotten further than me. Would there be any 
 interest in collaborating and turning this into a community 
 project?

 I think some goals would be:
 - A "runtime" (object.d) which provides either stubs for the 
 current compiler hooks, or implementations with minimal 
 dependencies
 - A "standard library" which does not require runtime support, 
 but can also be used when using Druntime (as a GC-free 
 alternative to Phobos)
 - Using MSVCRT on 32-bit as well
 - Options to enable/disable various runtime features? For 
 example, a project may want to use exceptions (which requires 
 ClassInfo) but not the GC.
 - Support for the Unilink linker? I think it is better at 
 generating smaller binaries that OPTLINK.

 There is also some ongoing work on the -betterC switch[2], 
 which I'm quite excited about, but it seems to be currently 
 blocked on waiting for the next DMD release as the changes[3] 
 might break a lot of code.

  [1]: https://github.com/CyberShadow/SlimD
  [2]: https://d.puremagic.com/issues/show_bug.cgi?id=11881
  [3]: https://github.com/D-Programming-Language/dmd/pull/2561

There was a minimal runtime library project somewhere on Bitbucket if I remember. I think the goal is the same. You could borrow some code from there perhaps. In general I totally agree with you. I wanted to do the same, but decided to wait for allocators...
Jan 26 2014
prev sibling next sibling parent "Dejan Lekic" <dejan.lekic gmail.com> writes:
On Saturday, 25 January 2014 at 22:15:29 UTC, Frank Bauer wrote:
 D is NOT a systems language. So no point in application level
 comparison with C. Set aside syntactic similarities.

 If your language does not have the C / C++ *semantics* of new /
 delete, or, better yet, owned pointers in Rust that 
 automatically
 free their memory when they go out of scope, but instead forces 
 a
 GC down your throat for some of the most basic array and 
 standard
 library functionality, you will never, ever write kernel level
 code that stands a chance against C, C++ or Rust.

 Other than that, a "D with owned pointers" and an opt-in GC
 relegated to a library implementation (maybe with some minimal
 language hooks) would be my language of choice. I love D's
 templates.

 I understand that this would be a complete redesign of the
 language and Phobos, and would break most of the D code out
 there. That's why it won't happen.

 Everybody seems to realize this, finally. Just have a look at 
 the
 last (long) GC thread. Comparisons of D with Java and C# all 
 over.

This post made me laugh. Thanks for the entertainment.
Jan 26 2014
prev sibling parent "JR" <zorael gmail.com> writes:
On Sunday, 26 January 2014 at 08:24:55 UTC, Mike wrote:
 ... But as a rude janitor that barges into the party, orders 
 the band to stop playing and the dancers to stop moving so he 
 can sweep the floor (i.e. GC) ...

Nominated for best stop-the-world GC analogy 2014. Cannot unsee. (un-think?)
Jan 26 2014