www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Disadvantages of building a compiler and library on top of a specific

reply "Frustrated" <c1514843 drdrb.com> writes:
Any time you hard code code something you are also creating
constraints.  If people build something on top of what you have
built they inherit those constrains.

Take building a house, once the foundation is laid and the
framework is added it becomes next to impossible to go back and
modify the foundation.

One big problem with D is that the both the runtime and library
are hard coded to use AGC. Switching these to use ARC or MMM are
just building different foundations but all creating very serious
constraints on the the users of D.

Because the runtime and library are created and maintained by
some dedicated and smart people, the "simplistic" way of memory
management does not need to be taken. (I realize it seems like
more work but it isn't given the n-fold benefit it creates for
all those that use it)

So, why not create a design pattern that allows one, at compile
time, to use any specific memory management technique they want
for the runtime and library(and user code)?

e.g., switches could be used to recompile the source when a
different pattern is desired or libraries could be created that
use different methods. e.g., phobos64GC.lib.

To make any of this work, and work efficiently, one must abstract
what is common between all memory management techniques.

Since allocation is always explicitly defined, it is easy to
handle. Which every  memory management technique is used, it
would be called/notified on allocation. The deallocation is the
tricky part. (reallocation is another potential issue)

For manual memory management the deallocation is explicitly
called by the user. This in and of itself is significant and
either allows the compiler to know that manual memory management
is used or to replace it with automatic management if desired.

e.g.,

{
auto a = new!strategy A;

release a; // <- explicit deallocation: replaced with
strategy.explicitDeallocation(a);
}

vs

{
auto a = new!strategy A;

// implicit deallocation: compiler inserts
strategy.scopeDeallocation(a);
}

implicit deallocation could necessarily be handled by the GC.
Basically if the programmer forgets to deallocate a type then it
is automatically handled(by say AGC or ARC or whatever). The
programmer, of course, should be allowed to deal with how this
actually behaves, such as make using a weak reference.

The key point here is that if we abstract the memory allocation
and deallocation schemes then many memory management techniques
should be possible(and possibly simultaneously with no hard coded
reliance on any specific technique.

Both manual and automatic schemes could be used simultaneously in
that if the user fails to release memory explicitly then
automatic releasing is used(AGC and/or ARC). This allows more
predicable memory management and in a perfect world, if the
programmer always properly released memory then no automatic
collections would occur. In the real world the programmer could
disable automatic collections and create weak references or use
automatic collections only on the debug build and log when they
occur to fix(e.g., he forgot to create a weak reference or
release memory explicitly). This could occur on production builds
too, logged, and reported. (so the production code might not be
ideal performance wise but be fixed in future builds).

Different allocation strategies have different ways have handling
how they deal with explicit and implicit calls. One strategy
could report all cases of implicit scope deallocation for the
programmer to look at. Another could completely ignore any
implicit deallocations which would turn off any automatic
collections = most performant but least safe.


One could define a global strategy and then local strategies
e.g., new A uses global strategy and new!myMMM A uses myMMM
strategy for the variable A. The interactions of strategies would
be relatively easy I think. If say, we have

class A
{
     B b;
     this() { b = new!strategy Y B; }
}

auto a = new!strategyX A;

Then on release of A, strategy X would call strategy Y's
deallocation for B. If it is "non-deterministic" then a
notification would be setup so that when b is actually released a
can be fully released. In some cases this could occur immediately
but is meant more for circular references.

e.g., if we have

class B
{
      A aa;
}

auto b = new!strategyZ B;
b.aa = a;

when we free a, b is subjected to freeing, which involves freeing
aa, which involves freeing a. Since a is already in the state of
being free'ed, it can be completely free'ed.

Of course, such a system might be slow if not properly designed
or may just be inherently slow.


Regardless, it is a strategy and other strategies might not work
this way(but there would still need to be some commonality
between them and it would be up to the compiler to optimize
things as much as possible.

The benefit of such a method is that one could test various
performance of different strategies and even, say, attempt to
move one automatic management(implicit deallocations) to explicit
to improve performance.

Basically the compiler could insert implicit deallocations
anywhere it things they are suppose to go... e.g., for local
variables that never have outside references. The allocation
strategy could do nothing(e.g., AGC) or immediately free(e.g.,
malloc) or reference count(e.g., ARC) or even throw an
error(e.g., MMM: "oops! I forget to deallocate or use  weak).

But what's great, is that by using a pattern for memory
allocation and deallocation, regardless if the above method is
the basis or not, D and phobos are no longer hard coded to one
scheme or another and a proper foundation is laid instead one
that is inflexible and limiting. There would no longer be
arguments about which memory management method is best and more
time with getting things done. Also, it would be very appealing
to the outside world because, say, they need to write some code
for the pic16 and need a very specific way of handling memory...
they'll much more likely be able to do it efficiently in D than
any other language... since not only does D offer complete
abstinence of built in management, it would also allow
"strategies" to be used.

Anyways, just an idea... hopefully good enough to get the point
across that we need a better way and we all can stop arguing over
specific details that are somewhat moot.
Feb 06 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 9:54 AM, Frustrated wrote:
 Any time you hard code code something you are also creating
 constraints.  If people build something on top of what you have
 built they inherit those constrains.

Nice thought.
 So, why not create a design pattern that allows one, at compile
 time, to use any specific memory management technique they want
 for the runtime and library(and user code)?

We have a very powerful mechanism for what could be called "statically deferred committal": templates. Phobos did a great job at not committing to e.g. range representation or string representation. It seems to me we have a clean shot at applying the wealth of knowledge and experience we accumulated with that, to deferring approaches to allocation.
 e.g., switches could be used to recompile the source when a
 different pattern is desired or libraries could be created that
 use different methods. e.g., phobos64GC.lib.

That won't even be necessary with templates. It would be necessary if we have some sort of global flag dictating a fundamental fork.
 To make any of this work, and work efficiently, one must abstract
 what is common between all memory management techniques.

 Since allocation is always explicitly defined, it is easy to
 handle. Which every  memory management technique is used, it
 would be called/notified on allocation. The deallocation is the
 tricky part. (reallocation is another potential issue)

Yah, std.allocator aims at formalizing that (in an untyped manner).
 For manual memory management the deallocation is explicitly
 called by the user. This in and of itself is significant and
 either allows the compiler to know that manual memory management
 is used or to replace it with automatic management if desired.

 e.g.,

 {
 auto a = new!strategy A;

 release a; // <- explicit deallocation: replaced with
 strategy.explicitDeallocation(a);
 }

 vs

 {
 auto a = new!strategy A;

 // implicit deallocation: compiler inserts
 strategy.scopeDeallocation(a);
 }

I don't think this works. The scope of "strategy" and "a" may be unrelated. More importantly a may be escaped (e.g. by passing it to functions etc) unbeknownst to "strategy", which in fact suggests that the strategy must be somehow known by "a" independently. (Stopped reading here.) Andrei
Feb 06 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 12:01 PM, Frustrated wrote:
 See the other post about this. scopeDeallocation meant to simply
 signal that the scope of a has ended but not necessarily there
 are no references to a.

So that's a struct destructor. Andrei
Feb 06 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 12:51 PM, Frustrated wrote:
 On Thursday, 6 February 2014 at 20:24:55 UTC, Andrei Alexandrescu
 wrote:
 On 2/6/14, 12:01 PM, Frustrated wrote:
 See the other post about this. scopeDeallocation meant to simply
 signal that the scope of a has ended but not necessarily there
 are no references to a.

So that's a struct destructor. Andrei

Well, except it hooks into the memory allocation strategy. I'm not saying that what I outlined above is perfect or the way to go but just an idea ;) If new had some way to pass an "interface" that contained the allocation strategy details and one had the notation like new!MyManualAllocator A; then I suppose A's destructor could call MyManualAllocator's "scopeDeallocator" method and you wouldn't need an implicit call there.

What is MyManualAllocator - type or value? I should emphasize that obsessing over the syntax is counterproductive. Call a blessed function and call it a day. Andrei
Feb 06 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Ola Fosheim Grøstad" " wrote in message 
news:fbsmcitanisyitajkvcb forum.dlang.org...

 Make a decision for where you want to go at the language specification 
 level. Forget about the current compiler/runtime. Where do you want to go?

Something tells me you're not a compilers guy. I want to go somewhere with an actual working compiler.
Feb 08 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Ola Fosheim Grøstad" " wrote in message 
news:frdycuzhjstlmgvgcjhy forum.dlang.org...

 Nothing I wrote prevents that. You just don't get optimal performance if 
 you base it off a backend that is optimized for a different language.

Not in theory, but it's like saying we shouldn't design the language based on semantics of existing processors, because it will lead to non-optimal performance on quantum computers. The fact these tools/hardware/backends already exist is worth a huge amount in producing a language that is useful, even if not optimally optimal.
Feb 08 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 08.02.2014 19:09, schrieb "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>":
 On Saturday, 8 February 2014 at 17:08:44 UTC, Daniel Murphy wrote:
  ...

 I think D could steal the VM/OS/embedded space with a very focused
 effort. C/C++ is ahead, but I believe D can surpass that if you go
 beyond C/C++ and forget about pleasing the Java/C# crowd.

Speaking from my Java/C# experience, D has a big up-hill battle against those ecosystems: - Tooling is great, we love our IDEs and just imagine having something like VisualVM to track down unwanted memory allocations and other contention points - Compilation to native code is also available, if one wishes to do so. - There are lots of embedded VM for both ecosystems capable to run in boards as small as a SIM card - High performance trading systems with low latency are moving away from C++ systems to JVM/.NET ones. Quite common in city, in London, for example. - The going native wave, has made both JVM guys think about including AOT as part of the standard JDK, add value types, make unsafe package official as part of the post Java 8 features. Whereas the .NET camp has a new JIT (RyuJIT) in the works, and a switch to full native deployment is being worked on (ProjectN). Triggered mostly by what seems to be the C++/CX failure. I wish D also finds its place in the mainstream, it is a hard fight nonetheless. -- Paulo
Feb 08 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
I like the idea, but instead of new!strategy I would prefer:
use  strategy(RefCount)
or
use  strategy(GC) // Default

We would have in druntime some functions which are tagged with 
 strategy(Whatever) (which are invoked if the use comes):
----
 strategy(C_Memory)
T[] new(T : U[], U)(size_t count) {
	return (cast(U*) calloc(count, T.sizeof))[0 .. count];
}

 strategy(C_Memory)
void delete(T : U[], U)(ref U[] arr) {
	free(arr.ptr);
	arr = null;
}
----

It's not perfect, just a thought.
Feb 06 2014
prev sibling next sibling parent "fra" <a b.it> writes:
On Thursday, 6 February 2014 at 18:18:49 UTC, Andrei Alexandrescu 
wrote:
 On 2/6/14, 9:54 AM, Frustrated wrote:
 {
 auto a = new!strategy A;

 // implicit deallocation: compiler inserts
 strategy.scopeDeallocation(a);
 }

I don't think this works. The scope of "strategy" and "a" may be unrelated. More importantly a may be escaped (e.g. by passing it to functions etc) unbeknownst to "strategy", which in fact suggests that the strategy must be somehow known by "a" independently.

control of the object is now passed to the "Strategy Manager", that could be a reference counter that decreases its counter, the GC that takes care of everything, and so on.
Feb 06 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Thursday, 6 February 2014 at 18:44:49 UTC, fra wrote:
 On Thursday, 6 February 2014 at 18:18:49 UTC, Andrei 
 Alexandrescu wrote:
 On 2/6/14, 9:54 AM, Frustrated wrote:
 {
 auto a = new!strategy A;

 // implicit deallocation: compiler inserts
 strategy.scopeDeallocation(a);
 }

I don't think this works. The scope of "strategy" and "a" may be unrelated. More importantly a may be escaped (e.g. by passing it to functions etc) unbeknownst to "strategy", which in fact suggests that the strategy must be somehow known by "a" independently.

control of the object is now passed to the "Strategy Manager", that could be a reference counter that decreases its counter, the GC that takes care of everything, and so on.

Yes, ScopeDeallocation would either be nop for manual allocation or a decrements of a reference counter. For example, if you were using manual allocation but backed by ARC or GC then scopeDeallocation would actually do something. If you wanted completely control then it could be a nop or a warning. Basically have the compiler fill in all the missing information that it can in "implicit calls" to the strategy method. These could be "overridden" to do whatever one wants depending on what the goal is. If an explicit call is made then the compiler won't add the implicit call.
Feb 06 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Thursday, 6 February 2014 at 18:41:25 UTC, Namespace wrote:
 I like the idea, but instead of new!strategy I would prefer:
 use  strategy(RefCount)
 or
 use  strategy(GC) // Default

 We would have in druntime some functions which are tagged with 
  strategy(Whatever) (which are invoked if the use comes):
 ----
  strategy(C_Memory)
 T[] new(T : U[], U)(size_t count) {
 	return (cast(U*) calloc(count, T.sizeof))[0 .. count];
 }

  strategy(C_Memory)
 void delete(T : U[], U)(ref U[] arr) {
 	free(arr.ptr);
 	arr = null;
 }
 ----

 It's not perfect, just a thought.

I'm sure there are a lot of ways to do it. I think the hard part is coming up with a cohesive and elegant solution that is efficient. By providing an allocation strategy we can deal with specific memory issues.
Feb 06 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Thursday, 6 February 2014 at 18:18:49 UTC, Andrei Alexandrescu
wrote:
 On 2/6/14, 9:54 AM, Frustrated wrote:
 Any time you hard code code something you are also creating
 constraints.  If people build something on top of what you have
 built they inherit those constrains.

Nice thought.
 So, why not create a design pattern that allows one, at compile
 time, to use any specific memory management technique they want
 for the runtime and library(and user code)?

We have a very powerful mechanism for what could be called "statically deferred committal": templates. Phobos did a great job at not committing to e.g. range representation or string representation. It seems to me we have a clean shot at applying the wealth of knowledge and experience we accumulated with that, to deferring approaches to allocation.
 e.g., switches could be used to recompile the source when a
 different pattern is desired or libraries could be created that
 use different methods. e.g., phobos64GC.lib.

That won't even be necessary with templates. It would be necessary if we have some sort of global flag dictating a fundamental fork.

Yeah, I was thinking templates could be used here to good effect.
 To make any of this work, and work efficiently, one must 
 abstract
 what is common between all memory management techniques.

 Since allocation is always explicitly defined, it is easy to
 handle. Which every  memory management technique is used, it
 would be called/notified on allocation. The deallocation is the
 tricky part. (reallocation is another potential issue)

Yah, std.allocator aims at formalizing that (in an untyped manner).

yeah, and I think it is definitely a step in the right direction... Just have to get the core and library to be along those lines.
 For manual memory management the deallocation is explicitly
 called by the user. This in and of itself is significant and
 either allows the compiler to know that manual memory 
 management
 is used or to replace it with automatic management if desired.

 e.g.,

 {
 auto a = new!strategy A;

 release a; // <- explicit deallocation: replaced with
 strategy.explicitDeallocation(a);
 }

 vs

 {
 auto a = new!strategy A;

 // implicit deallocation: compiler inserts
 strategy.scopeDeallocation(a);
 }

I don't think this works. The scope of "strategy" and "a" may be unrelated. More importantly a may be escaped (e.g. by passing it to functions etc) unbeknownst to "strategy", which in fact suggests that the strategy must be somehow known by "a" independently.

See the other post about this. scopeDeallocation meant to simply signal that the scope of a has ended but not necessarily there are no references to a. Would be used more in AGC and ARC than anything but could be used in other ways. The idea, I think, is to have the compiler insert as much implicit information as possible to handle stuff with the ability to override with explicit calls and to even override what the implicit calls do. It would sort of have the effect of adding "hooks" into the compiler to tell it what to do on allocations and deallocations.
Feb 06 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Thursday, 6 February 2014 at 20:24:55 UTC, Andrei Alexandrescu
wrote:
 On 2/6/14, 12:01 PM, Frustrated wrote:
 See the other post about this. scopeDeallocation meant to 
 simply
 signal that the scope of a has ended but not necessarily there
 are no references to a.

So that's a struct destructor. Andrei

Well, except it hooks into the memory allocation strategy. I'm not saying that what I outlined above is perfect or the way to go but just an idea ;) If new had some way to pass an "interface" that contained the allocation strategy details and one had the notation like new!MyManualAllocator A; then I suppose A's destructor could call MyManualAllocator's "scopeDeallocator" method and you wouldn't need an implicit call there. In either case though, it is implicit and done behind the scenes by the compiler. But if it's done by the destructor then it is, in some sense, "hidden" and it feels hard coded since we normally don't think of the user being able to control how an object is able to destruct itself. (usually this is done by the person that creates the object type) In some sense, destructors clean up their own mess they created but not make assumptions about how they were created. The problem then becomes, does new!MyManualAllocator A; flow through and A uses MyManualAllocator internally or does it use it's own or can we for A to use MyManualAllocator(or some other allocator)?
Feb 06 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Thursday, 6 February 2014 at 21:17:24 UTC, Andrei Alexandrescu
wrote:
 On 2/6/14, 12:51 PM, Frustrated wrote:
 On Thursday, 6 February 2014 at 20:24:55 UTC, Andrei 
 Alexandrescu
 wrote:
 On 2/6/14, 12:01 PM, Frustrated wrote:
 See the other post about this. scopeDeallocation meant to 
 simply
 signal that the scope of a has ended but not necessarily 
 there
 are no references to a.

So that's a struct destructor. Andrei

Well, except it hooks into the memory allocation strategy. I'm not saying that what I outlined above is perfect or the way to go but just an idea ;) If new had some way to pass an "interface" that contained the allocation strategy details and one had the notation like new!MyManualAllocator A; then I suppose A's destructor could call MyManualAllocator's "scopeDeallocator" method and you wouldn't need an implicit call there.

What is MyManualAllocator - type or value? I should emphasize that obsessing over the syntax is counterproductive. Call a blessed function and call it a day.

It would be more of an abstract type. Something special template aggregate that the compiler accesses to get code to "hook" in to the memory management parts of the code the compiler needs, but has delegated specifics to the "user". For example, When the compiler is parsing code and comes across the new keyword, it has hard coded what to do. Instead, if it delegated what to do to the code itself(that somebody writes external to the compiler) then it is more robust. It is exactly analogous to interfaces, classes vs non-oop programming. e.g., suppose we wanted a very generic compiler where the programmer could add his own keywords. Instead of hard coding the "actions" of the keywords we would delegate responsibility to the user and provide hooks. When the parser finds the keyword it simply calls code external to the compiler instead of hard coded internal code. Of course it can get complicated real quick but essentially that is what I am talking about with memory management here. The compiler delegates exactly what to do external code but provides the necessary hooks to properly deal with it. One can argue that we already have the ability to do that by overriding new and using destructors but these are not general enough as `new` is hard coded and destruction is not generic enough. The only way I can describe it properly is that it it would be nice to plug and play specific memory management allocators into the "compiler" so that almost anyone can achieve what they want. To do this requires more support from the compiler as is, it uses a hard coded version. It is exactly analogous to this: int x = 3; // <-- 3 is hard coded, not generic. Once compiled we can't change it. int x = file.read!int("settings.txt", 0); // <-- generic, x is not fixed at compile time to a specific value. If we need to change x we can do so. Now apply the same logic as above but to the compiler and memory management. Right now D is at the first case(D being X and the GC being 3) and we want to get to the second case(the text file being being a specific memory allocation method). file.read then is what needs to be come up with, which is the way to decouple the memory allocation scheme and the compiler's dependence on it. I have no solution to the above... just ideas that may lead to other ideas and hopefully a feasible solution.
Feb 06 2014
prev sibling next sibling parent =?UTF-8?B?Ik7DqW1ldGggUMOpdGVyIg==?= <hun.nemethpeter gmail.com> writes:
 It would be more of an abstract type. Something special template
 aggregate that the compiler accesses to get code to "hook" in to
 the memory management parts of  the code the compiler needs, but
 has delegated specifics to the "user".

It's intresting idea for me. If we mark a variable with a property, that how that memory is allocated then we got an marked AST node. And based on that mark we can perform AST manipulation during compile time. So if we marked a variable with ARC, then refcount decrease will injected during scope exit. If we mark a variable with GC, nothing will happen. We shoul allocate a separate memory area to refcounts, so that an RC variable can remain a pointer. These are just thoughts, nothing real proposal here... this is on my mind now..
Feb 07 2014
prev sibling next sibling parent "Frustrated" <c1514843 drdrb.com> writes:
On Friday, 7 February 2014 at 13:36:12 UTC, Németh Péter wrote:
 It would be more of an abstract type. Something special 
 template
 aggregate that the compiler accesses to get code to "hook" in 
 to
 the memory management parts of  the code the compiler needs, 
 but
 has delegated specifics to the "user".

It's intresting idea for me. If we mark a variable with a property, that how that memory is allocated then we got an marked AST node. And based on that mark we can perform AST manipulation during compile time. So if we marked a variable with ARC, then refcount decrease will injected during scope exit. If we mark a variable with GC, nothing will happen. We shoul allocate a separate memory area to refcounts, so that an RC variable can remain a pointer. These are just thoughts, nothing real proposal here... this is on my mind now..

Well, if one is providing hooks and provides the appropriate hooks then whatever strategy is used defines how it deals with it all. Basically the idea is to export the memory management work to external user code so it can be easily changed. In this case, for the most part, the compiler will not know that reference counting is be used(since it might not be) but scope hooks(or callbacks) need to be used where reference counting would need them.
Feb 07 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 07 Feb 2014 21:05:47 +0000
schrieb "Frustrated" <c1514843 drdrb.com>:

 On Friday, 7 February 2014 at 13:36:12 UTC, N=C3=A9meth P=C3=A9ter wrote:
 It would be more of an abstract type. Something special=20
 template
 aggregate that the compiler accesses to get code to "hook" in=20
 to
 the memory management parts of  the code the compiler needs,=20
 but
 has delegated specifics to the "user".

It's intresting idea for me. If we mark a variable with a property, that how that memory is=20 allocated then we got an marked AST node. And based on that=20 mark we can perform AST manipulation during compile time. So if=20 we marked a variable with ARC, then refcount decrease will=20 injected during scope exit. If we mark a variable with GC,=20 nothing will happen. We shoul allocate a separate memory area=20 to refcounts, so that an RC variable can remain a pointer.=20 These are just thoughts, nothing real proposal here... this is=20 on my mind now..

Well, if one is providing hooks and provides the appropriate hooks then whatever strategy is used defines how it deals with it all. =20 =20 Basically the idea is to export the memory management work to external user code so it can be easily changed. In this case, for the most part, the compiler will not know that reference counting is be used(since it might not be) but scope hooks(or callbacks) need to be used where reference counting would need them.

Can we just copy Rust already? Do you realize that you'll still need to have type constructors for your allocation schemes? A foo() { A a =3D new!myAlloc A(); return a; } void main() { A a =3D foo(); // All we know is we have a reference to A. // We need a type ctor to know we need to deallocate it // with myAlloc.scopeDesctructor. } --=20 Marco
Feb 07 2014
prev sibling next sibling parent "logicchains" <jonathan.t.barnard gmail.com> writes:
On Saturday, 8 February 2014 at 02:03:00 UTC, Marco Leise wrote:
 Can we just copy Rust already?

Assuming D was able to incorporate all of Rust's memory management features, what advantages would it offer over Rust apart from superior compile-time metaprogramming? Personally, I think working to improve the garbage collector could pay off in this regard; "Both Rust and D allow you to choose between using a garbage collector and using referencing counting, but D's garbage collector is a precise concurrent generational moving compacting collector, whereas Rust's is a non-parallel non-moving conservative collector that's orders of magnitude slower and uses much more memory." Currently, even Go has a better GC than D.
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 05:43:44 UTC, logicchains wrote:
 apart from superior compile-time metaprogramming? Personally, I 
 think working to improve the garbage collector could pay off in 
 this regard; "Both Rust and D allow you to choose between using 
 a garbage collector and using referencing counting, but D's 
 garbage collector is a precise concurrent generational moving 
 compacting collector, whereas Rust's is a non-parallel 
 non-moving conservative collector that's orders of magnitude 
 slower and uses much more memory." Currently, even Go has a 
 better GC than D.

Well, I think the main problem with D is that it is not a spec driven language, but an implementation driven language and that the implementation is based on compiler backends with very restrictive C semantics. D tries to bolt higher level functionality on top of that without opening up for efficient implementation of such features. For instance: 1. You could have GC free lambdas if you require that they only call functions that can work with fixed-size stack frames (so you allocate stack frames off a thread local heap where all allocation units are of a fixed size (say 1k)) and that they avoid recursion. 2. You could avoid lots of temporary allocations and copies if you could control what stack frames look like, by modifying the parent frame. 3. You could have zero overhead exception handling if you could get rid of the silly backwards-compatible driven design of Dwarf: - If you fully control code gen you can have dual return points with a fixed offset, so you can modify the return address on the stack by adding an offset before returning to indicate an error or just jumping. The penalty is low since predicted branches tend to be cheap. - If you don't need stack frames (by not having GC) you can maintain only frame pointers for function bodies that contains try blocks. Then jump straight to them on throw: Throw becomes: reg0 = exception_object JMP *(framepointer+CATCHADDR) Catch becomes: switch(reg0.someid){ ... default: framepointer = *(framepointer+NEXTPTR) JMP *(framepointer+CATCHADDR) } This would make for very efficient exception handling with no more setup cost than you have today for stack frames. I believe. It isn't reasonable to require that you write a backend from scratch, but I think it is reasonable to have a language spec that does not tie you to an implementation that will never go beyond C. If that makes adapting existing backends slightly less efficient, then that should be ok. It is more important to have a language spec that allows future compilers to be highly efficient. - If you want mandatory GC, ok, but then change the language spec to make it possible to write a powerful compiler. - If you 100% C++ parity/compatibility, great, but then make it just as easy (or easier) to write C++ type code without GC. Basically, you would have a language that consists primarily of syntactic sugar + some optimizing opportunities. - If you want better performance than C. Great! But don't limit the language to C-semantics in the backend. Make a decision for where you want to go at the language specification level. Forget about the current compiler/runtime. Where do you want to go?
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 15:57:17 UTC, Daniel Murphy wrote:
 I want to go somewhere with an actual working compiler.

Nothing I wrote prevents that. You just don't get optimal performance if you base it off a backend that is optimized for a different language. It is more important to have a coherent language spec that is targetting a usage domain than an optimal compiler for a suboptimal language design that is shoehorned into something it is not fit for.
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 17:08:44 UTC, Daniel Murphy wrote:
 Not in theory, but it's like saying we shouldn't design the 
 language based on semantics of existing processors, because it 
 will lead to non-optimal performance on quantum computers.

No, my focus is on what can run efficiently on hardware that exist and is becoming prevalent in the next 3-5 years. I want a system language focus that allow you to write efficient servers, that in the future will work well with cache level 1 transactional memory and other goodies that are coming. A language that makes it easy to write fast, robust OS-free servers with fixed memory allocation that you can upload as VMs so that you can have stable "game servers"/"web services" that are performant, stable, and cost efficient. No such language exist, a more focused D effort could take that niche and the future embedded space, I think. But you need to break away from C, C++, Java and C#. IMHO C-semantics is stuck in the 1960s. x86 is to some extent geared towards making it easy to make C-semantics more efficient, but you can easily get better performance by going to the machine code level in terms of structure. What is difficult today is specifying such structures, so people end up with more efficient C-semantics (like doing FSM in C rather than in a manner that is optimized for the hardware).
  The fact these tools/hardware/backends already exist is worth 
 a huge amount in producing a language that is useful, even if 
 not optimally optimal.

Sure, but that should not drive the language spec. That way you will never get the upper hand, you will never become more attractive than the competition. You can usually create reasonably efficient alternatives where the C-backend isn't cutting it in a proof-of-concept manner (like having extra data structures that you maintain on the side, or using DWARF for now, but admit that it is not your goal/standard!). I think the long term goal should be to have a backend that provides stuff that the C crowd does not have. Maybe when D is implemented in D you can move in that direction. D2 smells too much of "C++ with opinionated restrictions" with C#/Java stuff thrown in without really making OS level system programming more attractive. Unfortunately Go and Rust are currently also in that opinionated state that makes them atm not useful for systems level programming. And C/C++ is too unsuitable unless you have a lot of resources. But Go and Rust are more focused at a particular usage scenario (webserver/web browser) and their language design is thus focused. I think that is an advantage for them. I think D could steal the VM/OS/embedded space with a very focused effort. C/C++ is ahead, but I believe D can surpass that if you go beyond C/C++ and forget about pleasing the Java/C# crowd.
Feb 08 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 18:41:34 UTC, Paulo Pinto wrote:
 Speaking from my Java/C# experience, D has a big up-hill battle 
 against
 those ecosystems:

 - Tooling is great, we love our IDEs and just imagine having 
 something like VisualVM to track down unwanted memory 
 allocations and other contention points

 - Compilation to native code is also available, if one wishes 
 to do so.

I am sure you are right, and they are probably the better alternative if you have the economic margins and if stability is critical. The space I am thinking about is more that of scalable "artsy worlds", "freemium games" and "simple custom webservices/caches for mobile apps". It is important to keep costs down, both development and running cost because of the margins, so it should boot fast for scalability (and reboot after failure), easy to adapt and use memory well (for memcaching etc). The basic idea is to rent cheap cheap VMs close to the customers (in different countries) and be able to boot up more servers with little administration. I think C++ only works out if the design is ready before you code, and redesigning is painful. I fear Java and C# is a bit heavy in terms of memory and booting to run on cheap servers that should boot up fast. So I feel D and Go are currently the most promising alternatives, though I feel Go might be a bit difficult to work with and D is not quite there either. Maybe Rust will be ready in a year or two…
 - High performance trading systems with low latency are moving 
 away from
 C++ systems to JVM/.NET ones. Quite common in city, in London, 
 for example.

Yeah, they have the economic margins to purchase dedicated servers.
 - The going native wave, has made both JVM guys think about 
 including AOT as part of the standard JDK, add value types, 
 make unsafe package official as part of the post Java 8 
 features. Whereas the .NET camp has a new JIT (RyuJIT) in the 
 works, and a switch to full native deployment is being worked 
 on (ProjectN). Triggered mostly by what seems to be the C++/CX 
 failure.

 I wish D also finds its place in the mainstream, it is a hard 
 fight nonetheless.

It is, and I think you are quite right about the infrastructure part. On the client side I moved from Javascript to Dart and I think the productivity increased by a factor of 2 or something even though Dart compiles down to Javascript and has its own problems... But the IDE/library support really helps a lot. But actually, I don't think D needs to find its place in the mainstream. I think it needs to be the best option in its niche. And I think light weight, snappy servers that boot/reset in an instant is a niche it can grab.
Feb 08 2014