www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - More radical ideas about gc and reference counting

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Walter and I have had a long chat in which we figured our current 
offering of abstractions could be improved. Here are some thoughts. 
There's a lot of work ahead of us on that and I wanted to make sure 
we're getting full community buy-in and backup.

First off, we're considering eliminating destructor calls from within 
the GC entirely. It makes for a faster and better GC, but the real 
reason here is that destructors are philosophically bankrupt in a GC 
environment. I think there's no need to argue that in this community. 
The GC never guarantees calling destructors even today, so this decision 
would be just a point in the definition space (albeit an extreme one).

That means classes that need cleanup (either directly or by having 
fields that are structs with destructors) would need to garner that by 
other means, such as reference counting or manual. We're considering 
deprecating ~this() for classes in the future.

Also, we're considering a revamp of built-in slices, as follows. Slices 
of types without destructors stay as they are.

Slices T[] of structs with destructors shall be silently lowered into 
RCSlice!T, defined inside object.d. That type would occupy THREE words, 
one of which being a pointer to a reference count. That type would 
redefine all slice primitives to update the reference count accordingly.

RCSlice!T will not convert implicitly to void[]. Explicit cast(void[]) 
will be allowed, and will ignore the reference count (so if a void[] 
extracted from a T[] via a cast outlives all slices, dangling pointers 
will ensue).

I foresee any number of theoretical and practical issues with this 
approach. Let's discuss some of them here.


Thanks,

Andrei
Apr 30 2014
next sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 Walter and I have had a long chat in which we figured our 
 current offering of abstractions could be improved. Here are 
 some thoughts. There's a lot of work ahead of us on that and I 
 wanted to make sure we're getting full community buy-in and 
 backup.

 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).

An extreme one indeed, it would break a lot of my code. Every D project I wrote that does networking manages memory using a class that resides on the managed heap, but holds the actual wrapped data in the unmanaged heap. It has a number of advantages over other approaches, and I've posted it to the announce group, but the idea didn't seem to catch on. https://github.com/CyberShadow/ae/blob/master/sys/data.d I could migrate the concept to use reference counting (it was implemented in D1, before reference counting was possible), but that's my situation regarding breaking existing code.
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:31 PM, Vladimir Panteleev wrote:
 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:
 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this
 decision would be just a point in the definition space (albeit an
 extreme one).

An extreme one indeed, it would break a lot of my code. Every D project I wrote that does networking manages memory using a class that resides on the managed heap, but holds the actual wrapped data in the unmanaged heap.

So should I take it those classes all have destructors? -- Andrei
Apr 30 2014
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:
 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.

So should I take it those classes all have destructors? -- Andrei

(Yes, those destructors free the unmanaged memory.)
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:57 PM, Timon Gehr wrote:
 On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:
 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.

So should I take it those classes all have destructors? -- Andrei

(Yes, those destructors free the unmanaged memory.)

Thanks... that would need to change :o). -- Andrei
Apr 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:57 PM, Vladimir Panteleev wrote:
 On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei Alexandrescu wrote:
 An extreme one indeed, it would break a lot of my code. Every D project
 I wrote that does networking manages memory using a class that resides
 on the managed heap, but holds the actual wrapped data in the unmanaged
 heap.

So should I take it those classes all have destructors? --

One class, many instances (one for every chunk of data sent or received).

The question is, how comfortable are you with today's reality that some of those instances may be never destroyed? -- Andrei
Apr 30 2014
prev sibling next sibling parent reply "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 We're considering deprecating ~this() for classes in the future.

 [...]

 Slices T[] of structs with destructors shall be silently 
 lowered into RCSlice!T, defined inside object.d

I'm sure instead of outright deprecating destructor for class, we could just store them in a RCClass!C (which would be 2 words big). If you can get the scheme working with slices of structs with destructors, I don't see why the same scheme couldn't work with Class'es too.
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:33 PM, monarch_dodra wrote:
 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:
 We're considering deprecating ~this() for classes in the future.

 [...]

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d

I'm sure instead of outright deprecating destructor for class, we could just store them in a RCClass!C (which would be 2 words big).

That's an interesting idea, but the whole notion of converting to Object etc. makes the concept difficult (unless we define another root etc). Andrei
Apr 30 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 Walter and I have had a long chat in which we figured our 
 current offering of abstractions could be improved. Here are 
 some thoughts. There's a lot of work ahead of us on that and I 
 wanted to make sure we're getting full community buy-in and 
 backup.

Still no consideration for isolated.
 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.

Not sure how I feel about that. This has been proposed as a solution to some GC issue at last DConf and has been dismissed. Your post is unclear on why this decision has changed. There must be some new information or data that inform this decision. Or is it random ? Also, what about cycle in RC things ? Also, RC is good on top of GC, so you can collect loop, especially if we are going to RC automagically. That is a major issue. Finally, immutable is sharable accross thread. That mean, even if we bypass the type system, that RC must be atomic for immutable. As they convert automatically for co,st, that mean that all const code will be full of atomic increment/decrement. They are bad for the CPU, and cannot be optimized away. That mean we are back to having a const and a mutable version of functions for performance reasons.
 Also, we're considering a revamp of built-in slices, as 
 follows. Slices of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into RCSlice!T, defined inside object.d. That type 
 would occupy THREE words, one of which being a pointer to a 
 reference count. That type would redefine all slice primitives 
 to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[]) will be allowed, and will ignore the reference 
 count (so if a void[] extracted from a T[] via a cast outlives 
 all slices, dangling pointers will ensue).

This won't work. array have a lot of magic that cannot be created as library. ie: RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the compiler is concerned. This is why we still don't have any library that provides collection properly. One of the best is probably dcollection, and it has no solution for that problem. I'd like to see some core issue like that one before changing everything on top of it. When the whole thing is built on goo, readjusting the top to make it point upward is doomed to create the Pisa tower. Fine for tourists, but not very practical nor useful.
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:46 PM, deadalnix wrote:
 Still no consideration for isolated.

https://github.com/D-Programming-Language/dmd/pull/3452 may be of interest to you. -- Andrei
Apr 30 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 1:57 PM, deadalnix wrote:
 I already commented on past similar proposal. These are piling up more and more
 hacks to achieve roughly the same thing, badly. This makes it impossible to
 provide a 3rd party implementation that is compliant, is guaranteed to ends up
 in a nightmarish explosion of special cases, or to be vastly underpowered.

Actually, I think inference of the ability to implicitly and safely convert from mutable to immutable or shared to be very powerful. I also think it is superior than requiring the user to add ever more annotations, which some research (and my experience) shows that users are reluctant to do. I don't buy that it is hackish, etc., especially without more of a rationale as to why. A link to your previous comment would be useful.
Apr 30 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 6:50 PM, deadalnix wrote:
 On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:
 A link to your previous comment would be useful.

I've written several proposal regarding this. Please at least read them as what you just wrote only proves you are not well informed. Notably, the proposal do not require annotations from the users to do the kind of things are are currently being special cased.

A link would be nice.
Apr 30 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 11:57 PM, bearophile wrote:
 I agree the people are reluctant to add annotations, but I think it's also
 depends on the return of investment of the specific annotation. I am annotating
 all my code carefully with "pure", but the ROI of all those purity annotations
 is not great.

That's why I want to move more towards attribute inference.
 And I agree the piling of special cases is a not clean design. It's now simpler
 to ask the compiler if a certain assignment to immutable is allowed or not.
 Because I can't remember the supported cases.
 So I suggest a more principled and visible approach at the problem of ownership
 and uniqueness.

The "supported cases" are not random special case hacks. They are based on a solid principle - if the expression represents a unique reference to an object, then it is implicitly convertible to immutable or shared. For example, new int; Formerly, that created a pointer to a mutable int object. It could not be implicitly cast to a pointer to an immutable int. But we know that operator new returns the only pointer to what it created, so it should be convertible. This knowledge was simply added to the compiler. I don't see anything unprincipled about that, or hard to understand, or hackish, or whatever.
May 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:46 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:
 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.

Not sure how I feel about that.

"Ecstatic" is what you're looking for.
 This has been proposed as a solution to
 some GC issue at last DConf and has been dismissed. Your post is unclear
 on why this decision has changed. There must be some new information or
 data that inform this decision.

 Or is it random ?

I don't remember a proposal being made that made slices of structs with destructors distinct from other slices.
 Also, what about cycle in RC things?

Resource leaks may happen in programs in any language. I'm prepared to allow leaks in certain cases, and I posit here that such cases are rare (arrays of objects that embed references to the very array they're in).
 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC automagically.
 That is a major issue.

That stays, and it's useful. I'm talking destructors here.
 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.
 That mean we are back to having a const and a mutable version of
 functions for performance reasons.

I don't think that's the case. We could and should be able to only use refcounting for truly immutable slices. (Using a bit in the counter to indicate sharing comes to mind.)
 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

This won't work. array have a lot of magic that cannot be created as library. ie: RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the compiler is concerned.

We need to improve the language to allow for such. Did I mention it's not going to be easy? Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:24 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:
 I don't remember a proposal being made that made slices of structs
 with destructors distinct from other slices.

So the RC slice thing is what make all that worthwhile ?

I can't discuss this as I'm not sure what past discussions it refers.
 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC automagically.
 That is a major issue.

That stays, and it's useful. I'm talking destructors here.

Then the RCSlice do not provide any guarantee at all.

The guarantee it provides is once the refcount goes down to 0, elements will be destroyed.
 We need to improve the language to allow for such. Did I mention it's
 not going to be easy?

If I understand what you mentioned, RCSlice is what make to you the ditching of destructor worthwhile.

No, they are somewhat loosely related. Destructors are not called for arrays of struct _today_, so in that regard we're looking at improving a preexisting issue.
 The fact that this is not doable in
 current D should be a showstopper for this whole thread as all that is
 discussed here is dependent of what solution we choose for this problem
 (right now we have none).

This thread is a good place to discuss language improvements that make RCSlice possible. Andrei
Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:47 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:
 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.

How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable.

I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and interlocked refcounts go like 2, 4, 6, ... Then you do an unprotected read of the refcount. If it's odd, then it's impossible to having originated as an interlocked one. So proceed with simple increment. If it's even, do an interlocked increment. Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:08 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei Alexandrescu wrote:
 On 4/30/14, 2:47 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu wrote:
 Finally, immutable is sharable accross thread. That mean, even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code will be
 full of atomic increment/decrement. They are bad for the CPU, and
 cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.

How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable.

I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and interlocked refcounts go like 2, 4, 6, ... Then you do an unprotected read of the refcount. If it's odd, then it's impossible to having originated as an interlocked one. So proceed with simple increment. If it's even, do an interlocked increment. Andrei

I don't think I fully understand. Either all RC changes for a given type need to be atomic or none do, and that information is given by the type (everything that is immutable/const/shared). I don't see any feasible way of escaping this, or any advantage to a runtime convention like the odd/even trick above.

An object starting as shared or immutable would always need to be atomically refcounted. That information is statically known. For those we'd initialize the refcount to 2. An object starting as regular mutable would always be refcounted non-atomically. That's also known statically. So that constructor initializes the refcount to 1. Then a const object would dynamically choose the approach to refcounting depending on the counter's evenness. Andrei
Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:32 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:
 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or none do,
 and that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of escaping
 this, or any advantage to a runtime convention like the odd/even trick
 above.

If you CPU is decent you have some cache coherency protocol in place. This won't ensure that thing appears sequentially consistent, but you don't care here. You can proceed as follow in pseudo assembly : count = load count_addr need_atomic = count & 0x01 brtr atomic count = count + 2 store count count_addr br follow_up atomic: atomic_add count_addr 2 follow_up: // Code after increment goes here

brtr_atomic won't be needed on x86.
 Note that is working as count may not be the correct number in case of
 sharing, but will always have the same parity, so even reading the wrong
 value will make you branch properly and the value of count is not used
 to increment in the atomic block.

 I'm not happy with this solution, because:
   - You still have an atomic in there, and the compiler can't remove it.
 This reduce greatly the capability of the compiler to optimize. For
 instance, the compiler cannot optimize away redundant pairs of
 increment/decrement.

Only on const slices, architectures more relaxed than x86, and objects with destructors.
 That odd/even solution surely works, but ultimately do not solve the
 issue: if you want full speed, you'll have to provide both a const and a
 mutable version of the code, which defeat the purpose of const. Note
 that the exact same issue exists with inout.

I don't think it's all doom and gloom. Defining a destructor for an object suggests that the object is willing and able to give up a little speed for the sake of automation. I think it all dovetails nicely. Andrei
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:54 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 3:32 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:
 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or none do,
 and that information is given by the type (everything that is
 immutable/const/shared). I don't see any feasible way of escaping
 this, or any advantage to a runtime convention like the odd/even trick
 above.

If you CPU is decent you have some cache coherency protocol in place. This won't ensure that thing appears sequentially consistent, but you don't care here. You can proceed as follow in pseudo assembly : count = load count_addr need_atomic = count & 0x01 brtr atomic count = count + 2 store count count_addr br follow_up atomic: atomic_add count_addr 2 follow_up: // Code after increment goes here

brtr_atomic won't be needed on x86.

Sorry, I misread your code. I meant to say on x86 there's no need to do any handshake on the single-threaded case. -- Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:14 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei Alexandrescu wrote:
 Sorry, I misread your code. I meant to say on x86 there's no need to
 do any handshake on the single-threaded case. -- Andrei

You don't need it on most arch. Except alpha, I don't know any that would require it.

So much the better. How about ARM? All I know is it's very relaxed. -- Andrei
Apr 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:19 PM, froglegs wrote:
   Here is an alternative:

   1. classes are unique by default, they cannot be shared. No GC or RF
 required, and covers 90% of objects.

Where does 90% come from? On the contrary, it seems to me unique references are rather rare and fleeting. -- Andrei
Apr 30 2014
prev sibling parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-04-30 21:51:18 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and 
 interlocked refcounts go like 2, 4, 6, ...
 
 Then you do an unprotected read of the refcount. If it's odd, then it's 
 impossible to having originated as an interlocked one. So proceed with 
 simple increment. If it's even, do an interlocked increment.

Nice idea, although I'd add a twist to support polymorphism in class hierarchies: add a magic value (say zero) to mean "not reference-counted". When you instantiate an object that has a destructor, the reference counter is set to 1 or 2 depending on whether it's shared or not. If however the object has no destructor and is not reference counted, set the counter to the magic value. Then have the compiler assume all objects are reference counted. At runtime only those objects with a non-magic value as the reference count will actually increment/decrement the counter. Finally, let the the compiler understand some situations where objects are guarantied to have no destructors a destructor so it can omit automatic reference counting code in those cases. One such situation is when you have a reference to a final class with no destructor. We could also add a nodestructor attribute to forbid a class and all its descendants from having destructors. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Apr 30 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:57 PM, Andrei Alexandrescu wrote:
 RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the
 compiler is concerned.


There's always this hack: struct RCSlice(T){ static if(!is(T==const)) RCSlice!(const(StripImmutable!T)) upcast(){ return cast(typeof(return))this; } alias upcast this; // ... }
 We need to improve the language to allow for such. Did I mention it's
 not going to be easy?

Well, actually an easy way would be to have the compiler infer safe coercions. (A 'coercion' here is meant to denote a no-op implicit conversion, for example, conversion to const.) It would be enabled with an attribute on the template. Coercions between different instantiations of the same templated type would be safe and allowed if the memory layout does not change between them and the compiler can prove the coercion safe field-wise. There needn't be restrictions on non-field members. Types annotated like this would get the top-level mutability specifier stripped off if possible, during IFTI / when cast(). In particular, this would work: class C{} class D:C{} infer_coercions struct Slice(T){ size_t length; T* ptr; } void foo(T)(T arg){ /* ... */ } void main() safe{ auto a=[new D,new D]; auto s=Slice!D(a.length,a.ptr); Slice!(const(C)) sc=s; // ok, D* coercible to const(C)* const(Slice!C) cs=s; // ok, D* coercible to n-m-ind const(C*) sc=cs; // ok, D* is coercible to n-m-ind const(C)* // (length works because const(size_t) and size_t can be // freely corced between each other.) foo(cs); // actually calls foo!(Slice!(const(C))), // ==> no more magic in T[] } I.e. slices can be implemented in the library under this proposal. (The above just forwards the magical behaviour of T* to a user-defined type.) I think this fixes the issue in general, for example, for ranges: const rng1 = [1,2,3].map!(a=>2*a); const rng2 = rng1.filter!(a=>!!(a&1)); // ok Probably this should be slightly generalized, eg: infer_coercions struct S(T){ T field; } void foo(T)(T arg){} void main(){ S!(T[]) s; foo(s); // foo!(S!(const(T)[])) } However, then, whether to do const(S!T) => S!(const(T)) or const(S!T) => S!(TailConst!T) should maybe be specified on a per-parameter basis, because this is in general not easy to figure out for the compiler.
Apr 30 2014
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:24 AM, Timon Gehr wrote:
 I think this fixes the issue in general, for example, for ranges:

 const rng1 = [1,2,3].map!(a=>2*a);
 const rng2 = rng1.filter!(a=>!!(a&1)); // ok



 Probably this should be slightly generalized, eg:
 ...

(The generalization is actually needed for the range example to work.)
Apr 30 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:24 AM, Timon Gehr wrote:
      const(Slice!C) cs=s; // ok, D* coercible to n-m-ind const(C*)
      sc=cs; // ok, D* is coercible to n-m-ind const(C)*

(Ignore the 'n-m-ind', those are accidental leftovers from a half-baked version of the post.)
Apr 30 2014
prev sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/01/2014 12:48 AM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 22:24:29 UTC, Timon Gehr wrote:
 However, then, whether to do const(S!T) => S!(const(T)) or const(S!T)
 => S!(TailConst!T) should maybe be specified on a per-parameter basis,
 because this is in general not easy to figure out for the compiler.

That is the whole problem :D

Well, actually, just check whether there is a field that would need to change type, of the exact type of some parameter. If so, try to do the second thing for that parameter, otherwise try to do the first.
May 01 2014
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:21 PM, Andrei Alexandrescu wrote:
 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.
 ...

struct S{ ~this(){ /* ... */ } /* ... */ } class C{ S s; } ?
 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.
 ...

struct S{ ~this(){ /* ... */ } } class C{ S[] s; this(S[] s){ /* ... */ } } void main(){ new C(buildSs()); // memory leak } Also, cycles.
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 1:56 PM, Timon Gehr wrote:
 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?

By hand, as I mentioned. -- Andrei
Apr 30 2014
next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 1:56 PM, Timon Gehr wrote:
 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?
Apr 30 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:09 PM, Timon Gehr wrote:
 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 1:56 PM, Timon Gehr wrote:
 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 4/30/14, 2:09 PM, Timon Gehr wrote:
 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 1:56 PM, Timon Gehr wrote:
 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called?

That is correct. -- Andrei
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 4/30/14, 2:09 PM, Timon Gehr wrote:
 On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 1:56 PM, Timon Gehr wrote:
 struct S{
      ~this(){ /* ... */ }
      /* ... */
 }

 class C{
      S s;
 }

 ?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called?

That is correct. -- Andrei

I don't like the sound of that. I haven't found myself in a place where I needed to do something like this, but if I had to, I'd be very unhappy if struct dtors only work when they're not class members. Can we make them always work, and if necessary prohibit using them as class members?

Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote:
 How so? If we prohibit structs with dtors from being class members,
 then it could work. Of course, then you'd have to deal with the backlash
 from users who inevitably find some use case for it... but at least in
 theory it's possible.

Oh I see. I'd eliminated that ex officio because (a) it's a frequent case so lotsa breakage, and (b) we lack a replacement to recommend. -- Andrei
Apr 30 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 4:17 PM, H. S. Teoh via Digitalmars-d wrote:
 But if we're going to go in the direction of lack of dtor guarantees,
 then I might suggest we kill off struct dtors as well, since they also
 have their own set of pathologies.  Like passing structs with dtors by
 value (this includes returning them from a function) causing double
 destruction if the dtor isn't written carefully,

That sounds like a compiler bug.
 or putting structs with
 dtors into an array or std.container.*,

What's wrong with that? std.container.Array should destroy things properly, if not that's a library bug.
 and they stop working as one
 might expect.  Wasn't this whole fiasco the whole reason
 std.stdio.ByLine was recently rewritten to use ref counting? It relied
 on struct dtors before, and look how well that turned out.

But ref counting IS destructors.
 Basically, you can't allow something to have dtors, yet have no way to
 guarantee it will be called at the expected time.

I don't think so - it's navigating close enough to the "so let's not use cars anymore" fallacy. There are plenty many situations in which dtors are working fabulously well, and throwing them away because we can't guarantee they'll work absolutely well seems bad decision making to me.
 That kind of
 under-specification is the source of endless hard-to-trace bugs,
 gotchas, unfixable issues, and holes in the type system. If we're going
 to have dtors at all, let's do it *right*. Guarantee they always work,
 and reject all usages that break this guarantee (like putting a struct
 with dtor inside a class, putting it inside a dynamic array in GC
 memory, etc.). If we can't guarantee anything at all, then let's not
 have dtors at all.  Trying to stay in the gray zone in between does
 nothing but cause endless problems down the road. Not to mention
 language smells. What's the use of a feature that only sometimes works,
 where "sometimes" is un(der)specified, left to implementation, or
 depends on undefined behaviour?

A lot of other languages have such imperfections, and nobody raises a brow. Andrei
Apr 30 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 7:17 AM, H. S. Teoh via Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 04:33:23PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 No, allowing the user to specify a struct with a dtor, and put that
 struct in a class, and then saying "nyah nyah class dtors don't exist
 anymore so your struct will never destruct" -- *that* is bad decision
 making. At the very least, the compiler should complain loudly and
 clearly that structs with dtors should not be put inside a class. Isn't
 this what D philosophy is supposed to be? Things should *work* by
 default, and unexpected behaviour (dtor never gets called) should only
 happen when the user explicitly asks for it.

Problem is there's a lot of correct code that e.g. closes the File members properly etc. All that code would be disable together with the risky code.
 That kind of under-specification is the source of endless
 hard-to-trace bugs, gotchas, unfixable issues, and holes in the type
 system. If we're going to have dtors at all, let's do it *right*.
 Guarantee they always work, and reject all usages that break this
 guarantee (like putting a struct with dtor inside a class, putting it
 inside a dynamic array in GC memory, etc.). If we can't guarantee
 anything at all, then let's not have dtors at all.  Trying to stay in
 the gray zone in between does nothing but cause endless problems down
 the road. Not to mention language smells. What's the use of a feature
 that only sometimes works, where "sometimes" is un(der)specified,
 left to implementation, or depends on undefined behaviour?

A lot of other languages have such imperfections, and nobody raises a brow.

I thought D is here because we want to do better than that?

Better != perfection at all costs. Andrei
May 01 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/30/2014 4:17 PM, H. S. Teoh via Digitalmars-d wrote:
 If we're going
 to have dtors at all, let's do it *right*. Guarantee they always work,
 and reject all usages that break this guarantee (like putting a struct
 with dtor inside a class,

Seems to work when I try it: bar.d: ------------------------- import core.stdc.stdio; struct S { ~this() { printf("S.~this()\n"); } } class C { S s; } void main() { C c = new C(); c = null; } ------------------------- C:\cbx\mars>bar S.~this()
Apr 30 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 7:01 AM, H. S. Teoh via Digitalmars-d wrote:
 The proposal was to get rid of class dtors, in which case this code will
 no longer work. Is that really the direction we want to move in?

My point was that the compiler wasn't ignoring destructors for fields. Who should call the class destructor and when is a different issue.
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:
 I'm confused. Andrei's original proposal was to deprecate class dtors,
 with the view to eventually getting rid of them altogether. If indeed
 that's going to be the case, then wouldn't that mean that field dtors
 won't be invoked either (because otherwise, how would you know when to
 invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields and
 invoke their dtors, it would seem to me that having fields with dtors in
 a class (once class dtors no longer exist, according to Andrei's
 proposal) will be a dangerous thing to do, since the very guarantee by
 having a dtor in the first place (it will get called to clean up when
 the struct goes out of scope) is broken. If so, what's the point of
 having a dtor in the first place? It becomes no different from the
 manual cleanup that you have to do in C.

The thing is, GC is a terrible and unreliable method of managing non-memory resource lifetimes. Destructors for GC objects are not guaranteed to ever run. If you do have struct with a destructor as a field in a class, you've got, at minimum, suspicious code and a latent bug.
May 01 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 3:22 PM, H. S. Teoh via Digitalmars-d wrote:
 Exactly!!! This is why I said we should ban the use of structs with
 dtors as a field in a class. Isn't the D philosophy to be correct by
 default, and only allow potentially wrong/error-prone code if the user
 explicitly asks for it?  So this kind of error-prone usage should not be
 allowed by default.  Was my logic really that hard to follow?

Yes, we need to do something about it, no, I don't know what at the moment.
May 01 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:10 PM, Walter Bright wrote:
 On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:
 I'm confused. Andrei's original proposal was to deprecate class dtors,
 with the view to eventually getting rid of them altogether. If indeed
 that's going to be the case, then wouldn't that mean that field dtors
 won't be invoked either (because otherwise, how would you know when to
 invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields and
 invoke their dtors, it would seem to me that having fields with dtors in
 a class (once class dtors no longer exist, according to Andrei's
 proposal) will be a dangerous thing to do, since the very guarantee by
 having a dtor in the first place (it will get called to clean up when
 the struct goes out of scope) is broken. If so, what's the point of
 having a dtor in the first place? It becomes no different from the
 manual cleanup that you have to do in C.

The thing is, GC is a terrible and unreliable method of managing non-memory resource lifetimes. Destructors for GC objects are not guaranteed to ever run.

Agree.
 If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent bug.

Disagree. The simple matter here is that there's no "right" solution to counter what you deem as suspicious code. For example, we have a File struct. It is entirely legitimate for someone to want to keep a File member variable in a class object. They take care of opening and closing it appropriately. Again, this is 100% legit code to want to write. Coming to them from the perspective "you are doing it wrong and there is no recourse for achieving what you need to do" strikes me as odd. Andrei
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 5:43 PM, Andrei Alexandrescu wrote:
 For example, we have a File struct. It is entirely legitimate for someone to
 want to keep a File member variable in a class object. They take care of
opening
 and closing it appropriately. Again, this is 100% legit code to want to write.
 Coming to them from the perspective "you are doing it wrong and there is no
 recourse for achieving what you need to do" strikes me as odd.

I meant from the perspective of expecting the GC to close the file for you. If you manually run the destructor, such as from using delete, that wouldn't be a problem.
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 6:06 PM, Walter Bright wrote:
 On 5/1/2014 5:43 PM, Andrei Alexandrescu wrote:
 For example, we have a File struct. It is entirely legitimate for
 someone to
 want to keep a File member variable in a class object. They take care
 of opening
 and closing it appropriately. Again, this is 100% legit code to want
 to write.
 Coming to them from the perspective "you are doing it wrong and there
 is no
 recourse for achieving what you need to do" strikes me as odd.

I meant from the perspective of expecting the GC to close the file for you. If you manually run the destructor, such as from using delete, that wouldn't be a problem.

Or if you... close the file :o). OK, thanks for clarifying. -- Andrei
May 01 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 6:19 PM, Andrei Alexandrescu wrote:
 Or if you... close the file

Hmm. Never thought of that!
May 01 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:22 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via Digitalmars-d
wrote:
 On 5/1/2014 3:02 PM, H. S. Teoh via Digitalmars-d wrote:
 I'm confused. Andrei's original proposal was to deprecate class
 dtors, with the view to eventually getting rid of them altogether. If
 indeed that's going to be the case, then wouldn't that mean that
 field dtors won't be invoked either (because otherwise, how would you
 know when to invoke a field dtor?)?

 Without a clear definition of who is going to clean up those fields
 and invoke their dtors, it would seem to me that having fields with
 dtors in a class (once class dtors no longer exist, according to
 Andrei's proposal) will be a dangerous thing to do, since the very
 guarantee by having a dtor in the first place (it will get called to
 clean up when the struct goes out of scope) is broken. If so, what's
 the point of having a dtor in the first place? It becomes no
 different from the manual cleanup that you have to do in C.

The thing is, GC is a terrible and unreliable method of managing non-memory resource lifetimes. Destructors for GC objects are not guaranteed to ever run. If you do have struct with a destructor as a field in a class, you've got, at minimum, suspicious code and a latent bug.

Exactly!!! This is why I said we should ban the use of structs with dtors as a field in a class.

No.
 Isn't the D philosophy to be correct by
 default, and only allow potentially wrong/error-prone code if the user
 explicitly asks for it?

We can't propose correctness at the price of getting work done.
 So this kind of error-prone usage should not be
 allowed by default.  Was my logic really that hard to follow?

Your logic is flawed. I am sorry. Andrei
May 01 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/2/14, 3:09 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Thursday, 1 May 2014 at 22:23:46 UTC, H. S. Teoh via Digitalmars-d
 wrote:
 On Thu, May 01, 2014 at 03:10:04PM -0700, Walter Bright via
 Digitalmars-d wrote:
 The thing is, GC is a terrible and unreliable method of managing
 non-memory resource lifetimes. Destructors for GC objects are not
 guaranteed to ever run. If you do have struct with a destructor as a
 field in a class, you've got, at minimum, suspicious code and a latent
 bug.

Exactly!!! This is why I said we should ban the use of structs with dtors as a field in a class.

No, not in a class, but in any GC-managed object. It's unfortunate that class currently implies GC.

So now it looks like dynamic arrays also can't contain structs with destructors :o). -- Andrei
May 02 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-05-02 17:38, "Marc Schütz" <schuetzm gmx.net>" wrote:

 But my main point was actually this:
 Don't disallow destructors on classes just because they are classes, but
 disallow them when they are not guaranteed to be called (or calling them
 is troublesome because of multi-threading), i.e. GC.

Tango for D1 added a "dispose" method to Object. This method was called when "delete" or "scope" was used. -- /Jacob Carlborg
May 02 2014
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, May 02, 2014 at 11:44:47PM +0200, Jonathan M Davis via Digitalmars-d
wrote:
 On Fri, 02 May 2014 21:03:15 +0000
 monarch_dodra via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 
 On Friday, 2 May 2014 at 15:06:59 UTC, Andrei Alexandrescu wrote:
 So now it looks like dynamic arrays also can't contain structs
 with destructors :o). -- Andrei

Well, that's always been the case, and even worst, since in a dynamic array, destructor are guaranteed to *never* be run. Furthermore, given the "append causes relocation which duplicates", you are almost *guaranteed* to leak your destructors. You just can't keep track of the usage of a "naked" dynamic array. This usually comes as a great surprise to users in ".learn". It's also the reason why using "File[]" never ends well...

Heck, I probably knew that before, but I had completely forgotten. If you'd asked me yesterday whether struct destructors were run in dynamic arrays, I'd have said yes. And if someone like me doesn't remember that, would you expect the average D programmer to? The current situation is just plain bug-prone. Honestly, I really think that we need to figure out how to make it so that struct destructors are guaranteed to be run so long as the memory that they're in is collected. Without that, having destructors in structs anywhere other than directly on the stack is pretty much broken.

Thank you, that's what I've been trying to say. Having dtors sometimes run and sometimes not, is very bug-prone -- if for a seasoned D programmer, then how much more for an average D programmer? We need some kind of guarantees. The current situation sux. I might even say we'd have been better off having no dtors in the first place -- at least then it's consistent, you know you always have to cleanup. But the current situation of being neither here nor there, neither always cleaning up nor never, is not a good place to be in. I've to say that the more I look at this, the more I don't like this part of the language. :-/ T -- MASM = Mana Ada Sistem, Man!
May 02 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:06 AM, monarch_dodra wrote:
 "A class is allowed to have a destructor" <=> "A class can have members
 with destructors".

No equivalence, but implication. If a class has at least one member with a destructor, the compiler might need to generate a destructor for the class. -- Andrei
May 01 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:
 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the class.

And in fact that's what dmd does.
May 03 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/3/14, 12:40 PM, Walter Bright wrote:
 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:
 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the
 class.

And in fact that's what dmd does.

Which suggests a simple solution for calling destructors for structs and arrays: * Lower new for structs to return: new S; -> return &(new class { S member; }).member; * Lower array construction similarly. Then voila, the anonymous classes will destroy structs and arrays appropriately. Andrei
May 03 2014
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 5/3/2014 6:44 PM, Andrei Alexandrescu wrote:
 On 5/3/14, 12:40 PM, Walter Bright wrote:
 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:
 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for the
 class.

And in fact that's what dmd does.

Which suggests a simple solution for calling destructors for structs and arrays: * Lower new for structs to return: new S; -> return &(new class { S member; }).member; * Lower array construction similarly. Then voila, the anonymous classes will destroy structs and arrays appropriately.

Uhh, but doesn't this completely break as soon as class dtors go away?
May 03 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 3:54 PM, fra wrote:
 Can't do nothing but agree. Unless I'm missing something, no destructors
 means memory leaks every time I use non-class objects inside a class
 object.

If said non-class objects have destructors and are not cleaned up via other mechanisms.
 At least current behaviour makes sure that when memory is
 needed, everything gets cleaned up properly.

I think this was a false sense of security. There's no "making sure". Due to imprecision a destructor may or may not be called. Also, arrays of structs _never_ call destructors for their elements, which should shock you quite a bit more.
 Honestly, this sounds crazy
 to me.

s/crazy/radical/ Yes, radical. Because we want to uproot some bad weeds here. Andrei
Apr 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 2:26 PM, sclytrack wrote:
 It gets called when the gc runs.

No. No calls whatsoever while the gc runs. GC is notoriously bad for handling non-memory resources, and I see no reason to continue attempting it to do so after so much evidence to the contrary has been, um, collected :o). Andrei
Apr 30 2014
prev sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 10:06:17AM +0000, monarch_dodra via Digitalmars-d wrote:
 On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via Digitalmars-d
 wrote:
On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:

I don't like the sound of that. I haven't found myself in a >place
where I needed to do something like this, but if I had to, >I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary >prohibit
using them as class members?

Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei

How so? If we prohibit structs with dtors from being class members, then it could work.

Why would we prohibit structs with destructors from being class members? That don't make sense to me.

The proposal was to get rid of class dtors at some point. Which will introduce the problem of what to do when a class member is a struct with a dtor, since that dtor will never run.
 "A class is allowed to have a destructor" <=> "A class can have
 members with destructors".
 
 So we either kill off class destructor entirelly (which would mean no
 members with destructors) (But that seems like a bad idea), or we keep
 both.
 
 Or did I miss something in the argument?

Yes that's the gist of it. But Andrei seems to be convinced that class dtors are a bad idea, so to me, that also implies that class members with dtors are an equally bad (or worse) idea, so therefore they must be gotten rid of too. T -- All problems are easy in retrospect.
May 01 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 4/30/14, 2:09 PM, Timon Gehr wrote:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
On 4/30/14, 1:56 PM, Timon Gehr wrote:
struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called? T -- Never wrestle a pig. You both get covered in mud, and the pig likes it.
Apr 30 2014
prev sibling next sibling parent "sclytrack" <sclytrack fake.com> writes:
On Wednesday, 30 April 2014 at 21:17:23 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On 4/30/14, 2:09 PM, Timon Gehr wrote:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
On 4/30/14, 1:56 PM, Timon Gehr wrote:
struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called?

 T

Apr 30 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:
On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
On 4/30/14, 2:09 PM, Timon Gehr wrote:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
On 4/30/14, 1:56 PM, Timon Gehr wrote:
struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called?

That is correct. -- Andrei

I don't like the sound of that. I haven't found myself in a place where I needed to do something like this, but if I had to, I'd be very unhappy if struct dtors only work when they're not class members. Can we make them always work, and if necessary prohibit using them as class members? T -- People tell me I'm stubborn, but I refuse to accept it!
Apr 30 2014
prev sibling next sibling parent "fra" <a b.it> writes:
On Wednesday, 30 April 2014 at 22:49:01 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 02:30:05PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On 4/30/14, 2:15 PM, H. S. Teoh via Digitalmars-d wrote:
On Wed, Apr 30, 2014 at 02:13:32PM -0700, Andrei Alexandrescu 
via Digitalmars-d wrote:
On 4/30/14, 2:09 PM, Timon Gehr wrote:
On 04/30/2014 10:58 PM, Andrei Alexandrescu wrote:
On 4/30/14, 1:56 PM, Timon Gehr wrote:
struct S{
     ~this(){ /* ... */ }
     /* ... */
}

class C{
     S s;
}

?

By hand, as I mentioned. -- Andrei

I meant, is it going to be deprecated too?

No, that will continue to be allowed. -- Andrei

Then what is it going to do? S.~this will never get called?

That is correct. -- Andrei

I don't like the sound of that. I haven't found myself in a place where I needed to do something like this, but if I had to, I'd be very unhappy if struct dtors only work when they're not class members. Can we make them always work, and if necessary prohibit using them as class members? T

Can't do nothing but agree. Unless I'm missing something, no destructors means memory leaks every time I use non-class objects inside a class object. At least current behaviour makes sure that when memory is needed, everything gets cleaned up properly. Honestly, this sounds crazy to me.
Apr 30 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:

I don't like the sound of that. I haven't found myself in a place
where I needed to do something like this, but if I had to, I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary prohibit
using them as class members?

Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei

How so? If we prohibit structs with dtors from being class members, then it could work. Of course, then you'd have to deal with the backlash from users who inevitably find some use case for it... but at least in theory it's possible. But if we're going to go in the direction of lack of dtor guarantees, then I might suggest we kill off struct dtors as well, since they also have their own set of pathologies. Like passing structs with dtors by value (this includes returning them from a function) causing double destruction if the dtor isn't written carefully, or putting structs with dtors into an array or std.container.*, and they stop working as one might expect. Wasn't this whole fiasco the whole reason std.stdio.ByLine was recently rewritten to use ref counting? It relied on struct dtors before, and look how well that turned out. Basically, you can't allow something to have dtors, yet have no way to guarantee it will be called at the expected time. That kind of under-specification is the source of endless hard-to-trace bugs, gotchas, unfixable issues, and holes in the type system. If we're going to have dtors at all, let's do it *right*. Guarantee they always work, and reject all usages that break this guarantee (like putting a struct with dtor inside a class, putting it inside a dynamic array in GC memory, etc.). If we can't guarantee anything at all, then let's not have dtors at all. Trying to stay in the gray zone in between does nothing but cause endless problems down the road. Not to mention language smells. What's the use of a feature that only sometimes works, where "sometimes" is un(der)specified, left to implementation, or depends on undefined behaviour? T -- In a world without fences, who needs Windows and Gates? -- Christian Surchi
Apr 30 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Wednesday, 30 April 2014 at 23:19:18 UTC, H. S. Teoh via 
Digitalmars-d wrote:
 On Wed, Apr 30, 2014 at 03:55:38PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On 4/30/14, 3:47 PM, H. S. Teoh via Digitalmars-d wrote:

I don't like the sound of that. I haven't found myself in a 
place
where I needed to do something like this, but if I had to, 
I'd be
very unhappy if struct dtors only work when they're not class
members. Can we make them always work, and if necessary 
prohibit
using them as class members?

Then we're back to effectively class destructors. I think we've gathered quite a bit of evidence there are pernicious issues associated with them. -- Andrei

How so? If we prohibit structs with dtors from being class members, then it could work.

Why would we prohibit structs with destructors from being class members? That don't make sense to me. "A class is allowed to have a destructor" <=> "A class can have members with destructors". So we either kill off class destructor entirelly (which would mean no members with destructors) (But that seems like a bad idea), or we keep both. Or did I miss something in the argument?
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 03 May 2014 22:44:39 -0400
Nick Sabalausky via Digitalmars-d <digitalmars-d puremagic.com> wrote:

 On 5/3/2014 6:44 PM, Andrei Alexandrescu wrote:
 On 5/3/14, 12:40 PM, Walter Bright wrote:
 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:
 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for
 the class.

And in fact that's what dmd does.

Which suggests a simple solution for calling destructors for structs and arrays: * Lower new for structs to return: new S; -> return &(new class { S member; }).member; * Lower array construction similarly. Then voila, the anonymous classes will destroy structs and arrays appropriately.

Uhh, but doesn't this completely break as soon as class dtors go away?

Based on other comments, I think that Andrei has been convinced that class destructors can't go away at this point and that there certainly isn't any consensus that it would even be desirable for them to go away (though what he'd choose to do if we could break code willy-nilly, I don't know). So, this particular proposal is presumably done with the idea that class destructors are here to stay. Rather, it's trying to make it so that the destructors for structs on the heap get run unlike now - which is a much better direction to try and go IMHO. - Jonathan M Davis
May 03 2014
prev sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Sat, 03 May 2014 15:44:03 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 5/3/14, 12:40 PM, Walter Bright wrote:
 On 5/1/2014 7:59 AM, Andrei Alexandrescu wrote:
 If a class has at least one member with a
 destructor, the compiler might need to generate a destructor for
 the class.

And in fact that's what dmd does.

Which suggests a simple solution for calling destructors for structs and arrays: * Lower new for structs to return: new S; -> return &(new class { S member; }).member; * Lower array construction similarly. Then voila, the anonymous classes will destroy structs and arrays appropriately.

This might be a good approach, though I confess that it strikes me as rather weird to wrap structs in classes like that. It also leaves open the question of how to deal with structs that are newed directly rather than put in an array. _Those_ really need to be destroyed properly as well. And unless you're suggesting that S* effectively become a pointer to a struct within an anonymous class in all cases, not only would this not work for structs which were newed up directly on the heap, but I'd be worried about what would happen when someone did something like &arr[5] with an array of structs. Would they get a pointer to a class or a struct? It was my understanding that some of what Rainer Schutze did with his precise GC involved adding RTInfo which could make it possible to run struct destructors for structs that were newed up directly on the heap. If that is indeed the case, then I would think that we could use that for arrays of structs as well. - Jonathan M Davis
May 03 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:49:30 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 1:46 PM, deadalnix wrote:
 Still no consideration for isolated.

https://github.com/D-Programming-Language/dmd/pull/3452 may be of interest to you. -- Andrei

I already commented on past similar proposal. These are piling up more and more hacks to achieve roughly the same thing, badly. This makes it impossible to provide a 3rd party implementation that is compliant, is guaranteed to ends up in a nightmarish explosion of special cases, or to be vastly underpowered.
Apr 30 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei Alexandrescu 
wrote:
 An extreme one indeed, it would break a lot of my code. Every 
 D project
 I wrote that does networking manages memory using a class that 
 resides
 on the managed heap, but holds the actual wrapped data in the 
 unmanaged
 heap.

So should I take it those classes all have destructors? --

One class, many instances (one for every chunk of data sent or received).
Apr 30 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Wednesday, 30 April 2014 at 20:59:34 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 1:57 PM, Vladimir Panteleev wrote:
 On Wednesday, 30 April 2014 at 20:45:57 UTC, Andrei 
 Alexandrescu wrote:
 An extreme one indeed, it would break a lot of my code. 
 Every D project
 I wrote that does networking manages memory using a class 
 that resides
 on the managed heap, but holds the actual wrapped data in 
 the unmanaged
 heap.

So should I take it those classes all have destructors? --

One class, many instances (one for every chunk of data sent or received).

The question is, how comfortable are you with today's reality that some of those instances may be never destroyed? -- Andrei

I'm fine with it. The proxy objects are purposefully as small as possible, to make false references (and thus, memory leaks) unlikely, even on 32 bits. If I was concerned with it, I'd have switched to reference counting as soon as it was possible.
Apr 30 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu 
wrote:
 "Ecstatic" is what you're looking for.

Probably...
 This has been proposed as a solution to
 some GC issue at last DConf and has been dismissed. Your post 
 is unclear
 on why this decision has changed. There must be some new 
 information or
 data that inform this decision.

 Or is it random ?

I don't remember a proposal being made that made slices of structs with destructors distinct from other slices.

So the RC slice thing is what make all that worthwhile ?
 Also, RC is good on top of GC, so
 you can collect loop, especially if we are going to RC 
 automagically.
 That is a major issue.

That stays, and it's useful. I'm talking destructors here.

Then the RCSlice do not provide any guarantee at all.
 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).

This won't work. array have a lot of magic that cannot be created as library. ie: RCSlice!(const T) has nothing to do with const(RCSlice!T) as far as the compiler is concerned.

We need to improve the language to allow for such. Did I mention it's not going to be easy?

If I understand what you mentioned, RCSlice is what make to you the ditching of destructor worthwhile. The fact that this is not doable in current D should be a showstopper for this whole thread as all that is discussed here is dependent of what solution we choose for this problem (right now we have none).
Apr 30 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei Alexandrescu 
wrote:
 Finally, immutable is sharable accross thread. That mean, even 
 if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const code 
 will be
 full of atomic increment/decrement. They are bad for the CPU, 
 and cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.

How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable.
Apr 30 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 2:47 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei 
 Alexandrescu wrote:
 Finally, immutable is sharable accross thread. That mean, 
 even if we
 bypass the type system, that RC must be atomic for immutable.
 As they
 convert automatically for co,st, that mean that all const 
 code will be
 full of atomic increment/decrement. They are bad for the 
 CPU, and cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.

How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable.

I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and interlocked refcounts go like 2, 4, 6, ... Then you do an unprotected read of the refcount. If it's odd, then it's impossible to having originated as an interlocked one. So proceed with simple increment. If it's even, do an interlocked increment. Andrei

I don't think I fully understand. Either all RC changes for a given type need to be atomic or none do, and that information is given by the type (everything that is immutable/const/shared). I don't see any feasible way of escaping this, or any advantage to a runtime convention like the odd/even trick above.
Apr 30 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 22:08:23 UTC, John Colvin wrote:
 I don't think I fully understand.

 Either all RC changes for a given type need to be atomic or 
 none do, and that information is given by the type (everything 
 that is immutable/const/shared). I don't see any feasible way 
 of escaping this, or any advantage to a runtime convention like 
 the odd/even trick above.

If you CPU is decent you have some cache coherency protocol in place. This won't ensure that thing appears sequentially consistent, but you don't care here. You can proceed as follow in pseudo assembly : count = load count_addr need_atomic = count & 0x01 brtr atomic count = count + 2 store count count_addr br follow_up atomic: atomic_add count_addr 2 follow_up: // Code after increment goes here Note that is working as count may not be the correct number in case of sharing, but will always have the same parity, so even reading the wrong value will make you branch properly and the value of count is not used to increment in the atomic block. I'm not happy with this solution, because: - You still have an atomic in there, and the compiler can't remove it. This reduce greatly the capability of the compiler to optimize. For instance, the compiler cannot optimize away redundant pairs of increment/decrement. - You have a branch in there. Atomic are expensive, but branch as well. Especially since both are storing (and one atomically), so it can't be speculated. - If we start using that all over the place, the codegen will ends up being quite fat. That means less friendly cache behavior. That odd/even solution surely works, but ultimately do not solve the issue: if you want full speed, you'll have to provide both a const and a mutable version of the code, which defeat the purpose of const. Note that the exact same issue exists with inout.
Apr 30 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 22:24:29 UTC, Timon Gehr wrote:
 However, then, whether to do const(S!T) => S!(const(T)) or 
 const(S!T) => S!(TailConst!T) should maybe be specified on a 
 per-parameter basis, because this is in general not easy to 
 figure out for the compiler.

That is the whole problem :D
Apr 30 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei Alexandrescu 
wrote:
 Sorry, I misread your code. I meant to say on x86 there's no 
 need to do any handshake on the single-threaded case. -- Andrei

You don't need it on most arch. Except alpha, I don't know any that would require it.
Apr 30 2014
prev sibling next sibling parent "froglegs" <barf barf.com> writes:
  Here is an alternative:

  1. classes are unique by default, they cannot be shared. No GC 
or RF required, and covers 90% of objects.
  2. To share, retype as GC, the requirement being that it does 
not have a destructor.
Apr 30 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 30 April 2014 at 23:27:49 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 4:14 PM, deadalnix wrote:
 On Wednesday, 30 April 2014 at 23:05:29 UTC, Andrei 
 Alexandrescu wrote:
 Sorry, I misread your code. I meant to say on x86 there's no 
 need to
 do any handshake on the single-threaded case. -- Andrei

You don't need it on most arch. Except alpha, I don't know any that would require it.

So much the better. How about ARM? All I know is it's very relaxed. -- Andrei

It is expected that a barrier (dmb on ARM) is inserted when sharing information (in std.concurency for instance). So you should have some initialized value in there. It can be anything though. That mean that the load can get any value that was there since the last sharing done properly. It doesn't matter which one we get as they all have the same parity. This value can't be used in the atomic branch. That is why the pseudo-assembly code I posted do not use the counter value in the atomic branch.
Apr 30 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 30 April 2014 at 23:34:33 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 4:19 PM, froglegs wrote:
  Here is an alternative:

  1. classes are unique by default, they cannot be shared. No 
 GC or RF
 required, and covers 90% of objects.

Where does 90% come from? On the contrary, it seems to me unique references are rather rare and fleeting. -- Andrei

Yeah, 90% doesn't feel right to me. In C++ I try to use unique_ptr instead of shared_ptr wherever possible but I find I'm often not able to because my references need to be held in several places for efficiency. I'd say I'm more like 90% reference counted, 10% unique throughout my 300kloc codebase.
Apr 30 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Where does 90% come from? On the contrary, it seems to me 
 unique references are rather rare and fleeting. -- Andrei

I think unique references (linear types) are used rather commonly among the reference types of Rust. Bye, bearophile
Apr 30 2014
prev sibling next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 16:21:33 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Walter and I have had a long chat in which we figured our current  
 offering of abstractions could be improved. Here are some thoughts.  
 There's a lot of work ahead of us on that and I wanted to make sure  
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within  
 the GC entirely. It makes for a faster and better GC, but the real  
 reason here is that destructors are philosophically bankrupt in a GC  
 environment. I think there's no need to argue that in this community.  
 The GC never guarantees calling destructors even today, so this decision  
 would be just a point in the definition space (albeit an extreme one).

destructors are for cleaning up non-GC resources. File handles, malloc'd memory, etc. I don't see why these need to be eliminated. What about changing destructors so that they DON'T implicitly call member struct dtors? I'd prefer that, since struct dtors executed on the heap I agree are an issue. One can always call the dtor in the class dtor if necessary. Right now, you have no choice.
 That means classes that need cleanup (either directly or by having  
 fields that are structs with destructors) would need to garner that by  
 other means, such as reference counting or manual. We're considering  
 deprecating ~this() for classes in the future.

So essentially, any class with a dtor needs reference counting? How does one clean up a cycle of them?
 Slices T[] of structs with destructors shall be silently lowered into  
 RCSlice!T, defined inside object.d. That type would occupy THREE words,  
 one of which being a pointer to a reference count. That type would  
 redefine all slice primitives to update the reference count accordingly.

This is a possibility, but I think you would need to prove performance is not severely affected.
 I foresee any number of theoretical and practical issues with this  
 approach. Let's discuss some of them here.

I'd rather see a RC array type, which specifically supports reference counting for opt-in support. But we are getting WAY ahead of ourselves. A workable RC solution has to be proven first. I'd feel much more comfortable making such a change with a working, tested solution to look at. -Steve
Apr 30 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.

Virtually all GCs are known to be horrible at managing scarce resources (including memory itself). Giving the task of managing such to the GC is like making the cross-eyed guy be the shooter in https://www.youtube.com/watch?v=sen8Tn8CBA4. Andrei
Apr 30 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 6:51 PM, Andrei Alexandrescu wrote:
 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.

Virtually all GCs are known to be horrible at managing scarce resources (including memory itself).

Here the paren means "... when memory is scarce". GCs work well only in 3x+ the memory needed. -- Andrei
Apr 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/30/14, 7:20 PM, Steven Schveighoffer wrote:
 On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.

Virtually all GCs are known to be horrible at managing scarce resources (including memory itself).

The destructor can be a crutch, but it's not good to leave open resources when the user of your code has not cleaned them up manually.

Sounds like defensive programming to me.
 I can see no reason to disallow destruction from cleaning up resources
 that nobody else has managed to clean up.

Make memory management faster and better for everyone. I'm not saying it's reason enough, but it's a reason.
 The largest problem with D destructors comes from trying to clean up D
 objects in destructors of structs, that you never expected to be done
 via the GC.

 Cleaning up files and malloc'd memory is not an issue.

Not getting this... Andrei
Apr 30 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:10 AM, monarch_dodra wrote:
 On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer wrote:
 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that
 by other means, such as reference counting or manual. We're
 considering deprecating ~this() for classes in the future.

So essentially, any class with a dtor needs reference counting? How does one clean up a cycle of them?

Yeah, what he said. This has me very worried. Making cycles is actually incredibly easy. Any "inteligently" implemented "node-based" data structure, implemented with classes, such as a linked list, a tree or a graph, is virtually guaranteed to have a cycle somewhere...

In my experience classes with destructors are simpler (files, sockets)... Lists, trees, and graphs tend to be in-memory objects. But of course this is just speculation.
 *Or*

 is the idea that reference counting only works for finalization, but the
 memory is still managed by the GC? In that case, the destructors would
 leak, but the memory still be released?

 That's still bad, of course, but not "as" bad...

Yah, that would be the case. Essentially we're talking about making the entire GC heap passive. Andrei
May 01 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:
 On 4/30/2014 1:57 PM, deadalnix wrote:
 I already commented on past similar proposal. These are piling 
 up more and more
 hacks to achieve roughly the same thing, badly. This makes it 
 impossible to
 provide a 3rd party implementation that is compliant, is 
 guaranteed to ends up
 in a nightmarish explosion of special cases, or to be vastly 
 underpowered.

Actually, I think inference of the ability to implicitly and safely convert from mutable to immutable or shared to be very powerful. I also think it is superior than requiring the user to add ever more annotations, which some research (and my experience) shows that users are reluctant to do. I don't buy that it is hackish, etc., especially without more of a rationale as to why. A link to your previous comment would be useful.

I've written several proposal regarding this. Please at least read them as what you just wrote only proves you are not well informed. Notably, the proposal do not require annotations from the users to do the kind of things are are currently being special cased.
Apr 30 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
 destructors are for cleaning up non-GC resources. File handles, malloc'd
 memory, etc. I don't see why these need to be eliminated.

Virtually all GCs are known to be horrible at managing scarce resources (including memory itself).

The destructor can be a crutch, but it's not good to leave open resources when the user of your code has not cleaned them up manually. I can see no reason to disallow destruction from cleaning up resources that nobody else has managed to clean up. The largest problem with D destructors comes from trying to clean up D objects in destructors of structs, that you never expected to be done via the GC. Cleaning up files and malloc'd memory is not an issue. -Steve
Apr 30 2014
prev sibling next sibling parent "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.

I can't claim it should be done, but any time I've thought something should go in the destructor I was wrong and would end up with a close() or similar function.
Apr 30 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 Actually, I think inference of the ability to implicitly and 
 safely convert from mutable to immutable or shared to be very 
 powerful. I also think it is superior than requiring the user 
 to add ever more annotations, which some research (and my 
 experience) shows that users are reluctant to do.

I agree the people are reluctant to add annotations, but I think it's also depends on the return of investment of the specific annotation. I am annotating all my code carefully with "pure", but the ROI of all those purity annotations is not great. And I agree the piling of special cases is a not clean design. It's now simpler to ask the compiler if a certain assignment to immutable is allowed or not. Because I can't remember the supported cases. So I suggest a more principled and visible approach at the problem of ownership and uniqueness. Bye, bearophile
Apr 30 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 We're considering deprecating ~this() for classes in the future.

Such changes could happen in parallel to the (planned?) removal of some of the methods of Object. Bye, bearophile
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 13:21:33 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 Walter and I have had a long chat in which we figured our current 
 offering of abstractions could be improved. Here are some thoughts. 
 There's a lot of work ahead of us on that and I wanted to make sure 
 we're getting full community buy-in and backup.
 
 First off, we're considering eliminating destructor calls from within 
 the GC entirely. It makes for a faster and better GC, but the real 
 reason here is that destructors are philosophically bankrupt in a GC 
 environment. I think there's no need to argue that in this community. 
 The GC never guarantees calling destructors even today, so this
 decision would be just a point in the definition space (albeit an
 extreme one).

I really don't like the fact that struct destructors are not called by the GC, and if anything, I'd be inclined to argue for finding a way to guarantee that they get run rather than guaranteeing that they never get run. It's just far too easy to have a struct expect that its destructor will be run and then have issues when it's not run. But it would be better to define struct destructors as never getting run rather than having it be undefined as it is now.
 We're considering deprecating ~this() for classes in the future.

While it's not good to rely on finalizers, they're good to have as backup if the appropriate cleanup function doesn't get called like it's supposed to. They're not as critical as they'd be in Java, since we have structs, but I'd be disinclined to remove finalizers from D without a really good reason.
 Also, we're considering a revamp of built-in slices, as follows.
 Slices of types without destructors stay as they are.
 
 Slices T[] of structs with destructors shall be silently lowered into 
 RCSlice!T, defined inside object.d. That type would occupy THREE
 words, one of which being a pointer to a reference count. That type
 would redefine all slice primitives to update the reference count
 accordingly.
 
 RCSlice!T will not convert implicitly to void[]. Explicit
 cast(void[]) will be allowed, and will ignore the reference count (so
 if a void[] extracted from a T[] via a cast outlives all slices,
 dangling pointers will ensue).
 
 I foresee any number of theoretical and practical issues with this 
 approach. Let's discuss some of them here.

I'm really going to have to think about this one. It's such a radical change that I really don't know what to think about it. It will be interesting to see what others have to say about. - Jonathan M Davis
May 01 2014
prev sibling next sibling parent Jonathan M Davis via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Wed, 30 Apr 2014 14:00:31 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 On 4/30/14, 1:57 PM, Timon Gehr wrote:
 On 04/30/2014 10:45 PM, Andrei Alexandrescu wrote:
 An extreme one indeed, it would break a lot of my code. Every D
 project I wrote that does networking manages memory using a class
 that resides on the managed heap, but holds the actual wrapped
 data in the unmanaged heap.

So should I take it those classes all have destructors? -- Andrei

(Yes, those destructors free the unmanaged memory.)

Thanks... that would need to change :o). -- Andrei

And it doesn't even work now, because it's not guaranteed that finalizers get run. And IIRC, based on some of the discussions at dconf last year, dealing with the GC and unloading shared libraries would probably make the situation even worse. But not being able to rely on finalizers running does put us in a bit of a pickle, because it basically means that any case where you need a finalizer, you should probably be using reference counting rather than the GC. That would tend to mean that either classes are going to need to be wrapped in a struct that reference-counts them and/or they're going to need to be allocated with a custom allocator rather than the GC. This problem makes me think of C#'s using blocks where the object is created at the beginning of the using block, and it's dispose method is called when that block is exited (it may also be collected then, but I don't remember). I don't think that that's quite what we want, since there are plenty of cases where you want to pass a class around, so some kind of reference counting would be better, but we probably should consider having some kind of standard function similar to dispose so that there's a standard method to call when a class needs to be cleaned up. And that could tie into a struct in Phobos that we would have to do the reference counting by wrapping the class. - Jonathan M Davis
May 01 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 First off, we're considering eliminating destructor calls from 
 within the GC entirely. It makes for a faster and better GC, 
 but the real reason here is that destructors are 
 philosophically bankrupt in a GC environment. I think there's 
 no need to argue that in this community. The GC never 
 guarantees calling destructors even today, so this decision 
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.

I'm all for it, this will break code that was not correct in the first place, and was working by accident. Also, being called by the GC from any thread prevent many kinds of cleanup, such as those with a thread-wise bound context (CUDA, OpenGL, etc...).
May 01 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Wednesday, 30 April 2014 at 22:48:28 UTC, Andrei Alexandrescu 
wrote:
 On 4/30/14, 3:08 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 21:51:17 UTC, Andrei 
 Alexandrescu wrote:
 On 4/30/14, 2:47 PM, John Colvin wrote:
 On Wednesday, 30 April 2014 at 20:57:26 UTC, Andrei 
 Alexandrescu wrote:
 Finally, immutable is sharable accross thread. That mean, 
 even if we
 bypass the type system, that RC must be atomic for 
 immutable.
 As they
 convert automatically for co,st, that mean that all const 
 code will be
 full of atomic increment/decrement. They are bad for the 
 CPU, and
 cannot
 be optimized away.

Good point. I see that as a problem, albeit a solvable one.

How? Having lock; instructions implicitly appearing in normal looking slice code is unacceptable.

I'm thinking e.g. non-interlocked refcounts go like 1, 3, 5, ... and interlocked refcounts go like 2, 4, 6, ... Then you do an unprotected read of the refcount. If it's odd, then it's impossible to having originated as an interlocked one. So proceed with simple increment. If it's even, do an interlocked increment. Andrei

I don't think I fully understand. Either all RC changes for a given type need to be atomic or none do, and that information is given by the type (everything that is immutable/const/shared). I don't see any feasible way of escaping this, or any advantage to a runtime convention like the odd/even trick above.

An object starting as shared or immutable would always need to be atomically refcounted. That information is statically known. For those we'd initialize the refcount to 2. An object starting as regular mutable would always be refcounted non-atomically. That's also known statically. So that constructor initializes the refcount to 1. Then a const object would dynamically choose the approach to refcounting depending on the counter's evenness. Andrei

Ah ok, that makes sense.
May 01 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Thursday, 1 May 2014 at 01:04:08 UTC, Steven Schveighoffer 
wrote:
 That means classes that need cleanup (either directly or by 
 having fields that are structs with destructors) would need to 
 garner that by other means, such as reference counting or 
 manual. We're considering deprecating ~this() for classes in 
 the future.

So essentially, any class with a dtor needs reference counting? How does one clean up a cycle of them?

Yeah, what he said. This has me very worried. Making cycles is actually incredibly easy. Any "inteligently" implemented "node-based" data structure, implemented with classes, such as a linked list, a tree or a graph, is virtually guaranteed to have a cycle somewhere... *Or* is the idea that reference counting only works for finalization, but the memory is still managed by the GC? In that case, the destructors would leak, but the memory still be released? That's still bad, of course, but not "as" bad...
May 01 2014
prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
Removing class destructors would break my DQt library as it 
currently stands, which wraps C++ classes with D classes, Qt uses 
classes for polymorphism, for protected overrides and slots. I 
would have to consider some other means of wrapping C++. It might 
also be confusing for a few people.

So, having said that... I am completely in favour of removing 
class destructors. It's a simple principle. When you have to do 
less work, things can run faster. So I can see removing class 
destructors as a feature leading to reduced pause times. 
Destructors exist only for resource management. Doing resource 
management through GC... is a bad idea.
May 01 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 5:04 AM, w0rp wrote:
 Removing class destructors would break my DQt library as it currently
 stands, which wraps C++ classes with D classes, Qt uses classes for
 polymorphism, for protected overrides and slots. I would have to
 consider some other means of wrapping C++. It might also be confusing
 for a few people.

 So, having said that... I am completely in favour of removing class
 destructors. It's a simple principle. When you have to do less work,
 things can run faster. So I can see removing class destructors as a
 feature leading to reduced pause times. Destructors exist only for
 resource management. Doing resource management through GC... is a bad idea.

I've decided what I'll do. I'll implement both and let the user choose. Push policy up, implementation down! -- Andrei
May 01 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d
wrote:
 On Wed, 30 Apr 2014 13:21:33 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:
 

 First off, we're considering eliminating destructor calls from
 within the GC entirely. It makes for a faster and better GC, but the
 real reason here is that destructors are philosophically bankrupt in
 a GC environment. I think there's no need to argue that in this
 community.  The GC never guarantees calling destructors even today,
 so this decision would be just a point in the definition space
 (albeit an extreme one).

I really don't like the fact that struct destructors are not called by the GC, and if anything, I'd be inclined to argue for finding a way to guarantee that they get run rather than guaranteeing that they never get run. It's just far too easy to have a struct expect that its destructor will be run and then have issues when it's not run. But it would be better to define struct destructors as never getting run rather than having it be undefined as it is now.

+1.
 We're considering deprecating ~this() for classes in the future.

While it's not good to rely on finalizers, they're good to have as backup if the appropriate cleanup function doesn't get called like it's supposed to. They're not as critical as they'd be in Java, since we have structs, but I'd be disinclined to remove finalizers from D without a really good reason.

I'd like to hear an enumeration of those reasons as well. While in principle I agree with the sentiment to get rid of class dtors, I'm concerned about the rippling side-effects this will have throughout the rest of the language. Such as class members that are structs with dtors, which will mean that the struct dtors will never get called. Using another poster's argument: if dtors are used for resource management, then it's a bad idea to mix dtors with GC (i.e. classes). So in this sense I agree with getting rid of class dtors. But we have to consider what happens to the case where you stick a struct with a dtor into a class. I'm inclined to say that we should outright prohibit that, because again, the dtor (struct dtor this time) is used for resource management, and therefore shouldn't be mixed with a GC'd resource (i.e., classes). I find it unacceptable that the compiler will silently accept such a usage, and yet have no semantic guarantees (that dtor will run). This is the kind of behaviour I expect from C/C++, but not from D. Next thing you know, somebody is gonna start wrapping structs inside classes as a way of suppressing the dtor. I don't think that's the kind of thing the language should allow. I contend that it will do the language a great deal of good to outright ban such a kind of usage. (Structs with no dtors, OTOH, can be safely used as class members, obviously.) T -- It only takes one twig to burn down a forest.
May 01 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 7:30 AM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via Digitalmars-d
wrote:
 While it's not good to rely on finalizers, they're good to have as
 backup if the appropriate cleanup function doesn't get called like
 it's supposed to. They're not as critical as they'd be in Java, since
 we have structs, but I'd be disinclined to remove finalizers from D
 without a really good reason.

I'd like to hear an enumeration of those reasons as well.

1. Most scarce resources must be released eagerly. GC collections occur relatively rarely and are triggered by different signals (low on memory). 2. The notion of running the GC when one runs out of file handles or sockets is terribly inefficient. 3. Destructors are odd - they will run in a different thread than the one that created the object. They also are limited in surprising ways, i.e. may not allocate memory or block on other threads directly or indirectly. 4. Due to imprecision, there's no actual guarantee any given destructor will end up running. Leaving a scarce resources at the whim of a best-effort approach is poor design.
 While in principle I agree with the sentiment to get rid of class dtors,
 I'm concerned about the rippling side-effects this will have throughout
 the rest of the language. Such as class members that are structs with
 dtors, which will mean that the struct dtors will never get called.

 Using another poster's argument: if dtors are used for resource
 management, then it's a bad idea to mix dtors with GC (i.e. classes). So
 in this sense I agree with getting rid of class dtors. But we have to
 consider what happens to the case where you stick a struct with a dtor
 into a class. I'm inclined to say that we should outright prohibit that,

That can't happen. Andrei
May 01 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei
May 01 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 5/1/14, 12:52 PM, "Nordlw" wrote:
 into a class. I'm inclined to say that we should outright prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist. Andrei
May 01 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 3:17 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 02:29:22PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
Digitalmars-d wrote:
 On 5/1/14, 12:52 PM, "Nordlw" wrote:
 into a class. I'm inclined to say that we should outright
 prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist.

Argh, I feel like this discussion is going in circles. Do I really have to spell everything out just to make my point? 1) Today, a struct with a dtor will have the dtor invoked when the struct goes out of scope. 2) Today, the language allows you to put such a struct in a class as a member variable. 3) Today, the struct dtor of the class member will get invoked when the class dtor is invoked. 4) Andrei is proposing to deprecate class dtors, meaning that at some point they will no longer exist. 5) When that day comes, class dtors will no longer exist. 6) Since class dtors were the only thing that cleaned up the struct member variables by invoking their dtors, that means the struct dtor will *never* get invoked. Since the struct dtor will *never* get invoked, it makes no sense to define one in the first place.

Here's where the point derails. A struct may be preexisting; the decision to define a destructor for it and the decision to use polymorphism for an object that needs that structure are most of the time distinct. Andrei
May 01 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/2/14, 1:34 AM, Paolo Invernizzi wrote:
 On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu wrote:
 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via
 Digitalmars-d wrote:
 On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright
 prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist. Andrei

I have 165k lines of code to review for that change... I would not call it a minor breakage...

I didn't. I said a lot less that straight out disallowing struct members. -- Andrei
May 02 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 2:04 PM, "Nordlöw" wrote:
 So to check if I got this right; class dtors will, with your proposal,
 only be called by the GC when any of its struct members has dtors, right?

No, in this proposal the GC heap would be entirely passive. Given the pushback in this thread that prolly won't fly. Andrei
May 01 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/1/14, 2:21 PM, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright prohibit
 that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

class C { void finalizer(); private struct S { C owner; ~this() { owner.finalize(); } } private S s; } You aren't helping anything.

There's a misunderstanding somewhere. -- Andrei
May 01 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/2/14, 12:07 AM, deadalnix wrote:
 The breakage in the proposal is already massive. For instance, GtkD
 won't work. And other example have been presented.

Yah, prolly we can't go that far. -- Andrei
May 02 2014
prev sibling next sibling parent Benjamin Thaut <code benjamin-thaut.de> writes:
Am 01.05.2014 19:35, schrieb "Marc Schütz" <schuetzm gmx.net>":
 But conceptually, this is _not_ the same as classes! As others have
 mentioned, it's possible to created structs with `new`, or have them in
 dynamic arrays, as well as managing class objects manually.

 Maybe the language should have some way to distinguish between
 GC-managed and manually-managed objects, preferably in the type system.
 Then it could be statically checked whether an object is supposed to be
 GC-managed, and consequentially shouldn't have a destructor.

 The difference between classes and structs should then be reference vs.
 value semantics, and polymorphic vs static (which correlate nicely). It
 should, however, not imply whether the object is managed by the GC or
 not. Some kind of ownership mechanism would be more suited for that.

+1
May 03 2014
prev sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-01 17:35:36 +0000, "Marc Schtz" <schuetzm gmx.net> said:

 Maybe the language should have some way to distinguish between 
 GC-managed and manually-managed objects, preferably in the type system. 
 Then it could be statically checked whether an object is supposed to be 
 GC-managed, and consequentially shouldn't have a destructor.

Or turn the rule on its head: make it so having a destructor makes the heap memory block reference counted. With this adding a destructor always cause deterministic destruction. The compiler knows statically whether a struct has a destructor. For a class you need a runtime trick because the root object which can be either. Use a virtual call or a magic value in the reference count field to handle the reference count management. You also need a way to tag a class to be guarantied it has no derived class with a destructor (to provide a static proof for the compiler it can omit ARC code), perhaps disable ~this(). Then remains the problem of cycles. It could be a hard error if the destructor is safe (error thrown when the GC collects it). The destructor could be allowed to run (in any thread) if the destructor is system or trusted. The interesting thing with this is that the current D semantics are preserved, destructors become deterministic (except in the presence of cycles, which the GC will detect for you), and if you're manipulating pointers to pure memory (memory blocks having no destructor) there's no ARC overhead. And finally, no new pointer attributes; Walter will like this last one. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 03 2014
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-04 09:00:45 +0000, "Marc Schtz" <schuetzm gmx.net> said:

 On Saturday, 3 May 2014 at 11:12:56 UTC, Michel Fortin wrote:
 Or turn the rule on its head: make it so having a destructor makes the 
 heap memory block reference counted. With this adding a destructor 
 always cause deterministic destruction.
 
 The compiler knows statically whether a struct has a destructor. For a 
 class you need a runtime trick because the root object which can be 
 either. Use a virtual call or a magic value in the reference count 
 field to handle the reference count management. You also need a way to 
 tag a class to be guarantied it has no derived class with a destructor 
 (to provide a static proof for the compiler it can omit ARC code), 
 perhaps  disable ~this().
 
 Then remains the problem of cycles. It could be a hard error if the 
 destructor is  safe (error thrown when the GC collects it). The 
 destructor could be allowed to run (in any thread) if the destructor is 
  system or  trusted.
 
 The interesting thing with this is that the current D semantics are 
 preserved, destructors become deterministic (except in the presence of 
 cycles, which the GC will detect for you), and if you're manipulating 
 pointers to pure memory (memory blocks having no destructor) there's no 
 ARC overhead. And finally, no new pointer attributes; Walter will like 
 this last one.

This is certainly also an interesting idea, but I suspect it is bound to fail, simply because it involves ARC. Reference counting always makes things so much more complicated... See for example the cycles problem you mentioned: If you need a GC for that, you cannot guarantee that the objects will be collected, which was the reason to introduce ARC in the first place. Then there are the problems with shared vs. thread-local RC (including casting between the two), and arrays/slices of RC objects. And, of course, Walter doesn't like it ;-)

Arrays/slices of RC objects would have to be reference counted too. That was part of Andrei's proposal. Cycles break deterministic destruction and break the guaranty the destructor will be called because we're relying on the GC to free them, true. But today we have no guaranty at all, even when there's no cycle. It is possible to avoid cycles, and not that hard to do in a safe way when you have weak (auto-nulling) pointers (which D should have regardless of the memory management scheme). It's even easier if you use the GC to detect them. I'm proposing that GC calls to a safe destructor be a hard error so you can use that error to fix your cycles. (Note that cycles with no destructor are perfectly fine though.) As for shared vs. thread-local RC, Andrei proposed a solution for that earlier. http://forum.dlang.org/post/ljrr8l$2dt1$1 digitalmars.com I know Walter doesn't like to have pointers variables do magic stuff. This is why I'm confining ARC behaviour to objects, structs, and arrays having destructors. Andrei's initial proposal was to completely remove destructors for objects; all the pointers that would have survived that proposal are still RC-free in mine except for non-final classes when you haven't explicitly disabled the destructor. (An alternative to disabling the destructor would be to use a different root object for RC, I just find that less desirable.) Frankly, if we want destructors to work safely when on the heap, I don't see another way. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 04 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 00:07:42 -0400, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 4/30/14, 7:20 PM, Steven Schveighoffer wrote:
 On Wed, 30 Apr 2014 21:51:38 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 4/30/14, 6:04 PM, Steven Schveighoffer wrote:
 destructors are for cleaning up non-GC resources. File handles,  
 malloc'd
 memory, etc. I don't see why these need to be eliminated.

Virtually all GCs are known to be horrible at managing scarce resources (including memory itself).

The destructor can be a crutch, but it's not good to leave open resources when the user of your code has not cleaned them up manually.

Sounds like defensive programming to me.

Not at all. I've written programs that run for weeks at a time without ever crashing, who continually manage such resources via the GC. If the GC cleans up your object, but your object wasn't finalized, you would have to make that a hard error. i.e. crash. Still not talked about that I have seen, is the fact that with RC, we need a solution for cycles. If that's the GC then the GC WILL be cleaning up objects, even if they are ref counted.
 I can see no reason to disallow destruction from cleaning up resources
 that nobody else has managed to clean up.

Make memory management faster and better for everyone. I'm not saying it's reason enough, but it's a reason.

It doesn't make it better. The GC still is cleaning up the memory. Having it call close(fd) doesn't delay or make worse anything.
 The largest problem with D destructors comes from trying to clean up D
 objects in destructors of structs, that you never expected to be done
 via the GC.

 Cleaning up files and malloc'd memory is not an issue.

Not getting this...

class C { private int fd = -1; private ubyte[] buffer; this(string fname) { fd = open(fname, O_CREAT); buffer = (cast(ubyte*).malloc(100))[0..100]; } ~this() { if(fd != -1) { close(fd); fd = -1;} if(buffer.ptr) { .free(buffer.ptr); buffer = null} } } There is no problem with this code. It works correctly in all cases. It's not inefficient. It's not cumbersome or awkward or confusing. It's exactly what class destructors are for. However, this *is* a problem: class C { RCValue s; } I have no way to say how s will be destructed, unless I poison it somehow. Even still, s.~this() will be called when C.~this() is done. This can be run in a separate thread as another object that is calling s.~this() at the same time. Removing ~this() for classes just ADDS complication that we don't need. You are trying to fix the problem for the second situation by introducing a problem for the first that is unnecessary. Solutions for the second problem are not predicated on jettisoning class dtors. You can, for instance, try and run the dtor for the instance of C in the same thread where it is owned. You could, as you say, force C into a ref counted mechanism. But this doesn't mean you should prevent C from cleaning up its resources! Please understand that I'm not arguing for the status quo. What I'm arguing is that class destructors DO have some valid uses, and removing them would introduce new problems. I agree we need to find a solution to the ref counting + GC problem. -Steve
May 01 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 1 May 2014 at 15:28:25 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 7:30 AM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:37:28AM -0700, Jonathan M Davis via 
 Digitalmars-d wrote:
 While it's not good to rely on finalizers, they're good to 
 have as
 backup if the appropriate cleanup function doesn't get called 
 like
 it's supposed to. They're not as critical as they'd be in 
 Java, since
 we have structs, but I'd be disinclined to remove finalizers 
 from D
 without a really good reason.

I'd like to hear an enumeration of those reasons as well.

1. Most scarce resources must be released eagerly. GC collections occur relatively rarely and are triggered by different signals (low on memory). 2. The notion of running the GC when one runs out of file handles or sockets is terribly inefficient. 3. Destructors are odd - they will run in a different thread than the one that created the object. They also are limited in surprising ways, i.e. may not allocate memory or block on other threads directly or indirectly. 4. Due to imprecision, there's no actual guarantee any given destructor will end up running. Leaving a scarce resources at the whim of a best-effort approach is poor design.

These are good arguments against destructors for GC managed objects, IMO. But conceptually, this is _not_ the same as classes! As others have mentioned, it's possible to created structs with `new`, or have them in dynamic arrays, as well as managing class objects manually. Maybe the language should have some way to distinguish between GC-managed and manually-managed objects, preferably in the type system. Then it could be statically checked whether an object is supposed to be GC-managed, and consequentially shouldn't have a destructor. The difference between classes and structs should then be reference vs. value semantics, and polymorphic vs static (which correlate nicely). It should, however, not imply whether the object is managed by the GC or not. Some kind of ownership mechanism would be more suited for that.
May 01 2014
prev sibling next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
 into a class. I'm inclined to say that we should outright 
 prohibit that,

That can't happen.

Why is that? /Per
May 01 2014
prev sibling next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
 into a class. I'm inclined to say that we should outright 
 prohibit that,

That can't happen.

Why is that? /Per
May 01 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 5/1/14, 12:52 PM, "Nordlw" wrote:
into a class. I'm inclined to say that we should outright prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors? (Don't forget emplace'd class instances, which do not live in GC heap.) T -- Let's call it an accidental feature. -- Larry Wall
May 01 2014
prev sibling next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
 (1) Too much breakage, (2) would disallow a ton of correct 
 code, (3) no reasonable alternative to propose. We'd 
 essentially hang our users out to dry. -- Andrei

I misunderstood your usage of the word "can't". I thought it symbolized a practical/theoretical limitation, not design decision. So to check if I got this right; class dtors will, with your proposal, only be called by the GC when any of its struct members has dtors, right? For this to work the compiler will have to do some kind dtor inference of members, members of members etc, right?
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 12:52 PM, "Nordl=C3=B6w" wrote:
 into a class. I'm inclined to say that we should outright prohibit =




 that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) n=

 reasonable alternative to propose. We'd essentially hang our users out=

 to dry. -- Andrei

class C { void finalizer(); private struct S { C owner; ~this() { owner.finalize(); } } private S s; } You aren't helping anything. -Steve
May 01 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, May 01, 2014 at 02:29:22PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
On 5/1/14, 12:52 PM, "Nordlw" wrote:
into a class. I'm inclined to say that we should outright
prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist.

Argh, I feel like this discussion is going in circles. Do I really have to spell everything out just to make my point? 1) Today, a struct with a dtor will have the dtor invoked when the struct goes out of scope. 2) Today, the language allows you to put such a struct in a class as a member variable. 3) Today, the struct dtor of the class member will get invoked when the class dtor is invoked. 4) Andrei is proposing to deprecate class dtors, meaning that at some point they will no longer exist. 5) When that day comes, class dtors will no longer exist. 6) Since class dtors were the only thing that cleaned up the struct member variables by invoking their dtors, that means the struct dtor will *never* get invoked. Since the struct dtor will *never* get invoked, it makes no sense to define one in the first place. What's the point of defining a struct dtor if there is no guarantee it will get invoked? That completely defeats the purpose of a dtor. Do we really want D's dtors to have the semantics of "this *might* get called with the struct is destroyed, or it *might* not?" Then we might as well go back to C and do manual cleanups. And if Andrei's proposal to get rid of class dtors goes through, then the only way to guarantee dtor semantics is to prohibit the use of structs with dtors as class members. It's either that, or struct dtors will be left in the limbo of "maybe it will get called, maybe it won't". I don't see this as an advantageous position at all -- in fact, it's a regression from the current language, which, as Walter demonstrated, *does* invoke the struct dtor when the class goes out of scope. T -- We are in class, we are supposed to be learning, we have a teacher... Is it too much that I expect him to teach me??? -- RL
May 01 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Thu, 01 May 2014 17:33:56 -0400, Andrei Alexandrescu  =

<SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 2:21 PM, Steven Schveighoffer wrote:
 On Thu, 01 May 2014 16:03:06 -0400, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 On 5/1/14, 12:52 PM, "Nordl=C3=B6w" wrote:
 into a class. I'm inclined to say that we should outright prohibi=






 that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3)=



 no reasonable alternative to propose. We'd essentially hang our user=



 out to dry. -- Andrei

class C { void finalizer(); private struct S { C owner; ~this() { owner.finalize(); } } private S s; } You aren't helping anything.

There's a misunderstanding somewhere. -- Andrei

The above is or is not allowed under the new proposal? You said class dtors will cease to exist, but that you can have structs = = (with dtors?) embedded in them. I defined a de-facto destructor for a class, when none is allowed. Unles= s = you are planning on having classes not call the struct dtors? -Steve
May 01 2014
prev sibling next sibling parent "ed" <sillymongrel gmail.com> writes:
On Thursday, 1 May 2014 at 22:18:59 UTC, H. S. Teoh via 
Digitalmars-d wrote:
[snip]
 6) Since class dtors were the only thing that cleaned up the 
 struct
 member variables by invoking their dtors, that means the struct 
 dtor
 will *never* get invoked.

I might be mistaken but isn't it the case now that class dtors may never get invoked by the GC anyway? In which case having class dtors is pointless because one cannot rely on them. But dtors for structs I thought would still be useful. The dtor might not be called by the class dtor as it no longer exists, but it is called if I reassign the struct member by value. C c = new C(); // some code auto s = SomeStruct(/*with some parms*/); c.setStruct(s); // the existing c.struct dtor will be called, releasing whatever resources it needs Cheers, ed
May 01 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 1 May 2014 at 02:36:23 UTC, Walter Bright wrote:
 On 4/30/2014 6:50 PM, deadalnix wrote:
 On Thursday, 1 May 2014 at 01:20:37 UTC, Walter Bright wrote:
 A link to your previous comment would be useful.

I've written several proposal regarding this. Please at least read them as what you just wrote only proves you are not well informed. Notably, the proposal do not require annotations from the users to do the kind of things are are currently being special cased.

A link would be nice.

http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org Just made a new topic with code sample, so the idea can be grasped more easily (or so I hope).
May 01 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 1 May 2014 at 20:03:03 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright 
 prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

(1) is made of turbo lol. A huge amount of binding to C++ lib rely on destructor. The breakage in the proposal is already massive. For instance, GtkD won't work. And other example have been presented.
May 02 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 2 May 2014 at 00:45:42 UTC, Andrei Alexandrescu wrote:
 Here's where the point derails. A struct may be preexisting; 
 the decision to define a destructor for it and the decision to 
 use polymorphism for an object that needs that structure are 
 most of the time distinct.

 Andrei

Sheep eat grass. Boat float on water. These are completely distinct. Therefore, there won't be any issue if we ditch all the grass in the sea from that boat the carry sheeps around.
May 02 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu 
wrote:
 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei Alexandrescu 
 via Digitalmars-d wrote:
 On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright 
 prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist. Andrei

I have 165k lines of code to review for that change... I would not call it a minor breakage... /Paolo
May 02 2014
prev sibling next sibling parent reply "fra" <a b.it> writes:
On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu 
wrote:
 I think there's no need to argue that in this community. The GC 
 never guarantees calling destructors even today, so this 
 decision would be just a point in the definition space (albeit 
 an extreme one).

I think I (we) need a bit of clarification. Docs in http://dlang.org/class.html#destructors states that: "The garbage collector calls the destructor function when the object is deleted." As far as I understand, this means that destructors are always called when an instance is collected. Is this right? Doesn't this mean that destructors are guaranteed to run for unreferenced objects if we force the GC to do a full collect cycle?
May 02 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/2/14, 9:04 AM, fra wrote:
 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei Alexandrescu wrote:
 I think there's no need to argue that in this community. The GC never
 guarantees calling destructors even today, so this decision would be
 just a point in the definition space (albeit an extreme one).

I think I (we) need a bit of clarification. Docs in http://dlang.org/class.html#destructors states that: "The garbage collector calls the destructor function when the object is deleted." As far as I understand, this means that destructors are always called when an instance is collected. Is this right? Doesn't this mean that destructors are guaranteed to run for unreferenced objects if we force the GC to do a full collect cycle?

False pointers make it seem like unreferenced objects are in fact referenced, so fewer destructors will run than there should. -- Andrei
May 02 2014
prev sibling next sibling parent "monarch_dodra" <monarchdodra gmail.com> writes:
On Friday, 2 May 2014 at 16:20:47 UTC, Andrei Alexandrescu wrote:
 On 5/2/14, 9:04 AM, fra wrote:
 On Wednesday, 30 April 2014 at 20:21:33 UTC, Andrei 
 Alexandrescu wrote:
 I think there's no need to argue that in this community. The 
 GC never
 guarantees calling destructors even today, so this decision 
 would be
 just a point in the definition space (albeit an extreme one).

I think I (we) need a bit of clarification. Docs in http://dlang.org/class.html#destructors states that: "The garbage collector calls the destructor function when the object is deleted." As far as I understand, this means that destructors are always called when an instance is collected. Is this right? Doesn't this mean that destructors are guaranteed to run for unreferenced objects if we force the GC to do a full collect cycle?

False pointers make it seem like unreferenced objects are in fact referenced, so fewer destructors will run than there should. -- Andrei

Yeah, you have to read the "fine print": "collection implies destruction" *but* "no guarantees the collection will actually ever happen".
May 02 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 2 May 2014 at 20:59:46 UTC, monarch_dodra wrote:
 Yeah, you have to read the "fine print": "collection implies 
 destruction" *but* "no guarantees the collection will actually 
 ever happen".

That sound like the right balance. Also, make construction of object with destructor system as there is no way to ensure destructor won't resurrect the object or do some goofy thing with finalized reference it has.
May 02 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Friday, 2 May 2014 at 15:03:47 UTC, Andrei Alexandrescu wrote:
 On 5/2/14, 1:34 AM, Paolo Invernizzi wrote:
 On Thursday, 1 May 2014 at 21:29:19 UTC, Andrei Alexandrescu 
 wrote:
 On 5/1/14, 1:19 PM, H. S. Teoh via Digitalmars-d wrote:
 On Thu, May 01, 2014 at 01:03:06PM -0700, Andrei 
 Alexandrescu via
 Digitalmars-d wrote:
 On 5/1/14, 12:52 PM, "Nordlöw" wrote:
 into a class. I'm inclined to say that we should outright
 prohibit that,

That can't happen.

Why is that?

(1) Too much breakage, (2) would disallow a ton of correct code, (3) no reasonable alternative to propose. We'd essentially hang our users out to dry. -- Andrei

Isn't this what we're already doing by (eventually) getting rid of class dtors?

Not even close. (1) A lot less breakage, (2) disallowed code was already not guaranteed to work, (3) reasonable alternatives exist. Andrei

I have 165k lines of code to review for that change... I would not call it a minor breakage...

I didn't. I said a lot less that straight out disallowing struct members. -- Andrei

I would also add one point, just because it is not so obvious: I'll be more than happy to review my company code, if the proposed solutions about finalisation in class/struct turn out to take D one step forward being a better programming language. /Paolo
May 03 2014
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:
 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.
 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei

Honestly, that sounds like the entierly wrong apporach to me. Your approaching the problem in this way: "We can not implement a propper GC in D because the language design prevents us from doing so. So lets remove destructors to migate the issue of false pointers." While the approach should be. "The language does not allow to implement a propper GC (anything else then dirty mark & sweep), what needs to be changed to allow a implementation of a more sophisticated GC." Also let me tell you that at work we have a large C# codebase which heavily relies on resource management. So basically every class in there inherits from C#'s IDisposable interface which is used to manually call the finalizer on the class (but the C# GC will also call that finalizer!). Basically the entire codebase feels like manual memory management. You have to think about manually destroying every class and the entire advantage of having a GC, e.g. not having to think about memory management and thus beeing more productive, vanished. It really feels like writing C++ with C# syntax. Do we really want that for D? And what if I want unsafe slices of structs with destructors, for performance? Maybe I perfectly know that the memory behind the slice will outlive the slice, and I don't want the overhead of all the reference counthing behind it? If you actually deprecate ~this, there would be two options for me. 1) Migrate my entire codebase to some user defiend finalizer function (which doesn't have compiler support), which would be a lot of work. 2) Quit D. (which is becomeing more and more an option when reading the recent news group discussions.) -- Kind Regards Benjamin Thaut
May 03 2014
next sibling parent reply "froglegs" <barf barf.com> writes:
 feels like writing C++ with C# syntax.

Ahem. C++ has RAII k thanks. C++ > C# Proof.
May 03 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 03.05.2014 10:57, schrieb froglegs:
 feels like writing C++ with C# syntax.

Ahem. C++ has RAII k thanks. C++ > C# Proof.

Which does not work across threads, relies on stack allocations and has issues if cleaning a resource implies having a not handled exception on the destructor of the RAII class. I rather make use of "using" and FP resource handling via lambdas. -- Paulo
May 03 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/3/14, 1:49 AM, Benjamin Thaut wrote:
 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)

The entire idea of starting these discussions is to gather a sense of shared vision with the community on the best direction to follow. I'm trying to steer things so as to gain the maximum benefit from each potential breakage. In wake of that, it seems incongruous that first you ask for even larger breaking changes to the language, but then consider quitting it entirely on account of changes being too disruptive. Anyhow, just to clarify, it seems like eliminating destructor calls during GC is not a viable option. I'll define std.allocator to allow users to define such a GC if they so want, without prescribing either policy. Andrei
May 03 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, May 03, 2014 at 11:12:36AM -0700, Andrei Alexandrescu via Digitalmars-d
wrote:
[...]
 Anyhow, just to clarify, it seems like eliminating destructor calls
 during GC is not a viable option. I'll define std.allocator to allow
 users to define such a GC if they so want, without prescribing either
 policy.

Thank you. ;-) On that note, it would be very nice if we could clearly define when dtor calls will / will not happen. As someone else said, to an end-user of D it is hard to accept that basic language constructs like arrays and dtors, when used together, fail to function in an expected way (without understanding what goes on underneath the hood, e.g. why dtors of array elements may not get called). Ideally, the language should be designed such that these obvious combinations of language constructs should either Just Work(tm), or not be allowed, or require explicit annotation (so that the user knows something unexpected might happen). But, since we're not in an ideal world, the very least we could do is to clearly define exactly which combinations of language constructs may behave in an unexpected way, and document them up front in a prominent place. Otherwise, we risk turning off potential users -- I can just imagine a newbie to D writing something like "File[] files;" and then wondering why things don't work as expected, and then throwing in the towel and say "what a lousy language, let me move on to another one". And on that note, I'd like to say that my previous posts about prohibiting structs with dtors as class members (if we were to move in the direction of getting rid of class dtors altogether, which happily isn't the case anymore) were primarily motivated by the desire to see more of D obeying the principle of least surprise: two built-in language constructs, structs with dtors and class members, when combined together, *should* Just Work -- one expects that dtors will get called with the struct goes out of scope, by the very definition of a dtor, so one would expect they will still get cleaned up when they happen to be a class member. The fact that they don't is a surprise, which then requires some other way of warning the user that things aren't what they might be expected to be. Maybe my proposed solutions suck, but the underlying problem still needs to be addressed. The scope of the problem is smaller, now that we're no longer killing off class dtors, but nevertheless something needs to be done about it. I think this is one area where D could use a lot of improvement. There are currently a handful of glaring holes where built-in language constructs interact with each other in unexpected or buggy ways. While it may feel like mole-whacking (because of combinatorial explosion as you add features to the language), I think it's very important to address, since otherwise it gives new users an impression (not necessarily well-founded, but first impressions do matter) of sloppy language design. The seams show through, and it's not very nice. One example is the interaction of dtors with the GC. Another example is the interaction of const/immutable with AA's. Another is the interaction of disabled with .init and other generic code. It seems that almost every non-trivial use of these features is like navigating a minefield -- there are so many gotchas, unexpected behaviours, and implementation bugs, that it's embarrassing. Other areas include shared, though I don't have first-hand experience of that so I can't speak for it. While proposing radical changes may be exhilarating, in the long run that may do more harm than good. What these features need is some TLC and detailed fine-tuning within the current, already-existing framework. TL;DR: I'd like to see more attention paid to the details of how language features interact with each other, and fixing those issues, rather than inventing more radical new ideas that may or may not solve the problem, and more likely than not will introduce new problems to add to our already-long list of issues. T -- Amateurs built the Ark; professionals built the Titanic.
May 03 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/3/14, 8:48 PM, Caligo via Digitalmars-d wrote:
 On Sat, May 3, 2014 at 3:49 AM, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:

     2) Quit D. (which is becomeing more and more an option when reading
     the recent news group discussions.)

     --
     Kind Regards
     Benjamin Thaut


 I never thought I would say this, but I have begun to move away from D.
   I know others who are doing the same.  The advantages D has over other
 languages are slowly diminishing.  Plus, as it is, D is just as complex
 and complicated as C++, if not more.  Couple that with the never-ending
 bugs and weak development process, there isn't much confidence in
 choosing D.

 One of the main and crippling issues D has is its development process.
   Its inner circle, mainly Walter, Andrei and a few others, do not seem
 to have a good understanding of FOSS development process.  For example,
 Rust is only a few years old and much younger than D, but it has a
 greater number of contributors, and its rate of contributors seems to be
 growing faster.  D has failed at recruitment; I always read people
 (mainly the inner circle) mention the everlasting low-hanging fruits.
   Well, if the number of contributors was growing then the number of
 low-hanging fruits would be decreasing.  Andrei recently introduced the
 bounty system, which not only is an insult to those who contribute to
 FOSS, but it also goes to show that he really doesn't understand how and
 why people contribute to FOSS without ever asking to be compensated.

Mostly good points, but the bountysource program is an experiment by Facebook, not by myself. And (without me trying to speak on Facebook's behalf) it would be difficult to argue that Facebook doesn't understand FOSS or is out there to insult contributors. We're just experimenting with various angles.
 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner circle,
 or have to represent some corporation that's doing something with D.  I
 don't remember seeing a feature that was added to D/Phobos with some
 on/off switch that people could try in the next release, and then send
 in feedback.  You're in a much better position to make a decision about
 a feature if the users have actually used it and reported feedback.

The on/off switch may be a nice idea in the abstract but is hardly the perfect recipe to good language feature development; otherwise everybody would be using it, and there's not overwhelming evidence to that. (I do know it's been done a few times, such as the (in)famous "new scoping rule of the for statement" for C++ which has been introduced as an option by VC++.) I wonder how you've gotten the perception that one needs to be a member of the inner circle mafia to get things into D. Today's major contributors to D came from all over, without any preexisting relationship to anyone else, and their admission ticket has been getting work done. Could you please get into detail on how you view things? (I tried to look over your past posts to see a pattern of rejected contributions, but didn't find such.)
   Ahh, and don't get me started on Phobos review process; it's a joke,
 it's bogus and just pathetic.

Actually I'd love to get you started so I'd understand your angle better. I'm sure we can do a lot better. One good thing Phobos reviews have done since we initiated them has been to prevent bad artifacts to make it into the library. We'd love to make it better. From what I saw witnessing similar processes (C++, Boost, Python, Scala) - they all have some sense of awkward self-importance to them upon the first look. I think that's the way such things work.
 If D had a sound development process, I don't think we would be having
 the problems that we have today.

I've discussed development process with a number of people who participated at such. They mentioned that until you get teams paid to work on the respective system (OS, language, framework etc) it all works on the basis of people doing things to scratch an itch they have. The critical mass is attained when there are enough people to cover a large enough itching area :o). If telling people what to work on on their free time works, I haven't succeeded at it and don't know anyone who has. Andrei
May 03 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/4/14, 5:38 PM, Caligo via Digitalmars-d wrote:
 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:


     Mostly good points, but the bountysource program is an experiment by
     Facebook, not by myself. And (without me trying to speak on
     Facebook's behalf) it would be difficult to argue that Facebook
     doesn't understand FOSS or is out there to insult contributors.
     We're just experimenting with various angles.


 If the bounty system was such a great idea, then every FOSS project
 would be using it.

As I said: experiment.
 Now, hiring full-time engineers to work on a FOSS
 project, that's an entirely different issue.  Besides, if someone is
 trying to figure out how FOSS teams manage to become successful in
 regards to development and all the associated technical and social
 complexities, then all they have to do is study one of the million
 different FOSS projects out there.  Many well known FOSS contributors
 have actually documented their experience and knowledge of managing FOSS
 projects.

Great, a few representative links would be most welcome.
 Here is an idea:  include new features in DMD/Phobos as soon as they
 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days

This is nice, but on the face of it it's just this: an idea on how other people should do things on their free time. I'd have difficulty convincing people they should work that way. The kind of ideas that I noticed are successful are those that actually carry the work through and serve as good examples to follow. Andrei
May 04 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/4/14, 9:58 PM, Caligo via Digitalmars-d wrote:
 On Sun, May 4, 2014 at 11:09 PM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:


     Great, a few representative links would be most welcome.


 Here is a good starting point (it's a classic):

 http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar

Of course I've read that, and multiple times! I thought of mentioning it but concluded it would be trite. -- Andrei
May 04 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:
 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as good
 examples to follow.

There's imperfect but useful pull requests hanging around for years, extern(Obj-C) for instance, which may be useful as an experimental feature to many users, even if it's not ready for inclusion in the official feature list and support. I suspect it's (experimental) presence would stimulate further contribution towards D on iOS for instance; it may be an enabler for other potential contributors.

So it would be nice if you reviewed that code.
 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?

There would be of course room as long as there's be one or more champions for it. Would that be something you'd be interested in?
 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.

Interesting. Have you worked on a related proposal?
 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?

I'd say so, but we'd need initiative and quite a bit of work for such a platform. Would you be interested?
 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?

I agree - and that applies to you, too.
 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.

Contributions make it into the compiler and standard library if and they are properly motivated, well done, and reviewed by the core team which is literally self-appointed. The key to being on the core team is just reviewing contributions. Have you considered looking at submissions that are "hanging around for years"?
 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.

It's as easy as just reviewing stuff. Acta, non verba. Andrei
May 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:
 On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:
 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d


I don't really know anything about it... and that's not the point. I'm just suggesting by my prior email that some steps like creating an experimental space with a lower barrier to entry might encourage growth in the number of overall contributors, which I think was the basic flavour of the emails leading up to it.

Oh but that's very much the point. "Creating" is the right word (and the gerund form is delicious). In order for the "creating" to happen someone would need to do the "creation" - and of course the subsequent related work. You frame the matter as something "they" should do, when the reality is you're very much a part of "they". The idea of an experimental/unstable package has been discussed a number of times. The conclusion I seem to remember is that currently with dub available there's a really simple way to experiment with things in a comfortable and decentralized way. That said, it's not impossible that distributing things in an "std.experimental" package along with the official distribution gives them an extra umph. We could arrange for that to happen. Is this what you think would make the difference?
 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?

There would be of course room as long as there's be one or more champions for it. Would that be something you'd be interested in?

I have no horse in that race, but I see it come up all the time, and it is something I am passively interested in.

Same here. I'm too busy working in and on other aspects of D (which I believe are more important and more urgent) to have another project added to my plate. I'd be in your debt if you could clarify to me that AST macros are not something you believe I must work on right now.
 There's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

What makes dub inappropriate as an experimental space?
 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress? The
 DIP's shortcomings might be discovered by experimental users in the
 field? It's hard to know, but it's an example of the sort of things
 that may have a stifling effect on progress and contribution.

I think DIPs are a marked improvement over informal discussions on the mailing lists. For them to work well, two ingredients are needed: champions and reviewers. In some cases that did happen, in others it didn't.
 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.

Interesting. Have you worked on a related proposal?

Not really, I've initiated numerous discussions which always seems to end at AST macros. The only other semi-reasonable idea I've had is the concept that tagging mixin templates as UDA's might be a practical angle, but it doesn't really make clean sense, creates a syntactic special case and also doesn't seem powerful enough, so I'm not with any solid proposal that I can imagine within the current language framework. There are presently bigger issues that keep me awake at night.

Same here. So then I think you'd find it reasonable that I asked you to clarify these are not things you believe I need to work on. What I'm trying to get across here is that your and my role and opportunities are very much comparable. Anyone's clout within the D community is proportional to the work they're doing on D, and even Walter's role would be reduced to an honorific one if he ceased spending time working on D. My pedal is sealed to the floor already. I'm tapped out. Walter's too. Meaning that anything work you'd find fit to give us to do would need to come at other work's expense. In my case, that would mean overseeing four D projects at work and allocators on my free time. Walter's last piece of work is interfacing with C++ namespaces, which is very necessary for adoption. He and I have gotten pretty good at investing our time efficiently, but at some point there's only so many hours in a day and we need talented people - people like you - to help making things happen.
 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?

I'd say so, but we'd need initiative and quite a bit of work for such a platform. Would you be interested?

Well, in phobos, just approve 'exp' which has been raised countless times. I've got contributions that should be in exp, but instead, they're in limbo, and I've lost momentum and motivation since their completion is blocked by other issues, and I'm receiving no feedback from field testing.

I'm fine with adding an "exp" package to phobos if you believe that that's what's stopping things from happening. But do you really?
 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)

I could ask you the very same thing. What happened to std.serialization? Do you want me to work on that as well?
 In the compiler... I may be interested, but I don't have any such
 compiler feature in mind to motivate the effort.
 I have no idea what an experimental feature platform should look like
 in the compiler, and if it were to exist, I have no such feature in
 mind to make use of it, but I have raised examples of others that
 have.

Well then someone else would need to do all that work, so you'd need to convince them that it's a good idea.
 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?

I agree - and that applies to you, too.

Sure, but my point... is below.
 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.

Contributions make it into the compiler and standard library if and they are properly motivated, well done, and reviewed by the core team which is literally self-appointed. The key to being on the core team is just reviewing contributions. Have you considered looking at submissions that are "hanging around for years"?

Perhaps you misunderstood the point of my post. I've watched people make solid contributions that haven't gotten through. That is discouraging to others considering starting their own work, and for the person who has already put in the effort to continue to do so in the future.

Contributions haven't gotten through because there are not enough people to review them. People like you. We have an inflation of contributions, or if you wish a deflation of reviews.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?
 I think it needs an experimental place to live and have people make
 use of it for what it is. If it's blocked by other unpopular issues
 that aren't receiving attention, perhaps it's presence will manifest
 the appropriate motivation to see those other unpopular issues
 resolved at some point?

(I assume you refer to http://wiki.dlang.org/DIP43; couldn't find associated pull requests.) I don't know. I am not using Objective C, and I think it would be more efficient to have people who already have Objective C expertise to author and review the contribution. Would I be glad to see good interfacing of D with Objective C? Of course!
 My point is that successful OSS seems to be about enabling the
 lowest-friction contribution, and my feeling right now, is that the
 barrier to entry is high. Whether that's true or not I can't really
 comment, but it's a perception, and the mental barrier inhibiting the
 first step is perhaps the most significant barrier of all.

Forgive me but from where I stand you seem completely wrong. There are currently 190 open pull requests in our github projects. The primary reason that keeps them from being merged is good peer review from people like you and me. So currently the bottleneck is on the receiving side, not the initiating side of contributions. You can't claim incompetence on 190 distinct matters. You are fluent in D and a good engineer who could tell good design from bad.
 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.

Surely there must be something that should catch your fancy among the remaining 189 patches.
 Also, the authors themselves said they recognise it's
 not 'ready'. But is it 'experimental'?

Most likely. Have you tried to build their fork?
 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.

It's as easy as just reviewing stuff. Acta, non verba.

I've never felt I have any particular authority to comment on pulls that I have no experience or vested interest in. (I do occasionally comment on pull requests that I have some interest or knowledge in) I've also had some (hopefully useful) commentary in features that did make it; Win64, UDA's, some traits extensions, and lots of bug reports and fix confirmations.

That's a great start, thanks. Keep them coming.
 I'm plenty vocal and active on things I do feel I know about, but
 they're often pretty radical, unpopular, and rarely come even close to
 turning into code. I'm pretty certain that nothing left on my short
 list that I personally *really* care about will ever get pulled, even
 if I did do the work.
 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.

What can we do (short of pulling that) to be more encouraging of you to do good work? Andrei
May 06 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/6/2014 10:47 AM, Manu via Digitalmars-d wrote:
 On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
 I'm not even sure what the process it... if I go through and "LGTM" a
 bunch of pulls, does someone accept my judgement and click the merge
 button?
 You can see why I might not feel qualified to do such a thing?

You don't need to be qualified (although you certainly are) to review PR's. The process is anyone can review/comment on them. Non-language-changing PR's can be pulled by anyone on "Team DMD". Language changing PR's need to be approved by Andrei and I. "Team DMD" consists of people who have a consistent history of doing solid work reviewing PR's.
 I'll criticise at least 10 pull requests in the morning. I'll be
 curious to see what happens.

Great!
 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.

Surely there must be something that should catch your fancy among the remaining 189 patches.

We'll find out... when I wake up.

You don't actually have to understand the entire PR in order to do reviews. You can do micro reviews like "needs explanatory comment", "this line is bad style", "this section looks like a buffer overflow", "bad formatting", "don't use gets()", etc.
May 06 2014
parent luka8088 <luka8088 owave.net> writes:
On 6.5.2014. 20:10, Walter Bright wrote:
 On 5/6/2014 10:47 AM, Manu via Digitalmars-d wrote:
 On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
 I'm not even sure what the process it... if I go through and "LGTM" a
 bunch of pulls, does someone accept my judgement and click the merge
 button?
 You can see why I might not feel qualified to do such a thing?

You don't need to be qualified (although you certainly are) to review PR's. The process is anyone can review/comment on them. Non-language-changing PR's can be pulled by anyone on "Team DMD". Language changing PR's need to be approved by Andrei and I. "Team DMD" consists of people who have a consistent history of doing solid work reviewing PR's.

Interesting. This really needs to pointed out on the site.
May 09 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 12:18 PM, Wyatt wrote:
 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:
 It never occurred to you that people's libraries would be published as
 part of a centralised repository with a tool that manages dependencies?

Hate to be the cynic, but how in the world do you expect people to even know about Dub or code.dlang.org in the first place? I don't see it linked from any of the "obvious" places on dlang.org

It's an oversight - code.dlang.org is fairly recent so there are things to be still ironed out. Please submit a pull request to https://github.com/D-Programming-Language/dlang.org to include code.lang.org in the left navigation panel appropriately. -- Andrei
May 06 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 8:43 AM, Manu via Digitalmars-d wrote:
 On 6 May 2014 22:17, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d wrote:
 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It

 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.

This bit right here reminds me of something the Linux kernel has called "staging" [1]. It's basically what you describe: a subtree within main source tree for things to be publicly available while they finish baking, with the understanding that you're going to continue working on it and hopefully get it promoted to first-class citizen.
 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.

I know I and several other people are still interested in std.simd... :(

It hasn't gone away. It's a perfect example of where I got blocked by some things, and since it wasn't in an accessible location as an in-progress 'exp' module, it's barely used, and I've never gotten any feedback. Perhaps the blocks are lifted now (I haven't checked lately), but clearly I started working on other things and lost momentum, and I think it's safe to attribute this almost entirely to the fact it exists in my fork where nobody will find it, rather than 'exp', where people can still report usage experience, feedback, and keep me on track.

I can't seem to find simd on our dub site http://code.dlang.org/search?q=simd. Did you put it there under another name? Andrei
May 06 2014
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 06.05.2014 20:10, schrieb w0rp:
 On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d wrote:
 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps their
 developments?

 If people are pushing dub, then it should really be installed with DMD.

DUB should feel pretty natural on any operating system if you are a Python (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or PHP (PEAR) programmer. Although PyPi I know doesn't work as well on Windows. (It doesn't explain to you clearly how to install an MSVC compiler for PIP and what not.) I think it would be healthy to host supplementary modules first with DUB as third party modules, and then get them merged into Phobos after they have seen some real world use. That's sure to result in a faster turnaround time for going from experimentation to real usage.

That is the best approach. -- Paulo
May 06 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 06/05/14 19:57, Manu via Digitalmars-d wrote:

 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/

Doesn't Windows have NuGet? -- /Jacob Carlborg
May 06 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/5/2014 11:39 PM, Manu via Digitalmars-d wrote:
 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.

If you look, I have authored some pull requests that I think are very good, but are twisting in the wind and have for some time. Andrei has some, too. We all do. I understand that we'd all like our efforts to find quick approval and karma, but it isn't always going to happen, and we need to not let that discourage us.
May 06 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/6/2014 11:02 AM, w0rp wrote:
 Generally, championing something you believe in tends to require a great deal
of
 personal effort into trying to make it a reality.

That's right. Sometimes, we're lucky to find a champion step up and carry the flag for us, but 98% of the time we've got to champion it ourselves, or it just won't happen.
May 06 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-06 08:39, Manu via Digitalmars-d wrote:
's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress?

Implementing AST macros is probably quite a big investment in time. I'm currently working on other things but I will probably give it a try at some point.
 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)

To be honest, I got board and started to work on D/Objective-C instead, which I know you have interest in as well. But when that is done I will come back to std.serialization. If not sooner, I have a short attention span sometimes ;)
 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?

No, I'm working on it to making it ready. Feature wise I think it's complete. Or rather good enough for inclusion. It supports for more features than extern(C++) did when it was added. -- /Jacob Carlborg
May 06 2014
parent Jacob Carlborg <doob me.com> writes:
On 07/05/14 05:36, Manu via Digitalmars-d wrote:

 Haha, nice! I didn't realise that all my examples for hypothetical
 consideration came back to just you! :)

 So then, your take on an experimental space in the compiler for
 features that are still baking seems especially relevant.
 Am I talking shit, or do you think the idea has any value? Would it be
 valuable to get users testing these (quite large) developments while
 they're still baking?

I don't think not so much for D/Objective-C but definitely for AST macros and std.serialization. D/Objective-C is pretty straight forward, at least for the user facing parts. But it wouldn't hurt to test D/Objective-C. -- /Jacob Carlborg
May 07 2014
prev sibling next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
[...]
 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner
 circle, or have to represent some corporation that's doing something
 with D.

I'm sorry, but this is patently false. I am neither in the inner circle, nor do I represent any corporation, yet I've had many changes pulled into Phobos (including brand new code). I can't say I'm perfectly happy with the D development process either, but this kind of accusation is bordering on slander, and isn't helping anything. T -- Why are you blatanly misspelling "blatant"? -- Branden Robinson
May 04 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 05/05/14 00:55, H. S. Teoh via Digitalmars-d wrote:

 I'm sorry, but this is patently false. I am neither in the inner circle,
 nor do I represent any corporation, yet I've had many changes pulled
 into Phobos (including brand new code).

I think he's referring to language changes. Things that will require a DIP, i.e. not just adding a new __trait. -- /Jacob Carlborg
May 04 2014
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
05-May-2014 10:16, Arlen пишет:
 On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
 Digitalmars-d wrote:
 On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
 [...]
 Last but not least, currently there are two main ways for new features
 to make it into D/Phobos: you either have to belong to the inner
 circle, or have to represent some corporation that's doing something
 with D.

I'm sorry, but this is patently false. I am neither in the inner circle, nor do I represent any corporation, yet I've had many changes pulled into Phobos (including brand new code). I can't say I'm perfectly happy with the D development process either, but this kind of accusation is bordering on slander, and isn't helping anything. T

There is a lot of truth in what Caligo has said, but I would word that part of it differently. A couple years ago I submitted std.rational, but it didn't go anywhere. About a year later I discovered that someone else had done a similar thing, but it never made it into Phobos either.

The key to getting things done is persistence. Everybody is on their spare time, nobody aside from the author would be able to push it through. The process is not "I submit code and it finds its way into the standard library". It's rather getting people to try your stuff first and listening to them. Then with enough momentum and feedback one would go to review queue. Then start a review if nobody objects, then get into pass or postpone cycle, then survive the mess as the pull request goes into Phobos proper. Last but not least the burden of getting something into it is minor compared to tending the bugs and maintaining the stuff afterwards.
 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.

What that makes of some other open-source projects, that still traffic in patches over email :)
 The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.

Look at it this way - when something is simpler, it makes it that much harder to make the one and true version of it. Everybody knows what it is, and tries to put in some of his favorite sauce. The hardest things to push into Phobos are one-liners even if it makes a ton of things look better, more correct and whatnot. Anyhow I agree that Phobos development process (the one I know about most) is slow and imperfect largely due to the informal nature of participation. Some reviews were lively and great, some went in a gloomy silence with uncertain results without any good indication of the reason.
 --Arlen

-- Dmitry Olshansky
May 05 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/4/14, 11:16 PM, Arlen wrote:
 A couple years ago I submitted std.rational, but it didn't go
 anywhere.  About a year later I discovered that someone else had
 done a similar thing, but it never made it into Phobos either.
 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.  The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.

I looked into this (not sure to what extent it's representative of a pattern), and probably we could and should fix it. Looks like back in 2012 you've done the right things (http://goo.gl/kbYQJM) but for whatever reason there was not enough response from the community. Later on, Joseph Rushton Wakeling tried (http://goo.gl/XyQu3D) to put std.rational through the review process but things got stuck at https://github.com/D-Programming-Language/phobos/pull/1616 with support of traits by BigInt. I think the "needs to support BigInt" argument is not a blocker - we can release std.rational to only support built-in integers, and then adjust things later to expand support while keeping backward compatibility. I do think it's important that BigInt supports appropriate traits to be recognized as an integral-like type. If you, Joseph, or both would want to put std.rational again through the review process I think it should get a fair shake. I do agree that a lot of persistence is needed. Andrei
May 05 2014
prev sibling next sibling parent Caligo via Digitalmars-d <digitalmars-d puremagic.com> writes:
--047d7b6dd0844ea02504f89c5a89
Content-Type: text/plain; charset=UTF-8

On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:
 Mostly good points, but the bountysource program is an experiment by
 Facebook, not by myself. And (without me trying to speak on Facebook's
 behalf) it would be difficult to argue that Facebook doesn't understand
 FOSS or is out there to insult contributors. We're just experimenting with
 various angles.

If the bounty system was such a great idea, then every FOSS project would be using it. Now, hiring full-time engineers to work on a FOSS project, that's an entirely different issue. Besides, if someone is trying to figure out how FOSS teams manage to become successful in regards to development and all the associated technical and social complexities, then all they have to do is study one of the million different FOSS projects out there. Many well known FOSS contributors have actually documented their experience and knowledge of managing FOSS projects.
 The on/off switch may be a nice idea in the abstract but is hardly the
 perfect recipe to good language feature development; otherwise everybody
 would be using it, and there's not overwhelming evidence to that. (I do
 know it's been done a few times, such as the (in)famous "new scoping rule
 of the for statement" for C++ which has been introduced as an option by
 VC++.)

such a thing, #![feature(X,Y,Z)]. So does Haskell, with {-# feature #-}. Even Python has __future__, and many others.
 I wonder how you've gotten the perception that one needs to be a member of
 the inner circle mafia to get things into D. Today's major contributors to
 D came from all over, without any preexisting relationship to anyone else,
 and their admission ticket has been getting work done. Could you please get
 into detail on how you view things? (I tried to look over your past posts
 to see a pattern of rejected contributions, but didn't find such.)

are not members of a certain group. I was just trying to say that it's more likely for a _new_ feature to make it into D/Phobos if they are proposed by members of the inner circle or someone representing a corporation, probably because they become more noticeable, or because they get lost in all the forum noise, I don't know. I could be wrong, but that's just how I perceive it.
 Actually I'd love to get you started so I'd understand your angle better.
 I'm sure we can do a lot better. One good thing Phobos reviews have done
 since we initiated them has been to prevent bad artifacts to make it into
 the library. We'd love to make it better. From what I saw witnessing
 similar processes (C++, Boost, Python, Scala) - they all have some sense of
 awkward self-importance to them upon the first look. I think that's the way
 such things work.

arrive, and make them part of the official binary release so that the average D user can try them out. Make sure they are marked as unstable, and put a on/off switch on them (something like what Rust/Haskell have; not a compiler switch). If the feature receives no implementation bug reports for X consecutive days AND no design bug reports for Y consecutive days, then the feature is marked stable and officially becomes part of DMD/Phobos. The X and the Y can be decreased as D's number of users increases over the years. The whole idea is very much like farming: you are planting seeds. As the plants grow, some of them will not survive, others will be destroyed, and some of them will take years to grow. In any case, you harvest the fruits when they are ready. Here are good starting values for X and Y: X = 90 days Y = 180 days --047d7b6dd0844ea02504f89c5a89 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On S= un, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d <span di= r=3D"ltr">&lt;<a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_bla= nk">digitalmars-d puremagic.com</a>&gt;</span> wrote:<blockquote class=3D"g= mail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-l= eft:1ex"> <br> Mostly good points, but the bountysource program is an experiment by Facebo= ok, not by myself. And (without me trying to speak on Facebook&#39;s behalf= ) it would be difficult to argue that Facebook doesn&#39;t understand FOSS = or is out there to insult contributors. We&#39;re just experimenting with v= arious angles.</blockquote> <div><br></div><div>If the bounty system was such a great idea, then every = FOSS project would be using it. =C2=A0Now, hiring full-time engineers to wo= rk on a FOSS project, that&#39;s an entirely different issue. =C2=A0Besides= , if someone is trying to figure out how FOSS teams manage to become succes= sful in regards to development and all the associated technical and social = complexities, then all they have to do is study one of the million differen= t FOSS projects out there. =C2=A0Many well known FOSS contributors have act= ually documented their experience and knowledge of managing FOSS projects.<= /div> <div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8= ex;border-left:1px #ccc solid;padding-left:1ex"><div class=3D""> <br></div> The on/off switch may be a nice idea in the abstract but is hardly the perf= ect recipe to good language feature development; otherwise everybody would = be using it, and there&#39;s not overwhelming evidence to that. (I do know = it&#39;s been done a few times, such as the (in)famous &quot;new scoping ru= le of the for statement&quot; for C++ which has been introduced as an optio= n by VC++.)<br> <br></blockquote><div><br></div><div>No, it&#39;s nothing abstract, and it&= #39;s very practical and useful. =C2=A0Rust has such a thing, #![feature(X,= Y,Z)]. =C2=A0So does Haskell, with {-# feature #-}. =C2=A0Even Python has _= _future__, and many others.</div> <div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8= ex;border-left:1px #ccc solid;padding-left:1ex"> I wonder how you&#39;ve gotten the perception that one needs to be a member= of the inner circle mafia to get things into D. Today&#39;s major contribu= tors to D came from all over, without any preexisting relationship to anyon= e else, and their admission ticket has been getting work done. Could you pl= ease get into detail on how you view things? (I tried to look over your pas= t posts to see a pattern of rejected contributions, but didn&#39;t find suc= h.)<div class=3D""> <br> <br></div></blockquote><div><br></div><div>I wasn&#39;t trying to imply tha= t contributions are rejected if contributors are not members of a certain g= roup. =C2=A0I was just trying to say that it&#39;s more likely for a _new_ = feature to make it into D/Phobos if they are proposed by members of the inn= er circle or someone representing a corporation, probably because they beco= me more noticeable, or because they get lost in all the forum noise, I don&= #39;t know. =C2=A0I could be wrong, but that&#39;s just how I perceive it.<= /div> <div><br></div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"= margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class= =3D""> <br></div> Actually I&#39;d love to get you started so I&#39;d understand your angle b= etter. I&#39;m sure we can do a lot better. One good thing Phobos reviews h= ave done since we initiated them has been to prevent bad artifacts to make = it into the library. We&#39;d love to make it better. From what I saw witne= ssing similar processes (C++, Boost, Python, Scala) - they all have some se= nse of awkward self-importance to them upon the first look. I think that&#3= 9;s the way such things work.<div class=3D""> <br> <br></div></blockquote><div>=C2=A0</div><div>Here is an idea: =C2=A0include= new features in DMD/Phobos as soon as they arrive, and make them part of t= he official binary release so that the average D user can try them out. =C2= =A0Make sure they are marked as unstable, and put a on/off switch on them (= something like what Rust/Haskell have; not a compiler switch). =C2=A0If the= feature receives no implementation bug reports for X consecutive days AND = no design bug reports for Y consecutive days, then the feature is marked st= able and officially becomes part of DMD/Phobos. =C2=A0The X and the Y can b= e decreased as D&#39;s number of users increases over the years. =C2=A0The = whole idea is very much like farming: you are planting seeds. =C2=A0As the = plants grow, some of them will not survive, others will be destroyed, and s= ome of them will take years to grow. =C2=A0In any case, you harvest the fru= its when they are ready.</div> <div><br></div><div>=C2=A0Here are good starting values for X and Y:</div><= div>X =3D 90 days</div><div>Y =3D 180 days</div></div></div></div> --047d7b6dd0844ea02504f89c5a89--
May 04 2014
prev sibling next sibling parent Caligo via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a11c2c1d253d54204f89ffc13
Content-Type: text/plain; charset=UTF-8

On Sun, May 4, 2014 at 11:09 PM, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:
 Great, a few representative links would be most welcome.

http://en.wikipedia.org/wiki/The_Cathedral_and_the_Bazaar Here is an idea: include new features in DMD/Phobos as soon as they
 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days

This is nice, but on the face of it it's just this: an idea on how other people should do things on their free time. I'd have difficulty convincing people they should work that way. The kind of ideas that I noticed are successful are those that actually carry the work through and serve as good examples to follow. Andrei

--001a11c2c1d253d54204f89ffc13 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On S= un, May 4, 2014 at 11:09 PM, Andrei Alexandrescu via Digitalmars-d <span di= r=3D"ltr">&lt;<a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_bla= nk">digitalmars-d puremagic.com</a>&gt;</span> wrote:<blockquote class=3D"g= mail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-= left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> <div class=3D""> <br></div> Great, a few representative links would be most welcome.<div class=3D""><br=

(it&#39;s a classic):</div><div><br></div><div><a href=3D"http://en.wikiped= ia.org/wiki/The_Cathedral_and_the_Bazaar">http://en.wikipedia.org/wiki/The_= Cathedral_and_the_Bazaar</a><br> </div><div><br></div><div><br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(20= 4,204,204);border-left-style:solid;padding-left:1ex"><div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Here is an idea: =C2=A0include new features in DMD/Phobos as soon as they<b= r> arrive, and make them part of the official binary release so that the<br> average D user can try them out. =C2=A0Make sure they are marked as unstabl= e,<br> and put a on/off switch on them (something like what Rust/Haskell have;<br> not a compiler switch). =C2=A0If the feature receives no implementation bug= <br> reports for X consecutive days AND no design bug reports for Y<br> consecutive days, then the feature is marked stable and officially<br> becomes part of DMD/Phobos. =C2=A0The X and the Y can be decreased as D&#39= ;s<br> number of users increases over the years. =C2=A0The whole idea is very much= <br> like farming: you are planting seeds. =C2=A0As the plants grow, some of the= m<br> will not survive, others will be destroyed, and some of them will take<br> years to grow. =C2=A0In any case, you harvest the fruits when they are read= y.<br> <br> =C2=A0 Here are good starting values for X and Y:<br> X =3D 90 days<br> Y =3D 180 days<br> </blockquote> <br></div> This is nice, but on the face of it it&#39;s just this: an idea on how othe= r people should do things on their free time. I&#39;d have difficulty convi= ncing people they should work that way. The kind of ideas that I noticed ar= e successful are those that actually carry the work through and serve as go= od examples to follow.<span class=3D""><font color=3D"#888888"><br> <br> <br> Andrei<br> <br> </font></span></blockquote></div><br></div><div class=3D"gmail_extra">*sigh= *</div></div> --001a11c2c1d253d54204f89ffc13--
May 04 2014
prev sibling next sibling parent "Arlen" <_ gmail.com> writes:
On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
Digitalmars-d wrote:
 On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via 
 Digitalmars-d wrote:
 [...]
 Last but not least, currently there are two main ways for new 
 features
 to make it into D/Phobos: you either have to belong to the 
 inner
 circle, or have to represent some corporation that's doing 
 something
 with D.

I'm sorry, but this is patently false. I am neither in the inner circle, nor do I represent any corporation, yet I've had many changes pulled into Phobos (including brand new code). I can't say I'm perfectly happy with the D development process either, but this kind of accusation is bordering on slander, and isn't helping anything. T

There is a lot of truth in what Caligo has said, but I would word that part of it differently. A couple years ago I submitted std.rational, but it didn't go anywhere. About a year later I discovered that someone else had done a similar thing, but it never made it into Phobos either. Of course, it's not because we didn't belong to some "inner circle", but I think it has to do with the fact that D has a very poor development process. The point being, something as simple as a Rational library shouldn't take years for it to become part of Phobos, specially when people are taking the time to do the work. --Arlen
May 04 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, May 05, 2014 at 06:16:34AM +0000, Arlen via Digitalmars-d wrote:
 On Sunday, 4 May 2014 at 22:56:41 UTC, H. S. Teoh via
 Digitalmars-d wrote:
On Sat, May 03, 2014 at 10:48:47PM -0500, Caligo via Digitalmars-d wrote:
[...]
Last but not least, currently there are two main ways for new
features to make it into D/Phobos: you either have to belong to the
inner circle, or have to represent some corporation that's doing
something with D.

I'm sorry, but this is patently false. I am neither in the inner circle, nor do I represent any corporation, yet I've had many changes pulled into Phobos (including brand new code). I can't say I'm perfectly happy with the D development process either, but this kind of accusation is bordering on slander, and isn't helping anything.


 There is a lot of truth in what Caligo has said, but I would word
 that part of it differently.
 
 A couple years ago I submitted std.rational, but it didn't go
 anywhere.  About a year later I discovered that someone else had
 done a similar thing, but it never made it into Phobos either.
 Of course, it's not because we didn't belong to some "inner
 circle", but I think it has to do with the fact that D has a very
 poor development process.  The point being, something as simple
 as a Rational library shouldn't take years for it to become part
 of Phobos, specially when people are taking the time to do the
 work.

This wording is much more acceptable. ;-) While I think accusations of an "elite inner circle" are unfounded (and unfair), I do agree with the sentiment. I think some time ago there was some talk about very old pull requests that have been stuck at the bottom of the queue for months or even years, and nobody was looking at them. I don't know what came out of that talk, though -- apparently not very much. :-( OTOH, I did find that stubbornness and persistence help. If you keep pestering everybody about your new contribution, and keep pushing it even if people seem to ignore/dislike it, keep updating your pull even if it seems nobody cares, eventually somebody will take notice and do something about it. Of course, this is not ideal -- open source projects really should be actively welcoming new contributions, not merely passively accepting them -- but that's the way it is right now, and I'm not sure how to change that. Perhaps stubborn and persistent pestering of the PTBs until they change? Might help, you never know. ;-) T -- Unix was not designed to stop people from doing stupid things, because that would also stop them from doing clever things. -- Doug Gwyn
May 05 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 I think the "needs to support BigInt" argument is not a blocker 
 - we can release std.rational to only support built-in 
 integers, and then adjust things later to expand support while 
 keeping backward compatibility. I do think it's important that 
 BigInt supports appropriate traits to be recognized as an 
 integral-like type.

Bigints support is necessary for usable rationals, but I agree this can't block their introduction in Phobos if the API is good and adaptable to the successive support of bigints.
 If you, Joseph, or both would want to put std.rational again 
 through the review process I think it should get a fair shake. 
 I do agree that a lot of persistence is needed.

Rationals are rather basic (important) things, so a little of persistence is well spent here :-) Bye, bearophile
May 05 2014
prev sibling next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via Digitalmars-d wrote:
 Andrei Alexandrescu:
 
I think the "needs to support BigInt" argument is not a blocker - we
can release std.rational to only support built-in integers, and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt supports
appropriate traits to be recognized as an integral-like type.

Bigints support is necessary for usable rationals, but I agree this can't block their introduction in Phobos if the API is good and adaptable to the successive support of bigints.

Yeah, rationals without bigints will overflow very easily, causing many usability problems in user code.
If you, Joseph, or both would want to put std.rational again through
the review process I think it should get a fair shake. I do agree
that a lot of persistence is needed.

Rationals are rather basic (important) things, so a little of persistence is well spent here :-)

I agree, and support pushing std.rational through the queue. So, please don't give up, we need it get it in somehow. :) T -- I see that you JS got Bach.
May 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 5 May 2014 at 17:22:58 UTC, Marco Leise wrote:
 Am Mon, 5 May 2014 09:39:30 -0700
 schrieb "H. S. Teoh via Digitalmars-d"
 <digitalmars-d puremagic.com>:

 On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via 
 Digitalmars-d wrote:
 Andrei Alexandrescu:
 
I think the "needs to support BigInt" argument is not a 
blocker - we
can release std.rational to only support built-in integers, 
and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt 
supports
appropriate traits to be recognized as an integral-like 
type.

Bigints support is necessary for usable rationals, but I agree this can't block their introduction in Phobos if the API is good and adaptable to the successive support of bigints.

Yeah, rationals without bigints will overflow very easily, causing many usability problems in user code.
If you, Joseph, or both would want to put std.rational 
again through
the review process I think it should get a fair shake. I do 
agree
that a lot of persistence is needed.

Rationals are rather basic (important) things, so a little of persistence is well spent here :-)

I agree, and support pushing std.rational through the queue. So, please don't give up, we need it get it in somehow. :) T

That experimental package idea that was discussed months ago comes to my mind again. Add that thing as exp.rational and have people report bugs or shortcomings to the original author. When it seems to be usable by everyone interested it can move into Phobos proper after the formal review (that includes code style checks, unit tests etc. that mere users don't take as seriously).

And same objections still remain.
May 05 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Monday, 5 May 2014 at 00:44:43 UTC, Caligo via Digitalmars-d 
wrote:
 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via 
 Digitalmars-d <
 digitalmars-d puremagic.com> wrote:
 The on/off switch may be a nice idea in the abstract but is 
 hardly the
 perfect recipe to good language feature development; otherwise 
 everybody
 would be using it, and there's not overwhelming evidence to 
 that. (I do
 know it's been done a few times, such as the (in)famous "new 
 scoping rule
 of the for statement" for C++ which has been introduced as an 
 option by
 VC++.)

Rust has such a thing, #![feature(X,Y,Z)]. So does Haskell, with {-# feature #-}. Even Python has __future__, and many others.

Well, python __future__ it's not exactly that: it's for introducing changes that are impacting the actual codebase... It's some sort of extreme care for not braking anything out there. /Paolo
May 05 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/4/14, 5:38 PM, Caligo via Digitalmars-d wrote:
 On Sun, May 4, 2014 at 12:22 AM, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com <mailto:digitalmars-d puremagic.com>> wrote:
 Here is an idea:  include new features in DMD/Phobos as soon as they
 arrive, and make them part of the official binary release so that the
 average D user can try them out.  Make sure they are marked as unstable,
 and put a on/off switch on them (something like what Rust/Haskell have;
 not a compiler switch).  If the feature receives no implementation bug
 reports for X consecutive days AND no design bug reports for Y
 consecutive days, then the feature is marked stable and officially
 becomes part of DMD/Phobos.  The X and the Y can be decreased as D's
 number of users increases over the years.  The whole idea is very much
 like farming: you are planting seeds.  As the plants grow, some of them
 will not survive, others will be destroyed, and some of them will take
 years to grow.  In any case, you harvest the fruits when they are ready.

   Here are good starting values for X and Y:
 X = 90 days
 Y = 180 days

This is nice, but on the face of it it's just this: an idea on how other people should do things on their free time. I'd have difficulty convincing people they should work that way. The kind of ideas that I noticed are successful are those that actually carry the work through and serve as good examples to follow.

There's imperfect but useful pull requests hanging around for years, extern(Obj-C) for instance, which may be useful as an experimental feature to many users, even if it's not ready for inclusion in the official feature list and support. I suspect it's (experimental) presence would stimulate further contribution towards D on iOS for instance; it may be an enabler for other potential contributors. What about AST macros? It seems to me that this is never going to be explored and there are competing proposals, but I wonder if there's room for experimental implementations that anyone in the community can toy with? UDA's are super-useful, but they're still lacking the thing to really set them off, which is the ability to introduce additional boilerplate code at the site of the attribute. I reckon there's a good chance that creating a proper platform for experimental features would also have an advantage for community building and increase contribution in general. If new contributors can get in, have some fun, and start trying their ideas while also being able to share them with the community for feedback without fear they'll just be shot down and denied after all their work... are they not more likely to actually make a contribution in the first place? Once they've made a single contribution of any sort, are they then more likely to continue making other contributions in the future (having now taken the time to acclimatise themselves with the codebase)? I personally feel the perceived unlikeliness of any experimental contribution being accepted is a massive deterrence to making compiler contributions in the first place by anyone other than the most serious OSS advocates. I have no prior experience with OSS, and it's certainly a factor that's kept me at arms length.
May 05 2014
prev sibling next sibling parent reply "HaraldZealot" <harald_zealot tut.by> writes:
 That said, I really want my destructors, and would be very 
 upset to
 see them go. So... ARC?

Manu, can you direct me what is ARC? This abbreviation is very misgooglly.
May 05 2014
next sibling parent Jacob Carlborg <doob me.com> writes:
On 06/05/14 05:51, HaraldZealot wrote:

 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.

Automatic Reference Counting. Like regular RC but the compiler automatically inserts calls to release/free. -- /Jacob Carlborg
May 05 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available. -- /Jacob Carlborg
May 05 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d 
<digitalmars-d puremagic.com> said:

 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.

Note that iOS has no page swap. Apps just get killed if there isn't enough memory (after being sent a few low memory signals they can react on, clearing caches, etc.). Apps that takes a lot of memory cause other apps in the background to be killed (and later restarted when they come to the foreground). -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 06 2014
parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 14:30, Michel Fortin a crit :
 On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d
 <digitalmars-d puremagic.com> said:

 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.

Note that iOS has no page swap. Apps just get killed if there isn't enough memory (after being sent a few low memory signals they can react on, clearing caches, etc.). Apps that takes a lot of memory cause other apps in the background to be killed (and later restarted when they come to the foreground).

exact same state (or really close) they was before the kill.
May 06 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 08:33, Jacob Carlborg a écrit :
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

For me that working almost every days on mobile devices, that not a good thing to release the memory when there is some new allocations. Took an application with the notion of projects, here is a classical sequence users will do : 1) Launch the application 2) Open a project (heavy load on memory) 3) Do some modifications (adding few allocations) 4) Switch to the an other project Please consider those kind of restrictions (from iOS and/or Android) : - After memory warning if you don't release memory enough fast your application will be killed - If you lock the main thread for too much time your application will be killed Now how we implement transition between step 3 and 4 : - Saving the current project - Release manually all his memory and other resources can't be easily shared - At this point we are almost sure that memory and resources footprints are to the minimum amount (except leaks) - We can load the new project and allocate memory safely Now with the GC, we tend to saturate the memory of the device which will cause memory warning with a chance of crash of the application. It's slower than manual release cause of number of scans,... Even on devices like PC, GC's when they don't share memory pool between applications tends to reduce resources for other applications (bad for multitask OS). So with a GC for this kind of steps we need shutdown the GC, and force the collect where it's interesting. IMO, manual management is not the best, GC neither, but with good tools memory issues can be easy to find and fix. Apple have great tools for that on iOS, their memory analyser find leaks without any performance slow down just like if it's supported by the hardware!!!
May 06 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 12:58, Manu via Digitalmars-d a écrit :
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. The trouble is, GC runs more frequently as the amount of available free memory decreases, and trace&collect takes a long time, long enough that realtime software typically can't tolerate the pause. If the pause is super rare (minutes/hours apart), maybe it's acceptable, but if it's closer to every call to alloc (ie, when there is little/no free memory), then you realise it's fundamentally incompatible. Then consider that this is typically the default operating state of embedded systems; to have no free memory at runtime, and therefore D is effectively incompatible with a subset of computers... specifically, games consoles; a gigantic industry, and one of the few that still absolutely relies on native code where D is particularly compelling. If people want to call D a native systems language, this seems like a serious conflict with that idea. A major subset of the language is incompatible with realtime or embedded use (surely among the most compelling use cases remaining for native code these days!). Usually the argument is that I represent a niche, and therefore I should accept the restrictions of my environment, and revert to full manual memory management for my work, while letting everyone else have their fun. That is a broken argument, not only because it leaves very little motivation to leave C++, but also because regardless of whether I adopt such restrictions, library authors won't, and we don't live in the 1980's where I can write a few lines of self-contained assembly and call it a program. To say I can't depend on libraries is practically absurd. Games are gigantic software projects, and depend on heaps of libraries. The GC effectively eliminates libraries from embedded/realtime users, which I see as an absolute deal-breaker alone, irrespective of a further bunch more restrictions it places on that subset of users within their own code :/ I'll be happy with any solution that works, really. But as I see it, reference counting is the only garbage collection technology I know which will reliably clean up eagerly as things fall out of scope (also addressing this issue with destructors as bonus), and as such, it seems the logical solution. ARC is a demonstrable success in Obj-C, and works well in realtime and embedded systems. D has type system improvements which would theoretically allow for significant improvements over Obj-C.

D seems to mainly interests C++ developers that generally aren't interested by dynamically types language and based on a GC. But we always have the possibility to link C++ to a higher language for some part of applications. IMO system languages have to continue working with manual memory management.
May 06 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 13:39, Paulo Pinto a écrit :
 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and
 other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/

Is that VM full compatible with Java specifications and standard frameworks?
 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 --
 Paulo

Android works well, I love my nexus, it proves to me that it's possible to create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.
May 06 2014
next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 07/05/2014 05:58, Manu via Digitalmars-d a écrit :
 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Le 06/05/2014 13:39, Paulo Pinto a écrit :

create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.

Yeah, these are excellent points that I've tried and perhaps failed to articulate properly in the past. The amount of 'wasted' resources required to maintain a significant surplus on Android devices is ridiculous, and that's why I separated phones from other embedded systems. While phones can, and do, do this, other embedded systems don't. To say we need to leave half of the xbox/ps4 resources idle to soak up intermittency is completely unworkable. It's always important to remember too that the embedded market is by far the largest software market in the world.

4Mo of RAM. I sill was at school at this date, and it was hard to deal with so few memory, ROM and video memory was very limited. The project I worked on was only limited by memory cause of use of precomputed pictures (it was an old adventure game). On this console, game vendors have to pay for the ROM and backup memories, so we put in place some compressions and data packing logics. From this time I always take care of avoiding unnecessary absurd allocations.
May 07 2014
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:
 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...

by a very successful embedded RTOS, and compared to competition, runs silky smooth. Indeed iOS makes it a specific design goal that it should always feel silky smooth, never stuttery, they consider it a point of quality, and I completely agree. I don't know what other horse to back? ...

The problem when comparing iOS with Android, is that we aren't comparing ARC with GC. We are comparing a full OS, which we don't know how much ARC is actually used versus standard malloc/new with another OS, which has a so-and-so VM implementation, used mostly software rendering until version 4.1, and care for low end devices was only done in 4.4. If we add Windows Phone to the mix, then we have a .NET stack like Android (WP7 - GC/JIT) or in its sucessor (WP8) native code generation for .NET (GC) with a COM model for OS APIs (ARC). Both versions of Windows Phone run smoother that many Android phones, even the WP7 ones. Not saying you are not right, just that we need to look at the whole stack when comparing mobile OS, not just GC vs ARC. -- Paulo
May 09 2014
next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/05/2014 08:53, Paulo Pinto a écrit :
 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:
 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...

by a very successful embedded RTOS, and compared to competition, runs silky smooth. Indeed iOS makes it a specific design goal that it should always feel silky smooth, never stuttery, they consider it a point of quality, and I completely agree. I don't know what other horse to back? ...

The problem when comparing iOS with Android, is that we aren't comparing ARC with GC. We are comparing a full OS, which we don't know how much ARC is actually used versus standard malloc/new with another OS, which has a so-and-so VM implementation, used mostly software rendering until version 4.1, and care for low end devices was only done in 4.4. If we add Windows Phone to the mix, then we have a .NET stack like Android (WP7 - GC/JIT) or in its sucessor (WP8) native code generation for .NET (GC) with a COM model for OS APIs (ARC). Both versions of Windows Phone run smoother that many Android phones, even the WP7 ones. Not saying you are not right, just that we need to look at the whole stack when comparing mobile OS, not just GC vs ARC. -- Paulo

http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/ Just like Android phones, the battery is huge : 3400mAh It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz Only RAM seems reasonable : 1Go And it's maybe easier for Microsoft to do a GC with good performances cause they control everything on the OS. There is certainly some specifics memory management in the kernel related to the GC. When Google "just" put small pieces together based on a linux kernel. Yesterday I played with my nexus 5 at a game in augmented reality, it took 2hr 20 minutes to completely drain the battery (2300 mAh). Sadly this game doesn't exist on iOS but IMO an iPhone would do the same with a smaller battery (1440 mAh for iPhone 5s).
May 10 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:
 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 I don't know well WP8 models, but this one must run smoothly :
 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances cause
 they control everything on the OS. There is certainly some specifics memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.

They don't GC. MS are behind their C++/CX now, which introduced an ARC pointer type '^'.

.NET Applications on Windows Phone 8 make use of both (GC and COM RC). -- Paulo
May 10 2014
parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/05/2014 17:58, Manu via Digitalmars-d a écrit :
 On 11 May 2014 01:44, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:
 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:
 I don't know well WP8 models, but this one must run smoothly :

 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances
 cause
 they control everything on the OS. There is certainly some specifics
 memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.

They don't GC. MS are behind their C++/CX now, which introduced an ARC pointer type '^'.

.NET Applications on Windows Phone 8 make use of both (GC and COM RC).

I don't know any gamedev's who use .NET on WP8. C++/CX was a good step forward, and addressed one of the primary inhibitors to developer adoption of WP7. I don't know a single developer who ever released a WP7 app. WP7 was a ghost town compared to iOS and Android in terms of developer support. Lots of people I know are happily supporting WP8 now thanks to C++/CX.

WP7 force all gamedev using c++ to rewrite everything, it's completely infeasible. I heard Microsoft propose to paid some studios to port their products, but a lot refuse this offer.
May 10 2014
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 10.05.2014 15:28, schrieb Manu via Digitalmars-d:
 On 10 May 2014 16:53, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:
 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...

The only option I know that works is Obj-C's solution, as demonstrated by a very successful embedded RTOS, and compared to competition, runs silky smooth. Indeed iOS makes it a specific design goal that it should always feel silky smooth, never stuttery, they consider it a point of quality, and I completely agree. I don't know what other horse to back? ...

The problem when comparing iOS with Android, is that we aren't comparing ARC with GC. We are comparing a full OS, which we don't know how much ARC is actually used versus standard malloc/new with another OS, which has a so-and-so VM implementation, used mostly software rendering until version 4.1, and care for low end devices was only done in 4.4. If we add Windows Phone to the mix, then we have a .NET stack like Android (WP7 - GC/JIT) or in its sucessor (WP8) native code generation for .NET (GC) with a COM model for OS APIs (ARC). Both versions of Windows Phone run smoother that many Android phones, even the WP7 ones. Not saying you are not right, just that we need to look at the whole stack when comparing mobile OS, not just GC vs ARC.

Yes, granted. I shouldn't make that sort of comparison. What I am really trying to do with such comments, is to insist that it is used as a foundation for a very successful RTOS. To say "it's no good. ARC can't work. it can never be used as a foundation for a language. but, cycles!" and whatever, is kind of ignoring the evidence. It's interesting too that you bring Windows phone into the mix, because all the WP8 developers I know use microsoft's C++/CX, which has the new '^' ARC pointer type (as used intensively by the OS). Why did MS abandon GC? It looks like a similar sort of back-peddling from their commitment to GC as did Apple, when they started making a serious play for the embedded market.

They did not. A Windows Phone 8 application, if coded in .NET still uses a GC for the .NET world. http://blogs.msdn.com/b/abhinaba/archive/2012/11/08/windows-phone-8-evolution-of-the-runtime-and-application-compatibility.aspx ARC is used in C++/CX due to the COM nature of WinRT. After all the Windows Runtime is nothing more than COM with IInspectable as new derived interface from IUnknown. Which was actually the original design of .NET, COM+ VOS, before they decided to go MSIL. http://blogs.msdn.com/cfs-file.ashx/__key/communityserver-components-postattachments/00-10-32-72-38/Ext_2D00_VOS.pdf If anything, the native side has won the current internal politics war and they are steering the boat back to ahead of time compilation.
 C# implies massive GC related issues for realtime and embedded
 software. I have lots of colleagues who rant about it regularly, and
 my experience from 2 weekends back that I shared earlier in this
 thread is a scary slap in the face for me.

Where those experiences with Unity/Mono or proper .NET? Unity still uses Mono 2.6 from 2009, when Mono was under SuSE control. Entity to whom Unity has an agreement. http://unity3d.com/unity/faq#section-1828 ==> What version of Mono is used for 4.x? Can I use all .NET features? A new generational garbage collector was introduced in Mono with version 2.8 (2010) and improved in consequent releases. http://www.mono-project.com/Release_Notes_Mono_2.8 It became the default GC in the latest version, currently 3.2. -- Paulo
May 10 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei
May 10 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion?

Of course, and I've shown it numerous times, only for it to be blissfully ignored. Probably the most recent and influential work on it is http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf of which FIRST sentence is "Despite some clear advantages and recent advances, reference counting remains a poor cousin to high-performance tracing garbage collectors." The paper reviews previous work on the matter, ALL of which shows that RC is behind GC in speed, and presents (for the first time in 2013) new work that brings about parity of RC with GC.
 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.

As competent engineers we can't afford to be ignorant of the existing research and practice showing a much more complex tradeoff space than a black/white distinction. We also can't do the capital mistake of engineering - the beauty of the part at the expense of the beauty of the whole. Andrei
May 10 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/10/14, 10:03 AM, Manu via Digitalmars-d wrote:
 On 11 May 2014 02:15, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion?

Of course, and I've shown it numerous times, only for it to be blissfully ignored. Probably the most recent and influential work on it is http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf of which FIRST sentence is "Despite some clear advantages and recent advances, reference counting remains a poor cousin to high-performance tracing garbage collectors." The paper reviews previous work on the matter, ALL of which shows that RC is behind GC in speed, and presents (for the first time in 2013) new work that brings about parity of RC with GC.

I saw that paper before.

Then the right thing is to integrate it within one's worldview. It is difficult to reconcile your assertions and questions with the claim that you have read and internalized existing work on the topic.
 And I thought, if anything, it supported my
 argument in the context of D.

It doesn't seem that way to me. Your arguments so far seem to come from a staunch niche position that ignores factual realities and has no concern for the larger environment. Some of such arguments are factually incorrect, and just the minimal amends and qualifications are not sufficient.
 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?

I believe it is.
 Has anyone compared D's GC against 'the fastest tracing collectors'?
 What is the difference? Does D fall within 10%?

In all likelihood, no.
 If not, then chances are, according to this paper, RC would be
 superior to the existing GC in terms of performance,

I disagree.
 as well as
 reliability/predictability/functionality (destructors that work, not
 incompatible with embedded software).

I partially agree, but this is getting rather difficult to assess. Let's say I'd agree it's a valid viewpoint.
 If an awesome RC is implementable in D, but an awesome GC is not, then
 I can't see that even any performance argument can remain in the
 debate. They are talking specifically about a GC-backed-RC too, to
 handle cycles, so their implementation does not come at any cost to
 safety or added complexity for users, and has the key advantage that
 people like me may have the option to disable the cycle collector and
 take manual responsibility if desired.

 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.

As competent engineers we can't afford to be ignorant of the existing research and practice showing a much more complex tradeoff space than a black/white distinction. We also can't do the capital mistake of engineering - the beauty of the part at the expense of the beauty of the whole.

There is a black/white distinction though. It you can't make the freezing go away, it is _incompatible_ with certain classes of software. I don't care what the performance difference is when this is the case. If one option is incompatible and the other is not, then there is no tradeoff.

Of course it would be nice to have better support for reference counting. My understanding is that you advocate that reference counting would be compulsive, which would preclude the GC option. Andrei
May 10 2014
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:
 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?

I believe it is.

While it might be possible to implement a good GC in D it would require major changes in the language and its librariers. In my opinion it would be way more work to implement a propper GC than to implement ARC. Every state of the art GC requires percise knowdelge of _all_ pointers. And thats exactly what we currently don't have in D. Just to list a few issues with implementing this in D: 1) Calling C-Functions would require some kind of pinning API for the GC. But functions with C-calling convections are used ecessivly in druntime. So a lot of uneccessary pinning would happen within d-runtime just because of all the c-function hacks around the module system. 2) Most modern GC algorithms require write-barriers for pointers which are on the heap. This is a major problem for D, because structs can be both on the stack and on the heap. So if you call a method of a struct, the compiler can not know if the write barriers are actually neccessary (for heap based structs) or unneccssary (for stack based structs, e.g. inside classes). 3) Generating percise information of the location of pointers on the stack is going to be a major effort, because of structs, unions, exception handling and Ds ability to move struct instances freely around when dealing with exceptions. 4) As soon as you want to be fully percise you can not halt threads in random points, you will need to generate GC points on which the execution can be safely halted. This is going to require major effort again if it should be done efficiently. 5) As Ds strings are immutable(char)[] they can not be allocated on a thread local pool, because they must be freely shareable between threads, and there is currently no way to detect via the type system whean a immutable(char)[] is shared with another thread. So we would need to design a system were the sharing of memory blocks would be detected, so we could properly implement thread local GC pools. The same issue applies to __gshared. Honenstly, D doesn't currently have any GC support at all. The only GC support we have is that the GC knows all threads and can halt them. And D knows which function to call, to destroy class objects. So basically we have the same level of GC support as C or C++. You could just use the Boem GC, plug it into C or C++ and have a GC that is as good as the D one. So having a good GC in 5 years or having ARC in 1 year isn't a decision that is that hard for me. Kind Regards Benjamin Thaut
May 11 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 1:22 AM, Benjamin Thaut wrote:
 Honenstly, D doesn't currently have any GC support at all. The only GC support
 we have is that the GC knows all threads and can halt them. And D knows which
 function to call, to destroy class objects. So basically we have the same level
 of GC support as C or C++. You could just use the Boem GC, plug it into C or
C++
 and have a GC that is as good as the D one.

This is not quite correct. The Boehm GC knows nothing about the interiors of structs. The D one does (or at least has the capability of doing so using RTinfo). This means that the D collector can be 'mostly' precise, the imprecision would be for stack data. The Boehm collector cannot move objects around, the D one can.
May 11 2014
parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 11.05.2014 10:53, schrieb Walter Bright:
 On 5/11/2014 1:22 AM, Benjamin Thaut wrote:

 This is not quite correct. The Boehm GC knows nothing about the
 interiors of structs.

 The D one does (or at least has the capability of doing so using
 RTinfo). This means that the D collector can be 'mostly' precise, the
 imprecision would be for stack data.

Mostly percise doesn't help. Its either fully percise or beeing stuck with a impercise mark & sweep. Also D's GC doesn't use RTInfo currently. Additionally Rainer Schuetzes implementation showed that using RTInfo with the current GC makes the collection actually slower at the price of beeing 'mostly' percise. So I argue that the level of GC support we have is still at the level of C/C++.
 The Boehm collector cannot move objects around, the D one can.

Oh it can? Really? I would love to see how well interfacing with any C library works, if the D garbage collector actually does that. I bet GTKd would break the first second this is implemented. Or any other C library that exchanges data with D for that matter. The D garbage collector can not simply move around objects because we don't support pinning. Also I'm talking about what the D garbage collector currently actually does, and not what it could do. If we actually would implement arc, it would most likely take less time then a full blown proper gc and we would end up with better performance than we currently have. Beating the 300% slowdown which the current GC imposes is not that hard. If we however keep arguing what a GC could do, we will be stuck with the impercise mark & sweep forever. (or at least the next 5 years)
May 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 2:48 AM, Benjamin Thaut wrote:
 Mostly percise doesn't help. Its either fully percise or beeing stuck with a
 impercise mark & sweep.

This is not correct. It helps because most of the false pointers will be in the heap, and the heap will be accurately scanned, nearly eliminating false references to garbage.
 Also D's GC doesn't use RTInfo currently.

Yes, but that is not an issue with the language design.
 Additionally Rainer Schuetzes implementation showed that using RTInfo with the
current GC
 makes the collection actually slower at the price of beeing 'mostly' percise.

As I recall, his implementation showed dramatic improvements in memory consumption. (All GC schemes are tradeoffs.)
 The Boehm collector cannot move objects around, the D one can.


Yes. D, for example, requires that objects not be self-referential for this reason.
 I would love to see how well interfacing with any C library
 works, if the D garbage collector actually does that.

The current D GC does not do that, but it can. It will work fine with C libraries as long as a pointer to the data is retained on the stack, as it naturally will be by passing it as a stack parameter to a C function. Roots found on the stack would automatically pin the object. I know this works because I implemented it for a Java VM back in the 90's. Someone wrote a paper about a similar scheme calling it a "mostly copying collector".
 Also I'm talking about what the D garbage collector currently actually does,
and
 not what it could do.

I think it would be more useful to talk about what it could do without language changes.
 If we actually would implement arc, it would most likely
 take less time then a full blown proper gc and we would end up with better
 performance than we currently have. Beating the 300% slowdown which the current
 GC imposes is not that hard.

 If we however keep arguing what a GC could do, we will be stuck with the
 impercise mark & sweep forever. (or at least the next 5 years)

I believe the performance claims of pervasive ARC are unjustifiably optimistic and based on unproven technology that has never been implemented anywhere.
May 11 2014
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 11.05.2014 22:33, Walter Bright wrote:
 The Boehm collector cannot move objects around, the D one can.


Yes. D, for example, requires that objects not be self-referential for this reason.

I don't think the GC would have problems with fixing up internal pointers to the object itself. self-referential is prohibited to allow moving structures by memcpy, e.g. as return value.
May 11 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 1:45 PM, Rainer Schuetze wrote:
 On 11.05.2014 22:33, Walter Bright wrote:
 The Boehm collector cannot move objects around, the D one can.


Yes. D, for example, requires that objects not be self-referential for this reason.

I don't think the GC would have problems with fixing up internal pointers to the object itself. self-referential is prohibited to allow moving structures by memcpy, e.g. as return value.

The earlier D design did not have postblit, and so there was no fixing up of internal pointers.
May 11 2014
prev sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 12.05.2014 06:57, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Sunday, 11 May 2014 at 20:45:08 UTC, Rainer Schuetze wrote:
 On 11.05.2014 22:33, Walter Bright wrote:
 The Boehm collector cannot move objects around, the D one can.


Yes. D, for example, requires that objects not be self-referential for this reason.

I don't think the GC would have problems with fixing up internal pointers to the object itself. self-referential is prohibited to allow moving structures by memcpy, e.g. as return value.

Does this mean that you cannot safely implement something as basic as a circular linked list?

Depends on your implementation, but I would not expect that the nodes of your linked list could be subject to being moved (mostly happens on the stack AFAICT), as pointers to the node from other nodes would become invalid, too.
May 12 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:
 It doesn't matter where the false pointers are. The largest issue with false
 pointers is not how many false pointers there are. It only matters how large
the
 block is that it "points" at. The larger your blocks get, the more likely they
 are "pointed" at by the stack. On 32-bit systems, allocate 1/256th of your
 memory space (i.e. 16.5MB), and the likelihood of random data on the stack
 pointing at it is roughly 1/256. This problem is just about eliminated with
 64-bit pointers.

Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's work best when the size of the objects being allocated is very small relative to the size of the heap space. Fortunately, it's a mathematical inevitability that large allocations relative to the GC size are rare, and so it isn't much of a pain to handle them manually.
 And in fact, even if it's forbidden, "requires" is too strong a word -- there
is
 no static or runtime prevention of this.

It's still forbidden. Andrei wrote a template that will verify this at runtime, but I don't recall its name.
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:32 PM, Steven Schveighoffer wrote:
 It's still forbidden. Andrei wrote a template that will verify this at
 runtime, but I don't recall its name.

Can you cite the spec where it says it's forbidden? Forgotten templates are not a convincing argument. Regardless, Java can use a moving GC, and allows self references. The idea that self references prevent a moving GC is simply false. If you think about it a bit, you will understand why.

I see this is not specified in the documentation. Not sure what happened here, but I'll have to think about it.
May 12 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 1:22 AM, Benjamin Thaut wrote:
 2) Most modern GC algorithms require write-barriers for pointers which are on
 the heap. This is a major problem for D, because structs can be both on the
 stack and on the heap. So if you call a method of a struct, the compiler can
not
 know if the write barriers are actually neccessary (for heap based structs) or
 unneccssary (for stack based structs, e.g. inside classes).

I know about write barriers. The thing about D is, far fewer objects are allocated on the GC heap than in languages like Java, which don't have stack allocation at all. The write barrier would have to be there for every write through a pointer. This is justifiable where every pointer points to the GC heap, but I really do not believe it is justifiable for D, where only a minority does. D could not be performance competitive with C++ if write barriers were added. D also cannot be performance competitive with C++ if pervasive ARC is used and memory safety is retained. Rust is attempting to solve this problem by using 'borrowed' pointers, but this is unproven technology, see my reply to Manu about it.
May 11 2014
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 11.05.2014 23:43, schrieb sclytrack:
 On Sunday, 11 May 2014 at 11:42:37 UTC, ponce wrote:
 On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:
 D also cannot be performance competitive with C++ if pervasive ARC is
 used and memory safety is retained. Rust is attempting to solve this
 problem by using 'borrowed' pointers, but this is unproven
 technology, see my reply to Manu about it.

I work in a C++ shop and as I see it, resource management is becoming a solved problem: - resource owners holds a std::unique_ptr<T> on them. Resource release is taken care by C++ destructors normally. That means to be exception-safe, each resource type must better have its class. - resource users eventually "borrow" the resource by taking a raw pointer out of the unique pointer. What Rust would do with lifetimes here is ensure that the resource is still there since move semantics seems pervasive in this language. In C++ we ensure the resource holder outlive the users. - std::shared_ptr is not needed with such constraints. This means no cycles in the object graph. TBH I have yet to find a dependency scheme that can't be done that way. When I use D I can't help but think that releasing resources feels more manual and error-prone ("oops that resource should have been a struct not a class" and such traps). I do not have huge concerns about D GC, but I would be glad to have more support for owned pointers (ie. Unique!T in Phobos or better). I have no idea how to make it safe ie. ensure the resource outlive its users.

I like this owner/unique, borrow thing. is managed (currently reference counted) ~ is owner & is borrow fn example3() -> int { let mut x = ~X {f: 3}; let y = &x.f; x = ~X {f: 4}; // Error reported here. *y } According to Debian Rust is still too experimental to be packaged. http://web.mit.edu/rust-lang_v0.8/doc/tutorial-borrowed-ptr.html servo is written in Rust. https://github.com/mozilla/servo There is very little use of " ", it's mostly "&" and "~". Heck I didn't find any while casually browsing the code. It's like they are not using it at all. I don't know Rust, it is the first day I look (read documentation) at it.

Have you searched for RC<>() and GC<>() as well? The is gone from Rust and replaced by library types. -- Paulo
May 11 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/11/14, 2:49 PM, ponce wrote:
 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". Heck I
 didn't find any   while casually browsing the code. It's like they are
 not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.

The right tally here would include bald pointers as well. Also, I'm quite surprised by the confidence of this assertion. -- Andrei
May 11 2014
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 11.05.2014 10:22, Benjamin Thaut wrote:
 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:
 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?

I believe it is.

While it might be possible to implement a good GC in D it would require major changes in the language and its librariers. In my opinion it would be way more work to implement a propper GC than to implement ARC. Every state of the art GC requires percise knowdelge of _all_ pointers. And thats exactly what we currently don't have in D.

I think most garbage collectors can work with a number of false pointers. The referenced memory area has to be treated as pinned and cannot be moved. Limiting the false pointers to stack and registers seems like a compromise, though most of the stack could even be annotated. Code for this does already exist in the debug info generation, though I suspect stack tracing could be unreliable. Here's my current stance on the GC discussions: I agree that the current GC is pretty lame, even if it were precise. "Stop-the-World" with complete tracing does not work for any interactive application that uses more than a few hundred MB of garbage collected memory (with or without soft-realtime requirements). Other applications with larger allocation requirements are easily dominated by collection time. Proposing to use manual memory management instead is admitting failure to me. For a reasonable GC I currently see 2 possible directions: 1. Use a scheme that takes a snapshot of the heap, stack and registers at the moment of collection and do the actual collection in another thread/process while the application can continue to run. This is the way Leandro Lucarellas concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork" that doesn't exist on every OS/architecture. A manual copy of the memory won't scale to very large memory, though it might be compressed to possible pointers. Worst case it will need twice as much memory as the current heap. It would be very interesting how far we can push this model on the supported platforms. 2. Change the compiler to emit (library defined) write barriers for modifications of (possible) pointers. This will allow to experiment with more sophisticated GC algorithms (if the write barrier is written in D, we might also need pointers without barriers to implement it). I know Walter is against this, and I am also not sure if this adds acceptable overhead, but we don't have proof of the opposite, too. As we all know, the usual eager reference counting with atomic operations is not memory-safe, so my current favorite is "concurrent buffered reference counting" (see chapter 18.2/3 "The garbage collection handbook" by Richard Jones et al): reference count modifications are not performed directly by the write barrier, but it just logs the operation into a thread local buffer. This is then processed by a collector thread which also detects cycles (only on candidates which had their reference count decreased during the last cycle). Except for very large reference chains this scales with the number of executed pointer modifications and not with the heap size.
May 11 2014
next sibling parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 11.05.2014 21:16, John Colvin wrote:
 On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:
 For a reasonable GC I currently see 2 possible directions:
 Coventry
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the
 memory won't scale to very large memory, though it might be compressed
 to possible pointers. Worst case it will need twice as much memory as
 the current heap.

a) Can't we do our own specialised alternative, instead of doing something as heavyweight as fork?

If we don't want to make a copy with memcpy, we have to rely on the OS paging. I don't know linux too well, but on Windows the idea would be to use VirtualProtect with PAGE_WRITECOPY. This could allow control on page size granularity over what would be mapped to another process.
 b) Surely that worst case is a very unlikely case, given a precise,
 concurrent garbage collector.

Preciseness has little impact because copy-on-write protection works on page level. You will also pay for the page fault if you only write non-pointer data into the page. It would help to sort memory chunks without pointers (they have attribute NO_SCAN) into separate pages that don't need page protection.
 c) Does this actually help in the 99% cpu, 99% memory, 16ms per frame
 setting that Manu talks about?

You'll need more memory, but given enough idle-time for a lower priority collector thread, a higher priority thread could run almost undisturbed. IIRC this meant about 40ms, as all threads need to be stopped for the time the fork is running.
 It would be very interesting how far we can push this model on the
 supported platforms.

 2. Change the compiler to emit (library defined) write barriers for
 modifications of (possible) pointers. This will allow to experiment
 with more sophisticated GC algorithms (if the write barrier is written
 in D, we might also need pointers without barriers to implement it). I
 know Walter is against this, and I am also not sure if this adds
 acceptable overhead, but we don't have proof of the opposite, too.

Adding optional write-barriers would be great. It would allow us to do real experiments and stop the discussion being so hypothetical. It doesn't do any harm to add the capability, surely?

Sure, but it is also not easy ;-) I guess we would need some inspiration from existing implementation in other languages how to do it efficiently.
May 11 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 11:18 AM, Rainer Schuetze wrote:
 1. Use a scheme that takes a snapshot of the heap, stack and registers at the
 moment of collection and do the actual collection in another thread/process
 while the application can continue to run. This is the way Leandro Lucarellas
 concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies
 on "fork" that doesn't exist on every OS/architecture. A manual copy of the
 memory won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the current
 heap.

 It would be very interesting how far we can push this model on the supported
 platforms.

This is worth doing even if it is only possible on a subset of platforms. D should not be constrained to the worst of all the platforms. My recollection is that Leandro ran into some serious reentrancy problems in the Linux API. Was this ever resolved?
May 11 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11-May-2014 23:16, John Colvin пишет:
 On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:
 On 11.05.2014 10:22, Benjamin Thaut wrote:
 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:
 The next sentence goes on to list the advantages of RC (issues we have
 wrestled with, like destructors), and then goes on to say the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?

I believe it is.

While it might be possible to implement a good GC in D it would require major changes in the language and its librariers. In my opinion it would be way more work to implement a propper GC than to implement ARC. Every state of the art GC requires percise knowdelge of _all_ pointers. And thats exactly what we currently don't have in D.

I think most garbage collectors can work with a number of false pointers. The referenced memory area has to be treated as pinned and cannot be moved. Limiting the false pointers to stack and registers seems like a compromise, though most of the stack could even be annotated. Code for this does already exist in the debug info generation, though I suspect stack tracing could be unreliable. Here's my current stance on the GC discussions: I agree that the current GC is pretty lame, even if it were precise. "Stop-the-World" with complete tracing does not work for any interactive application that uses more than a few hundred MB of garbage collected memory (with or without soft-realtime requirements). Other applications with larger allocation requirements are easily dominated by collection time. Proposing to use manual memory management instead is admitting failure to me. For a reasonable GC I currently see 2 possible directions: Coventry 1. Use a scheme that takes a snapshot of the heap, stack and registers at the moment of collection and do the actual collection in another thread/process while the application can continue to run. This is the way Leandro Lucarellas concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork" that doesn't exist on every OS/architecture. A manual copy of the memory won't scale to very large memory, though it might be compressed to possible pointers. Worst case it will need twice as much memory as the current heap.

a) Can't we do our own specialised alternative, instead of doing something as heavyweight as fork?

Provided heap was allocated as one big mmap-ed region (or File mapping in Win32) it should be straightforward to remap it as Copy-on-Write. This would though reduce virtual memory available in half (if not more provided we'd want to leave some space for C's malloc). Would be cool to find a way to the allocation of thread stacks... else we'd have to copy them while threads are frozen. -- Dmitry Olshansky
May 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 3:42 PM, Dmitry Olshansky wrote:
 Provided heap was allocated as one big mmap-ed region (or File mapping in
Win32)
 it should be straightforward to remap it as Copy-on-Write. This would though
 reduce virtual memory available in half (if not more provided we'd want to
leave
 some space for C's malloc).

This is not an issue for 64 bit machines. I don't think it would be a big problem if we have an improved GC for 64 bit machines only. 64 bits have other advantages for a GC - very few false pointers.
May 11 2014
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On 05/11/2014 08:18 PM, Rainer Schuetze wrote:
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.

There is a problem with this scheme, copy-on-write is extremely expensive when a mutation happens. That's one page fault (context switch) + copying a whole page + mapping the new page. It's much worse with huge pages (2MB page size).
May 12 2014
parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 13.05.2014 00:15, Martin Nowak wrote:
 On 05/11/2014 08:18 PM, Rainer Schuetze wrote:
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.

There is a problem with this scheme, copy-on-write is extremely expensive when a mutation happens. That's one page fault (context switch) + copying a whole page + mapping the new page.

I agree that this might be critical, but it is a one time cost per page. It seems unrealistic to do this with user mode exceptions, but the OS should have this optimized pretty well.
 It's much worse
 with huge pages (2MB page size).

How common are huge pages nowadays?
May 12 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Steven Schveighoffer"  wrote in message 
news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...

 If a thread is doing only reads, or is  only touching non-Scanned memory, 
 it continues.

How long do threads usually go without touching the stack?
May 13 2014
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:
 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where references to
 objects of a particular heap can be placed), exempting certain threads
 from GC completely, ...

This comes up from time to time, but to me it is very blurry how this can work in reality. Considering how "shared" is supposed to be used to be useful (do some locking, then cast away "shared") there is no guarantee by the language that any object is actually thread local (no references from other threads). Working with immutable (e.g. strings) is shared by design.
May 12 2014
next sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 13.05.2014 13:09, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:
 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:
 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where references to
 objects of a particular heap can be placed), exempting certain threads
 from GC completely, ...

This comes up from time to time, but to me it is very blurry how this can work in reality. Considering how "shared" is supposed to be used to be useful (do some locking, then cast away "shared") there is no guarantee by the language that any object is actually thread local (no references from other threads). Working with immutable (e.g. strings) is shared by design.

Yes, but only a part of the data is shared. I suspect the majority of the data in typical programs will be thread-local. If you use a message passing model, you can improve that even further (though it requires a way to move an object to another thread's heap). This way, you can - in the best case - avoid the shared heap completely.

Possible, but this is with a language without shared data (including immutable), not D. Safely transferring a large object tree from one thread to another will be very expensive. If you try to optimize that, the D compiler won't help you to guarantee memory safety. Actually I don't have much experience with "shared" (I usually use __gshared if I need a shared global). Maybe I understand the idea better if you show what happens with this code when run with thread local GC: class C { C next; } shared(C) list; C newC() { return new C; // allocated on the local heap? } void prependToList(C c) { synchronized(list) { C lst = cast(C) list; c.next = lst; // anything special happening here? list = cast(shared(C)) c; // and here? } } void callMeFromMultipleThreads() { prependToList(newC()); }
May 13 2014
parent Rainer Schuetze <r.sagitario gmx.de> writes:
On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:
 class C { C next; }

 shared(C) list;

 C newC()
 {
     return new C; // allocated on the local heap?

Yes, because it doesn't say "new shared(C)".
 }

 void prependToList(C c)
 {
     synchronized(list)
     {
         C lst = cast(C) list;
         c.next = lst;  // anything special happening here?
         list = cast(shared(C)) c;  // and here?

The casts are a problem. You're basically lying to the compiler, so this cannot work. Instead, it should be something like: auto newC() { return new isolated(C); } void prependToList(isolated(C) c) { synchronized(list) { // transfer to shared heap // afterwards, c is no longer usable ("consumed") shared(C) tmp = c.makeShared(); tmp.next = list; list = tmp; } }

Sorry for the late reply. I currently don't know of a feasible way to replace casting away "shared" if you want to do more than just "tmp.next = list;", for example call "bool check(C c);" on tmp or list? I can't implement this with a shared object, because any operations on "shared" tend to not exist (ok, maybe I could for this very simple example, so consider C contains a container to operate on). Also, the object is already locked, I should not have to do it again in a function if it only accepts a shared C. How can I prepend something to the list that is not isolated? I guess I cannot, as it might create references to same object both shared and unshared. My gut feeling tells me that this ownership handling through the type system might work, but it is very restrictive and does not really contribute to the first goal of D listed on the front page: "convenience".
 This uses the "isolated" concept suggested by deadalnix here:
 http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org?page=1

     }
 }

 void callMeFromMultipleThreads()
 {
     prependToList(newC());
 }


May 16 2014
prev sibling parent reply Rainer Schuetze <r.sagitario gmx.de> writes:
On 13.05.2014 14:20, Wyatt wrote:
 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:
 This comes up from time to time, but to me it is very blurry how this
 can work in reality.

there concerns I've missed that make that not applicable?

I just read the first chapters, and according to that, existing local gcs needs write barriers, so we are back to my second proposal. The implementation in the paper even adds read barriers.
 Considering how "shared" is supposed to be used to be useful (do some
 locking, then cast away "shared") there is no guarantee by the
 language that any object is actually thread local (no references from
 other threads). Working with immutable (e.g. strings) is shared by
 design.

I'm not seeing much in the documentation, but from what I can tell (per the FAQ), shared in D just guarantees it's on the global heap?

To me, shared is a type modifier that doesn't implicitely convert to anything else without casting. If a global/static variable is shared, it is placed into the data segment of the executable image, not TLS. The heap is not involved here.
 -Wyatt

 [0]
 https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf

May 13 2014
parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/14/2014 06:41 PM, Wyatt wrote:
 To me, shared is a type modifier that doesn't implicitely convert to
 anything else without casting.

Interesting, maybe this should be clarified better. Having skimmed back over chapter 13 of TDPL, my understanding of its semantics are that it only really enforces atomicity and execution order. Also, this bit from near the beginning of 13.12 states: "For all numeric types and function pointers, shared-qualified values are convertible implicitly to and from unqualified values." That sounds kind of at-odds with your interpretation...? :/

Not too much. Those are trivial cases (one copies the entire qualified data, so obviously one is free to change qualifiers as one wishes).
May 14 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:
 I can't think of many situations where that wouldn't be the case. What
 sort of software is it not an issue to experience intermittent
 freezing?

Batch programs, for example a compiler. Essentially any program that is not interactive and does not have hard realtime requirements, which is (possibly) most programs.
 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,

Compiler improvements to improve on ARC are the same technology as used to improve GC. You're essentially arguing that one is easy pickings and the other is impractically difficult, but they're actually about the same level.
May 10 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:
 This is truly a niche usage case though,

Come on! Like about 80% of the programs on any linux box? Like the OCR program I run? A payroll processing program? Any scientific numerical analysis program? Engineering programs?
 If you're prepared to label the largest entertainment industry on the
 planet a niche,

I'm not doing that.
 How many libs does DMD link?

We've gone over this before. You were concerned that the libraries you linked with were incompetently written, and implied that if ARC was pervasive, they would be competently written. I can guarantee you, however, that ARC leaves plenty of opportunity for incompetence :-)
 I'm hard pressed to think of any software I see people using every day
 which isn't realtime in some sense.

Programs you "see" by definition have a user interface. But an awful lot of programs are not seen, but that doesn't mean they aren't there and aren't running. See my list above.
 Compiler improvements to improve on ARC are the same technology as used to
 improve GC.

 You're essentially arguing that one is easy pickings and the other is
 impractically difficult, but they're actually about the same level.

Are they? This is the first I've heard such a claim. If that's the case,

Yes.
 then that why is there an argument?

Because if this was an easy problem, it would have been solved. In particular, if the ARC overhead was easily removed by simple compiler enhancements, why hasn't ARC taken the world by storm? It's not like ARC was invented yesterday.
 That work just needs to be done,

That's a massive understatement. This is PhD research topic material, not something I can churn out in a week or two if only I had a more positive attitude :-)
 but by all prior
 reports I've heard, awesome GC is practically incompatible with D for
 various reasons.

There is no such thing as a GC which would satisfy your requirements.
 No matter how awesome it is, it seems conceptually
 incompatible with my environment.

My point!
 I keep saying, I'm more than happy to be proved wrong (really!), but
 until someone can, then it's unfair to dismiss my arguments so
 easily...

I believe I have answered your arguments, not dismissed them.
 and I just don't think it's that unreasonable. ARC is an
 extremely successful technology, particularly in the
 compiled/native/systems language space (OC, C++/CX,

What was dismissed is the reality pointed out many times that those systems resolve the perf problems of ARC by providing numerous means of manually escaping it, with the resulting desecration of soundness guarantees.
 Rust).

Rust is not an extremely successful technology. It's barely even been implemented.
 Is there
 actually any evidence of significant GC success in this space?
 Successes all seem to be controlled VM based languages like Java and
 C#; isolated languages with no intention to interact with existing
 native worlds. There must be good reason for that apparent separation
 in trends?

There are many techniques for mitigating GC problems in D, techniques that are not available in Java or C#. You can even do shared_ptr<> in D. You can use nogc to guarantee the GC pause troll isn't going to pop up unexpectedly. There are a bunch of other techniques, too. ARC simply is not a magic, no problem solution one can use without careful thought in large, complex systems. (Of course, neither is GC nor any other memory management scheme.)
May 10 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/10/14, 11:27 PM, Manu via Digitalmars-d wrote:
 On 11 May 2014 14:57, Walter Bright via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:
 This is truly a niche usage case though,

Come on! Like about 80% of the programs on any linux box? Like the OCR program I run? A payroll processing program? Any scientific numerical analysis program? Engineering programs?

Linux programs aren't at risk of being ported from C any time soon.

Is this seriously being aired as an argument? -- Andrei
May 11 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/10/2014 11:27 PM, Manu via Digitalmars-d wrote:
 Because if this was an easy problem, it would have been solved. In
 particular, if the ARC overhead was easily removed by simple compiler
 enhancements, why hasn't ARC taken the world by storm? It's not like ARC was
 invented yesterday.

And as far as I can tell, it has, at least in this (native/compiled/systems) space. O-C, C++/CX, Rust... where are the counter-examples?

Again, O-C and C++/CX ARC are not memory safe because in order to make it perform they provide unsafe escapes from it. Neither even attempts pervasive ARC. Rust is simply not an example of proven technology. We cannot even discuss this if we cannot agree on basic, objective facts.
 It's **workable**.

Nobody has demonstrated that pervasive ARC is both performant and memory safe. Have you ever written some code using RC in O-C or C++/CX, and disassembled it to see what it looks like? Do you realize that every decrement must happen inside an exception handler? Now imagine that for all code that deals with pointers? ------------- A Comment on Rust ------------------ This is based on my very incomplete knowledge of Rust, i.e. just reading a few online documents on it. If I'm wrong, please correct me. Rust's designers apparently are well aware of the performance cost of pervasive ARC. Hence, they've added the notion of a "borrowed" pointer, which is an escape from ARC. The borrowed pointer is made memory safe by: 1. Introducing restrictions on what can be done with a borrowed pointer so the compiler can determine its lifetime. I do not know the extent of these restrictions. 2. Introducing an annotation to distinguish a borrowed pointer from an ARC pointer. If you don't use the annotation, you get pervasive ARC with all the poor performance that entails. Implicit in borrowed pointers is Rust did not solve the problem of having the compiler eliminate unnecessary inc/dec. My experience with pointer annotations to improve performance is pretty compelling - almost nobody adds those annotations. They get their code to work with the default, and never get around to annotating it. This, of course, provided me with a large opportunity to kick ass in the performance dept. because I would use them, but that didn't help when I had to use other peoples' code. People who have added annotations to Java have seen the same result. They can't get regular programmers to use them. The annotations have their downsides even if you make the effort to use them. Since they are a different type from ARC pointers, you cannot have a data structure, say a tree, that contains both (without having a tag to say which one it is). They do not mix. A function taking one type of pointer cannot be called with the other type. Worse, these effects are transitive, making a function hierarchy rather inflexible. Are these valid concerns with Rust? I haven't written any Rust code, and I haven't heard of a whole lot of Rust code being written. So I don't know. We'll see.
May 11 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:

 Again, O-C and C++/CX ARC are not memory safe because in order to make 
 it perform they provide unsafe escapes from it.

But D could provide memory-safe escapes. If we keep the current GC to collect cycles, we could also allow raw pointers managed by the GC alongside ARC. Let's say we have two kinds of pointers: rc+gc pointers (the default) and gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only pointer, the compiler emits code that disables destruction via the reference counting. This makes the GC solely responsible for destructing and deallocating that memory block. You can still assign the pointer to a rc+gc pointer later on, but the reference count is no longer reliable which is why RC-based destruction has been disabled. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 11 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 5:52 AM, Michel Fortin wrote:
 On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:

 Again, O-C and C++/CX ARC are not memory safe because in order to make it
 perform they provide unsafe escapes from it.

But D could provide memory-safe escapes. If we keep the current GC to collect cycles, we could also allow raw pointers managed by the GC alongside ARC. Let's say we have two kinds of pointers: rc+gc pointers (the default) and gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only pointer, the compiler emits code that disables destruction via the reference counting. This makes the GC solely responsible for destructing and deallocating that memory block. You can still assign the pointer to a rc+gc pointer later on, but the reference count is no longer reliable which is why RC-based destruction has been disabled.

Yes, you can make it memory safe by introducing another pointer type, as Rust does. But see my comments about this scheme in the message you replied to. (Yes, I understand your proposal is a variation on that scheme.)
May 11 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-11 19:37:30 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 5/11/2014 5:52 AM, Michel Fortin wrote:
 On 2014-05-11 08:29:13 +0000, Walter Bright <newshound2 digitalmars.com> said:
 
 Again, O-C and C++/CX ARC are not memory safe because in order to make it
 perform they provide unsafe escapes from it.

But D could provide memory-safe escapes. If we keep the current GC to collect cycles, we could also allow raw pointers managed by the GC alongside ARC. Let's say we have two kinds of pointers: rc+gc pointers (the default) and gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only pointer, the compiler emits code that disables destruction via the reference counting. This makes the GC solely responsible for destructing and deallocating that memory block. You can still assign the pointer to a rc+gc pointer later on, but the reference count is no longer reliable which is why RC-based destruction has been disabled.

Yes, you can make it memory safe by introducing another pointer type, as Rust does. But see my comments about this scheme in the message you replied to. (Yes, I understand your proposal is a variation on that scheme.)

It is a variation on that scheme, but with one significant difference: those two pointer types can mix and there's no restriction on assignments from one type to the other. There's therefore no transitive effect and no complicated ownership rule to understand. This obviously does not address all your concerns with ARC (which I'll admit most are valid), but this "ARC isn't memory-safe" argument has to stop. It does not make sense. One doesn't need to sacrifice memory safety to use ARC, neither is that sacrifice necessary for having islands of non-ARC code. That's what I was trying to point out in my previous post. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 1:43 PM, Michel Fortin wrote:
 It is a variation on that scheme, but with one significant difference: those
two
 pointer types can mix and there's no restriction on assignments from one type
to
 the other. There's therefore no transitive effect and no complicated ownership
 rule to understand.

 This obviously does not address all your concerns with ARC (which I'll admit
 most are valid), but this "ARC isn't memory-safe" argument has to stop. It does
 not make sense. One doesn't need to sacrifice memory safety to use ARC, neither
 is that sacrifice necessary for having islands of non-ARC code. That's what I
 was trying to point out in my previous post.

As long as C++/CX and O-C are brought out here as proven, successful examples for D to emulate here, and there is no acknowledgement that they are not remotely memory safe, I need to continue to point this out. Rust is memory safe, but is an unproven design. Your proposal still relies on a GC to provide the memory safety, and has no inherent protection against GC pauses. Your idea has a lot of merit, but it is a hybrid ARC/GC system.
May 11 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-05-11 21:41:10 +0000, Walter Bright <newshound2 digitalmars.com> said:

 Your proposal still relies on a GC to provide the memory safety, and 
 has no inherent protection against GC pauses. Your idea has a lot of 
 merit, but it is a hybrid ARC/GC system.

If you thread carefully you can disable GC collections at runtime and not suffer GC pauses. If you have no cycles and no gc_only pointers, then you won't run out of memory. I think if this thread has proven something, it's that people need to be able to choose their memory management policy when the default is unsatisfactory. I'm trying to find a way to do that, a way to disable one side or the other if it is poisonous to your particular application. It is a hybrid system I'm suggesting, no doubt. It'd also be an interesting experiment, if someone wants to take it.
 As long as C++/CX and O-C are brought out here as proven, successful 
 examples for D to emulate here, and there is no acknowledgement that 
 they are not remotely memory safe, I need to continue to point this out.

You should not say that ARC is not safe then, you should say instead that ARC in those languages has to be supplemented with unsafe code to be fast enough. That statement I can agree with. Taking the shortcut saying simply "ARC is unsafe" is misleading. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
May 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 8:44 PM, Michel Fortin wrote:
 As long as C++/CX and O-C are brought out here as proven, successful examples
 for D to emulate here, and there is no acknowledgement that they are not
 remotely memory safe, I need to continue to point this out.

You should not say that ARC is not safe then, you should say instead that ARC in those languages has to be supplemented with unsafe code to be fast enough.

And I did, probably at least a dozen times in this thread, including quotes you replied to specifically: "Again, O-C and C++/CX ARC are not memory safe because in order to make it perform they provide unsafe escapes from it." I don't know how I could be clearer. In the spirit of being clear, I am opposed to any replacement for the GC that: 1. makes it impossible to meet or exceed C++ performance levels 2. is not memory safe 3. implies shipping multiple builds of Phobos, each using a different memory management scheme I am not opposed to ARC as a supplement, even if it is system only, in particular one that can interact with O-C.
May 11 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 9:47 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 14:41:10 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 Your proposal still relies on a GC to provide the memory safety,
 […] it is a hybrid ARC/GC system.

But I thought ARC cannot be designed without GC to resolve cycles.

It can be, there are various schemes to deal with that, including "don't create cycles". GC is just one of them. http://en.wikipedia.org/wiki/Reference_counting#Dealing_with_reference_cycles
May 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 3:18 AM, Marco Leise wrote:
 Your were arguing against Michel Fortin's proposal on the
 surface, when your requirement cannot even be fulfilled
 theoretically it seems.

Lots of people use ARC without a GC.
 Which could mean that you don't like
 the idea of replacing D's GC with an ARC solution.

I don't like the idea of replacing D's GC with ARC. But for different reasons.
May 12 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 7:55 AM, w0rp wrote:
 Python people don't tend to talk about speed that much.

:-)
May 11 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/11/2014 10:29 AM, Walter Bright wrote:
 ...

 ------------- A Comment on Rust ------------------

 This is based on my very incomplete knowledge of Rust, i.e. just reading
 a few online documents on it. If I'm wrong, please correct me.
 ...

Well, region-based memory management is not new, and Rust's approach is natural, even more so when given some background. Have you ever read up on type systems? Eg: http://www.cis.upenn.edu/~bcpierce/tapl/ http://www.cis.upenn.edu/~bcpierce/attapl/
 Rust's designers apparently are well aware of the performance cost of
 pervasive ARC. Hence, they've added the notion of a "borrowed" pointer,
 which is an escape from ARC.

It is not an escape from ARC per se. It is a way to write type safe code which is not dependent on the allocation strategy of the processed data. (One can e.g. safely borrow mutable data as immutable and the type system ensures that during the time of the borrow, the data does not mutate.)
 The borrowed pointer is made memory safe by:

 1. Introducing restrictions on what can be done with a borrowed pointer
 so the compiler can determine its lifetime. I do not know the extent of
 these restrictions.
 ...

The type system tracks lifetimes across function and data structure boundaries. The main restriction is that such a pointer cannot be escaped. (But borrowed pointers may still be stored in data structures.)
 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...

No, this is completely inaccurate: Both choices are explicit, and reference counting is just one of the possible memory management schemes. (Hence borrowed pointers are used unless there is a reason not to.)
 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.
 ...

Avoiding inc/dec is not what justifies borrowed pointers.
 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.

The 'default' way to pass by reference is by borrowed pointer.
 ...
 They do not mix. A function taking one type of
 pointer cannot be called with the other type.
 ...

A function taking a borrowed pointer can be called with a reference counted pointer. Abstracting over allocation strategy is the point of borrowing.
 Worse, these effects are transitive, making a function hierarchy rather
 inflexible.

 Are these valid concerns with Rust?

I don't think they are.
May 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 8:54 AM, Timon Gehr wrote:
 It is not an escape from ARC per se. It is a way to write type safe code which
 is not dependent on the allocation strategy of the processed data. (One can
e.g.
 safely borrow mutable data as immutable and the type system ensures that during
 the time of the borrow, the data does not mutate.)

That's clearly an additional benefit of the borrowed pointer notion. But have you examined generated Rust code for the cost of inc/dec? I haven't, but I don't see any way they could avoid this (very expensive) cost without borrowed pointers.
 The type system tracks lifetimes across function and data structure boundaries.
 The main restriction is that such a pointer cannot be escaped. (But borrowed
 pointers may still be stored in data structures.)

The idea must be that the borrowed pointer lifetime cannot exceed the lifetime of the lender. The thing is, if the compiler is capable of figuring out these lifetimes by examining the code, then there would be no need for the programmer to specify a pointer as borrowed - the compiler could infer it. Hence, there have got to be significant restrictions to the point where the compiler could figure out the rest.
 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...


Yes, one is & and the other is . This does not change my point. Back in the bad old DOS days, there was near* and *. The choices were explicit, but the results were there anyway - people overwhelmingly chose the more general *, performance be damned.
 and reference
 counting is just one of the possible memory management schemes. (Hence borrowed
 pointers are used unless there is a reason not to.)

Yes, I can see it supports other schemes, but I predict that RC will dominate, and other schemes will be about as pervasive as using Boehm's GC with C++.
 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.


I find that very hard to believe. It implies that there is little cost to inc/dec, or the compiler is clever enough to eliminate the bulk of the inc/dec pairs.
 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.


Time will tell how well having the most restrictive pointer type be the default works out.
 They do not mix. A function taking one type of
 pointer cannot be called with the other type.

pointer. Abstracting over allocation strategy is the point of borrowing.

Right, and a function in DOS taking a * would accept a near*. But the other way did not work, and so people wrote their functions using *, performance was thrown under the bus.
 Are these valid concerns with Rust?


Perhaps you're right. But my experience with DOS programming is programmers preferred convenience and reusable functions, and hence used plain * everywhere. And like I said, this provided a huge competitive advantage for me. The great boon with 32 bit code was the elimination of special pointer types, now it was easy to write fast, reusable functions, and the effort involved in creating efficient data structures dropped by probably half. It seems clear that the decisions of borrow/managed are going to pervade Rust code. How well is that going to work with complex programs and data structures? How well with average programmers? How well is it going to work when some leaf has to change from borrowed to managed, and the ripple effects of that change? These are all unknowns, not proven successful technology. My experience with a similar system (16 bit C) does not make it look promising, as far as performance goes.
May 11 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/11/2014 10:05 PM, Walter Bright wrote:
 On 5/11/2014 8:54 AM, Timon Gehr wrote:
 It is not an escape from ARC per se. It is a way to write type safe
 code which
 is not dependent on the allocation strategy of the processed data.
 (One can e.g.
 safely borrow mutable data as immutable and the type system ensures
 that during
 the time of the borrow, the data does not mutate.)

That's clearly an additional benefit of the borrowed pointer notion. But have you examined generated Rust code for the cost of inc/dec? I haven't, but I don't see any way they could avoid this (very expensive) cost without borrowed pointers. ...

Sure, but performance is the additional benefit.
 The type system tracks lifetimes across function and data structure
 boundaries.
 The main restriction is that such a pointer cannot be escaped. (But
 borrowed
 pointers may still be stored in data structures.)

The idea must be that the borrowed pointer lifetime cannot exceed the lifetime of the lender. The thing is, if the compiler is capable of figuring out these lifetimes by examining the code,

There are explicit lifetime annotations in function signatures.
 then there would be no need for the programmer to
 specify a pointer as borrowed - the compiler could infer it.

Not modularly. (Yes, global lifetime analysis exists, but this is not what they are doing.)
 Hence, there have got to be significant restrictions to the point where the
 compiler could figure out the rest.
 ...

It is simply not true that type systems are inherently restricted to checking trivial properties. They can be made as strong as mathematical logic without much fuss.
 2. Introducing an annotation to distinguish a borrowed pointer from an
 ARC pointer. If you don't use the annotation, you get pervasive ARC with
 all the poor performance that entails.
 ...


Yes, one is & and the other is .

No, actually currently one is & and the other is RC<T> AFAIK.
 This does not change my point.

In my opinion it actually does, to the point of rendering it invalid.
 Back in
 the bad old DOS days, there was near* and *. The choices were explicit,

Explicit among near* and *. I.e. between adding 'near' and leaving it out. Also, if you add 'near' it will not be compatible with pointers that do not have it.
 but the results were there anyway - people overwhelmingly chose the more
 general *, performance be damned.
 ...

RC<T> is not more general. It cannot refer to stack-allocated data, for instance.
 and reference
 counting is just one of the possible memory management schemes. (Hence
 borrowed
 pointers are used unless there is a reason not to.)

Yes, I can see it supports other schemes, but I predict that RC will dominate, and other schemes will be about as pervasive as using Boehm's GC with C++. ...

The point is, borrowing does not depend on the allocation scheme, as long as it is safe.
 Implicit in borrowed pointers is Rust did not solve the problem of
 having the compiler eliminate unnecessary inc/dec.


I find that very hard to believe. It implies that there is little cost to inc/dec, or the compiler is clever enough to eliminate the bulk of the inc/dec pairs. ...

Sure, borrowing is very lightweight, but ultimately what is most important is that it solves the problem of multiple incompatible pointer types and makes the type system more expressive as well.
 My experience with pointer annotations to improve performance is pretty
 compelling - almost nobody adds those annotations. They get their code
 to work with the default, and never get around to annotating it.


Time will tell how well having the most restrictive pointer type be the default works out. ...

A function that uses none of the specific pointer capabilities is more general, so what other choice of 'default' makes sense?
 They do not mix. A function taking one type of
 pointer cannot be called with the other type.

counted pointer. Abstracting over allocation strategy is the point of borrowing.

Right, and a function in DOS taking a * would accept a near*. But the other way did not work, and so people wrote their functions using *, performance was thrown under the bus.
 Are these valid concerns with Rust?


Perhaps you're right. But my experience with DOS programming is programmers preferred convenience and reusable functions, and hence used plain * everywhere. And like I said, this provided a huge competitive advantage for me. ...

Convenience and reusable functions means using borrowed pointers whenever possible.
 The great boon with 32 bit code was the elimination of special pointer
 types, now it was easy to write fast, reusable functions, and the effort
 involved in creating efficient data structures dropped by probably half.

 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.

But they are often obvious.
 How well is that going to work with complex programs and data
 structures? How well with average programmers? How well is it going to
 work when some leaf has to change from borrowed to managed, and the
 ripple effects of that change?

 These are all unknowns, not proven successful technology.

They are using Rust to write a safe and performant web browser while developing the language.
 My experience with a similar system (16 bit C) does not make it look
 promising, as far as performance goes.

Borrowed pointers are not even superficially similar to near*. They are compatible with everything else, because they can store data that was borrowed from anywhere else.
May 11 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 1:59 PM, Timon Gehr wrote:
 On 05/11/2014 10:05 PM, Walter Bright wrote:
 That's clearly an additional benefit of the borrowed pointer notion. But
 have you examined generated Rust code for the cost of inc/dec? I
 haven't, but I don't see any way they could avoid this (very expensive)
 cost without borrowed pointers.


One constant theme in this thread, one I find baffling, is the regular dismissal of the performance implications of inc/dec. Borrowed pointers are not necessary to support raw pointers - this can be (and is in some systems) supported by simply wrapping the raw pointer with a dummy reference count. The reason for borrowed pointers is performance. Rust would be non-viable without them. I strongly suggest writing a snippet in [[insert your favorite proven technology RC language here]] and disassembling the result, and have a look at what inc/dec entails.
 The thing is, if the compiler is capable of figuring out these lifetimes
 by examining the code,

There are explicit lifetime annotations in function signatures.

Yes, because the compiler cannot figure it out itself, so the programmer has to annotate.
 It is simply not true that type systems are inherently restricted to checking
 trivial properties. They can be made as strong as mathematical logic without
 much fuss.

Again, Rust would not need borrowed pointers nor the annotations for them if this knowledge could be deduced by the compiler. Heck, if the compiler can deduce lifetimes accurately, you can get rid of GC and RC, and just have the compiler insert malloc/free in the right spots. Note that there is a Java version that does this partway, sometimes it will replace a GC object with a stack allocated one if it is successful in deducing that the object lifetime does not exceed the lifetime of the function.
 Yes, one is & and the other is  .

No, actually currently one is & and the other is RC<T> AFAIK.

Then Rust changed again. The document I read on borrowed pointers was likely out of date, though it had no date on it.
 RC<T> is not more general. It cannot refer to stack-allocated data, for
instance.

So there is no general pointer type that has an unbounded lifetime?
 Sure, borrowing is very lightweight, but ultimately what is most important is
 that it solves the problem of multiple incompatible pointer types and makes the
 type system more expressive as well.

Adding more pointer types makes a type system more expressive, by definition.
 A function that uses none of the specific pointer capabilities is more general,
 so what other choice of 'default' makes sense?

A function that doesn't have restrictions on what can be done with the pointers passed to it. Borrowed pointers have restrictions on their usage - this is explicitly stated in the Rust documentation.
 Convenience and reusable functions means using borrowed pointers whenever
possible.

Of course. And writing fast code means making your functions fast whenever possible!
 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.

But they are often obvious.

I've written a lot of ref counted code in the past, enough to know that such is a pretty optimistic statement. Dealing with was a significant and ongoing drain on my time, especially when the programs and data structures got more complex.
 They are using Rust to write a safe and performant web browser while developing
 the language.

Sure. But that browser hasn't been released yet. Consider that I've written safe and performant code in D, but others tell me I am unique and that I cannot expect average programmers to get it right. I repeatedly point out to Manu that he can write performant code in D that does not suffer from GC stalls, and he repeatedly replies that he has to work with average programmers who are not capable of doing this. So while I have no doubt that the Mozilla team may be very effective at using Rust and making it shine, that may not be transferable to the larger community.
 Borrowed pointers are not even superficially similar to near*. They are
 compatible with everything else, because they can store data that was borrowed
 from anywhere else.

As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure? The similarity is that there are one way conversions from one to the other, and one of the types is more general. I infer from your other statements about Rust that it doesn't actually have a general pointer type.
May 11 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:
 What? You've never offered me a practical solution.

I have, you've just rejected them.
 What do I do?

1. you can simply do C++ style memory management. shared_ptr<>, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times.
May 12 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:12 AM, Dicebot wrote:
 I think this is more of library writing culture problem than engineering
 problem. High quality library shouldn't rely on any internal allocations at
all,
 deferring this decision to user code. Otherwise you will eventually have
 problems, GC or not.

Consider my PR: https://github.com/D-Programming-Language/phobos/pull/2149 This is exactly what it does - it 'pushes' the decisions about allocating memory up out of the library to the user. I suspect a great deal of storage allocation can be removed from Phobos with this technique, without sacrificing performance, flexibility, or memory safety. (In fact, it improves on performance and flexibility!) I also agree with your larger point that if you are relying on an unknown library for time critical code, and that library was not designed with time criticality guarantees in mind, you're going to have nothing but trouble. Regardless of GC or RC.
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 10:07 AM, Dicebot wrote:
 We have already had discussion where I did state that current  nogc
 implementation is not robust enough and failed to explain the use case for
 weaker  nogc clearly. Conclusion was that we should return to this topic after
 Don's DConf talk ;)

Sure - next week!
May 12 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-12 19:14, Dicebot wrote:

 It lacks any good static reflection though. And this stuff is damn
 addictive when you try it of D caliber.

It has macros, that basically requires great support for static reflection to be usable. -- /Jacob Carlborg
May 12 2014
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 05/13/2014 03:56 PM, Dicebot wrote:
 On Monday, 12 May 2014 at 19:32:49 UTC, Jacob Carlborg wrote:
 On 2014-05-12 19:14, Dicebot wrote:

 It lacks any good static reflection though. And this stuff is damn
 addictive when you try it of D caliber.

It has macros, that basically requires great support for static reflection to be usable.

Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html those are not full-blown AST macros like ones you have been proposing, more like hygienic version of C macros.

https://github.com/huonw/brainfuck_macro/blob/master/lib.rs
May 13 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-05-13 15:56, Dicebot wrote:

 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.

Hmm, I haven't looked at Rust macros that much. -- /Jacob Carlborg
May 13 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/13/2014 09:07 PM, Jacob Carlborg wrote:
 On 2014-05-13 15:56, Dicebot wrote:

 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.

Hmm, I haven't looked at Rust macros that much.

Again, the following is an example of Rust macros in action. A bf program is compiled to Rust code at compile time. https://github.com/huonw/brainfuck_macro/blob/master/lib.rs Compile-time computations create an AST which is then spliced. Seems full-blown enough to me and not at all like C macros.
May 13 2014
parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
14-May-2014 17:16, Dicebot пишет:
 On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:
 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:
 On 2014-05-13 15:56, Dicebot wrote:

 Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been proposing,
 more like hygienic version of C macros.

Hmm, I haven't looked at Rust macros that much.

Again, the following is an example of Rust macros in action. A bf program is compiled to Rust code at compile time. https://github.com/huonw/brainfuck_macro/blob/master/lib.rs Compile-time computations create an AST which is then spliced. Seems full-blown enough to me and not at all like C macros.

Comments look promising but unfortunately I can't understand a single line of actual code :) Mine rust-fu is too weak.

This is curious: http://burntsushi.net/rustdoc/regex/ -- Dmitry Olshansky
May 14 2014
parent Jacob Carlborg <doob me.com> writes:
On 14/05/14 23:47, Dmitry Olshansky wrote:

 This is curious:
 http://burntsushi.net/rustdoc/regex/

It seems they have compile time regular expressions too. -- /Jacob Carlborg
May 14 2014
prev sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 12/05/2014 19:14, Dicebot a écrit :
 On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote:
 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the
 language ;)

Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about "good coding practices". It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber.

All compile time things of D are marvelous. This with the compile time and the language less error prone make me want D. I am not sure I need safety so much. It's nice but not mandatory for any of my projects. The only one which has to be safe is DQuick.
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 2:28 PM, Xavier Bigand wrote:
 All compile time things of D are marvelous.
 This with the compile time and the language less error prone make me want D.
 I am not sure I need safety so much. It's nice but not mandatory for any of my
 projects. The only one which has to be safe is DQuick.

Safety becomes a big concern when you're developing code as part of a team.
May 12 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/11/2014 10:57 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot store a
 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do™.

This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use, and cannot (without tagging) mix memory allocation schemes. For example, let's say a compiler has internally a single hash table of strings. With a GC, those strings can be statically allocated, or on the GC heap, or anything with a lifetime longer than the table's. But I don't see how this could work in Rust.
May 12 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/12/2014 10:54 AM, Walter Bright wrote:
 On 5/11/2014 10:57 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do™.

This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use,

Global variables are banned in Rust code outside of unsafe blocks.
 and cannot (without tagging) mix memory allocation schemes.
 ...

Tagging won't help with all memory allocation schemes.
 For example, let's say a compiler has internally a single hash table of
 strings. With a GC, those strings can be statically allocated, or on the
 GC heap, or anything with a lifetime longer than the table's.
 But I don't see how this could work in Rust.

It's possible if you don't make the table global. (OTOH in D this is not going to work at all.)
May 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 5:15 AM, Timon Gehr wrote:
 On 05/12/2014 10:54 AM, Walter Bright wrote:
 On 5/11/2014 10:57 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do™.

This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use,

Global variables are banned in Rust code outside of unsafe blocks.

Global can also mean assigning through a reference passed as a parameter.
May 12 2014
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/12/2014 06:37 PM, Walter Bright wrote:
 On 5/12/2014 5:15 AM, Timon Gehr wrote:
 On 05/12/2014 10:54 AM, Walter Bright wrote:
 On 5/11/2014 10:57 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot
 store a
 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do™.

This means that a global data structure in Rust has to decide what memory allocation scheme its contents must use,

Global variables are banned in Rust code outside of unsafe blocks.

Global can also mean assigning through a reference passed as a parameter.

Do you mean the table is not actually global but passed by parameter, or that the global table is accessed in unsafe code and then passed by parameter or something else?
May 12 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 12:36 PM, Timon Gehr wrote:
 Do you mean the table is not actually global but passed by parameter,

Yes. But note that the distinction between the two is often blurry. Under the hood on some systems, global data is accessed via the equivalent of a hidden parameter.
May 12 2014
prev sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 05/12/2014 02:50 AM, Walter Bright wrote:
 On 5/11/2014 1:59 PM, Timon Gehr wrote:
 On 05/11/2014 10:05 PM, Walter Bright wrote:
 That's clearly an additional benefit of the borrowed pointer notion. But
 have you examined generated Rust code for the cost of inc/dec? I
 haven't, but I don't see any way they could avoid this (very expensive)
 cost without borrowed pointers.


One constant theme in this thread, one I find baffling, is the regular dismissal of the performance implications of inc/dec.

Irrelevant, I'm not doing that. (And again, reference counting is not the only allocation mechanism in Rust. AFAICT, most allocations use owned pointers.)
 Borrowed pointers are not necessary to support raw pointers  - this can be
(and is in some
 systems) supported by simply wrapping the raw pointer with a dummy
 reference count.
 ...

I have no idea what this part is trying to bring across.
 The reason for borrowed pointers is performance.

No, it is safety. Raw pointers give you all of the performance.
 Rust would be non-viable without them.

True in that it would fail to meet its design goals. Rust provides a tracing garbage collector as well, so it is at least as viable as D regarding performance of safe memory management. (Probably more, actually, because it does not provide precision-unfriendly constructs such as undiscriminated unions.)
 I strongly suggest writing a snippet in [[insert your favorite proven
 technology RC language here]] and disassembling the result, and have a
 look at what inc/dec entails.
 ...

I don't have trouble seeing the cost of reference counting. (And it's you who claimed that this is going to be the only memory allocation scheme in use in Rust code.)
 The thing is, if the compiler is capable of figuring out these lifetimes
 by examining the code,

There are explicit lifetime annotations in function signatures.

Yes, because the compiler cannot figure it out itself, so the programmer has to annotate. ...

You are saying 'if the compiler is capable of figuring out these lifetimes by examining the code, then ...' and then you are saying that the compiler is incapable of figuring them out itself. What is it that we are arguing here? Are you saying the Rust compiler should infer all memory management automatically or that it cannot possibly do that, or something else?
 It is simply not true that type systems are inherently restricted to
 checking
 trivial properties. They can be made as strong as mathematical logic
 without
 much fuss.

Again, Rust would not need borrowed pointers nor the annotations for them if this knowledge could be deduced by the compiler. Heck, if the compiler can deduce lifetimes accurately,

It does not deduce anything. It checks that borrowed pointers do not outlive their source. Lifetime parameters are used to transport the required information across function signatures.
 you can get rid of GC and RC,
 and just have the compiler insert malloc/free in the right spots.
 ...

That's a form of GC, and I already acknowledged that global region inference exists, but noted that this is not what is used.
 Note that there is a Java version that does this partway, sometimes it
 will replace a GC object with a stack allocated one if it is successful
 in deducing that the object lifetime does not exceed the lifetime of the
 function.
 ...

I know. Also, inference is harder to control and less efficient than simply making the relevant information part of type signatures. http://en.wikipedia.org/wiki/Region-based_memory_management#Region_inference "This work was completed in 1995[9] and integrated into the ML Kit, a version of ML based on region allocation in place of garbage collection. This permitted a direct comparison between the two on medium-sized test programs, yielding widely varying results ("between 10 times faster and four times slower") depending on how "region-friendly" the program was; compile times, however, were on the order of minutes."
 Yes, one is & and the other is  .

No, actually currently one is & and the other is RC<T> AFAIK.

Then Rust changed again. The document I read on borrowed pointers was likely out of date, though it had no date on it. ...

Yes, most documents on Rust are at least slightly out of date.
 RC<T> is not more general. It cannot refer to stack-allocated data,
 for instance.

So there is no general pointer type that has an unbounded lifetime? ...

How can it be general and have an unbounded lifetime and be safe?
 Sure, borrowing is very lightweight, but ultimately what is most
 important is
 that it solves the problem of multiple incompatible pointer types and
 makes the
 type system more expressive as well.

Adding more pointer types makes a type system more expressive, by definition. ...

No. Also, this is irrelevant, because I was highlighting the _importance_ of the fact that it does in this case.
 A function that uses none of the specific pointer capabilities is more
 general,
 so what other choice of 'default' makes sense?

A function that doesn't have restrictions on what can be done with the pointers passed to it.

You mean a function that only accepts GC-managed pointers? (Not having _any_ restrictions at all does not make any sense.) I don't see how this is better, because such a function is usable in less contexts.
 Borrowed pointers have restrictions on their
 usage - this is explicitly stated in the Rust documentation.
 ...

Values of every type have restrictions on their usage. If your function requires less capabilities from it's arguments, then it is usable in more cases. Why is this not the most sensible thing to do whenever possible?
 Convenience and reusable functions means using borrowed pointers
 whenever possible.

Of course. And writing fast code means making your functions fast whenever possible! ...

What is your point here?
 It seems clear that the decisions of borrow/managed are going to pervade
 Rust code.

But they are often obvious.

I've written a lot of ref counted code in the past, enough to know that such is a pretty optimistic statement. Dealing with was a significant and ongoing drain on my time, especially when the programs and data structures got more complex. ...

You said you don't have any experience writing Rust code, and claimed that it is not a proven technology, so I fail to see how you can have any relevant experience with borrowing.
...

 Borrowed pointers are not even superficially similar to near*. They are
 compatible with everything else, because they can store data that was
 borrowed
 from anywhere else.

As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure?

Sure. If you are going to escape a pointer, you need to think about how that memory is managed. That's how it is. And there are many options: You could simply transfer ownership, you might use reference counting, or a tracing GC.
 The similarity is
 that there are one way conversions from one to the other, and one of the
 types is more general.

And the difference is that the one you can convert to is faster here.
 I infer from your other statements about Rust
 that it doesn't actually have a general pointer type.

Memory can be borrowed no matter how it was allocated. That's what I call general. How do you fill that term?
May 12 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 4:35 AM, bearophile wrote:
 I suggested to add an optional method named "onGC" to unions that if present is
 called at run-time by the GC to know what's the real type of stored data, to
 make tracing more precise.

Unions of pointers are so rare in actual code that treating them conservatively is not a big problem.
May 12 2014
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote:
 I just searched through my code, and 7 out of 12 unions had pointers.

Relative number of objects with unions, not declarations with unions!
May 12 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/12/2014 10:25 AM, bearophile wrote:
 Walter Bright:

 Unions of pointers are so rare in actual code that treating them
 conservatively is not a big problem.

std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough.

BTW, the RTinfo can be used to discriminate unions.
May 12 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/12/14, 10:25 AM, bearophile wrote:
 Walter Bright:

 Unions of pointers are so rare in actual code that treating them
 conservatively is not a big problem.

std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough.

I, too, felt the need of onGC() - actually preGC() - in my allocators implementation. Specifically, a thread-local freelist would save a pointer to the root in thread-local storage (i.e. a traditional D global variable). That would thread through a number of free nodes available for allocation. When a GC cycle occurs, it's okay if the list stays referenced; the GC will consider it "used" and won't do anything in particular about it. However, the GC cycle is a good opportunity to clean these freelists and offer the memory for other size classes, seeing as the freelists may grow unreasonably large and then just hold memory for no good reason. A hook that nulls all freelist heads just as the collection process starts would be helpful. Andrei
May 12 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
12-May-2014 22:08, Andrei Alexandrescu пишет:
 On 5/12/14, 10:25 AM, bearophile wrote:
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.

One word - weak pointers. Then head of freelist is weak and can be collected at whim.
 Andrei

-- Dmitry Olshansky
May 12 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/12/14, 11:17 AM, Dmitry Olshansky wrote:
 12-May-2014 22:08, Andrei Alexandrescu пишет:
 On 5/12/14, 10:25 AM, bearophile wrote:
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.

One word - weak pointers. Then head of freelist is weak and can be collected at whim.

Of course. My point here is that here you need simpler support than full-blown weak pointers. -- Andrei
May 12 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/12/14, 12:59 PM, bearophile wrote:
 Andrei Alexandrescu:

 I, too, felt the need of onGC() - actually preGC() - in my allocators
 implementation.
 ...
 A hook that nulls all freelist heads just as the collection process
 starts would be helpful.

How is this going to help increase tracing precision of unions (and Algebraic built on top of unions)?

How did I give the impression it has anything to do with unions? -- Andrei
May 12 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:
 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:
 On 2014-05-13 15:56, Dicebot wrote:

 Judging by 
 http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been 
 proposing,
 more like hygienic version of C macros.

Hmm, I haven't looked at Rust macros that much.

Again, the following is an example of Rust macros in action. A bf program is compiled to Rust code at compile time. https://github.com/huonw/brainfuck_macro/blob/master/lib.rs Compile-time computations create an AST which is then spliced. Seems full-blown enough to me and not at all like C macros.

I really don't know, but understanding is that there are two types of macros in Rust. There's the simple "hygienic macro" one, which has some documentation: http://static.rust-lang.org/doc/0.10/guide-macros.html. Then there's the other, newer kind of macros (they may be called "procedural macros"... or not) and those are basically undocumented at this point. My understanding is that the newer kind of macros can do literally anything, like order you a pizza at compile-time.
May 13 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 13 May 2014 at 21:16:55 UTC, Timon Gehr wrote:
 On 05/13/2014 09:07 PM, Jacob Carlborg wrote:
 On 2014-05-13 15:56, Dicebot wrote:

 Judging by 
 http://static.rust-lang.org/doc/0.6/tutorial-macros.html
 those are not full-blown AST macros like ones you have been 
 proposing,
 more like hygienic version of C macros.

Hmm, I haven't looked at Rust macros that much.

Again, the following is an example of Rust macros in action. A bf program is compiled to Rust code at compile time. https://github.com/huonw/brainfuck_macro/blob/master/lib.rs Compile-time computations create an AST which is then spliced. Seems full-blown enough to me and not at all like C macros.

Comments look promising but unfortunately I can't understand a single line of actual code :) Mine rust-fu is too weak.
May 14 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/05/2014 01:31, Francesco Cattoglio a écrit :
 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:
 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)

Second, 16.6ms is the time you have to do everything between frames. This means that in that timeframe you have to: -update your game state. -possibly process all network I/O. -prepare the rendering pipeline for the next frame. Updating the game state can imply make computations on lots of stuff: physics, animations, creation and deletion of entities and particles, AI logic... pick your poison. At every frame you will have an handful of objects being destroyed and a few resources that might go forgotten. One frame would probably only need very little objects collected. But given some times the amount of junk can grow out of control easily. Your code will end up stuttering at some point (because of random collections at random times), and this can be really bad.

As I know AAA game engine are reputed to do zero allocations during the frame computation, but I think it less the case noways cause of the dynamism of scene and huge environments that are streamed. I recently fix a performance issue due to a code design that force destruction of walls (I am working on an architecture application) before creating them back when the user move them. gprof, show me that this point took around 30% of CPU time in a frame, and only allocations/destructions was about 5%. This 5% percents contains destructions of object in the architecture part and the 3D engine, same for construction. Construction also add operation like new geometry updload, so I don't think place of new and delete was high without the job made by constructors and destructors. Reserving memory (malloc) isn't really an issue IMO, but operations related to constructions and destructions of objects can be expensive.
May 10 2014
parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/05/2014 15:25, Manu via Digitalmars-d a écrit :
 On 10 May 2014 19:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Le 10/05/2014 01:31, Francesco Cattoglio a écrit :

 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:
 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)

I'll try to guess: if you want something running at 60 Frames per Second, 16.6ms is the time you have to do everything between frames. This means that in that timeframe you have to: -update your game state. -possibly process all network I/O. -prepare the rendering pipeline for the next frame. Updating the game state can imply make computations on lots of stuff: physics, animations, creation and deletion of entities and particles, AI logic... pick your poison. At every frame you will have an handful of objects being destroyed and a few resources that might go forgotten. One frame would probably only need very little objects collected. But given some times the amount of junk can grow out of control easily. Your code will end up stuttering at some point (because of random collections at random times), and this can be really bad.

As I know AAA game engine are reputed to do zero allocations during the frame computation, but I think it less the case noways cause of the dynamism of scene and huge environments that are streamed.

Running a game in a zero-alloc environment was a luxury that ended about 10 years ago, for reasons that you say. Grand Theft Auto proved that you don't need to have loading screens, and now it's a basic requirement. We also have a lot more physics, environmental destruction, and other dynamic behaviour. It was also something we achieved in the past with extreme complexity, and the source of many (most?) bugs. We still allocated technically, but we had to micro-manage every single little detail, mini pools and regions for everything. Processors are better now, in the low-frequency code, we can afford to spend a tiny bit of time using a standard allocation model. The key is the growing separation between low-frequency and high-frequency code. On a 33mhz Playstation, there wasn't really much difference between the worlds. Now there is, and we can afford to allow 'safety' into the language at the cost of a little memory management. We just can't have that cost include halting threads all the time for lengthy collect processes. I know this, because we already use RC extensively in games anyway; DirectX uses COM, and most resources use manual RC because we need things to release eagerly, and for destructors to work properly. I don't see how ARC would add any significant cost to the manual RC that is basically standard for many years. It would add simplicity and safety, which are highly desirable.
 I recently fix a performance issue due to a code design that force
 destruction of walls (I am working on an architecture application) before
 creating them back when the user move them. gprof, show me that this point
 took around 30% of CPU time in a frame, and only allocations/destructions
 was about 5%. This 5% percents contains destructions of object in the
 architecture part and the 3D engine, same for construction. Construction
 also add operation like new geometry updload, so I don't think place of new
 and delete was high without the job made by constructors and destructors.
 Reserving memory (malloc) isn't really an issue IMO, but operations related
 to constructions and destructions of objects can be expensive.

It's better to have eager destructors (which empowered you with the ability to defer them if you need to), than to not have destructors at all, which has been the center if this last few days argument. It's not hard to defer resource destruction to a background thread in an eager-destruction model.

graph with cost create by gprof, and it reveals to me the easiest points need to be modified a cache insertion. I didn't just differs the destruction, cause the majority of times walls stay the same with a new position, in that case I can directly reuse the same geometry I just have to update it. The other case is when a wall is merged to another one (when you put 2 rooms one against each other). One other issue we have, is the usage of containers of std that are really slow in debug (from 60fps to 15fps for parts runnings smoothly), on Android it's hard to build some part of application with optimization and an other one with.
May 10 2014
prev sibling next sibling parent Mike Parker <aldacron gmail.com> writes:
On 5/10/2014 6:05 AM, Wyatt wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 Let's also bear in mind that Java's GC is worlds ahead of D's.


Yes. Given all the man hours that have gone into it over the years, it would be surprising if it weren't. Actually, though, there isn't really one collector that ships with the JRE anymore. There are different implementations, each using a different strategy, that can be selected at JVM startup. Furthermore, each collector is tunable so if you aren't satisfied with the default collector out of the box, you can use the JVM's instrumentation to find the most optimal match for your app's usage patterns. See [1] for some details. [1] http://www.infoq.com/articles/Java_Garbage_Collection_Distilled
May 10 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 12 May 2014 at 10:51:33 UTC, Marco Leise wrote:

 Time will tell if all "well written" D libraries will be  nogc
 to move the question of allocations to the user.

If there was such a thing as "weakly &nogc" then we would could do this //some library function void foo(OR, IR)(OR o, IR i) weak-nogc { //take things from i and put them in o } allocations would be possible during the execution of foo, but *only* in the implementations of OR and IR, which means that the developer gets the control and guarantees they need, but doesn't have to explicitly pre-allocate (which might not even be possible). I don't see how it would work with UFCS though...
May 12 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 17:03:18 UTC, Walter Bright wrote:
 On 5/12/2014 2:12 AM, Dicebot wrote:
 I think this is more of library writing culture problem than 
 engineering
 problem. High quality library shouldn't rely on any internal 
 allocations at all,
 deferring this decision to user code. Otherwise you will 
 eventually have
 problems, GC or not.

Consider my PR: https://github.com/D-Programming-Language/phobos/pull/2149 This is exactly what it does - it 'pushes' the decisions about allocating memory up out of the library to the user. I suspect a great deal of storage allocation can be removed from Phobos with this technique, without sacrificing performance, flexibility, or memory safety. (In fact, it improves on performance and flexibility!)

We have already had discussion where I did state that current nogc implementation is not robust enough and failed to explain the use case for weaker nogc clearly. Conclusion was that we should return to this topic after Don's DConf talk ;)
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 03:44, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/12/2014 10:31 AM, Manu via Digitalmars-d wrote:
 I just searched through my code, and 7 out of 12 unions had pointers.

Relative number of objects with unions, not declarations with unions!

Ah, well I have 3 different tree/graph structures with unions, and tree/graph nodes have a tendency to accumulate many instances.
May 12 2014
prev sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 13:37:05 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:

 On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy  
 <yebbliesnospam gmail.com> wrote:

 "Steven Schveighoffer"  wrote in message  
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...

 If a thread is doing only reads, or is  only touching non-Scanned  
 memory, it continues.

How long do threads usually go without touching the stack?

The stack would have to be COW.

No sorry, the stack only contains roots, so it can be marked as "safe to use" pretty quickly. -Steve
May 13 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Gr=C3=B8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:

 On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:
 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:
 pointing at it is roughly 1/256. This problem is just about eliminat=



 with
 64-bit pointers.


Not generally true. This presumes that the heap is not in the lower =

 region of the address space and that you don't use 64 bit ints on the =

 stack.

I was thinking in terms of purely a random number happening to point at = = heap data. Practically speaking, I don't know the true likelihood based = on = the heap address scheme of 64-bit OSes, but I know that we always have a= = complainer who will try and do an array-append test on 32-bit code, and = = end up exhausting memory unexpectedly. -Steve
May 12 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 17:32:09 -0400, Steven Schveighoffer  
<schveiguy yahoo.com> wrote:


 The workaround is simply to keep it around, but that's not always a  
 scalable solution.

Sorry, actually you can free it. That's the correct workaround. -Steve
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 21:22:09 UTC, Steven Schveighoffer 
wrote:
 On Mon, 12 May 2014 14:14:28 -0400, Ola Fosheim Grøstad 
 <ola.fosheim.grostad+dlang gmail.com> wrote:

 On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:
 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:
 pointing at it is roughly 1/256. This problem is just about 
 eliminated with
 64-bit pointers.


Not generally true. This presumes that the heap is not in the lower region of the address space and that you don't use 64 bit ints on the stack.

I was thinking in terms of purely a random number happening to point at heap data. Practically speaking, I don't know the true likelihood based on the heap address scheme of 64-bit OSes, but

Wicked topic. In AMD64 mode hi-mem is usually reserved for kernel etc. Traditionally the unixy heap grew from low towards high addresses: http://en.wikipedia.org/wiki/Sbrk But that is legacy. I think mmap is it… :-P And layout is randomized to reduce the effect of buffer overflow etc. :-(
 I know that we always have a complainer who will try and do an 
 array-append test on 32-bit code, and end up exhausting memory 
 unexpectedly.

Uhuh. Not focusing on precise collection gets ugly.
May 12 2014
prev sibling parent Jerry <jlquinn optonline.net> writes:
Rainer Schuetze <r.sagitario gmx.de> writes:

 On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:
 class C { C next; }

 shared(C) list;

 C newC()
 {
     return new C; // allocated on the local heap?

Yes, because it doesn't say "new shared(C)".
 }

 void prependToList(C c)
 {
     synchronized(list)
     {
         C lst = cast(C) list;
         c.next = lst;  // anything special happening here?
         list = cast(shared(C)) c;  // and here?

The casts are a problem. You're basically lying to the compiler, so this cannot work. Instead, it should be something like: auto newC() { return new isolated(C); } void prependToList(isolated(C) c) { synchronized(list) { // transfer to shared heap // afterwards, c is no longer usable ("consumed") shared(C) tmp = c.makeShared(); tmp.next = list; list = tmp; } }

Sorry for the late reply. I currently don't know of a feasible way to replace casting away "shared" if you want to do more than just "tmp.next = list;", for example call "bool check(C c);" on tmp or list? I can't implement this with a shared object, because any operations on "shared" tend to not exist (ok, maybe I could for this very simple example, so consider C contains a container to operate on). Also, the object is already locked, I should not have to do it again in a function if it only accepts a shared C. How can I prepend something to the list that is not isolated? I guess I cannot, as it might create references to same object both shared and unshared. My gut feeling tells me that this ownership handling through the type system might work, but it is very restrictive and does not really contribute to the first goal of D listed on the front page: "convenience".

Reading this, I had a thought. What if synchronized temporarily stripped the shared qualifier from list? At the end of the synchronized block, shared is restored. Then: void prependToList(C c) { synchronized(list) { // list is now C, not shared(C) c.next = list; // ok list = c; // ok check(list); // ok, not a shared object here } // list is now shared again. move list to shared heap if not already } I'm sure I'm missing something, but using shared within a synchronized block might be saner if this worked... Jerry
May 17 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 14:04, Manu via Digitalmars-d a écrit :
 On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/ http://www.is2t.com/products/ http://www.mountaineer.org/netmf-for-stm32/ Just a small sample of the partners providing the said support.

True, much embedded work isn't realtime. There are a lot of purpose devices where performance is not particularly important, probably correctness is, hence Java may be popular in this environment, and safe D may have a great application here. In another realm, if we're talking about really small systems (microcontrollers with memory in the megabytes), the tendency to rely on libraries is reduced significantly, since you probably can't afford the memory for the libs. This environment is the one where it's realistic to say "tag main() with nogc, and work from there", ie, ban the GC throughout the whole project. However, another very popular use for embedded systems IS realtime software. Games consoles are the obvious one here, but also POS/retail devices, kiosks, televisions, PVR's/dvd/bluray and other entertainment related devices, cars and automobile HUD's, music players, advertising displays, etc. There's no shortage of realtime devices in the embedded space. Notably, I didn't say 'phones'. Although I think they do generally fall into this category, I think they're drifting away. Since they run full OS stack's, it's typical to have unknown amounts of free memory for user-space apps and virtual memory managers that can page swap, so having excess memory overhead probably isn't such a big deal. It's still a major performance hazard though. Stuttering realtime applications is never a professional look, and Android suffers chronically in this department compared to iOS.

almost every time you switch applications, the previous one see his memory collected and swapped on the Flash (maybe also compressed).
 I spent my last weekend trying to get a PS4 game (built with Unity;
 uses mono as a scripting environment) running at an acceptable frame
 rate. I didn't succeed, and a significant factor was the JS/C# garbage
 collector. We'll probably need to cut content from the game, such that
 it leaves plenty of 'excess' resource available to absorb the spikes
 it causes when they occur. What a waste of the machine!
 Note, C#'s GC is much better than D's. It does seem to run in the
 realm of single-digit milliseconds.

May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 May 2014 10:50, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 They are using Rust to write a safe and performant web browser while
 developing
 the language.

Sure. But that browser hasn't been released yet. Consider that I've written safe and performant code in D, but others tell me I am unique and that I cannot expect average programmers to get it right. I repeatedly point out to Manu that he can write performant code in D that does not suffer from GC stalls, and he repeatedly replies that he has to work with average programmers who are not capable of doing this.

What? You've never offered me a practical solution. You tell me I have to sacrifice any dependency on libraries (ridiculous), and all the modern conveniences and safety of automatic memory management to do it! Indeed, average programmers are a real-life practical problem, and it's not just them, I also appreciate the convenience offered personally. I only have one life, I *really* appreciate saving time on mundane and otherwise inconsequential tasks. Tell me some other reasons why I would be attracted to D? Take that all away, and what's the point? Automating some boilerplate is nice, but it's not the motivating reason for a wholesale adoption. You haven't told me how I can use the GC (or whatever memory management scheme, I really don't care) in the low frequency code (again, read: almost all code ever), and not have it interfere with the high frequency code. This is the fundamental problem with the GC. I can't use it ***ANYWHERE***, including any libraries I link. You can't isolate a GC, it's effects are not localised. If you could, maybe it'd be more workable... but even if it were properly concurrent and didn't halt the realtime threads when it collected, it's still totally impractical for any background thread to freeze for 10s-100s of ms while it runs a collect because I received a network packet which needs to be sent somewhere for processing or whatever. What do I do?
 So while I have no doubt that the Mozilla team may be very effective at
 using Rust and making it shine, that may not be transferable to the larger
 community.

May 12 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d 
wrote:
 You haven't told me how I can use the GC (or whatever memory
 management scheme, I really don't care) in the low frequency 
 code
 (again, read: almost all code ever), and not have it interfere 
 with
 the high frequency code.

You will like Don's talk this year ;) Game dev code in not just any low level code though.
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 May 2014 17:24, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 May 2014 at 07:12:29 UTC, Manu via Digitalmars-d wrote:
 You haven't told me how I can use the GC (or whatever memory
 management scheme, I really don't care) in the low frequency code
 (again, read: almost all code ever), and not have it interfere with
 the high frequency code.

You will like Don's talk this year ;)

I'm super-disappointed I can't make it this year! We were evicted from our house, have to move, and I can't bail for a week and leave that all on my mrs while she kicks along the fulltime job :(
 Game dev code in not just any low level code though.

It's not just low-level code. It's also very high level code. Gamedev is a unique (and extremely interesting) field which combines and tightly integrates almost every imaginable field of software engineering. I can't think of any that are left out, or any other industries that also share this property. I argue, if you can make a modern high-end game in a language, it can do anything.
May 12 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Monday, 12 May 2014 at 00:50:24 UTC, Walter Bright wrote:
 On 5/11/2014 1:59 PM, Timon Gehr wrote:
 Borrowed pointers are not even superficially similar to near*. 
 They are
 compatible with everything else, because they can store data 
 that was borrowed
 from anywhere else.

As long as those pointers don't escape. Am I right in that one cannot store a borrowed pointer into a global data structure?

Perhaps: struct Test { n: &'static int, // [1] m: int } static val: int = 123; static mut t: Test = Test { n: &'static val, m: 0 }; fn main() { unsafe { // [2] let p = &mut t.m; *p = 456; println!("{} {}", *t.n, t.m); // prints: 123 456 } } [1]: In order to create a static instance of 'Test', the 'n' field (which is a borrowed pointer) must be specified as to be pointing at a static immutable (int) variable. [2]: Any use of static mutable data requires the use of an 'unsafe' block (similar to trusted in D)
May 12 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Monday, 12 May 2014 at 08:10:43 UTC, Tommi wrote:
 Perhaps: [..]

Somewhat surprisingly to me, you can later on change the borrowed pointer in the mutable static 'Test' to point at a mutable static int: struct Test { n: &'static int } static old: int = 111; static mut new: int = 222; static mut t: Test = Test { n: &'static old }; fn main() { unsafe { println!("{}", *t.n); // prints: 111 t.n = &new; // Can point to a different static println!("{}", *t.n); // prints: 222 // ...but can't point to a local, e.g: // let v = 123; // t.n = &v; // error: `v` does not live long enough new = 333; println!("{}", *t.n); // prints: 333 } }
May 12 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Sun, 11 May 2014 16:33:04 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/11/2014 2:48 AM, Benjamin Thaut wrote:
 Mostly percise doesn't help. Its either fully percise or beeing stuck  
 with a
 impercise mark & sweep.

This is not correct. It helps because most of the false pointers will be in the heap, and the heap will be accurately scanned, nearly eliminating false references to garbage.

It doesn't matter where the false pointers are. The largest issue with false pointers is not how many false pointers there are. It only matters how large the block is that it "points" at. The larger your blocks get, the more likely they are "pointed" at by the stack. On 32-bit systems, allocate 1/256th of your memory space (i.e. 16.5MB), and the likelihood of random data on the stack pointing at it is roughly 1/256. This problem is just about eliminated with 64-bit pointers.
 Yes. D, for example, requires that objects not be self-referential for  
 this reason.

As previously stated, self referencing does not preclude GC moving. This statement is simply false, you can self reference in D for objects. You cannot for structs, but not because of a possibility for the moving GC, but because of the requirement to be able to move a struct instance. And in fact, even if it's forbidden, "requires" is too strong a word -- there is no static or runtime prevention of this. -Steve
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 17:52:18 UTC, Walter Bright wrote:
 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:
 pointing at it is roughly 1/256. This problem is just about 
 eliminated with
 64-bit pointers.


Not generally true. This presumes that the heap is not in the lower region of the address space and that you don't use 64 bit ints on the stack.
 Generally, it is a bad idea to allocate such large blocks on 
 the GC heap. GC's work best when the size of the objects being 
 allocated is very small relative to the size of the heap space.

Generally not true. This is a deficiency of not having a smart allocator / precise scanning that use available meta information properly (obtained statically or by profiling).
 Fortunately, it's a mathematical inevitability that large 
 allocations relative to the GC size are rare, and so it isn't 
 much of a pain to handle them manually.

Programmer pain is not measured in number of instances, but in terms of model complexity. Ola.
May 12 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 02:12:44 -0400, Rainer Schuetze <r.sagitario gmx.de>  
wrote:

 On 13.05.2014 00:15, Martin Nowak wrote:
 On 05/11/2014 08:18 PM, Rainer Schuetze wrote:
 1. Use a scheme that takes a snapshot of the heap, stack and registers
 at the moment of collection and do the actual collection in another
 thread/process while the application can continue to run. This is the
 way Leandro Lucarellas concurrent GC works
 (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork"
 that doesn't exist on every OS/architecture. A manual copy of the  
 memory
 won't scale to very large memory, though it might be compressed to
 possible pointers. Worst case it will need twice as much memory as the
 current heap.

There is a problem with this scheme, copy-on-write is extremely expensive when a mutation happens. That's one page fault (context switch) + copying a whole page + mapping the new page.

I agree that this might be critical, but it is a one time cost per page. It seems unrealistic to do this with user mode exceptions, but the OS should have this optimized pretty well. > It's much worse > with huge pages (2MB page size). How common are huge pages nowadays?

I know this is coming from a position of extreme ignorance, but why do we have to do copy on write? What about pause on write? In other words, if a thread tries to write to a page that's being used by the collector, it pauses the thread until the page is no longer being used by the GC. This doesn't fix everything, but it's at least as good as today's GC, which preemptively pauses threads. My ignorance is that I have no idea if this is even possible, and I also have no idea how it would affect performance. -Steve
May 13 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer 
wrote:
 I know this is coming from a position of extreme ignorance, but 
 why do we have to do copy on write? What about pause on write?

Not sure how that will help? Pointers may still escape collection? (but you get that with transactional memory, on the cache level)
May 13 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 13 May 2014 at 16:14:07 UTC, Ola Fosheim Grøstad 
wrote:
 On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer 
 wrote:
 Not sure how that will help? Pointers may still escape 
 collection?

(as in not being traced, leading to freeing live memory)
May 13 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/5/14, 11:07 PM, HaraldZealot wrote:
 I have to say that all this discussion (more precisely the understanding
 on the side of key developers) make me very upset.

This is a misunderstanding of the situation. This is brainstorming. There has to be a public place in which ideas can be discussed freely, no matter how radical. Otherwise we'd need to create a cabal and only present finalized decisions to the public. We can't get to a community censorship so strong so as to disallow unencumbered discussion of radical ideas. All those who got up in arms about this thread, I compel you to reconsider. Andrei
May 06 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 5/6/14, 8:19 AM, HaraldZealot wrote:
 This is a misunderstanding of the situation. This is brainstorming.
 There has to be a public place in which ideas can be discussed freely,
 no matter how radical.

This is another misunderstanding :)). Not radicalism upset me, but proposal to create one more hole instead exist hole from man, that touched me solidness in code by his book. It were surprising. But it's only emotion.

Leave emotion aside and use reasoning. -- Andrei
May 06 2014
prev sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 06/05/2014 06:05, Manu via Digitalmars-d a écrit :
 On 6 May 2014 13:51, HaraldZealot via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?

Manu, can you direct me what is ARC? This abbreviation is very misgooglly.

Automatic reference counting, and solution used by Apple in Obj-C. There has been massive debate on the topic already, but it's generally been dismissed.

coworkers like that. Me I really love the Memory Analyzer. Fork?
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 13:51, HaraldZealot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?

Manu, can you direct me what is ARC? This abbreviation is very misgooglly.

Automatic reference counting, and solution used by Apple in Obj-C. There has been massive debate on the topic already, but it's generally been dismissed.
May 05 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
I have to say that all this discussion (more precisely the 
understanding on the side of key developers) make me very upset.

It's good that Andrei agreed with impossibility of the 
harebrained disallowing of the class destructors. But I was very 
surprise, that so thought go to such head, because such solution 
contradicts the D spirit totally. There are many language which 
are very popular and have many dark moments in their design. I 
(and I think not only me) go to the D not for its popularity, but 
for its clarity, power and SANITY (that bases on strong 
guaranties). The strong solutions found on the strong decision 
makes D itself. (Examples of such strong solutions: 
immutabilities, default unshareness, struct and class as distinct 
being). And way that leads us in state where stucts have dtors 
and classes haven't but allow struct with dtor as member and 
people have to call struct dtor manually, isn't D way. Because 
such way relies on programmers discipline, but how Andrei has 
written "If there one thing that decades of computing have taught 
us, it must be that discipline-oriented programming does not 
scale."[TDPL, p. 399].

Our negative filling flood out may be sane from psychologically 
view, but neither sane nor constructive for D future. For solving 
problem it's need its formulate. We have to state that current 
state (lack of structs' dtors call guaranty) is insane, the 
harebrained disallowing of the class destructors is insane too. 
And what is sane? If I properly understand philosophy of D, we 
need semiautomated (not full) resource manager with strong 
guaranty and good performance, and which automated mode covers 
the most part of use-case. It is the target. Garbage collection 
or reference counting or any possible third way is a detail 
therefor task and mean not a target. And one task, that lays on 
the way to target, is minimal rape of D2 language (even if 
solution will be D3), so IMO dtors (perhaps only for structs) 
must survive.

I notice that I view only part of problem, can anybody link or 
describe me completely state and problems of current garbage 
collection and other resource management? It help me in finding 
of existence solution (at least theoretical).

---
Alaksiej Stankievič
May 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 6 May 2014 at 06:07:41 UTC, HaraldZealot wrote:
 I notice that I view only part of problem, can anybody link or 
 describe me completely state and problems of current garbage 
 collection and other resource management? It help me in finding 
 of existence solution (at least theoretical).

A precise scanning GC is the only robust general solution. RC with weak pointers can only account for a subset of all possible models. But I agree with you, the language should be redesigned to have a GC friendly set of D constructs where FFI is followed by programmer guaranteed postconditions (specified by library authors). In other words the nogc appoach is not sufficient, a gc approach is needed.
May 05 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:
 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:
 This is nice, but on the face of it it's just this: an idea on how other
 people should do things on their free time. I'd have difficulty
 convincing
 people they should work that way. The kind of ideas that I noticed are
 successful are those that actually carry the work through and serve as
 good
 examples to follow.

There's imperfect but useful pull requests hanging around for years, extern(Obj-C) for instance, which may be useful as an experimental feature to many users, even if it's not ready for inclusion in the official feature list and support. I suspect it's (experimental) presence would stimulate further contribution towards D on iOS for instance; it may be an enabler for other potential contributors.

So it would be nice if you reviewed that code.

I don't really know anything about it... and that's not the point. I'm just suggesting by my prior email that some steps like creating an experimental space with a lower barrier to entry might encourage growth in the number of overall contributors, which I think was the basic flavour of the emails leading up to it.
 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?

There would be of course room as long as there's be one or more champions for it. Would that be something you'd be interested in?

I have no horse in that race, but I see it come up all the time, and it is something I am passively interested in. There's at least one DIP which received little attention afaict, it's an example of something that I think would probably manifest into code in an experimental space, but clearly couldn't be accepted as a language feature without lots of field time. In lieu of an experimental space, there will be no action. It's an interesting example actually. I think lots feel the DIP isn't really an effective solution, but nobody has the motivation or ideas to refine it. The DIP author clearly has no motivation to test it experimentally, but perhaps that's what it needs to progress? The DIP's shortcomings might be discovered by experimental users in the field? It's hard to know, but it's an example of the sort of things that may have a stifling effect on progress and contribution.
 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.

Interesting. Have you worked on a related proposal?

Not really, I've initiated numerous discussions which always seems to end at AST macros. The only other semi-reasonable idea I've had is the concept that tagging mixin templates as UDA's might be a practical angle, but it doesn't really make clean sense, creates a syntactic special case and also doesn't seem powerful enough, so I'm not with any solid proposal that I can imagine within the current language framework. There are presently bigger issues that keep me awake at night.
 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?

I'd say so, but we'd need initiative and quite a bit of work for such a platform. Would you be interested?

Well, in phobos, just approve 'exp' which has been raised countless times. I've got contributions that should be in exp, but instead, they're in limbo, and I've lost momentum and motivation since their completion is blocked by other issues, and I'm receiving no feedback from field testing. What happened to std.serislisation? There was motion there a year or so back... I was looking forward to it, and did some minor reviewing at the time. I wonder if that's an interesting case study? (I haven't looked) In the compiler... I may be interested, but I don't have any such compiler feature in mind to motivate the effort. I have no idea what an experimental feature platform should look like in the compiler, and if it were to exist, I have no such feature in mind to make use of it, but I have raised examples of others that have.
 Once they've made a single contribution of any sort, are they then
 more likely to continue making other contributions in the future
 (having now taken the time to acclimatise themselves with the
 codebase)?

I agree - and that applies to you, too.

Sure, but my point... is below.
 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.

Contributions make it into the compiler and standard library if and they are properly motivated, well done, and reviewed by the core team which is literally self-appointed. The key to being on the core team is just reviewing contributions. Have you considered looking at submissions that are "hanging around for years"?

Perhaps you misunderstood the point of my post. I've watched people make solid contributions that haven't gotten through. That is discouraging to others considering starting their own work, and for the person who has already put in the effort to continue to do so in the future. The Obj-C thing as an example. Granted, it's a huge feature and has extensive implications. The Authors have said themselves that they agree it's not 'ready' for inclusion... so, what? It sits and rots? I think it needs an experimental place to live and have people make use of it for what it is. If it's blocked by other unpopular issues that aren't receiving attention, perhaps it's presence will manifest the appropriate motivation to see those other unpopular issues resolved at some point? My point is that successful OSS seems to be about enabling the lowest-friction contribution, and my feeling right now, is that the barrier to entry is high. Whether that's true or not I can't really comment, but it's a perception, and the mental barrier inhibiting the first step is perhaps the most significant barrier of all. I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything about it, other than I'd really like to use D on iOS and that's a major hurdle. Also, the authors themselves said they recognise it's not 'ready'. But is it 'experimental'?
 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.

It's as easy as just reviewing stuff. Acta, non verba.

I've never felt I have any particular authority to comment on pulls that I have no experience or vested interest in. (I do occasionally comment on pull requests that I have some interest or knowledge in) I've also had some (hopefully useful) commentary in features that did make it; Win64, UDA's, some traits extensions, and lots of bug reports and fix confirmations. I'm plenty vocal and active on things I do feel I know about, but they're often pretty radical, unpopular, and rarely come even close to turning into code. I'm pretty certain that nothing left on my short list that I personally *really* care about will ever get pulled, even if I did do the work. There's a perfectly good pull there for not-virtual-by-default. No amount of beating will get that horse through, despite almost unanimous community support. That was... extremely discouraging, to say the least.
May 05 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 16:28, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 06/05/14 05:51, HaraldZealot wrote:

 Manu, can you direct me what is ARC? This abbreviation is very
 misgooglly.

Automatic Reference Counting. Like regular RC but the compiler automatically inserts calls to release/free.

And further, more importantly, automatically *eliminates* redundant calls to add/dec ref, which I think has much greater potential in D's type system than in Obj-C.
May 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 6 May 2014 at 06:39:19 UTC, Ola Fosheim Grøstad wrote:
 a GC friendly set of D constructs where FFI is followed by 
 programmer guaranteed postconditions (specified by library 
 authors).

Btw "postcondition" is the wrong term. In order to be robust need to specify how far into the GC heap a FFI can go while it is live. Consider callbacks from C to D. You don't know the layout of the stack frame, so you have to scan the stack assuming anything unknown is a pointer. But that is not sufficient, because the C code may be holding references to the GC heap in a malloc'ed struct. So, you have to scan all reachable malloc'ed memory too. At the end of the day some discipline is required for system level programming because the robust solutions are very expensive.
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
wrote:
 On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:
 Walter and I have had a long chat in which we figured our 
 current
 offering of abstractions could be improved. Here are some 
 thoughts.
 There's a lot of work ahead of us on that and I wanted to 
 make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls 
 from within
 the GC entirely. It makes for a faster and better GC, but the 
 real
 reason here is that destructors are philosophically bankrupt 
 in a GC
 environment. I think there's no need to argue that in this 
 community.

 The GC never guarantees calling destructors even today, so 
 this decision
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having
 fields that are structs with destructors) would need to 
 garner that by
 other means, such as reference counting or manual. We're 
 considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as 
 follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into
 RCSlice!T, defined inside object.d. That type would occupy 
 THREE words,
 one of which being a pointer to a reference count. That type 
 would
 redefine all slice primitives to update the reference count 
 accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).

 I foresee any number of theoretical and practical issues with 
 this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei

Honestly, that sounds like the entierly wrong apporach to me. Your approaching the problem in this way: "We can not implement a propper GC in D because the language design prevents us from doing so. So lets remove destructors to migate the issue of false pointers." While the approach should be. "The language does not allow to implement a propper GC (anything else then dirty mark & sweep), what needs to be changed to allow a implementation of a more sophisticated GC."

Couldn't agree more. Abandoning destructors is a disaster. Without destructors, you effectively have manual memory management, or rather, manual 'resource' management, which is basically the same thing, even if you have a GC. It totally undermines the point of memory management as a foundational element of the language if most things are to require manual release/finalisation/destruction or whatever you wanna call it.
 Also let me tell you that at work we have a large C# codebase 
 which heavily
 relies on resource management. So basically every class in 
 there inherits
 from C#'s IDisposable interface which is used to manually call 
 the finalizer
 on the class (but the C# GC will also call that finalizer!). 
 Basically the
 entire codebase feels like manual memory management. You have 
 to think about
 manually destroying every class and the entire advantage of 
 having a GC,
 e.g. not having to think about memory management and thus 
 beeing more
 productive, vanished. It really feels like writing C++ with C# 
 syntax. Do we
 really want that for D?

This is interesting to hear someone else say this. I have always found C# - an alleged GC language - to result in extensive manual memory management in practise too. I've ranted enough about it already, but I have come to the firm conclusion that the entire premise of a mark&sweep GC is practically corrupt. Especially in D. Given this example that you raise with C#, and my own experience that absolutely parallels your example, I realise that GC's failure extends into far more cases than just the ones I'm usually representing. I also maintain that GC isn't future-proof in essence. Computers grow exponentially, and GC performance inversely tracks the volume of memory in the system. Anything with an exponential growth curve is fundamentally not future-proof. I predict a 2025 Wikipedia entry: "GC was a cute idea that existed for a few years in the early 2000's while memory ranged in the 100's mb - few gb's, but quickly became unsustainable as computer technology advanced".

Java Azul VM GC was already handling 1 TB in 2010. http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf GC is not the only way of doing automatic memory management, but this ongoing discussion steams more from D's current GC status and respective phobia in C world, and less from what a modern GC is capable of. -- Paulo
May 06 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Paulo Pinto:

 Java Azul VM GC was already handling 1 TB in 2010.

 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern GC
 is capable of.

On the other hand I don't think you will have available in the D world the amount of work done on the Azul GC. This is the problem with looking at Java. Bye, bearophile
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 07:34:13 UTC, bearophile wrote:
 Paulo Pinto:

 Java Azul VM GC was already handling 1 TB in 2010.

 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, 
 but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern 
 GC
 is capable of.

On the other hand I don't think you will have available in the D world the amount of work done on the Azul GC. This is the problem with looking at Java. Bye, bearophile

Hence my remark that this whole discussion is more about finding working alternatives in the D world and less about what industrial class GCs are capable of. I was just doing a heads up, as people tend to misunderstand GCs. -- Paulo
May 06 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
 That experimental package idea that was discussed months ago
 comes to my mind again. Add that thing as exp.rational and
 have people report bugs or shortcomings to the original
 author. When it seems to be usable by everyone interested it
 can move into Phobos proper after the formal review (that
 includes code style checks, unit tests etc. that mere users
 don't take as seriously).

 As long as there is nothing even semi-official, it is tempting
 to write such a module from scratch in a quick&dirty fashion
 and ignore existing work.
 The experimental package makes it clear that this code is
 eventually going to the the official way and home brewed stuff
 wont have a future. Something in the standard library is much
 less likely to be reinvented. On the other hand, once a module
 is in Phobos proper, it is close to impossible to change the
 API to accommodate for a new use case. That's why I think the
 most focused library testing and development can happen in the
 experimental phase of a module. The longer it is, the more
 people will have tried it in their projects before formal
 review, which would greatly improve informed decisions.
 The original std.rationale proposal could have been in active
 use now for months!

+1 The same idea came to my mind yesterday too (that signalize that isn't void idea). And GNU/Linux community has proof that such model works. I mean Archlinux as example (but as I think other example exists). There are core, extra, community repositories and AUR (arch user repository). Today AUR contains 48489 packages and 11814 of which are orphan. But even orphanity isn't a sign of useless. Possibility to write PKGBUILD and place it in AUR has everybody and it is not such difficult. Only small part of these package goes to official repository, but this part EXISTS. And in my opinion this is the most rapid way for evolution. For example archlinux has the best support of D language through abundance of linux distributives (http://pastebin.com/tPKWS4ga and this is official repositories not in AUR). Oh, I've forgotten to mention all sorts of testing repositories. And D community have all means for evolute such way. We have dub and http://code.dlang.org/. It rests to do a little: create package (in D sense) which will contain packages that installed through dub; and some UI to vote, orphan and adopt package. There are example as how that I see: On http://code.dlang.org/ we can found package "rationals" (for example) with mark expstd. Than we install package with dub to some environment (it may be fakeroot, peruser import path or even perproject directory (the last likes me the least)) and address this package as "import expstd.rationals;" in our D projects. It's were good when dub or code.dlang.org counts installation. When some package counts enough installation and enough history (includes bug tracking), it become candidate to include in phobos. All trusted users (in therm of Archlinux) receive strong signal about and one of them review such package and become its auxiliary comaintainer. No one package can exist in official repository without maintainer. And some words about why "expstd". I think that with such package model comes time to add not only phobos in "namespace". There are many useful libraries which may not become part of phobos (linear algebra for example, or even GUI bindings), but should have some standardization and highway to distribution. I suggest package "extra" (in D sense) and "bind" (for deimos and perhaps not only deimos packages). Naturally they start their life as "expextra" and "expbind". In the future we will need some other package (in D sense), but for start that three are enough. As Andrei said "acta non verba". I promise that I make all, considering my life environment, to help and start this project. P.S. This theme will become independent. And this is yet another one argue, that forum forms not ideal to represent news and current state of community. Important things sink often in other topics. P.P.S. My apologies for my poor English.
May 06 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
 The major issue with the garbage collector is that it's not 
 guaranteed to run a collection. When a collection is run the GC 
 will call the destructors for the objects it collects. If 
 there's no guarantee a collection is run there can be no 
 guarantee that destructors are called. A collection is usually 
 run when allocating new memory and there's not enough memory 
 available.

So it seems that I have understood more than I've thought. If really problem rely on this than an idea came to my mind yesterday. What about separate destructors call and garbage collections mechanism? This idea reflects current D state with postblit constructor in structures. Raw memory allocation, default initialization (with T.init field) and after all that calling constructor (especially postblit). In therms of destructors it sound as: after becoming object or structure needless some automatic mechanism call cascading destructor, but memory still not fried, and GC collects memory in its time. Yes it's sounded monstrous, but perhaps when which mechanism will done only self work, each of them can be lighter. And you always can call some kind of finalizator manually for performance (some kind of flag can exist to protect from double call of destructors).
May 06 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 6 May 2014 at 06:07:41 UTC, HaraldZealot wrote:
 I have to say that all this discussion (more precisely the 
 understanding on the side of key developers) make me very upset.

 It's good that Andrei agreed with impossibility of the 
 harebrained disallowing of the class destructors. But I was 
 very surprise, that so thought go to such head, because such 
 solution contradicts the D spirit totally. There are many 
 language which are very popular and have many dark moments in 
 their design. I (and I think not only me) go to the D not for 
 its popularity, but for its clarity, power and SANITY (that 
 bases on strong guaranties). The strong solutions found on the 
 strong decision makes D itself. (Examples of such strong 
 solutions: immutabilities, default unshareness, struct and 
 class as distinct being). And way that leads us in state where 
 stucts have dtors and classes haven't but allow struct with 
 dtor as member and people have to call struct dtor manually, 
 isn't D way. Because such way relies on programmers discipline, 
 but how Andrei has written "If there one thing that decades of 
 computing have taught us, it must be that discipline-oriented 
 programming does not scale."[TDPL, p. 399].

Actually, IMO the clarity and sanity you talk about are a strong argument _for_ the deprecation of destructors for GC objects: Premise 1: Destructors are for things that _must_ be done when an object is destroyed. Destroying (or leaking) an object without calling its destructor can (and often will) lead to wrong program behavior and is therefore an error. Premise 2: Tracing GCs are by their nature non-deterministic and can not guarantee proper destruction of objects. This is true even if full type information about the managed objects is available (which is currently the case only for classes), as long as they are non-precise (which, realistically, in a systems programming language they will always be at least to some degree). Conclusion: It's an error to have objects with destructors be managed by the GC. Per Andrei's quote about "discipline-oriented programming", this should therefore be made impossible. (Apart from that, there are tons of things that you aren't allowed to do in a destructor called from the current GC, but there's currently no way to detect them statically => again "discipline-oriented programming".) Of course, how exactly this is achieved is up for debate. Also, maybe one of the premises is false...
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. The trouble is, GC runs more frequently as the amount of available free memory decreases, and trace&collect takes a long time, long enough that realtime software typically can't tolerate the pause. If the pause is super rare (minutes/hours apart), maybe it's acceptable, but if it's closer to every call to alloc (ie, when there is little/no free memory), then you realise it's fundamentally incompatible. Then consider that this is typically the default operating state of embedded systems; to have no free memory at runtime, and therefore D is effectively incompatible with a subset of computers... specifically, games consoles; a gigantic industry, and one of the few that still absolutely relies on native code where D is particularly compelling. If people want to call D a native systems language, this seems like a serious conflict with that idea. A major subset of the language is incompatible with realtime or embedded use (surely among the most compelling use cases remaining for native code these days!). Usually the argument is that I represent a niche, and therefore I should accept the restrictions of my environment, and revert to full manual memory management for my work, while letting everyone else have their fun. That is a broken argument, not only because it leaves very little motivation to leave C++, but also because regardless of whether I adopt such restrictions, library authors won't, and we don't live in the 1980's where I can write a few lines of self-contained assembly and call it a program. To say I can't depend on libraries is practically absurd. Games are gigantic software projects, and depend on heaps of libraries. The GC effectively eliminates libraries from embedded/realtime users, which I see as an absolute deal-breaker alone, irrespective of a further bunch more restrictions it places on that subset of users within their own code :/ I'll be happy with any solution that works, really. But as I see it, reference counting is the only garbage collection technology I know which will reliably clean up eagerly as things fall out of scope (also addressing this issue with destructors as bonus), and as such, it seems the logical solution. ARC is a demonstrable success in Obj-C, and works well in realtime and embedded systems. D has type system improvements which would theoretically allow for significant improvements over Obj-C.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 17:16, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d
 wrote:
 Java Azul VM GC was already handling 1 TB in 2010.

Is D compatible with this GC? And how does it go in an environment with 128mb of ram, and no free memory?
 http://qconsf.com/sf2010/dl/qcon-sanfran-2010/slides/GilTene_GCNirvanaHighThroughputAndLowLatencyTogether.pdf

 GC is not the only way of doing automatic memory management, but
 this ongoing discussion steams more from D's current GC status
 and respective phobia in C world, and less from what a modern GC
 is capable of.

It's not a 'phobia', it's an incompatibility. Show me a 'modern' GC that's compatible with realtime+embedded, and also compatible with D, and then we can talk... I'm sure there's a very good reason nobody has raised such a technology for consideration. Framing the C world in this way is completely unfair. I've spent years here trying to reconcile my relationship with the GC. I've actually defended the GC on numerous occasions. I don't want to reject the GC, it's very convenient... but I think it's reached the point where I have to face the music. I want to know the plan, and if it remains incompatible with my industry, then I can't reasonably continue to advocate D in my field, can I? If it were my language, I know what I'd do. It may require some radical changes, and I would do it because I simply had _no other choice_. Please, make suggestions! It's been years, and I'm still waiting to hear another realistic proposal.
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/ http://www.is2t.com/products/ http://www.mountaineer.org/netmf-for-stm32/ Just a small sample of the partners providing the said support. -- Paulo
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link or describe
 me completely state and problems of current garbage collection and other
 resource management? It help me in finding of existence solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/ http://www.is2t.com/products/ http://www.mountaineer.org/netmf-for-stm32/ Just a small sample of the partners providing the said support.

True, much embedded work isn't realtime. There are a lot of purpose devices where performance is not particularly important, probably correctness is, hence Java may be popular in this environment, and safe D may have a great application here. In another realm, if we're talking about really small systems (microcontrollers with memory in the megabytes), the tendency to rely on libraries is reduced significantly, since you probably can't afford the memory for the libs. This environment is the one where it's realistic to say "tag main() with nogc, and work from there", ie, ban the GC throughout the whole project. However, another very popular use for embedded systems IS realtime software. Games consoles are the obvious one here, but also POS/retail devices, kiosks, televisions, PVR's/dvd/bluray and other entertainment related devices, cars and automobile HUD's, music players, advertising displays, etc. There's no shortage of realtime devices in the embedded space. Notably, I didn't say 'phones'. Although I think they do generally fall into this category, I think they're drifting away. Since they run full OS stack's, it's typical to have unknown amounts of free memory for user-space apps and virtual memory managers that can page swap, so having excess memory overhead probably isn't such a big deal. It's still a major performance hazard though. Stuttering realtime applications is never a professional look, and Android suffers chronically in this department compared to iOS. I spent my last weekend trying to get a PS4 game (built with Unity; uses mono as a scripting environment) running at an acceptable frame rate. I didn't succeed, and a significant factor was the JS/C# garbage collector. We'll probably need to cut content from the game, such that it leaves plenty of 'excess' resource available to absorb the spikes it causes when they occur. What a waste of the machine! Note, C#'s GC is much better than D's. It does seem to run in the realm of single-digit milliseconds.
May 06 2014
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d 
wrote:
 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It
 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.

This bit right here reminds me of something the Linux kernel has called "staging" [1]. It's basically what you describe: a subtree within main source tree for things to be publicly available while they finish baking, with the understanding that you're going to continue working on it and hopefully get it promoted to first-class citizen.
 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.

I know I and several other people are still interested in std.simd... :( -Wyatt [1] http://lwn.net/Articles/324279/
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 12:05:10 UTC, Manu via Digitalmars-d 
wrote:
 On 6 May 2014 21:39, Paulo Pinto via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
 wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/ http://www.is2t.com/products/ http://www.mountaineer.org/netmf-for-stm32/ Just a small sample of the partners providing the said support.

True, much embedded work isn't realtime. There are a lot of purpose devices where performance is not particularly important, probably correctness is, hence Java may be popular in this environment, and safe D may have a great application here. In another realm, if we're talking about really small systems (microcontrollers with memory in the megabytes), the tendency to rely on libraries is reduced significantly, since you probably can't afford the memory for the libs. This environment is the one where it's realistic to say "tag main() with nogc, and work from there", ie, ban the GC throughout the whole project. However, another very popular use for embedded systems IS realtime software. Games consoles are the obvious one here, but also POS/retail devices, kiosks, televisions, PVR's/dvd/bluray and other entertainment related devices, cars and automobile HUD's, music players, advertising displays, etc. There's no shortage of realtime devices in the embedded space.

I can provide more examples for JVMs in all areas you list. However those JVMs do cheat a bit, as they provide a fine grain control over GC and also allow for region based allocations and immortal memory.
 Notably, I didn't say 'phones'. Although I think they do 
 generally
 fall into this category, I think they're drifting away. Since 
 they run
 full OS stack's, it's typical to have unknown amounts of free 
 memory
 for user-space apps and virtual memory managers that can page 
 swap, so
 having excess memory overhead probably isn't such a big deal. 
 It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.

 I spent my last weekend trying to get a PS4 game (built with 
 Unity;
 uses mono as a scripting environment) running at an acceptable 
 frame
 rate. I didn't succeed, and a significant factor was the JS/C# 
 garbage
 collector. We'll probably need to cut content from the game, 
 such that
 it leaves plenty of 'excess' resource available to absorb the 
 spikes
 it causes when they occur. What a waste of the machine!
 Note, C#'s GC is much better than D's. It does seem to run in 
 the
 realm of single-digit milliseconds.

Just as info regarding your experience, please note that both Android and Unity have not so good GC implementations. Dalvik's GC is hardly touched since Android 2.3, hopefully it will be improved with the new ART runtime. Unity's Mono implementation is a legacy one, based on the old Novell Mono implementation as Unity is not willing to pay the royalties Xamarin is asking for modern Mono implementations. As for the gaming world, I don't really have the required experience to talk much about it. Just doing the heads up that D GC != GC that many of us use. -- Paulo
May 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 6 May 2014 at 11:35:53 UTC, Manu via Digitalmars-d 
wrote:
 Please, make suggestions! It's been years, and I'm still 
 waiting to hear another realistic proposal.

1. Use nogc and C++ style allocation for hard real time threads (render logic). 2. Use segmented GC for soft real time threads (world logic). Lobby for more restrictions/better runtime support from the compiler for fast precise single-pass regional (local) garbage collection. As a bonus you will get faster exception handling too if identifying stack frames is faster. Lobby for restrictions: - C may only call back to nogc D-functions so you can collect without C-frames on the stack. - Allow omitting framepointers, global call optimization and stack optimizations for nogc nothrow threads. etc. The key to performance is to allow specifying constraints that the compiler and runtime can assume to hold. The problem with the D community is a lack of priority of system level programming which is at odds with "safe", "no discipline needed" and "Java/C# convinience". No constraints == no performance.
May 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 6 May 2014 at 12:21:55 UTC, Paulo Pinto wrote:
 As for the gaming world, I don't really have the required 
 experience to talk much about it. Just doing the heads up that 
 D GC != GC that many of us use.

I very much appreciate the links you provide about different GC solutions, and GC can be appropriate for real time situations where you don't want to maximize resource utilization or where you have predictable downtime to collect. However the approach taken by appliances (where the application reliability is more important than getting the highest possible execution speed) or business solutions such as Azul Zing (where scalability is more important than hardware cost) is not really suitable for system level programming. It is suitable for a wide range of applications, that is true. You can have GC with predictable destructors, but you have to design the language and runtime around it. You can have real time GC, but you need a dedicated GC and structure your program to realize the potential. I think the only realistic road for D is to imposing/infer constraints that makes certain that execution paths is compatible with desirable runtime properties and available runtime support for that specific thread. There is no silver bullet, but the implementation driven design that D seems to be undergoing is affecting the resulting language/run-time design in a bad way, IMO.
May 06 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 May 2014 at 14:29:01 UTC, Andrei Alexandrescu wrote:
 On 5/5/14, 11:07 PM, HaraldZealot wrote:
 I have to say that all this discussion (more precisely the 
 understanding
 on the side of key developers) make me very upset.

This is a misunderstanding of the situation. This is brainstorming. There has to be a public place in which ideas can be discussed freely, no matter how radical. Otherwise we'd need to create a cabal and only present finalized decisions to the public. We can't get to a community censorship so strong so as to disallow unencumbered discussion of radical ideas. All those who got up in arms about this thread, I compel you to reconsider. Andrei

+1 with(devilsAdvocateMode) { I think people are a bit used to the situation you describe, hence the misunderstanding w.r.t. the approach you're trying here }
May 06 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
 This is a misunderstanding of the situation. This is 
 brainstorming. There has to be a public place in which ideas 
 can be discussed freely, no matter how radical.

This is another misunderstanding :)). Not radicalism upset me, but proposal to create one more hole instead exist hole from man, that touched me solidness in code by his book. It were surprising. But it's only emotion.
May 06 2014
prev sibling next sibling parent "HaraldZealot" <harald_zealot tut.by> writes:
 This is another misunderstanding :)). Not radicalism upset me, 
 but proposal to create one more hole instead exist hole from 
 man, that touched me solidness in code by his book. It were 
 surprising. But it's only emotion.

But yes, It may possible as troll mode in brainstorming.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 22:30, Michel Fortin via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2014-05-06 12:04:55 +0000, Manu via Digitalmars-d
 <digitalmars-d puremagic.com> said:

 Notably, I didn't say 'phones'. Although I think they do generally
 fall into this category, I think they're drifting away. Since they run
 full OS stack's, it's typical to have unknown amounts of free memory
 for user-space apps and virtual memory managers that can page swap, so
 having excess memory overhead probably isn't such a big deal. It's
 still a major performance hazard though. Stuttering realtime
 applications is never a professional look, and Android suffers
 chronically in this department compared to iOS.

Note that iOS has no page swap. Apps just get killed if there isn't enough memory (after being sent a few low memory signals they can react on, clearing caches, etc.). Apps that takes a lot of memory cause other apps in the background to be killed (and later restarted when they come to the foreground).

I'm just saying why those platforms aren't likely to suffer the no-free-memory environment, and therefore can't really be considered proper embedded systems like video games consoles.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 6 May 2014 22:17, Wyatt via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via Digitalmars-d wrote:
 The Obj-C thing as an example. Granted, it's a huge feature and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? It

 sits and rots?  I think it needs an experimental place to live
 and have people make use of it for what it is.

This bit right here reminds me of something the Linux kernel has called "staging" [1]. It's basically what you describe: a subtree within main source tree for things to be publicly available while they finish baking, with the understanding that you're going to continue working on it and hopefully get it promoted to first-class citizen.
 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that nothing
 left on my short list that I personally *really* care about will
 ever get pulled, even if I did do the work.

I know I and several other people are still interested in std.simd... :(

It hasn't gone away. It's a perfect example of where I got blocked by some things, and since it wasn't in an accessible location as an in-progress 'exp' module, it's barely used, and I've never gotten any feedback. Perhaps the blocks are lifted now (I haven't checked lately), but clearly I started working on other things and lost momentum, and I think it's safe to attribute this almost entirely to the fact it exists in my fork where nobody will find it, rather than 'exp', where people can still report usage experience, feedback, and keep me on track.
May 06 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu wrote:
 On 5/6/14, 8:43 AM, Manu via Digitalmars-d wrote:
 On 6 May 2014 22:17, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 06:39:45 UTC, Manu via 
 Digitalmars-d wrote:
 The Obj-C thing as an example. Granted, it's a huge feature 
 and
 has extensive implications. The Authors have said themselves
 that they agree it's not 'ready' for inclusion... so, what? 
 It

 sits and rots?  I think it needs an experimental place to 
 live
 and have people make use of it for what it is.

This bit right here reminds me of something the Linux kernel has called "staging" [1]. It's basically what you describe: a subtree within main source tree for things to be publicly available while they finish baking, with the understanding that you're going to continue working on it and hopefully get it promoted to first-class citizen.
 I'm plenty vocal and active on things I do feel I know about,
 but they're often pretty radical, unpopular, and rarely come
 even close to turning into code. I'm pretty certain that 
 nothing
 left on my short list that I personally *really* care about 
 will
 ever get pulled, even if I did do the work.

I know I and several other people are still interested in std.simd... :(

It hasn't gone away. It's a perfect example of where I got blocked by some things, and since it wasn't in an accessible location as an in-progress 'exp' module, it's barely used, and I've never gotten any feedback. Perhaps the blocks are lifted now (I haven't checked lately), but clearly I started working on other things and lost momentum, and I think it's safe to attribute this almost entirely to the fact it exists in my fork where nobody will find it, rather than 'exp', where people can still report usage experience, feedback, and keep me on track.

I can't seem to find simd on our dub site http://code.dlang.org/search?q=simd. Did you put it there under another name? Andrei

I don't think it's there. I would love if it was, it would be a great addition to the D ecosystem.
May 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:
 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu
 I can't seem to find simd on our dub site 
 http://code.dlang.org/search?q=simd. Did you put it there 
 under another name?

 Andrei

I don't think it's there. I would love if it was, it would be a great addition to the D ecosystem.

Which only confirms my suspicion that those who complain about lack of experimental Phobos package are likely to simply not use dub.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:
 On 6 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/5/14, 8:19 PM, Manu via Digitalmars-d wrote:
 On 5 May 2014 14:09, Andrei Alexandrescu via Digitalmars-d

So it would be nice if you reviewed that code.

I don't really know anything about it... and that's not the point. I'm just suggesting by my prior email that some steps like creating an experimental space with a lower barrier to entry might encourage growth in the number of overall contributors, which I think was the basic flavour of the emails leading up to it.

Oh but that's very much the point. "Creating" is the right word (and the gerund form is delicious). In order for the "creating" to happen someone would need to do the "creation" - and of course the subsequent related work. You frame the matter as something "they" should do, when the reality is you're very much a part of "they".

Throughout this email, you seem to have completely misunderstood my intent. I haven't asked you to do anything. I originally replied in support of prior comments that raised yet again the idea of having an experimental space with significantly relaxed contribution policy. I was supporting the notion that an agreement to the existence of an experimental space might be a valuable thing. It requires nothing more than discussion and agreement that it is actually something the community thinks is a good idea.
 The idea of an experimental/unstable package has been discussed a number of
 times. The conclusion I seem to remember is that currently with dub
 available there's a really simple way to experiment with things in a
 comfortable and decentralized way.

Is that something that's general knowledge, or a strategy that people are already using? This is the first time the idea has ever occurred to me.
 That said, it's not impossible that distributing things in an
 "std.experimental" package along with the official distribution gives them
 an extra umph. We could arrange for that to happen. Is this what you think
 would make the difference?

Yes. This is basically the entire point of my post.
 What about AST macros? It seems to me that this is never going to be
 explored and there are competing proposals, but I wonder if there's
 room for experimental implementations that anyone in the community can
 toy with?

There would be of course room as long as there's be one or more champions for it. Would that be something you'd be interested in?

I have no horse in that race, but I see it come up all the time, and it is something I am passively interested in.

Same here. I'm too busy working in and on other aspects of D (which I believe are more important and more urgent) to have another project added to my plate. I'd be in your debt if you could clarify to me that AST macros are not something you believe I must work on right now.

I haven't suggested you need to work on anything. Other people with ideas on the topic have appeared to be interested in working on it, and it seems they need a place where they can produce the experimental system and make it available to everyone. I wonder if that would motivate them to do it. You can see how there's a distinction between contributors getting a system to an experimental point and then letting people have at it while it bakes, and getting it to the point it's perfect and ready for formal review and acceptance, prior to people really having convenient access to it to try it out? I'm not sure how you perceive that I'm saying you need to do something in this scenario, other than consider some changes in contribution policy as part of a committee.
 There's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

What makes dub inappropriate as an experimental space?

I don't know. But at face value, I'd suggest that it's a concept that's quite foreign to Windows users. It never occurred to me until right now. If that's a popular approach, then the the dub package listing probably needs greater visibility. It well maintained? dub should almost certainly be bundled with DMD if this is to become a standard approach. This only addresses libs though, not experimental compiler features. That needs an agreed policy for enabling/disabling such features, and I don't see how that can escape living in the DMD repo?
 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress? The
 DIP's shortcomings might be discovered by experimental users in the
 field? It's hard to know, but it's an example of the sort of things
 that may have a stifling effect on progress and contribution.

I think DIPs are a marked improvement over informal discussions on the mailing lists. For them to work well, two ingredients are needed: champions and reviewers. In some cases that did happen, in others it didn't.

Wholesale inclusion is different than getting something in as an experimental feature though. If there's a proper policy for experimental features, and there is a much lower standard against which they are included, then there'll possibly be more people experimenting. 10x so for compiler features, since most people don't build their own compiler; unless experimental compiler features are included, they'll never be tested by end-users. I'm not suggesting any change to the DIP system, it's fine, and a good starting point for new development. But I think there's a lot of DIP's there which may be interesting features, and I wonder if there would be a difference in perceived friction if they knew there was an experimental stage available where the community can initially try new things out, offer useful feedback, and motivating the push to the goalpost. Ideally, experimental features should be available to all D users for maximum possible user testing. Only people on this forum would ever find someone's personal fork, merge with their local source and compile themselves a compiler. Most people even here would never do that.
 UDA's are super-useful, but they're still lacking the thing to really
 set them off, which is the ability to introduce additional boilerplate
 code at the site of the attribute.

Interesting. Have you worked on a related proposal?

Not really, I've initiated numerous discussions which always seems to end at AST macros. The only other semi-reasonable idea I've had is the concept that tagging mixin templates as UDA's might be a practical angle, but it doesn't really make clean sense, creates a syntactic special case and also doesn't seem powerful enough, so I'm not with any solid proposal that I can imagine within the current language framework. There are presently bigger issues that keep me awake at night.

Same here. So then I think you'd find it reasonable that I asked you to clarify these are not things you believe I need to work on.

I'm not asking you to work on anything.
 What I'm trying to get across here is that your and my role and
 opportunities are very much comparable. Anyone's clout within the D
 community is proportional to the work they're doing on D, and even Walter's
 role would be reduced to an honorific one if he ceased spending time working
 on D.

 My pedal is sealed to the floor already. I'm tapped out. Walter's too.
 Meaning that anything work you'd find fit to give us to do would need to
 come at other work's expense. In my case, that would mean overseeing four D
 projects at work and allocators on my free time. Walter's last piece of work
 is interfacing with C++ namespaces, which is very necessary for adoption. He
 and I have gotten pretty good at investing our time efficiently, but at some
 point there's only so many hours in a day and we need talented people -
 people like you - to help making things happen.

I'm not sure what you think I'm asking you to do. I'm just suggesting to consider a change in policy wrt experimental contributions. My suggestions are: Allow people to commit to an 'exp' phobos package; reduce the approval burden for people that want to make experimental modules available. This is a no-action action. Just a decision that it's a good idea or not. I imagine criteria for inclusion as an experimental package might be something like; as long as a decent amount of people agree that it is a feature that should eventually be in phobos, and that the initial API is sensible and reasonably conformant, let it exist in exp while it bakes, prior to the lengthy formal review process. Allow people to commit experimental features to the compiler, like Obj-C, so people can try it. There will need to be some agreement on how experimental compiler features are to be enabled/disabled in a standard way, and obviously approval would be subject to reasonable community approval. Again, I don't think that implies any work on your part. The difference is, there's no massive threads and debates about what something should or shouldn't be, and people can get to work, try things, make it available, get feedback, etc, prior to a formal review process. I'd also suggest that experimental code in phobos or the compiler not be a required burden for GDC/LDC if it runs into trouble with their update process. Policy should allow them to stub it out, and refer it back to the exp author.
 I reckon there's a good chance that creating a proper platform for
 experimental features would also have an advantage for community
 building and increase contribution in general. If new contributors can
 get in, have some fun, and start trying their ideas while also being
 able to share them with the community for feedback without fear
 they'll just be shot down and denied after all their work... are they
 not more likely to actually make a contribution in the first place?

I'd say so, but we'd need initiative and quite a bit of work for such a platform. Would you be interested?

Well, in phobos, just approve 'exp' which has been raised countless times. I've got contributions that should be in exp, but instead, they're in limbo, and I've lost momentum and motivation since their completion is blocked by other issues, and I'm receiving no feedback from field testing.

I'm fine with adding an "exp" package to phobos if you believe that that's what's stopping things from happening. But do you really?
 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)

I could ask you the very same thing. What happened to std.serialization? Do you want me to work on that as well?

I haven't asked you to work on anything! I just raised it as a potential example where something might have gone off the rails because there wasn't an effective experimental place for it to go while baking.
 In the compiler... I may be interested, but I don't have any such
 compiler feature in mind to motivate the effort.
 I have no idea what an experimental feature platform should look like
 in the compiler, and if it were to exist, I have no such feature in
 mind to make use of it, but I have raised examples of others that
 have.

Well then someone else would need to do all that work, so you'd need to convince them that it's a good idea.

I don't know what work you refer to. Maybe there is work to do to support this? I don't know. It seems more like a policy decision to me.
 I personally feel the perceived unlikeliness of any experimental
 contribution being accepted is a massive deterrence to making compiler
 contributions in the first place by anyone other than the most serious
 OSS advocates.

Contributions make it into the compiler and standard library if and they are properly motivated, well done, and reviewed by the core team which is literally self-appointed. The key to being on the core team is just reviewing contributions. Have you considered looking at submissions that are "hanging around for years"?

Perhaps you misunderstood the point of my post. I've watched people make solid contributions that haven't gotten through. That is discouraging to others considering starting their own work, and for the person who has already put in the effort to continue to do so in the future.

Contributions haven't gotten through because there are not enough people to review them. People like you. We have an inflation of contributions, or if you wish a deflation of reviews.

And again, maybe they would have more inertia if they made it 'in' at an earlier point, as an experimental, and they built a community of users behind the feature proving it's value, and providing feedback and motivation? Experimentals shouldn't require extensive reviews. Basic sanity checks and code quality compliance are much easier for any uninvested 3rd party to review/verify.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?
 I think it needs an experimental place to live and have people make
 use of it for what it is. If it's blocked by other unpopular issues
 that aren't receiving attention, perhaps it's presence will manifest
 the appropriate motivation to see those other unpopular issues
 resolved at some point?

(I assume you refer to http://wiki.dlang.org/DIP43; couldn't find associated pull requests.) I don't know. I am not using Objective C, and I think it would be more efficient to have people who already have Objective C expertise to author and review the contribution. Would I be glad to see good interfacing of D with Objective C? Of course!

But, the question is, should it be accepted as an 'experimental' (unfinished) feature, that people can access on some flag or whatever? This particular one has been resurrected multiple times, but always gets stuck. Would there be more pressure to finish the job if people were actively using it, despite the fact it's yet incomplete?
 My point is that successful OSS seems to be about enabling the
 lowest-friction contribution, and my feeling right now, is that the
 barrier to entry is high. Whether that's true or not I can't really
 comment, but it's a perception, and the mental barrier inhibiting the
 first step is perhaps the most significant barrier of all.

Forgive me but from where I stand you seem completely wrong. There are currently 190 open pull requests in our github projects. The primary reason that keeps them from being merged is good peer review from people like you and me. So currently the bottleneck is on the receiving side, not the initiating side of contributions.

This is probably true. I can imagine that I'd feel much more confident and qualified to review features for inclusion as experimentals, since it should only require quality compliance and sanity checking, and not deep understanding of the feature itself, potential interaction with other features or code that I'm unfamiliar with, etc.
 You can't claim incompetence on 190 distinct matters. You are fluent in D
 and a good engineer who could tell good design from bad.

I'm not even sure what the process it... if I go through and "LGTM" a bunch of pulls, does someone accept my judgement and click the merge button? You can see why I might not feel qualified to do such a thing? I'll criticise at least 10 pull requests in the morning. I'll be curious to see what happens.
 I can't review the Obj-C patch. 1, it's huge, 2, I don't know anything
 about it, other than I'd really like to use D on iOS and that's a
 major hurdle.

Surely there must be something that should catch your fancy among the remaining 189 patches.

We'll find out... when I wake up.
 Also, the authors themselves said they recognise it's
 not 'ready'. But is it 'experimental'?

Most likely. Have you tried to build their fork?

I download DMD from dlang.org. It would be nice if end-users could try it out under these conventional usage scenarios. Are they welcome to have the feature enabled on a switch or pragma or something, and merge into trunk such that it is available to end-users in the next release?
 I have no prior experience with OSS, and it's certainly
 a factor that's kept me at arms length.

It's as easy as just reviewing stuff. Acta, non verba.

I've never felt I have any particular authority to comment on pulls that I have no experience or vested interest in. (I do occasionally comment on pull requests that I have some interest or knowledge in) I've also had some (hopefully useful) commentary in features that did make it; Win64, UDA's, some traits extensions, and lots of bug reports and fix confirmations.

That's a great start, thanks. Keep them coming.
 I'm plenty vocal and active on things I do feel I know about, but
 they're often pretty radical, unpopular, and rarely come even close to
 turning into code. I'm pretty certain that nothing left on my short
 list that I personally *really* care about will ever get pulled, even
 if I did do the work.
 There's a perfectly good pull there for not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely discouraging, to
 say the least.

What can we do (short of pulling that) to be more encouraging of you to do good work?

Weigh in on policy relating to an experimental feature space in the compiler, and in phobos? How should experimental compiler features be enabled? pragma? switches? How should experimental libraries be presented? phobos 'exp'? dub? These are just conversations and decisions. It's this simple; I (and apparently others, it wasn't my idea) think an experimental space would be good. Do you agree? It's an idea that has always been rejected in the past. My logic is, such a thing may theoretically reduce friction and lower the bar for new development, and maybe stimulate new contributions and ideally contributors.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
Missed one... >_<

On 7 May 2014 01:46, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/5/14, 11:39 PM, Manu via Digitalmars-d wrote:
 Well, in phobos, just approve 'exp' which has been raised countless
 times. I've got contributions that should be in exp, but instead,
 they're in limbo, and I've lost momentum and motivation since their
 completion is blocked by other issues, and I'm receiving no feedback
 from field testing.

I'm fine with adding an "exp" package to phobos if you believe that that's what's stopping things from happening. But do you really?

Maybe. Surely I've demonstrated that I think it's a definite possibility, and worth a shot. But I don't see why this is my question to answer. This requires consensus. It's been raised and rejected numerous times in the past... why? And why would those that rejected it previously have changed their mind right now? This isn't meant to be a discussion between the 2 of us.
May 06 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Tuesday, 6 May 2014 at 15:43:22 UTC, Manu via Digitalmars-d 
wrote:
 I think it's safe to attribute this almost entirely to
 the fact it exists in my fork where nobody will find it, rather 
 than
 'exp', where people can still report usage experience, 
 feedback, and
 keep me on track.

As other people said, the dub registry is that 'exp' place.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 May 2014 03:32, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:
 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu
 I can't seem to find simd on our dub site
 http://code.dlang.org/search?q=simd. Did you put it there under another
 name?

 Andrei

I don't think it's there. I would love if it was, it would be a great addition to the D ecosystem.

Which only confirms my suspicion that those who complain about lack of experimental Phobos package are likely to simply not use dub.

As I said before, dub has never even occurred to me. No windows user is likely to naturally think to use a package manager :/ It's sitting in my phobos fork. Isn't that where everyone keeps their developments? If people are pushing dub, then it should really be installed with DMD.
May 06 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 May 2014 at 17:47:31 UTC, Manu via Digitalmars-d 
wrote:
 Is that something that's general knowledge, or a strategy that 
 people
 are already using? This is the first time the idea has ever 
 occurred
 to me.

 I don't know. But at face value, I'd suggest that it's a concept
 that's quite foreign to Windows users. It never occurred to me 
 until
 right now.

It never occurred to you that people's libraries would be published as part of a centralised repository with a tool that manages dependencies? It's pretty common-place in a variety of languages. (https://rubygems.org/ https://pypi.python.org/pypi http://www.cpan.org/ etc...). With regard to experimental modules, it's good to get them some real-world exposure first via dub even if there was some "staging" package in phobos.
 If that's a popular approach, then the the dub package listing
 probably needs greater visibility. It well maintained?

 dub should almost certainly be bundled with DMD if this is to 
 become a
 standard approach.

It's been discussed a few times. I think greater stability was wanted before bundling.
 This only addresses libs though, not experimental compiler 
 features.
 That needs an agreed policy for enabling/disabling such 
 features, and
 I don't see how that can escape living in the DMD repo?

Good point.
May 06 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 6 May 2014 at 18:00:14 UTC, Walter Bright wrote:
 On 5/5/2014 11:39 PM, Manu via Digitalmars-d wrote:
 There's a perfectly good pull there for 
 not-virtual-by-default. No
 amount of beating will get that horse through, despite almost
 unanimous community support. That was... extremely 
 discouraging, to
 say the least.

If you look, I have authored some pull requests that I think are very good, but are twisting in the wind and have for some time. Andrei has some, too. We all do. I understand that we'd all like our efforts to find quick approval and karma, but it isn't always going to happen, and we need to not let that discourage us.

Generally, championing something you believe in tends to require a great deal of personal effort into trying to make it a reality.
May 06 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:
 On 7 May 2014 03:32, Dicebot via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 15:52:10 UTC, John Colvin wrote:
 On Tuesday, 6 May 2014 at 15:48:59 UTC, Andrei Alexandrescu
 I can't seem to find simd on our dub site
 http://code.dlang.org/search?q=simd. Did you put it there 
 under another
 name?

 Andrei

I don't think it's there. I would love if it was, it would be a great addition to the D ecosystem.

Which only confirms my suspicion that those who complain about lack of experimental Phobos package are likely to simply not use dub.

As I said before, dub has never even occurred to me. No windows user is likely to naturally think to use a package manager :/

Well, dub and code.dlang.org is a good way for them to start :p
 It's sitting in my phobos fork. Isn't that where everyone keeps 
 their
 developments?

not once they're done enough to be useful. See all of the projects: http://code.dlang.org/
May 06 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:
 As I said before, dub has never even occurred to me. No windows 
 user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps 
 their
 developments?

 If people are pushing dub, then it should really be installed 
 with DMD.

DUB should feel pretty natural on any operating system if you are a Python (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or PHP (PEAR) programmer. Although PyPi I know doesn't work as well on Windows. (It doesn't explain to you clearly how to install an MSVC compiler for PIP and what not.) I think it would be healthy to host supplementary modules first with DUB as third party modules, and then get them merged into Phobos after they have seen some real world use. That's sure to result in a faster turnaround time for going from experimentation to real usage.
May 06 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d 
wrote:
 Which only confirms my suspicion that those who complain about 
 lack of
 experimental Phobos package are likely to simply not use dub.

As I said before, dub has never even occurred to me. No windows user is likely to naturally think to use a package manager :/ It's sitting in my phobos fork. Isn't that where everyone keeps their developments?

It is usual way to handle Phobos proposals that are going into review queue. If you have a stand-alone module that is ready to be experimented with, you make an own repo for it and provide as dub package. This makes possible for users to fetch it easily and work with latest master version if desired without waiting for new DMD release. I can't comment about Windows user mentality but seems like other language package managers are used quite a lot on Windows. Suggestions about making it more recognized are welcome.
 If people are pushing dub, then it should really be installed 
 with DMD.

It is the plan right now and I remember getting a conceptual agreement from Andrei about it. Problem is there are several things that need to be stabilized in it before such distribution may happen and Sonke is pretty much only person working on it continuously AFAICS. Any effort that is needed to maintain "exp.*" is better to be put into getting dub into desired state. I can volunteer to manage actual dub inclusion process but problem is code needs some work before it (and I have been doing close to 0 D programming other than my job lately).
May 06 2014
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:
 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool that 
 manages dependencies?

Hate to be the cynic, but how in the world do you expect people to even know about Dub or code.dlang.org in the first place? I don't see it linked from any of the "obvious" places on dlang.org, and I can't even find a single _mention_ that we apparently have a package manager. Nothing in the FAQ about "Contributing to D". And as if all that wasn't enough, the "Links" page still points to digitalmars.com. From a normal user's standpoint, they simply don't exist.
 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).

Ruby: "Libraries" at the top on the home page links to https://www.ruby-lang.org/en/libraries/, which explains gem and links to rubygems Python: Not super easy to see, but "PyPI" _is_ linked in the top bar. Perl: "CPAN" in the top bar links to http://www.perl.org/cpan.html, which explains cpan ...and I think you can see where this is going. I love my package manager, but I'm going to have to agree with Manu's bewilderment here. -Wyatt
May 06 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Tuesday, 6 May 2014 at 19:18:08 UTC, Wyatt wrote:
 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:
 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool that 
 manages dependencies?

Hate to be the cynic, but how in the world do you expect people to even know about Dub or code.dlang.org in the first place? I don't see it linked from any of the "obvious" places on dlang.org, and I can't even find a single _mention_ that we apparently have a package manager. Nothing in the FAQ about "Contributing to D". And as if all that wasn't enough, the "Links" page still points to digitalmars.com. From a normal user's standpoint, they simply don't exist.
 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).

Ruby: "Libraries" at the top on the home page links to https://www.ruby-lang.org/en/libraries/, which explains gem and links to rubygems Python: Not super easy to see, but "PyPI" _is_ linked in the top bar. Perl: "CPAN" in the top bar links to http://www.perl.org/cpan.html, which explains cpan ...and I think you can see where this is going. I love my package manager, but I'm going to have to agree with Manu's bewilderment here. -Wyatt

I believe this is the case. If I did not stay vaguely informed of the latest trends with the various popular dynamic languages, i.e., Python, Ruby, NodeJS, basically anything popular with the Silicon Valley hipsters (no offense to users of those languages), I'd have no idea what a language-based package manager is, much less why you'd want to use one when the various *nix have it built in. If you work in the game industry, then your primary language is probably C++ with a bit of Lua on the side. Do those languages even *have* package managers? Even if you understand how that all works, Dub isn't advertised at all, as you said.
May 06 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On that note, 
http://code.dlang.org/?sort=updated&category=library.std_aspirant.
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 May 2014 04:10, w0rp via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Tuesday, 6 May 2014 at 17:57:11 UTC, Manu via Digitalmars-d wrote:
 As I said before, dub has never even occurred to me. No windows user
 is likely to naturally think to use a package manager :/
 It's sitting in my phobos fork. Isn't that where everyone keeps their
 developments?

 If people are pushing dub, then it should really be installed with DMD.

DUB should feel pretty natural on any operating system if you are a Python (PyPi), Ruby (Gems), JavaScript (NPM), Perl (CPAN), or PHP (PEAR) programmer. Although PyPi I know doesn't work as well on Windows. (It doesn't explain to you clearly how to install an MSVC compiler for PIP and what not.) I think it would be healthy to host supplementary modules first with DUB as third party modules, and then get them merged into Phobos after they have seen some real world use. That's sure to result in a faster turnaround time for going from experimentation to real usage.

I'm not a user of any of those languages or cultures. Web feels like an alien world to me. I don't think suggesting that 'I should have known better' is good defence for something that not obvious or intuitive to end-users. You can embarrass me, but that doesn't improve the situation for users who aren't present on this forum. I'm not unhappy to use DUB to this end, it just never occurred to me. I suggest it's only practical after these changes though: 1: dub is bundled with DMD 2: The paths for installed dub packages should be either automatic, or requested/configured during the DMD installer. 3: DMD needs to know where to find DUB packages by default. I think a massive bonus would be: VisualD/Mono-D integrate a repo lister with an 'install package' button, and a 'feedback/bug report' button (linking to the package repo?) for each package. **many integration, such awesome, wow** Has anyone done a dub gui already?
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 May 2014 04:55, Jacob Carlborg via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 2014-05-06 08:39, Manu via Digitalmars-d wrote:
 's at least one DIP which received little attention afaict, it's
 an example of something that I think would probably manifest into code
 in an experimental space, but clearly couldn't be accepted as a
 language feature without lots of field time.
 In lieu of an experimental space, there will be no action.

 It's an interesting example actually. I think lots feel the DIP isn't
 really an effective solution, but nobody has the motivation or ideas
 to refine it. The DIP author clearly has no motivation to test it
 experimentally, but perhaps that's what it needs to progress?

Implementing AST macros is probably quite a big investment in time. I'm currently working on other things but I will probably give it a try at some point.
 What happened to std.serislisation? There was motion there a year or
 so back... I was looking forward to it, and did some minor reviewing
 at the time. I wonder if that's an interesting case study? (I haven't
 looked)

To be honest, I got board and started to work on D/Objective-C instead, which I know you have interest in as well. But when that is done I will come back to std.serialization. If not sooner, I have a short attention span sometimes ;)
 Perhaps you misunderstood the point of my post. I've watched people
 make solid contributions that haven't gotten through. That is
 discouraging to others considering starting their own work, and for
 the person who has already put in the effort to continue to do so in
 the future.
 The Obj-C thing as an example. Granted, it's a huge feature and has
 extensive implications. The Authors have said themselves that they
 agree it's not 'ready' for inclusion... so, what? It sits and rots?

No, I'm working on it to making it ready. Feature wise I think it's complete. Or rather good enough for inclusion. It supports for more features than extern(C++) did when it was added. -- /Jacob Carlborg

Haha, nice! I didn't realise that all my examples for hypothetical consideration came back to just you! :) So then, your take on an experimental space in the compiler for features that are still baking seems especially relevant. Am I talking shit, or do you think the idea has any value? Would it be valuable to get users testing these (quite large) developments while they're still baking?
May 06 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Le 06/05/2014 13:39, Paulo Pinto a écrit :

create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.

Yeah, these are excellent points that I've tried and perhaps failed to articulate properly in the past. The amount of 'wasted' resources required to maintain a significant surplus on Android devices is ridiculous, and that's why I separated phones from other embedded systems. While phones can, and do, do this, other embedded systems don't. To say we need to leave half of the xbox/ps4 resources idle to soak up intermittency is completely unworkable. It's always important to remember too that the embedded market is by far the largest software market in the world.
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 22:48:02 UTC, Meta wrote:
 On Tuesday, 6 May 2014 at 19:18:08 UTC, Wyatt wrote:
 On Tuesday, 6 May 2014 at 18:02:46 UTC, John Colvin wrote:
 It never occurred to you that people's libraries would be 
 published as part of a centralised repository with a tool 
 that manages dependencies?

Hate to be the cynic, but how in the world do you expect people to even know about Dub or code.dlang.org in the first place? I don't see it linked from any of the "obvious" places on dlang.org, and I can't even find a single _mention_ that we apparently have a package manager. Nothing in the FAQ about "Contributing to D". And as if all that wasn't enough, the "Links" page still points to digitalmars.com. From a normal user's standpoint, they simply don't exist.
 It's pretty common-place in a variety of languages. 
 (https://rubygems.org/ https://pypi.python.org/pypi 
 http://www.cpan.org/ etc...).

Ruby: "Libraries" at the top on the home page links to https://www.ruby-lang.org/en/libraries/, which explains gem and links to rubygems Python: Not super easy to see, but "PyPI" _is_ linked in the top bar. Perl: "CPAN" in the top bar links to http://www.perl.org/cpan.html, which explains cpan ...and I think you can see where this is going. I love my package manager, but I'm going to have to agree with Manu's bewilderment here. -Wyatt

I believe this is the case. If I did not stay vaguely informed of the latest trends with the various popular dynamic languages, i.e., Python, Ruby, NodeJS, basically anything popular with the Silicon Valley hipsters (no offense to users of those languages), I'd have no idea what a language-based package manager is, much less why you'd want to use one when the various *nix have it built in. If you work in the game industry, then your primary language is probably C++ with a bit of Lua on the side. Do those languages even *have* package managers? Even if you understand how that all works, Dub isn't advertised at all, as you said.

On Windows NuGet supports all Visual Studio languages, including C++, since Visual Studio 2010. For Lua there are LuaRocks and LuaDist. A *nix package manager is brain dead idea for software development as it ties the language libraries to the specific OS one is using. Good luck getting packages if the author did not consider your OS. Specially if they are only available in binary format, as it is standard in the enterprise world. With a language pakage manager I can produce package XPTO that will work on all OS, it won't conflict with the system packages, specially important on servers used for CI of multiple projects. -- Paulo
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 7 May 2014 at 03:58:38 UTC, Manu via Digitalmars-d 
wrote:
 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Le 06/05/2014 13:39, Paulo Pinto a écrit :

possible to create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.

Yeah, these are excellent points that I've tried and perhaps failed to articulate properly in the past. The amount of 'wasted' resources required to maintain a significant surplus on Android devices is ridiculous, and that's why I separated phones from other embedded systems. While phones can, and do, do this, other embedded systems don't. To say we need to leave half of the xbox/ps4 resources idle to soak up intermittency is completely unworkable. It's always important to remember too that the embedded market is by far the largest software market in the world.

You ignored my link about Aicas products for real time, embedded industrial systems. -- Paulo
May 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 6 May 2014 at 22:07:15 UTC, Xavier Bigand wrote:
 Le 06/05/2014 13:39, Paulo Pinto a écrit :
 On Tuesday, 6 May 2014 at 10:58:14 UTC, Manu via Digitalmars-d 
 wrote:
 On 6 May 2014 16:33, Jacob Carlborg via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 06/05/14 08:07, HaraldZealot wrote:

 I notice that I view only part of problem, can anybody link 
 or describe
 me completely state and problems of current garbage 
 collection and
 other
 resource management? It help me in finding of existence 
 solution (at
 least theoretical).

The major issue with the garbage collector is that it's not guaranteed to run a collection. When a collection is run the GC will call the destructors for the objects it collects. If there's no guarantee a collection is run there can be no guarantee that destructors are called. A collection is usually run when allocating new memory and there's not enough memory available.

I think it's also an important consideration that GC is incompatible with low-memory and real-time environments. ...

I guess outside the gaming world, embedded and real-time seem to be getting lots of Java and .NET love: https://www.aicas.com/cms/

Is that VM full compatible with Java specifications and standard frameworks?

Yes. That is the point of having a Java VM certification process. It still baffles me that many aren't aware Sun/Oracle JVM is *only* the reference implementation.
 http://www.is2t.com/products/

 http://www.mountaineer.org/netmf-for-stm32/

 Just a small sample of the partners providing the said support.


 --
 Paulo

Android works well, I love my nexus, it proves to me that it's possible to create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.

MicroEJ VMs can run in ARM Cortex-M systems. Do you know what that means? A VM taking 28 Kbytes of flash, less than 1.5 Kbytes RAM with a boot time of just 2 ms, with a 120 MHz CPU. Android Dalvik VM sucks. It has not been improved since Android 2.3, other than additional tweaks on 4.3, and Google only did improve it to get it to a good enough state. Hopefully ART will fix this. -- Paulo
May 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 7 May 2014 at 06:50:34 UTC, Paulo Pinto wrote:
 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific OS
 one is using.

The difference is that you are more vulnerable by getting software from language specific repositories.
May 07 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 7 May 2014 at 07:26:34 UTC, Ola Fosheim Grøstad 
wrote:
 On Wednesday, 7 May 2014 at 06:50:34 UTC, Paulo Pinto wrote:
 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific 
 OS
 one is using.

The difference is that you are more vulnerable by getting software from language specific repositories.

This is for software development, not end user software. -- Paulo
May 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 7 May 2014 at 12:47:05 UTC, Paulo Pinto wrote:
 This is for software development, not end user software.

Which makes avoiding trojans or rogue libraries even more critical. I avoid sources with low volume, no auditing and questionable authentication for commercial development. Basically it means downloading from the original author or using a distribution like Debian... (or creating a separate account, which I have done for D).
May 07 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 7 May 2014 at 20:09:07 UTC, Xavier Bigand wrote:
 Le 07/05/2014 05:58, Manu via Digitalmars-d a écrit :
 On 7 May 2014 08:07, Xavier Bigand via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Le 06/05/2014 13:39, Paulo Pinto a écrit :

it's possible to create really smooth applications based completely on Java (not 100% of that) but if we compare the Nexus 5 to iPhone 4 : Memory : 2 GB RAM vs 512 MB RAM CPU : Quad-core 2.3 GHz Krait 400 vs 1 GHz Cortex-A8 Battery : Li-Po 2300 mAh battery vs Li-Po 1420 mAh battery And compared to an iPhone 5s Memory : 2 GB RAM vs 1 GB RAM CPU : Quad-core 2.3 GHz Krait 400 vs Dual-core 1.3 GHz Cyclone Battery : Li-Po 2300 mAh battery vs Li-Po 1560 mAh battery It's maybe not really significant but the majority of Android devices that have acceptable performances have a lot of memory, a quad cores CPU and an heavy battery. So that cool Java can run smoothly but at which price? I think the margin of Apple produce is unbelievable.

Yeah, these are excellent points that I've tried and perhaps failed to articulate properly in the past. The amount of 'wasted' resources required to maintain a significant surplus on Android devices is ridiculous, and that's why I separated phones from other embedded systems. While phones can, and do, do this, other embedded systems don't. To say we need to leave half of the xbox/ps4 resources idle to soak up intermittency is completely unworkable. It's always important to remember too that the embedded market is by far the largest software market in the world.

have only 4Mo of RAM. I sill was at school at this date, and it was hard to deal with so few memory, ROM and video memory was very limited. The project I worked on was only limited by memory cause of use of precomputed pictures (it was an old adventure game). On this console, game vendors have to pay for the ROM and backup memories, so we put in place some compressions and data packing logics. From this time I always take care of avoiding unnecessary absurd allocations.

4MB?! That is a world of pleasure. Try to cram a Z80 application into 48 KB. :) The main problem nowadays is not automatic memory management, in whatever form, be it GC, RC, compiler dataflow, dependent types or whatever. The problem is how many developers code like the memory was infinite without pausing a second to think about their data structures and algorithms. Just yesterday I have re-written a Java application that in the application hot path does zero allocations on the code under our control. It requires execution analysis tooling, and thinking how to write the said code. That's it. .. Paulo
May 07 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 8 May 2014 16:11, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Wednesday, 7 May 2014 at 20:09:07 UTC, Xavier Bigand wrote:

 4MB?! That is a world of pleasure.

 Try to cram a Z80 application into 48 KB. :)

I've never heard of a Z80 program running a tracing GC. (I have used refcounting on a Z80 though...)
 The main problem nowadays is not automatic memory management, in
 whatever form, be it GC, RC, compiler dataflow, dependent types
 or whatever.

 The problem is how many developers code like the memory was
 infinite without pausing a second to think about their data
 structures and algorithms.

Okay, please don't be patronising. Let's assume the programmers you're talking to aren't incompetent. This discussion's existence should prove that your theory is incorrect. Let's consider your argument though, it sounds EXACTLY like the sort of programmer that would invent and rely on a tracing GC! The technology practically implies a presumption of infinite (or significant excess) memory. Such an excess suggests either the software isn't making full use of the system it's running on, or the programmers writing the software are like you say.
 Just yesterday I have re-written a Java application that in the
 application hot path does zero allocations on the code under our
 control.

 It requires execution analysis tooling, and thinking how to write the said
 code.
 That's it.

Congratulations. But I'm not sure how that helps me? You know my arguments, how does this example of yours address them? What were the operating and target hardware requirements of your project? How much memory did you have? How much memory did you use? What is the frequency of runtime resource loading/swapping? What was the pause period and the consequence when you did collect? Let's also bear in mind that Java's GC is worlds ahead of D's. I am getting very tired of repeating myself and having my points basically ignored, or dismissed with something like "my project which doesn't actually share those requirements works fine" (not that I'm saying you did that just now; I don't know, you need to tell me more about your project). I'd really like to establish as fact or fiction whether tracing GC is _practically_ incompatible with competitive embedded/realtime environments where pushing the hardware to the limits is a desire. (Once upon a time, this was a matter of pride for all software engineers) If people can prove that I'm making it all up, and "my concerns are invalid if I just ...", or whether my points are actually true. It doesn't matter what awesome GC research is out there if it's incompatible with D and/or small devices that may or may not have a robust operating system. D is allegedly a systems language, and while there is no such definition, my own take is that means it shouldn't be incompatible with, or discourage certain classes of computers or software by nature, otherwise it becomes a niche language. Please, prove me wrong. Show me how tracing collection can satisfy the basic requirements I've raised on countless prior posts, or practical workarounds that you would find reasonable if you were to consider working within those restrictions yourself and still remain compelling enough to adopt D in your corporation in the first place (implying a massive risk, and cost in retraining all the staff and retooling). I don't know how to reconcile the problem with the existing GC, and I am not happy to sacrifice large parts of the language for it. I've made the argument before that sacrificing large parts of the language as a 'work-around' is, in essence, sacrificing practically all libraries. That is a truly absurd notion; to suggest that anybody should take advice to sacrifice access to libraries is being unrealistic. I refer again to my example from last weekend. I was helping my mates try and finish up their PS4 release. It turns out, the GC is very erratic, causing them massive performance problems, and they're super stressed about this. Naturally, the first thing I did was scolded them for being stupid enough to use C# on a game in the first place, but then as I tried to search for practical options, I started to realise the gravity of the situation, and I'm really glad I'm not wearing their shoes! I've made the argument in the past that this is the single most dangerous class of issue to encounter; one that emerges as a problem only at the very end of the project and the issues are fundamental and distributed, beyond practical control. Massive volumes of code are committed, and no time or budget exists to revisit and repair the work that was already signed off. The only practical solution in this emergency situation is to start cutting assets (read: eliminate your competitive edge against competition), and try and get the system resource usage down to that half-ish of the system, as demonstrated from the prior android vs iphone comparison. It comes to this; how can a GC ever work in a memory limited environment? If the frequency of collection is almost 100% of allocations, then GC must practically be banned everywhere and all the associated language features lost. How can I rely on library code to never allocate? As I see it, this problem is basically synonymous with the current issue of lazy/unreliable destruction. They would both be solved by addressing this fundamental issue. And the only solution I know is to use RC instead. Eager release of memory is required to make sure the consecutive alloc has free memory to allocate, problem solved, and destructors work too! :)
May 09 2014
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d 
wrote:
 Let's also bear in mind that Java's GC is worlds ahead of D's.

 I am getting very tired of repeating myself and having my points
 basically ignored, or dismissed with something like "my project
 which doesn't actually share those requirements works fine" (not
 that I'm saying you did that just now; I don't know, you need to
 tell me more about your project).  I'd really like to establish
 as fact or fiction whether tracing GC is _practically_
 incompatible with competitive embedded/realtime environments
 where pushing the hardware to the limits is a [requirement].

our GC isn't good, but I've seen a lot of handwaving. The pattern lately has involved someone talking about ARC being insufficient, leading somehow to Manu asserting sufficient GC is impossible (why?), and everyone kind of ignoring it after that. I've been digging into research on the subject while I wait for test scripts to run, and my gut feeling is it's definitely possible to get GC at least into striking distance, but I'm not nearly an expert on this area. (Some of these are dead clever, though! I just read this one today: https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf)
 I don't know how to reconcile the problem with the existing GC,
 and I am not happy to sacrifice large parts of the language for
 it.  I've made the argument before that sacrificing large parts
 of the language as a 'work-around' is, in essence, sacrificing
 practically all libraries. That is a truly absurd notion; to
 suggest that anybody should take advice to sacrifice access to
 libraries is being unrealistic.

and saying to just not use major language features (I believe I recall slices were in that list?) really doesn't sit well with me either. But conversely, Manu, something has been bothering me: aren't you restricted from using most libraries anyway, even in C++? "Decent" or "acceptable" performance isn't anywhere near "maximum", so shouldn't any library code that allocates in any language be equally suspect? So from that standpoint, isn't any library you use in any language going to _also_ be tuned for performance in the hot path? Maybe I'm barking up the wrong tree, but I don't recall seeing this point addressed. More generally, I feel like we're collectively missing some important context: What are you _doing_ in your 16.6ms timeslice? I know _I'd_ appreciate a real example of what you're dealing with without any hyperbole. What actually _must_ be done in that timeframe? Why must collection run inside that window? What must be collected when it runs in that situation? (Serious questions.) See, in the final-by-default discussions, you clearly explained the issues and related them well to concerns that are felt broadly, but this... yeah, I don't really have any context for this, when D would already be much faster than the thirty years of C navel lint (K&R flavour!) that I grapple in my day job. -Wyatt
May 09 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:
 But conversely, Manu, something has been bothering me: aren't 
 you restricted from using most libraries anyway, even in C++?  
 "Decent" or "acceptable" performance isn't anywhere near 
 "maximum", so shouldn't any library code that allocates in any 
 language be equally suspect?  So from that standpoint, isn't 
 any library you use in any language going to _also_ be tuned 
 for performance in the hot path?  Maybe I'm barking up the 
 wrong tree, but I don't recall seeing this point addressed.

 More generally, I feel like we're collectively missing some 
 important context:  What are you _doing_ in your 16.6ms 
 timeslice?  I know _I'd_ appreciate a real example of what 
 you're dealing with without any hyperbole.  What actually 
 _must_ be done in that timeframe?  Why must collection run 
 inside that window?  What must be collected when it runs in 
 that situation?  (Serious questions.)

Second, 16.6ms is the time you have to do everything between frames. This means that in that timeframe you have to: -update your game state. -possibly process all network I/O. -prepare the rendering pipeline for the next frame. Updating the game state can imply make computations on lots of stuff: physics, animations, creation and deletion of entities and particles, AI logic... pick your poison. At every frame you will have an handful of objects being destroyed and a few resources that might go forgotten. One frame would probably only need very little objects collected. But given some times the amount of junk can grow out of control easily. Your code will end up stuttering at some point (because of random collections at random times), and this can be really bad.
May 09 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 May 2014 07:05, Wyatt via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:

 I've been digging into research on the subject while I wait for test scripts
 to run, and my gut feeling is it's definitely possible to get GC at least
 into striking distance, but I'm not nearly an expert on this area.

 (Some of these are dead clever, though! I just read this one today:
 https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf)

Well, that's a nice paper. But I've seen heaps of people paste heaps of papers, and nobody EVER come along and say "this, we could do this in D". I'm happy to be proven wrong, in fact, there's nothing I'd like more. I'm not an expert on GC (and I don't really want to be), but I've been trolling this forum for years and the conversation emerges regularly. As far as I can tell, there is a de facto agreement that any of those potentially awesome GC's are not actually feasible in the context of D. This is perhaps my biggest mistake in my commitment to D; I came along and identified it as a red-flag on day 1, but I saw there was lots of discussion and activity on the matter. I assumed I shouldn't worry about it, it would sort itself out... Years later, still, nobody seems to have any idea what to do, at least, such that it would be addressed in a manner acceptable to my work. The only option I know that works is Obj-C's solution, as demonstrated by a very successful embedded RTOS, and compared to competition, runs silky smooth. Indeed iOS makes it a specific design goal that it should always feel silky smooth, never stuttery, they consider it a point of quality, and I completely agree. I don't know what other horse to back?
 I don't know how to reconcile the problem with the existing GC,
 and I am not happy to sacrifice large parts of the language for
 it.  I've made the argument before that sacrificing large parts
 of the language as a 'work-around' is, in essence, sacrificing
 practically all libraries. That is a truly absurd notion; to
 suggest that anybody should take advice to sacrifice access to
 libraries is being unrealistic.

just not use major language features (I believe I recall slices were in that list?) really doesn't sit well with me either. But conversely, Manu, something has been bothering me: aren't you restricted from using most libraries anyway, even in C++?

No, and this is where everyone seems to completely lose the point. Just because high performance/realtime code has time critical parts, that tends not to be much of your code. It is small by volume, and received disproportionate attention by coders. It's fine, forget about that bit, except for that it needs to be able to run uninterrupted. _Most_ of your code is ancillary logic and glue, which typically runs in response to events, and even though it's execution frequency is super-low, it's still often triggered in realtime thread (just not very often). There are also many background threads you employ to do low priority tasks where the results aren't an immediate requirement. Some of these tasks include: resource management, loading and preparing data, communications/networking, processing low-frequency work; almost all of these tasks make heavy use of 3rd party libraries, and allocate. You can't have an allocation stop the world, because it stops the realtime threads too, at least, under any mythical GC scheme I'm aware of that's been proposed as a potential option for D?
 "Decent" or "acceptable"
 performance isn't anywhere near "maximum", so shouldn't any library code
 that allocates in any language be equally suspect?  So from that standpoint,
 isn't any library you use in any language going to _also_ be tuned for
 performance in the hot path?  Maybe I'm barking up the wrong tree, but I
 don't recall seeing this point addressed.

A library which is a foundation of a realtime system will employ realtime practises. Those are not the libraries I'm worried about. Most libraries that are useful aren't those libraries. They are tool libs, and they are typically written to be simple and maintainable, and usually by a PC developer, with no real consideration towards specific target applications.
 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any hyperbole.

It doesn't matter what I'm doing in my 16ms timeslice most of the time. I'm running background threads, and also triggering occasional low frequency events in the realtime thread. Again, most code by volume is logic and glue, it is not typically serviced intensively like the core realtime systems, and most often, by the junior programmers... I appreciate that I haven't successfully articulated the function of this code, but that is because to describe "what I'm doing" would be to give you a million lines of code to nit-pick through. Almost anything you can imagine, is the answer, as long as it's reasonably short such that it's not worth the synchronisation costs of queueing it with a parallel job manager or whatever. This logic and glue needs to have access to all the conveniences of the language for productivity and maintainability reasons, and typically, if you execute only one or 2 of these bits of code per frame, it will have no meaningful impact on performance... unless it allocates, triggers a collect, and freezes the system. I repeat, the juniors... D has lots of safety features to save programmers from themselves, and I don't consider it a workable option, or goal for the language, to suggest we should abandon them. ARC overhead would have no meaningful impact on performance, GC may potentially freeze execution. I am certain I would never notice ARC overhead on a profiler, and if I did, there are very simple methods to shift it elsewhere in the few specific circumstances it emerges.
 What actually _must_ be done in that timeframe?  Why must collection run
 inside that window?  What must be collected when it runs in that situation?
 (Serious questions.)

Anything can and does happen in low-frequency event logic. Collection 'must' run in that window in the event an allocation exists, and there is no free memory, which is the likely scenario. strings? closures? array initialisations were a problem (i'm not sure if that was considered a bug and fixed though?). even some should-be stack allocations are allocated when the compiler thinks it's a requirement for safety. string interaction with C libs is good source of allocations, but there are many. Or even small transient allocations, temp's to do small amounts of work which would otherwise be released upon scope exit. Banning that sort of practise throws a massive spanner in conventional software engineering practise that everyone is familiar with.
 See, in the final-by-default discussions, you clearly explained the issues
 and related them well to concerns that are felt broadly, but this... yeah, I
 don't really have any context for this, when D would already be much faster
 than the thirty years of C navel lint (K&R flavour!) that I grapple in my
 day job.

I appreciate your argument. I realise it's why I've had so little effect... I just can't easily articulate specific instances, because they are basically unknowable until they happen, and they're hidden by literally millions of loc. Implicit allocations appear all the time, and if you deliberately ban them, you quickly find all those language features don't work and your code gets a lot more difficult to write and maintain. It's a practicality, risk aversion, and maintenance issue. If you had to tag main() with nogc, and then write a millions-of-loc program in a team of 60, D becomes so much less compelling argument than otherwise. D has 2 major advantages over C++ as I see. 1, meta, and that's great, but you don't sit and write meta all the time. 2, productivity and correctness, which I find to be the more compelling case to be made to adopt D. It affects all programmers all day, every day, and we lose many aspects of the languages offering if we tag nogc, which includes libs.
May 09 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Saturday, 10 May 2014 at 07:08:04 UTC, Andrei Alexandrescu 
wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.

This is very, very, very wrong. -- Andrei

I've seen this discussion ("it's almost performance-free", "it's a performance killer") so many times, I can't even say who has the burden of proof anymore.
May 10 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Saturday, 10 May 2014 at 07:42:05 UTC, Francesco Cattoglio 
wrote:
 On Saturday, 10 May 2014 at 07:08:04 UTC, Andrei Alexandrescu 
 wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.

This is very, very, very wrong. -- Andrei

I've seen this discussion ("it's almost performance-free", "it's a performance killer") so many times, I can't even say who has the burden of proof anymore.

I wish that someone would take the time and implement ARC in D. That's the only way to prove anything. If you implement it and you can provide clear evidence for its advantages, then that just ends all discussions.
May 10 2014
prev sibling next sibling parent "Paolo Invernizzi" <paolo.invernizzi no.address> writes:
On Saturday, 10 May 2014 at 06:53:07 UTC, Paulo Pinto wrote:
 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:
 On 10 May 2014 07:05, Wyatt via Digitalmars-d

The problem when comparing iOS with Android, is that we aren't comparing ARC with GC. We are comparing a full OS, which we don't know how much ARC is actually used versus standard malloc/new with another OS, which has a so-and-so VM implementation, used mostly software rendering until version 4.1, and care for low end devices was only done in 4.4. If we add Windows Phone to the mix, then we have a .NET stack like Android (WP7 - GC/JIT) or in its sucessor (WP8) native code generation for .NET (GC) with a COM model for OS APIs (ARC). Both versions of Windows Phone run smoother that many Android phones, even the WP7 ones. Not saying you are not right, just that we need to look at the whole stack when comparing mobile OS, not just GC vs ARC. -- Paulo

+1 -- Paolo
May 10 2014
prev sibling next sibling parent "Nick B" <nick.barbalich gmail.com> writes:
On Saturday, 10 May 2014 at 08:18:30 UTC, w0rp wrote:

 I've seen this discussion ("it's almost performance-free", 
 "it's a performance killer") so many times, I can't even say 
 who has the burden of proof anymore.

I wish that someone would take the time and implement ARC in D. That's the only way to prove anything. If you implement it and you can provide clear evidence for its advantages, then that just ends all discussions.

How hard would this be, exactly ? Perhaps then, and only then, could you make a apples to apples comparison ? Nick
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 May 2014 19:07, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Le 10/05/2014 01:31, Francesco Cattoglio a écrit :

 On Friday, 9 May 2014 at 21:05:18 UTC, Wyatt wrote:
 But conversely, Manu, something has been bothering me: aren't you
 restricted from using most libraries anyway, even in C++? "Decent" or
 "acceptable" performance isn't anywhere near "maximum", so shouldn't
 any library code that allocates in any language be equally suspect?
 So from that standpoint, isn't any library you use in any language
 going to _also_ be tuned for performance in the hot path?  Maybe I'm
 barking up the wrong tree, but I don't recall seeing this point
 addressed.

 More generally, I feel like we're collectively missing some important
 context:  What are you _doing_ in your 16.6ms timeslice?  I know _I'd_
 appreciate a real example of what you're dealing with without any
 hyperbole.  What actually _must_ be done in that timeframe?  Why must
 collection run inside that window?  What must be collected when it
 runs in that situation?  (Serious questions.)

I'll try to guess: if you want something running at 60 Frames per Second, 16.6ms is the time you have to do everything between frames. This means that in that timeframe you have to: -update your game state. -possibly process all network I/O. -prepare the rendering pipeline for the next frame. Updating the game state can imply make computations on lots of stuff: physics, animations, creation and deletion of entities and particles, AI logic... pick your poison. At every frame you will have an handful of objects being destroyed and a few resources that might go forgotten. One frame would probably only need very little objects collected. But given some times the amount of junk can grow out of control easily. Your code will end up stuttering at some point (because of random collections at random times), and this can be really bad.

As I know AAA game engine are reputed to do zero allocations during the frame computation, but I think it less the case noways cause of the dynamism of scene and huge environments that are streamed.

Running a game in a zero-alloc environment was a luxury that ended about 10 years ago, for reasons that you say. Grand Theft Auto proved that you don't need to have loading screens, and now it's a basic requirement. We also have a lot more physics, environmental destruction, and other dynamic behaviour. It was also something we achieved in the past with extreme complexity, and the source of many (most?) bugs. We still allocated technically, but we had to micro-manage every single little detail, mini pools and regions for everything. Processors are better now, in the low-frequency code, we can afford to spend a tiny bit of time using a standard allocation model. The key is the growing separation between low-frequency and high-frequency code. On a 33mhz Playstation, there wasn't really much difference between the worlds. Now there is, and we can afford to allow 'safety' into the language at the cost of a little memory management. We just can't have that cost include halting threads all the time for lengthy collect processes. I know this, because we already use RC extensively in games anyway; DirectX uses COM, and most resources use manual RC because we need things to release eagerly, and for destructors to work properly. I don't see how ARC would add any significant cost to the manual RC that is basically standard for many years. It would add simplicity and safety, which are highly desirable.
 I recently fix a performance issue due to a code design that force
 destruction of walls (I am working on an architecture application) before
 creating them back when the user move them. gprof, show me that this point
 took around 30% of CPU time in a frame, and only allocations/destructions
 was about 5%. This 5% percents contains destructions of object in the
 architecture part and the 3D engine, same for construction. Construction
 also add operation like new geometry updload, so I don't think place of new
 and delete was high without the job made by constructors and destructors.
 Reserving memory (malloc) isn't really an issue IMO, but operations related
 to constructions and destructions of objects can be expensive.

It's better to have eager destructors (which empowered you with the ability to defer them if you need to), than to not have destructors at all, which has been the center if this last few days argument. It's not hard to defer resource destruction to a background thread in an eager-destruction model.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 May 2014 16:53, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am 10.05.2014 08:27, schrieb Manu via Digitalmars-d:
 On 10 May 2014 07:05, Wyatt via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On Friday, 9 May 2014 at 16:12:00 UTC, Manu via Digitalmars-d wrote:
 ...

The only option I know that works is Obj-C's solution, as demonstrated by a very successful embedded RTOS, and compared to competition, runs silky smooth. Indeed iOS makes it a specific design goal that it should always feel silky smooth, never stuttery, they consider it a point of quality, and I completely agree. I don't know what other horse to back? ...

The problem when comparing iOS with Android, is that we aren't comparing ARC with GC. We are comparing a full OS, which we don't know how much ARC is actually used versus standard malloc/new with another OS, which has a so-and-so VM implementation, used mostly software rendering until version 4.1, and care for low end devices was only done in 4.4. If we add Windows Phone to the mix, then we have a .NET stack like Android (WP7 - GC/JIT) or in its sucessor (WP8) native code generation for .NET (GC) with a COM model for OS APIs (ARC). Both versions of Windows Phone run smoother that many Android phones, even the WP7 ones. Not saying you are not right, just that we need to look at the whole stack when comparing mobile OS, not just GC vs ARC.

Yes, granted. I shouldn't make that sort of comparison. What I am really trying to do with such comments, is to insist that it is used as a foundation for a very successful RTOS. To say "it's no good. ARC can't work. it can never be used as a foundation for a language. but, cycles!" and whatever, is kind of ignoring the evidence. It's interesting too that you bring Windows phone into the mix, because all the WP8 developers I know use microsoft's C++/CX, which has the new '^' ARC pointer type (as used intensively by the OS). Why did MS abandon GC? It looks like a similar sort of back-peddling from their commitment to GC as did Apple, when they started making a serious play for the embedded market. C# implies massive GC related issues for realtime and embedded software. I have lots of colleagues who rant about it regularly, and my experience from 2 weekends back that I shared earlier in this thread is a scary slap in the face for me.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion? I have many colleagues who work on phones. The patterns I see are that the iOS guys who use Obj-C never complain about it, and the GC guys are often struggling with it. The bias in gamedev is strongly towards iOS too, so that should be a more populous group to draw samples from.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 I don't know well WP8 models, but this one must run smoothly :
 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances cause
 they control everything on the OS. There is certainly some specifics memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.

They don't GC. MS are behind their C++/CX now, which introduced an ARC pointer type '^'.
May 10 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d 
wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion? I have many colleagues who work on phones. The patterns I see are that the iOS guys who use Obj-C never complain about it, and the GC guys are often struggling with it. The bias in gamedev is strongly towards iOS too, so that should be a more populous group to draw samples from.

You conflate freezing and performance. ARC easily can be slower but is has benefit of slowdown distributed more evenly thus not obviously notable.
May 10 2014
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d 
wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion?

It is well known that reference counting can perform poorly. The updates damage the cache and atomic reference counts will stall pipelines. Of course, there are things that can be done to mitigate the impact, but the same is true for GC. Just Google for reference counting performance.
 I have many colleagues who work on phones. The patterns I see 
 are that
 the iOS guys who use Obj-C never complain about it, and the GC 
 guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should 
 be a
 more populous group to draw samples from.

Of course. In game dev a large frame time spike is worse, and more obvious/noticeable than a consistent and mostly predictable performance hit.
May 10 2014
prev sibling next sibling parent "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Saturday, 10 May 2014 at 13:50:05 UTC, Dicebot wrote:
 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via 
 Digitalmars-d wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, 
 GC may
 potentially freeze execution. I am certain I would never 
 notice ARC
 overhead on a profiler, and if I did, there are very simple 
 methods to
 shift it elsewhere in the few specific circumstances it 
 emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion? I have many colleagues who work on phones. The patterns I see are that the iOS guys who use Obj-C never complain about it, and the GC guys are often struggling with it. The bias in gamedev is strongly towards iOS too, so that should be a more populous group to draw samples from.

You conflate freezing and performance. ARC easily can be slower but is has benefit of slowdown distributed more evenly thus not obviously notable.

It should be noted that reference counting can also cause large stalls when a root object is freed, causing a cascade of decrements-to-zero, which would have been handled better by a GC.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 00:23, Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 10 May 2014 at 13:50:05 UTC, Dicebot wrote:
 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d wrote:

It should be noted that reference counting can also cause large stalls when a root object is freed, causing a cascade of decrements-to-zero, which would have been handled better by a GC.

That problem is sooo easy to deal with by contrast. And the programmer maintains control.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 00:20, Peter Alexander via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Saturday, 10 May 2014 at 13:33:40 UTC, Manu via Digitalmars-d wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion?

It is well known that reference counting can perform poorly. The updates damage the cache and atomic reference counts will stall pipelines. Of course, there are things that can be done to mitigate the impact, but the same is true for GC. Just Google for reference counting performance.

I agree that acceptable performance for ARC requires compiler support. What doesn't? Acceptable performance in general requires extensive compiler support (in the form of optimisation). Why should this be considered any differently? D is thread-local by default. That's a blessing for RC, the requirement for atomics will be minimal. D also has pure, which helps a lot. Hopefully one day we'll have 'scope' fleshed out to make proper escaping guarantees in non-pure functions too, that would really help. There are strategies to reduce impact on the dcache. It just seems like a much more approachable problem with actual options available than the mythical awesome GC.
 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.

Of course. In game dev a large frame time spike is worse, and more obvious/noticeable than a consistent and mostly predictable performance hit.

I can't think of many situations where that wouldn't be the case. What sort of software is it not an issue to experience intermittent freezing? Is it shown that cumulative RC fiddling costs significantly exceed a GC collect pass? Recall too that D has significant opportunity to improve on ARC as implemented by other languages, and D's GC is much worse than most, so if you considered the same competitive difference in the context of D, how would it look then? It's funny actually, D offers many advantages towards ARC that other languages don't enjoy... but by contrast, D barely supports the notion of a GC it seems, or surely we'd have had more people experimenting with cool advanced implementations by now?
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 01:44, Paulo Pinto via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am 10.05.2014 15:37, schrieb Manu via Digitalmars-d:
 On 10 May 2014 19:43, Xavier Bigand via Digitalmars-d

 <digitalmars-d puremagic.com> wrote:
 I don't know well WP8 models, but this one must run smoothly :

 http://www.nokia.com/fr-fr/mobiles/telephone-portable/lumia1320/fiche-technique/

 Just like Android phones, the battery is huge : 3400mAh
 It's the same for CPU : Qualcomm Snapdragon™ S4, dual core 1,7 GHz
 Only RAM seems reasonable : 1Go

 And it's maybe easier for Microsoft to do a GC with good performances
 cause
 they control everything on the OS. There is certainly some specifics
 memory
 management in the kernel related to the GC. When Google "just" put small
 pieces together based on a linux kernel.

They don't GC. MS are behind their C++/CX now, which introduced an ARC pointer type '^'.

.NET Applications on Windows Phone 8 make use of both (GC and COM RC).

I don't know any gamedev's who use .NET on WP8. C++/CX was a good step forward, and addressed one of the primary inhibitors to developer adoption of WP7. I don't know a single developer who ever released a WP7 app. WP7 was a ghost town compared to iOS and Android in terms of developer support. Lots of people I know are happily supporting WP8 now thanks to C++/CX.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 02:15, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/10/14, 6:33 AM, Manu via Digitalmars-d wrote:
 On 10 May 2014 17:08, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 On 5/9/14, 11:27 PM, Manu via Digitalmars-d wrote:
 ARC overhead would have no meaningful impact on performance, GC may
 potentially freeze execution. I am certain I would never notice ARC
 overhead on a profiler, and if I did, there are very simple methods to
 shift it elsewhere in the few specific circumstances it emerges.

This is very, very, very wrong. -- Andrei

Is there any evidence to support that assertion?

Of course, and I've shown it numerous times, only for it to be blissfully ignored. Probably the most recent and influential work on it is http://research.microsoft.com/pubs/202163/rcix-oopsla-2013.pdf of which FIRST sentence is "Despite some clear advantages and recent advances, reference counting remains a poor cousin to high-performance tracing garbage collectors." The paper reviews previous work on the matter, ALL of which shows that RC is behind GC in speed, and presents (for the first time in 2013) new work that brings about parity of RC with GC.

I saw that paper before. And I thought, if anything, it supported my argument in the context of D. The next sentence goes on to list the advantages of RC (issues we have wrestled with, like destructors), and then goes on to say the recent awesome RC is within 10% of "the fastest tracing collectors". Are you suggesting that D's GC is among 'the fastest tracing collectors'? Is such a GC possible in D? Has anyone compared D's GC against 'the fastest tracing collectors'? What is the difference? Does D fall within 10%? If not, then chances are, according to this paper, RC would be superior to the existing GC in terms of performance, as well as reliability/predictability/functionality (destructors that work, not incompatible with embedded software). If an awesome RC is implementable in D, but an awesome GC is not, then I can't see that even any performance argument can remain in the debate. They are talking specifically about a GC-backed-RC too, to handle cycles, so their implementation does not come at any cost to safety or added complexity for users, and has the key advantage that people like me may have the option to disable the cycle collector and take manual responsibility if desired.
 I have many colleagues who work on phones. The patterns I see are that
 the iOS guys who use Obj-C never complain about it, and the GC guys
 are often struggling with it.
 The bias in gamedev is strongly towards iOS too, so that should be a
 more populous group to draw samples from.

As competent engineers we can't afford to be ignorant of the existing research and practice showing a much more complex tradeoff space than a black/white distinction. We also can't do the capital mistake of engineering - the beauty of the part at the expense of the beauty of the whole.

There is a black/white distinction though. It you can't make the freezing go away, it is _incompatible_ with certain classes of software. I don't care what the performance difference is when this is the case. If one option is incompatible and the other is not, then there is no tradeoff. There are also technical advantages to RC too, like destructors that work, which I find very compelling.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 08:40, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:
 I can't think of many situations where that wouldn't be the case. What
 sort of software is it not an issue to experience intermittent
 freezing?

Batch programs, for example a compiler.

Indeed, that was the only one that came to my mind. GC might be better for shell apps because it is unlikely that a collect will ever occur, it'll usually terminate before it gets to it, but there's certainly no hard-dependency on GC in this use case. This is truly a niche usage case though, and yet, it also has no particular incompatibility with any choice of technology, only a slight preference towards GC for the reason that you never bother freeing memory in DMD. If you're prepared to label the largest entertainment industry on the planet a niche, then this sort of work is a hyper-niche... and regardless, not actually incompatible with any particular option. I'd like to consider this form of software for a moment though. Assuming the GC was a library offered beside the language, DDMD for instance would be able to make particularly easy and convenient use of it. Shell apps by comparison _are_ generally small and self-contained. It's far more reasonable to suggest that a shell app can adopt specific and custom allocation patterns when the code is relatively small, function is specific, the life is short, the probability of running into incompatible 3rd party libraries is almost zero by comparison. How many libs does DMD link? This thought has actually left me slightly concerned, that as much as Andrei is accusing me of unreasonably arguing my "staunch niche position that ignores factual realities" (a somewhat offensive and dismissive notion), that there might be some silent, but significant (perhaps even unconscious) element of the same in reverse.
 Essentially any program that is not interactive and does not have hard
 realtime requirements, which is (possibly) most programs.

I'm hard pressed to think of any software I see people using every day which isn't realtime in some sense. The majority of relevant software today is on embedded devices, and visceral touch-based UI is key to their impression of quality.
 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,

Compiler improvements to improve on ARC are the same technology as used to improve GC. You're essentially arguing that one is easy pickings and the other is impractically difficult, but they're actually about the same level.

Are they? This is the first I've heard such a claim. If that's the case, then that why is there an argument? That work just needs to be done, and we can experiment with both targets... I can fairly easily visualise the path to ARC, but by all prior reports I've heard, awesome GC is practically incompatible with D for various reasons. There were discussions about different advanced options by the experts at dconf last year, which seemed to hit various impasses. And I'm still not even sure if 'awesome GC' will solve my problems? Will it? No matter how awesome it is, it seems conceptually incompatible with my environment. I keep saying, I'm more than happy to be proved wrong (really!), but until someone can, then it's unfair to dismiss my arguments so easily... and I just don't think it's that unreasonable. ARC is an extremely successful technology, particularly in the compiled/native/systems language space (OC, C++/CX, Rust). Is there actually any evidence of significant GC success in this space? Successes all seem to be controlled VM based languages like Java and C#; isolated languages with no intention to interact with existing native worlds. There must be good reason for that apparent separation in trends?
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 03:54, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/10/14, 10:03 AM, Manu via Digitalmars-d wrote:
 There is a black/white distinction though.
 It you can't make the freezing go away, it is _incompatible_ with
 certain classes of software.
 I don't care what the performance difference is when this is the case.
 If one option is incompatible and the other is not, then there is no
 tradeoff.

Of course it would be nice to have better support for reference counting. My understanding is that you advocate that reference counting would be compulsive, which would preclude the GC option.

As I've said many times, I'm really not married to ARC... it just makes sense. In the years I've been hanging around here, I'm yet to see anyone present a realistic path for D towards an acceptable GC solution. If anybody can do that, I'll reconsider my position. But it seems it's becoming less and less likely to me. What I know is that there is a solution which has a proven and extremely successful track record in native languages and embedded/realtime systems, and even solves ancillary problems like destructors while at it. But most importantly, it doesn't exclude any classes of software by nature. If ARC were a foundation for D, I wonder if there's opportunity for programs that may elect to disable the RC and leave the work exclusively to the backing cycle collector, in the same way that I might disable the cycle collector and take manual responsibility for weak references...? I may argue from a biased perspective, but placing more weight on some particular negative qualities of a technology is not unreasonable when they represent a showstopper. Which part of my issue with the GC is factually incorrect? All it would take is for anyone competent to tell me how a magic GC (which is compatible with D) will stop freezing in the low/no-memory environment, and I'll shut up. Even controlling it... I was initially wishing for something like an incremental garbage collector, which I could allocate a maximum periodic time-slice (1ms or less) on a regular basis. Not only was it agreed that this is basically impossible (a bunch of collection ground-work would need to be repeated at the start of each micro-collect, probably representing more time than the total time-slice... iirc), but it almost certainly loses any raw performance advantage over ARC in such a case anyway, and it becomes moot.
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 08:40, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/10/2014 8:54 AM, Manu via Digitalmars-d wrote:

 Recall too that D has significant opportunity to
 improve on ARC as implemented by other languages,

You're essentially arguing that one is easy pickings and the other is impractically difficult, but they're actually about the same level.

That's not actually what I was arguing at all. I was arguing that D's ARC offering would likely be superior to any that have come before, due to inherent advantages in the language, and by all reports, D is incapable of having the worlds best GC implemented successfully for various reasons. If it is possible to implement the worlds best GC in D, it's still not a done deal though. We still need to consider; does the worlds best GC remain incompatible with my workload at a fundamental level? What do we do about destructors? If ARC is within 10% of the worlds best GC, and both are equally attainable in D's context, is 10% a fair price to pay for a more inclusive technology? Who is it that will declare that 10% difference a deal-breaker? (they're obviously not here now, since D's existing GC is not in that ballpark) With them in mind, and upon an ARC platform, can we offer those users the GC as a library? Or perhaps a mechanism to disable RC code generation in their local code and falling back upon the cycle-collecting GC already in place to pick up the task?
May 10 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 14:57, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/10/2014 8:58 PM, Manu via Digitalmars-d wrote:
 This is truly a niche usage case though,

Come on! Like about 80% of the programs on any linux box? Like the OCR program I run? A payroll processing program? Any scientific numerical analysis program? Engineering programs?

Linux programs aren't at risk of being ported from C any time soon. And If the GC is a library, shell apps probably represent the set where it is the most convenient to make use of a GC lib. Accounting software is UI based software. It's just of a sort that people aren't likely to notice or complain if it stutters occasionally. Engineering programs are typically realtime in some way. Most productivity software of that sort (CAD, ART/design, etc) is highly interactive. Tools like OCR as a shell app, refer to the short-lived shell app point. OCR as a feature of an art package (like photoshop), again, highly interactive productivity software.
 How many libs does DMD link?

We've gone over this before. You were concerned that the libraries you linked with were incompetently written, and implied that if ARC was pervasive, they would be competently written. I can guarantee you, however, that ARC leaves plenty of opportunity for incompetence :-)

It's not about incompetence, it's about incompatibility. If I can't tolerate a collect, not only do I sacrifice extensive and productive parts of the language, this means a library which I have no control over is banned from the GC outright. That is entirely unrealistic. I can approach a performance hazard which manifests locally, as with ARC. I have no way to address the sort that manifests in random places, at random times, for reasons that are outside of my control, interferes with the entire application, and increases in frequency as the free memory decreases (read: as I get closer to shipping day).
 I'm hard pressed to think of any software I see people using every day
 which isn't realtime in some sense.

Programs you "see" by definition have a user interface. But an awful lot of programs are not seen, but that doesn't mean they aren't there and aren't running. See my list above.

Sure, and of the subset of those which run in a time-critical environment, they are the best candidates to make use of GC as a lib. As I see, most of those apps these days are more likely to be written in a language like python, so it's hard to make any raw performance argument about that category of software in general. It would be an extremely small subset where it is a consideration. Conversely, all realtime/UI based/user-facing software cares about performance and stuttering.
 then that why is there an argument?

Because if this was an easy problem, it would have been solved. In particular, if the ARC overhead was easily removed by simple compiler enhancements, why hasn't ARC taken the world by storm? It's not like ARC was invented yesterday.

And as far as I can tell, it has, at least in this (native/compiled/systems) space. O-C, C++/CX, Rust... where are the counter-examples? I don't think comparing D to Java/C# is a very good comparison compared to the other native languages which do truly exist on the same playing field.
 That work just needs to be done,

That's a massive understatement. This is PhD research topic material, not something I can churn out in a week or two if only I had a more positive attitude :-)

That's okay, I am just looking for direction. I want to see a commitment to a path I can get behind. Not a path that, however many years from now, I still have good reason to believe it won't satisfy my requirements. All my energy in the meantime would be a waste in that case.
 but by all prior
 reports I've heard, awesome GC is practically incompatible with D for
 various reasons.

There is no such thing as a GC which would satisfy your requirements.

Then... the argument is finished. You have a choice. You may choose to explore the more inclusive technology (which also solves some other outstanding language problems, like destructors), or you confirm D is married to GC, and I consider my options and/or future involvement in that context.
 No matter how awesome it is, it seems conceptually
 incompatible with my environment.

My point!

My point too!
 and I just don't think it's that unreasonable. ARC is an
 extremely successful technology, particularly in the
 compiled/native/systems language space (OC, C++/CX,

What was dismissed is the reality pointed out many times that those systems resolve the perf problems of ARC by providing numerous means of manually escaping it, with the resulting desecration of soundness guarantees.

You've said that, but I don't think there's any hard evidence of that. It is an option, of course, which is extremely valuable to have, but I don't see any evidence that it is a hard requirement. Andrei's paper which asserted that modern ARC fell within 10% of "the fastest GC" didn't make that claim with the caveat that "extensive unsafe escaping was required to produce these results".
 Rust).

Rust is not an extremely successful technology. It's barely even been implemented.

Maybe... but they probably had extensive arguments on the same issues, and their findings should surely be included among the others. It's definitely modern, and I'm sure it was considered from a modern point of view, which I think is meaningful.
 Is there
 actually any evidence of significant GC success in this space?
 Successes all seem to be controlled VM based languages like Java and
 C#; isolated languages with no intention to interact with existing
 native worlds. There must be good reason for that apparent separation
 in trends?

There are many techniques for mitigating GC problems in D, techniques that are not available in Java or C#. You can even do shared_ptr<> in D. You can use nogc to guarantee the GC pause troll isn't going to pop up unexpectedly. There are a bunch of other techniques, too.

RC is no good without compiler support. One second you're arguing precisely this case that useful RC requires extensive compiler support to be competitive (I completely agree), and then you flip about and I hear the "use RefCounted!" argument again (which also has no influence on libraries I depend on). I've argued before that nogc has no practical effect, unless you tag it on main(), and then D is as good as if it didn't have memory management at all. Non-C libraries are practically eliminated. Sounds realistic in a shell app perhaps, but not in a major software package. D's appeal depends largely on it's implicit memory management, and convenience/correctness oriented constructs that it enables. Both these suggestions only have any effect over my local code, which ignores the library problem again.
 ARC simply is not a magic, no problem solution one can use without careful
 thought in large, complex systems. (Of course, neither is GC nor any other
 memory management scheme.)

I have never claimed it's magic. It's **workable**. GC is apparently not, as you admitted a few paragraphs above. The key difference is that ARC cost is localised, which presents many options. GC cost is unpredictable, and gets progressively worse as environments become more and more like mine. Short of banning memory management program-wide (absurd, it's 2014), or having such an excess (waste) of available resources that I'm sabotaging competitive distinction, theres not really workable options. If it's true that ARC falls within 10% of the best GC's, surely it must be considered a serious option, especially considering we've started talking about ideas like "maybe we should make things with destructors lower to use ARC"? Performance, it turns out, is apparently much more similar than I had imagined, which would lead me to factor that out as a significant consideration. Which is the more _inclusive_ option? And unless D is capable of 'worlds fastest GC', then ARC would apparently be a speed improvement over the current offering too.
May 10 2014
prev sibling next sibling parent reply Benjamin Thaut <code benjamin-thaut.de> writes:
Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:
 Does ~this() actually work, or just usually work?
 Do you call your destructors manually like C#?

Back when I actually used D's GC, it did usually work. There were a few workarounds because destructors would be called in the wrong thread, but other than that it did work for me. By now I call my destructors manually like in C++. I do full manual memory management (I have a modified version of druntime that does not even contain the GC anymore). I always liked D because of its metaprogramming capabilities and not because of the automatic memory management. So I use D as a C++ with better meta programming. But this also means that I do not use phobos. I actually tried using only parts of phobos, but the dependency hell in phobos results in all modules being pulled in as soon as you use one central one, so I stopped using most of phobos alltogether. The annoying part, as you already pointed out, is that I can't use any third party libraries. So I'm living in my own little D world. So I would be disappointed if classes suddently don't have destructors anymore. Most annoying would be the lack of compiler generated field destructors (e.g. automatically calling the destructor of each struct in the class). I could work around that, but it would be another anoyance over C++ and as they are piling up it becomes more and more viable to just go back to C++.
 I support the notion that if the GC isn't removed as a foundational
 feature of D, then destructors should probably be removed from D.
 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?

I think ARC could work, but should be extended with some sort of ownership notation. Often a block of memory (e.g. an array of data) is exclusivly owned by a single object. So it would be absolutly uneccessary to reference count that block of memory. Instead I would want something like Rust has, borrowed pointers. (We actually already have that, "scope" but its not defined nor implemented for anything but delegates) Kind Regards Benjamin Thaut
May 11 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 11.05.2014 12:57, schrieb w0rp:
 The vast majority of software, at least as far as I can see, use web
 services. That makes up the vast majority of software on my Android
 phone. Garbage collection is definitely applicable for web servers, so
 there is a huge area where D and a garbage collector can apply nicely. I
 think the arguement that the vast majority of software should be real
 time now is very weak, I wouldn't argue that. I would simply argue that
 garbage collection isn't applicable to real time software, because that
 is a given.

 I'm really not sure how anything but a manual memory management
 allocation scheme could work with real time software. It seems to me
 that if you are writing software where any cost in time is absolutely
 critical, and you must know exactly when you are allocated and
 deallocating, then the best you can hope to do is to write these things
 yourself.

 I don't think it's possible for a computer out there to manage time for
 you at the most fundamental level, managing memory. If I was to write a
 real time application, I would not interact with a garbage collector and
 use primarily small data structures on a stack. If I needed to allocate
 objects on a heap, I would use something I could resize and destroy
 pretty manually, or at least in a scoped manner, like std::vector. I
 can't see how garbage collection or automatic reference counting would
 help me. I would want to have primarily scoped or unique references to
 data, not shared references.

Apparently the customers of Aicas, Aonix, IBM and IS2T have a different opinion. -- Paulo
May 11 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 11 May 2014 17:52, Benjamin Thaut via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:

 I support the notion that if the GC isn't removed as a foundational
 feature of D, then destructors should probably be removed from D.

 That said, I really want my destructors, and would be very upset to
 see them go. So... ARC?

I think ARC could work, but should be extended with some sort of ownership notation. Often a block of memory (e.g. an array of data) is exclusivly owned by a single object. So it would be absolutly uneccessary to reference count that block of memory. Instead I would want something like Rust has, borrowed pointers. (We actually already have that, "scope" but its not defined nor implemented for anything but delegates)

Indeed, I also imagine that implementation of 'scope' would allow for a really decent ARC experience. D already has some advantages over other languages, but that one would be big.
May 11 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 11 May 2014 at 09:53:59 UTC, Manu via Digitalmars-d 
wrote:
 On 11 May 2014 17:52, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am 06.05.2014 05:40, schrieb Manu via Digitalmars-d:

 I support the notion that if the GC isn't removed as a 
 foundational
 feature of D, then destructors should probably be removed 
 from D.

 That said, I really want my destructors, and would be very 
 upset to
 see them go. So... ARC?

I think ARC could work, but should be extended with some sort of ownership notation. Often a block of memory (e.g. an array of data) is exclusivly owned by a single object. So it would be absolutly uneccessary to reference count that block of memory. Instead I would want something like Rust has, borrowed pointers. (We actually already have that, "scope" but its not defined nor implemented for anything but delegates)

Indeed, I also imagine that implementation of 'scope' would allow for a really decent ARC experience. D already has some advantages over other languages, but that one would be big.

Yes, together with an opImplicitCast of some sort it could probably even be implemented as a pure library type (std.typecons.RefCounted). std.typecons.scoped can also benefit from this. This would allow safe implicit casting to non-RC types, i.e. passing to a function that accepts a non-RC scope parameter. This is extremely important if we want to be able to use RC types only in some cases; otherwise it would need to "infect" everything in order to be safe.
May 11 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
The vast majority of software, at least as far as I can see, use 
web services. That makes up the vast majority of software on my 
Android phone. Garbage collection is definitely applicable for 
web servers, so there is a huge area where D and a garbage 
collector can apply nicely. I think the arguement that the vast 
majority of software should be real time now is very weak, I 
wouldn't argue that. I would simply argue that garbage collection 
isn't applicable to real time software, because that is a given.

I'm really not sure how anything but a manual memory management 
allocation scheme could work with real time software. It seems to 
me that if you are writing software where any cost in time is 
absolutely critical, and you must know exactly when you are 
allocated and deallocating, then the best you can hope to do is 
to write these things yourself.

I don't think it's possible for a computer out there to manage 
time for you at the most fundamental level, managing memory. If I 
was to write a real time application, I would not interact with a 
garbage collector and use primarily small data structures on a 
stack. If I needed to allocate objects on a heap, I would use 
something I could resize and destroy pretty manually, or at least 
in a scoped manner, like std::vector. I can't see how garbage 
collection or automatic reference counting would help me. I would 
want to have primarily scoped or unique references to data, not 
shared references.
May 11 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Tuesday, 6 May 2014 at 03:40:47 UTC, Manu via Digitalmars-d 
wrote:
 On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:
 Walter and I have had a long chat in which we figured our 
 current
 offering of abstractions could be improved. Here are some 
 thoughts.
 There's a lot of work ahead of us on that and I wanted to 
 make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls 
 from within
 the GC entirely. It makes for a faster and better GC, but the 
 real
 reason here is that destructors are philosophically bankrupt 
 in a GC
 environment. I think there's no need to argue that in this 
 community.

 The GC never guarantees calling destructors even today, so 
 this decision
 would be just a point in the definition space (albeit an 
 extreme one).

 That means classes that need cleanup (either directly or by 
 having
 fields that are structs with destructors) would need to 
 garner that by
 other means, such as reference counting or manual. We're 
 considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as 
 follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently 
 lowered into
 RCSlice!T, defined inside object.d. That type would occupy 
 THREE words,
 one of which being a pointer to a reference count. That type 
 would
 redefine all slice primitives to update the reference count 
 accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit 
 cast(void[])
 will be allowed, and will ignore the reference count (so if a 
 void[]
 extracted from a T[] via a cast outlives all slices, dangling 
 pointers
 will ensue).

 I foresee any number of theoretical and practical issues with 
 this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei

Honestly, that sounds like the entierly wrong apporach to me. Your approaching the problem in this way: "We can not implement a propper GC in D because the language design prevents us from doing so. So lets remove destructors to migate the issue of false pointers." While the approach should be. "The language does not allow to implement a propper GC (anything else then dirty mark & sweep), what needs to be changed to allow a implementation of a more sophisticated GC."

Couldn't agree more. Abandoning destructors is a disaster. Without destructors, you effectively have manual memory management, or rather, manual 'resource' management, which is basically the same thing, even if you have a GC. It totally undermines the point of memory management as a foundational element of the language if most things are to require manual release/finalisation/destruction or whatever you wanna call it.
 Also let me tell you that at work we have a large C# codebase 
 which heavily
 relies on resource management. So basically every class in 
 there inherits
 from C#'s IDisposable interface which is used to manually call 
 the finalizer
 on the class (but the C# GC will also call that finalizer!). 
 Basically the
 entire codebase feels like manual memory management. You have 
 to think about
 manually destroying every class and the entire advantage of 
 having a GC,
 e.g. not having to think about memory management and thus 
 beeing more
 productive, vanished. It really feels like writing C++ with C# 
 syntax. Do we
 really want that for D?

This is interesting to hear someone else say this. I have always found C# - an alleged GC language - to result in extensive manual memory management in practise too. I've ranted enough about it already, but I have come to the firm conclusion that the entire premise of a mark&sweep GC is practically corrupt. Especially in D. Given this example that you raise with C#, and my own experience that absolutely parallels your example, I realise that GC's failure extends into far more cases than just the ones I'm usually representing. I also maintain that GC isn't future-proof in essence. Computers grow exponentially, and GC performance inversely tracks the volume of memory in the system. Anything with an exponential growth curve is fundamentally not future-proof. I predict a 2025 Wikipedia entry: "GC was a cute idea that existed for a few years in the early 2000's while memory ranged in the 100's mb - few gb's, but quickly became unsustainable as computer technology advanced".
 And what if I want unsafe slices of structs with destructors, 
 for
 performance? Maybe I perfectly know that the memory behind the 
 slice will
 outlive the slice, and I don't want the overhead of all the 
 reference
 counthing behind it?

 If you actually deprecate ~this, there would be two options 
 for me.
 1) Migrate my entire codebase to some user defiend finalizer 
 function (which
 doesn't have compiler support), which would be a lot of work.

Does ~this() actually work, or just usually work? Do you call your destructors manually like C#?
 2) Quit D. (which is becomeing more and more an option when 
 reading the
 recent news group discussions.)

I'm starting to fear the same outcome for myself. I don't have any idea how to reconcile this problem in my working environment, and very little community sympathy. I'm not seeing real solutions emerging, and the only one I can imagine that's even theoretically possible is wildly unpopular (ARC). For years, I just (naively?) assumed that the GC was immature, and would improve with time. Never gave it much thought; assumed there were people much smarter than me with a plan... I can't imagine any way out of this without significant breaking changes to the type system. Probably a new pointer type at least. This thread is starting to convince me that the GC should probably be repositioned as a convenience library provided *beside* the language, rather than a foundation of the language. It should be exclusively opt-in, not opt-out, and upon opting-in, you accept the associated problems. The revelation that established GC languages like C# are actually broken too hadn't occurred to me until now, but if I had to nominate the single biggest disaster in C# from my experience, that's certainly it; it's built on a GC, but isn't really compatible with it either. I'm constantly cleaning up manually in C#, which leaves very little value in the feature, and a definite tendency to produce unreliability by users who aren't experts on garbage collection and presume it should 'just work'. I support the notion that if the GC isn't removed as a foundational feature of D, then destructors should probably be removed from D. That said, I really want my destructors, and would be very upset to see them go. So... ARC?

You make some good arguments for ARC, but I think most of the discussion here is lacking in expertise. We need at least one serious memory management expert, possibly academic, to really set straight what is and isn't possible in D.
May 11 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:
 D also cannot be performance competitive with C++ if pervasive 
 ARC is used and memory safety is retained. Rust is attempting 
 to solve this problem by using 'borrowed' pointers, but this is 
 unproven technology, see my reply to Manu about it.

I work in a C++ shop and as I see it, resource management is becoming a solved problem: - resource owners holds a std::unique_ptr<T> on them. Resource release is taken care by C++ destructors normally. That means to be exception-safe, each resource type must better have its class. - resource users eventually "borrow" the resource by taking a raw pointer out of the unique pointer. What Rust would do with lifetimes here is ensure that the resource is still there since move semantics seems pervasive in this language. In C++ we ensure the resource holder outlive the users. - std::shared_ptr is not needed with such constraints. This means no cycles in the object graph. TBH I have yet to find a dependency scheme that can't be done that way. When I use D I can't help but think that releasing resources feels more manual and error-prone ("oops that resource should have been a struct not a class" and such traps). I do not have huge concerns about D GC, but I would be glad to have more support for owned pointers (ie. Unique!T in Phobos or better). I have no idea how to make it safe ie. ensure the resource outlive its users.
May 11 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:
 On 11.05.2014 10:22, Benjamin Thaut wrote:
 Am 10.05.2014 19:54, schrieb Andrei Alexandrescu:
 The next sentence goes on to list the advantages of RC 
 (issues we have
 wrestled with, like destructors), and then goes on to say 
 the recent
 awesome RC is within 10% of "the fastest tracing collectors".
 Are you suggesting that D's GC is among 'the fastest tracing
 collectors'? Is such a GC possible in D?

I believe it is.

While it might be possible to implement a good GC in D it would require major changes in the language and its librariers. In my opinion it would be way more work to implement a propper GC than to implement ARC. Every state of the art GC requires percise knowdelge of _all_ pointers. And thats exactly what we currently don't have in D.

I think most garbage collectors can work with a number of false pointers. The referenced memory area has to be treated as pinned and cannot be moved. Limiting the false pointers to stack and registers seems like a compromise, though most of the stack could even be annotated. Code for this does already exist in the debug info generation, though I suspect stack tracing could be unreliable. Here's my current stance on the GC discussions: I agree that the current GC is pretty lame, even if it were precise. "Stop-the-World" with complete tracing does not work for any interactive application that uses more than a few hundred MB of garbage collected memory (with or without soft-realtime requirements). Other applications with larger allocation requirements are easily dominated by collection time. Proposing to use manual memory management instead is admitting failure to me. For a reasonable GC I currently see 2 possible directions: Coventry 1. Use a scheme that takes a snapshot of the heap, stack and registers at the moment of collection and do the actual collection in another thread/process while the application can continue to run. This is the way Leandro Lucarellas concurrent GC works (http://dconf.org/2013/talks/lucarella.html), but it relies on "fork" that doesn't exist on every OS/architecture. A manual copy of the memory won't scale to very large memory, though it might be compressed to possible pointers. Worst case it will need twice as much memory as the current heap.

a) Can't we do our own specialised alternative, instead of doing something as heavyweight as fork? b) Surely that worst case is a very unlikely case, given a precise, concurrent garbage collector. c) Does this actually help in the 99% cpu, 99% memory, 16ms per frame setting that Manu talks about?
 It would be very interesting how far we can push this model on 
 the supported platforms.

 2. Change the compiler to emit (library defined) write barriers 
 for modifications of (possible) pointers. This will allow to 
 experiment with more sophisticated GC algorithms (if the write 
 barrier is written in D, we might also need pointers without 
 barriers to implement it). I know Walter is against this, and I 
 am also not sure if this adds acceptable overhead, but we don't 
 have proof of the opposite, too.

Adding optional write-barriers would be great. It would allow us to do real experiments and stop the discussion being so hypothetical. It doesn't do any harm to add the capability, surely?
 As we all know, the usual eager reference counting with atomic 
 operations is not memory-safe, so my current favorite is 
 "concurrent buffered reference counting" (see chapter 18.2/3 
 "The garbage collection handbook" by Richard Jones et al): 
 reference count modifications are not performed directly by the 
 write barrier, but it just logs the operation into a thread 
 local buffer. This is then processed by a collector thread 
 which also detects cycles (only on candidates which had their 
 reference count decreased during the last cycle). Except for 
 very large reference chains this scales with the number of 
 executed pointer modifications and not with the heap size.

May 11 2014
prev sibling next sibling parent "sclytrack" <sclytrack fake.com> writes:
On Sunday, 11 May 2014 at 11:42:37 UTC, ponce wrote:
 On Sunday, 11 May 2014 at 08:59:42 UTC, Walter Bright wrote:
 D also cannot be performance competitive with C++ if pervasive 
 ARC is used and memory safety is retained. Rust is attempting 
 to solve this problem by using 'borrowed' pointers, but this 
 is unproven technology, see my reply to Manu about it.

I work in a C++ shop and as I see it, resource management is becoming a solved problem: - resource owners holds a std::unique_ptr<T> on them. Resource release is taken care by C++ destructors normally. That means to be exception-safe, each resource type must better have its class. - resource users eventually "borrow" the resource by taking a raw pointer out of the unique pointer. What Rust would do with lifetimes here is ensure that the resource is still there since move semantics seems pervasive in this language. In C++ we ensure the resource holder outlive the users. - std::shared_ptr is not needed with such constraints. This means no cycles in the object graph. TBH I have yet to find a dependency scheme that can't be done that way. When I use D I can't help but think that releasing resources feels more manual and error-prone ("oops that resource should have been a struct not a class" and such traps). I do not have huge concerns about D GC, but I would be glad to have more support for owned pointers (ie. Unique!T in Phobos or better). I have no idea how to make it safe ie. ensure the resource outlive its users.

I like this owner/unique, borrow thing. is managed (currently reference counted) ~ is owner & is borrow fn example3() -> int { let mut x = ~X {f: 3}; let y = &x.f; x = ~X {f: 4}; // Error reported here. *y } According to Debian Rust is still too experimental to be packaged. http://web.mit.edu/rust-lang_v0.8/doc/tutorial-borrowed-ptr.html servo is written in Rust. https://github.com/mozilla/servo There is very little use of " ", it's mostly "&" and "~". Heck I didn't find any while casually browsing the code. It's like they are not using it at all. I don't know Rust, it is the first day I look (read documentation) at it.
May 11 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". Heck 
 I didn't find any   while casually browsing the code. It's like 
 they are not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.
May 11 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Sunday, 11 May 2014 at 21:49:06 UTC, ponce wrote:
 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I didn't find any   while casually browsing the code. 
 It's like they are not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.

I guess that after burning their fingers with auto_ptr, people still have to realize that unique_ptr might be good to use.
May 11 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Sunday, 11 May 2014 at 22:06:55 UTC, Francesco Cattoglio wrote:
 On Sunday, 11 May 2014 at 21:49:06 UTC, ponce wrote:
 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I didn't find any   while casually browsing the code. 
 It's like they are not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.

I guess that after burning their fingers with auto_ptr, people still have to realize that unique_ptr might be good to use.

sorry, my bad, I read it reversed :D
May 11 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 11 May 2014 at 20:45:08 UTC, Rainer Schuetze wrote:
 On 11.05.2014 22:33, Walter Bright wrote:
 The Boehm collector cannot move objects around, the D one 
 can.


Yes. D, for example, requires that objects not be self-referential for this reason.

I don't think the GC would have problems with fixing up internal pointers to the object itself. self-referential is prohibited to allow moving structures by memcpy, e.g. as return value.

Does this mean that you cannot safely implement something as basic as a circular linked list?
May 11 2014
prev sibling next sibling parent "Tommi" <tommitissari hotmail.com> writes:
On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 I like this owner/unique, borrow thing.

   is managed (currently reference counted)
 ~ is owner
 & is borrow

I like it too. But a few notes: 1) The managed pointer T has been deprecated and you should use the standard library types Gc<T> and Rc<T> instead. 2) The owned pointer ~T has been largely removed from the language and you should use the standard library type Box<T> instead. The basic idea is that if a function needs to have ownership of its argument, the function should take its argument by value. And if the function doesn't need the ownership, it should take its argument either by a mutable or immutable reference (they don't like to call it "borrowed pointer" anymore, it's called simply a "reference" now). Owned types get moved by default when you pass them to a function that takes its argument by value. You call the 'clone' method to make a copy of a variable of an owned type.
May 12 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 11 May 2014 at 18:18:41 UTC, Rainer Schuetze wrote:
 For a reasonable GC I currently see 2 possible directions:

 1. Use a scheme that takes a snapshot of the heap, stack and 
 registers at the moment of collection and do the actual 
 collection in another thread/process while the application can 
 continue to run. This is the way Leandro Lucarellas concurrent 
 GC works (http://dconf.org/2013/talks/lucarella.html), but it 
 relies on "fork" that doesn't exist on every OS/architecture. A 
 manual copy of the memory won't scale to very large memory, 
 though it might be compressed to possible pointers. Worst case 
 it will need twice as much memory as the current heap.

 It would be very interesting how far we can push this model on 
 the supported platforms.

 2. Change the compiler to emit (library defined) write barriers 
 for modifications of (possible) pointers. This will allow to 
 experiment with more sophisticated GC algorithms (if the write 
 barrier is written in D, we might also need pointers without 
 barriers to implement it). I know Walter is against this, and I 
 am also not sure if this adds acceptable overhead, but we don't 
 have proof of the opposite, too.

 As we all know, the usual eager reference counting with atomic 
 operations is not memory-safe, so my current favorite is 
 "concurrent buffered reference counting" (see chapter 18.2/3 
 "The garbage collection handbook" by Richard Jones et al): 
 reference count modifications are not performed directly by the 
 write barrier, but it just logs the operation into a thread 
 local buffer. This is then processed by a collector thread 
 which also detects cycles (only on candidates which had their 
 reference count decreased during the last cycle). Except for 
 very large reference chains this scales with the number of 
 executed pointer modifications and not with the heap size.

I'm surprised that you didn't include: 3. Thread-local GC, isolated zones (restricting where references to objects of a particular heap can be placed), exempting certain threads from GC completely, ...
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 13 May 2014 at 06:12:46 UTC, Rainer Schuetze wrote:
 On 13.05.2014 00:15, Martin Nowak wrote:

 There is a problem with this scheme, copy-on-write is extremely
 expensive when a mutation happens. That's one page fault 
 (context
 switch) + copying a whole page + mapping the new page.

I agree that this might be critical, but it is a one time cost per page. It seems unrealistic to do this with user mode exceptions, but the OS should have this optimized pretty well.

As I pointed out this won't help dynamic games that easily can touch 50000 pages per frame if you use a single global allocator. 2000 cycles * 50K = 100K = frame drop. What's worse, if you are low on memory you will start to swap to disk (or compress pages). So that means you have to optimize for collections, use dedicated allocators that keep dynamic data on the same pages etc... Basically you get the disadvantage of manual memory management and in the worst case the memory requirements of a copying GC without the benefits...
May 13 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:
 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:
 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where 
 references to
 objects of a particular heap can be placed), exempting certain 
 threads
 from GC completely, ...

This comes up from time to time, but to me it is very blurry how this can work in reality. Considering how "shared" is supposed to be used to be useful (do some locking, then cast away "shared") there is no guarantee by the language that any object is actually thread local (no references from other threads). Working with immutable (e.g. strings) is shared by design.

Yes, but only a part of the data is shared. I suspect the majority of the data in typical programs will be thread-local. If you use a message passing model, you can improve that even further (though it requires a way to move an object to another thread's heap). This way, you can - in the best case - avoid the shared heap completely.
May 13 2014
prev sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:
 This comes up from time to time, but to me it is very blurry 
 how this can work in reality.

Are there concerns I've missed that make that not applicable?
 Considering how "shared" is supposed to be used to be useful 
 (do some locking, then cast away "shared") there is no 
 guarantee by the language that any object is actually thread 
 local (no references from other threads). Working with 
 immutable (e.g. strings) is shared by design.

I'm not seeing much in the documentation, but from what I can tell (per the FAQ), shared in D just guarantees it's on the global heap? -Wyatt [0] https://research.microsoft.com/en-us/um/people/simonpj/papers/parallel/local-gc.pdf
May 13 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 2 May 2014 at 00:45:42 UTC, Andrei Alexandrescu wrote:
 Here's where the point derails. A struct may be preexisting; 
 the decision to define a destructor for it and the decision to 
 use polymorphism for an object that needs that structure are 
 most of the time distinct.

 Andrei

I wonder how common the pattern of putting a struct with a destructor in a class actually is. It might be a case for defining data structures with GC allocation rather than reference counting (as in std.container). I suppose a choice of allocator will change this quite a lot. That which is allocated with reference counting could uniquely hold its container member and then call the destructor when it dies, or similar. I find it kind of a funny thing to put something like a File inside of a class. I have always seen the mix of GC and resource management as more managing resources like Files in scopes and reading data from the resources which turn into objects in memory which are garbage collected. Not allocating garbage collected objects which contain resources.
May 03 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 2 May 2014 at 20:59:46 UTC, monarch_dodra wrote:
 Yeah, you have to read the "fine print": "collection implies 
 destruction" *but* "no guarantees the collection will actually 
 ever happen".

Which make destructors dangerous constructs. It means you now risk getting random bugs-reports after deployment. Acceptable for a hobby language, not acceptable for a systems programming language.
May 03 2014
prev sibling next sibling parent Caligo via Digitalmars-d <digitalmars-d puremagic.com> writes:
--001a11c2081065ca5004f88ae4ef
Content-Type: text/plain; charset=UTF-8

On Sat, May 3, 2014 at 3:49 AM, Benjamin Thaut via Digitalmars-d <
digitalmars-d puremagic.com> wrote:
 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)

 --
 Kind Regards
 Benjamin Thaut

I never thought I would say this, but I have begun to move away from D. I know others who are doing the same. The advantages D has over other languages are slowly diminishing. Plus, as it is, D is just as complex and complicated as C++, if not more. Couple that with the never-ending bugs and weak development process, there isn't much confidence in choosing D. One of the main and crippling issues D has is its development process. Its inner circle, mainly Walter, Andrei and a few others, do not seem to have a good understanding of FOSS development process. For example, Rust is only a few years old and much younger than D, but it has a greater number of contributors, and its rate of contributors seems to be growing faster. D has failed at recruitment; I always read people (mainly the inner circle) mention the everlasting low-hanging fruits. Well, if the number of contributors was growing then the number of low-hanging fruits would be decreasing. Andrei recently introduced the bounty system, which not only is an insult to those who contribute to FOSS, but it also goes to show that he really doesn't understand how and why people contribute to FOSS without ever asking to be compensated. Last but not least, currently there are two main ways for new features to make it into D/Phobos: you either have to belong to the inner circle, or have to represent some corporation that's doing something with D. I don't remember seeing a feature that was added to D/Phobos with some on/off switch that people could try in the next release, and then send in feedback. You're in a much better position to make a decision about a feature if the users have actually used it and reported feedback. Ahh, and don't get me started on Phobos review process; it's a joke, it's bogus and just pathetic. If D had a sound development process, I don't think we would be having the problems that we have today. --001a11c2081065ca5004f88ae4ef Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On S= at, May 3, 2014 at 3:49 AM, Benjamin Thaut via Digitalmars-d <span dir=3D"l= tr">&lt;<a href=3D"mailto:digitalmars-d puremagic.com" target=3D"_blank">di= gitalmars-d puremagic.com</a>&gt;</span> wrote:<blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x"> 2) Quit D. (which is becomeing more and more an option when reading the rec= ent news group discussions.)<span class=3D"HOEnZb"><font color=3D"#888888">= <br> <br> -- <br> Kind Regards<br> Benjamin Thaut<br> </font></span></blockquote></div><br></div><div class=3D"gmail_extra">I nev= er thought I would say this, but I have begun to move away from D. =C2=A0I = know others who are doing the same. =C2=A0The advantages D has over other l= anguages are slowly diminishing. =C2=A0Plus, as it is, D is just as complex= and complicated as C++, if not more. =C2=A0Couple that with the never-endi= ng bugs and weak development process, there isn&#39;t much confidence in ch= oosing D.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">One of the = main and crippling issues D has is its development process. =C2=A0Its inner= circle, mainly Walter, Andrei and a few others, do not seem to have a good= understanding of FOSS development process. =C2=A0For example, Rust is only= a few years old and much younger than D, but it has a greater number of co= ntributors, and its rate of contributors seems to be growing faster. =C2=A0= D has failed at recruitment; I always read people (mainly the inner circle)= mention the everlasting low-hanging fruits. =C2=A0Well, if the number of c= ontributors was growing then the number of low-hanging fruits would be decr= easing. =C2=A0Andrei recently introduced the bounty system, which not only = is an insult to those who contribute to FOSS, but it also goes to show that= he really doesn&#39;t understand how and why people contribute to FOSS wit= hout ever asking to be compensated.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">Last but no= t least, currently there are two main ways for new features to make it into= D/Phobos: you either have to belong to the inner circle, or have to repres= ent some corporation that&#39;s doing something with D. =C2=A0I don&#39;t r= emember seeing a feature that was added to D/Phobos with some on/off switch= that people could try in the next release, and then send in feedback. =C2= =A0You&#39;re in a much better position to make a decision about a feature = if the users have actually used it and reported feedback. =C2=A0Ahh, and do= n&#39;t get me started on Phobos review process; it&#39;s a joke, it&#39;s = bogus and just pathetic.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">If D had a = sound development process, I don&#39;t think we would be having the problem= s that we have today.</div><div class=3D"gmail_extra"><br></div><div class= =3D"gmail_extra"> <br></div></div> --001a11c2081065ca5004f88ae4ef--
May 03 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 3 May 2014 at 11:12:56 UTC, Michel Fortin wrote:
 On 2014-05-01 17:35:36 +0000, "Marc Schütz" <schuetzm gmx.net> 
 said:

 Maybe the language should have some way to distinguish between 
 GC-managed and manually-managed objects, preferably in the 
 type system. Then it could be statically checked whether an 
 object is supposed to be GC-managed, and consequentially 
 shouldn't have a destructor.

Or turn the rule on its head: make it so having a destructor makes the heap memory block reference counted. With this adding a destructor always cause deterministic destruction. The compiler knows statically whether a struct has a destructor. For a class you need a runtime trick because the root object which can be either. Use a virtual call or a magic value in the reference count field to handle the reference count management. You also need a way to tag a class to be guarantied it has no derived class with a destructor (to provide a static proof for the compiler it can omit ARC code), perhaps disable ~this(). Then remains the problem of cycles. It could be a hard error if the destructor is safe (error thrown when the GC collects it). The destructor could be allowed to run (in any thread) if the destructor is system or trusted. The interesting thing with this is that the current D semantics are preserved, destructors become deterministic (except in the presence of cycles, which the GC will detect for you), and if you're manipulating pointers to pure memory (memory blocks having no destructor) there's no ARC overhead. And finally, no new pointer attributes; Walter will like this last one.

This is certainly also an interesting idea, but I suspect it is bound to fail, simply because it involves ARC. Reference counting always makes things so much more complicated... See for example the cycles problem you mentioned: If you need a GC for that, you cannot guarantee that the objects will be collected, which was the reason to introduce ARC in the first place. Then there are the problems with shared vs. thread-local RC (including casting between the two), and arrays/slices of RC objects. And, of course, Walter doesn't like it ;-)
May 04 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
Initial post in this thread makes focus on a change that does not 
fix anything and implies silent semantical breakage. I am glad 
Andrei has reconsidered it but feels like key problem is not that 
proposal itself was bad but that it does not even try to solve.

Real issue being deterministic destruction and/or deallocation of 
polymorphic objects. Class destructors are bad for that indeed. 
But I want some better replacement before prohibiting anything.

There is a very practical use case that highlights existing 
problem, it was discussed few months ago in exception performance 
thread. To speed up throwing of non-const exceptions it is 
desirable to use pool of exception objects. However currently 
there is no place where to put code for releasing object back 
into pool. One cannot use destructor because it will never be 
called if pool keeps the reference. One cannot use reference 
counting struct wrapper because breaks polymorphic catching of 
exceptions.

Any good solution to be proposed should be capable of solving 
this problem.
May 05 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 5 May 2014 09:39:30 -0700
schrieb "H. S. Teoh via Digitalmars-d"
<digitalmars-d puremagic.com>:

 On Mon, May 05, 2014 at 03:55:12PM +0000, bearophile via Digitalmars-d wrote:
 Andrei Alexandrescu:
 
I think the "needs to support BigInt" argument is not a blocker - we
can release std.rational to only support built-in integers, and then
adjust things later to expand support while keeping backward
compatibility. I do think it's important that BigInt supports
appropriate traits to be recognized as an integral-like type.

Bigints support is necessary for usable rationals, but I agree this can't block their introduction in Phobos if the API is good and adaptable to the successive support of bigints.

Yeah, rationals without bigints will overflow very easily, causing many usability problems in user code.
If you, Joseph, or both would want to put std.rational again through
the review process I think it should get a fair shake. I do agree
that a lot of persistence is needed.

Rationals are rather basic (important) things, so a little of persistence is well spent here :-)

I agree, and support pushing std.rational through the queue. So, please don't give up, we need it get it in somehow. :) T

That experimental package idea that was discussed months ago comes to my mind again. Add that thing as exp.rational and have people report bugs or shortcomings to the original author. When it seems to be usable by everyone interested it can move into Phobos proper after the formal review (that includes code style checks, unit tests etc. that mere users don't take as seriously). As long as there is nothing even semi-official, it is tempting to write such a module from scratch in a quick&dirty fashion and ignore existing work. The experimental package makes it clear that this code is eventually going to the the official way and home brewed stuff wont have a future. Something in the standard library is much less likely to be reinvented. On the other hand, once a module is in Phobos proper, it is close to impossible to change the API to accommodate for a new use case. That's why I think the most focused library testing and development can happen in the experimental phase of a module. The longer it is, the more people will have tried it in their projects before formal review, which would greatly improve informed decisions. The original std.rationale proposal could have been in active use now for months! -- Marco
May 05 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 4 May 2014 19:00, via Digitalmars-d <digitalmars-d puremagic.com> wrote:
 On Saturday, 3 May 2014 at 11:12:56 UTC, Michel Fortin wrote:
 On 2014-05-01 17:35:36 +0000, "Marc Schütz" <schuetzm gmx.net> said:

 Maybe the language should have some way to distinguish between GC-managed
 and manually-managed objects, preferably in the type system. Then it could
 be statically checked whether an object is supposed to be GC-managed, and
 consequentially shouldn't have a destructor.

Or turn the rule on its head: make it so having a destructor makes the heap memory block reference counted. With this adding a destructor always cause deterministic destruction. The compiler knows statically whether a struct has a destructor. For a class you need a runtime trick because the root object which can be either. Use a virtual call or a magic value in the reference count field to handle the reference count management. You also need a way to tag a class to be guarantied it has no derived class with a destructor (to provide a static proof for the compiler it can omit ARC code), perhaps disable ~this(). Then remains the problem of cycles. It could be a hard error if the destructor is safe (error thrown when the GC collects it). The destructor could be allowed to run (in any thread) if the destructor is system or trusted. The interesting thing with this is that the current D semantics are preserved, destructors become deterministic (except in the presence of cycles, which the GC will detect for you), and if you're manipulating pointers to pure memory (memory blocks having no destructor) there's no ARC overhead. And finally, no new pointer attributes; Walter will like this last one.

This is certainly also an interesting idea, but I suspect it is bound to fail, simply because it involves ARC. Reference counting always makes things so much more complicated... See for example the cycles problem you mentioned: If you need a GC for that, you cannot guarantee that the objects will be collected, which was the reason to introduce ARC in the first place.

So specify that improper weak reference attribution may lead to interference with proper execution of destructors. People generally understand this, and at least they'd have such a tool to make their code behave correctly. Perhaps even have rules that things with destructors create static errors if they are used in a way that may create circular references when effective weak attribution is not detected by the compiler (if such a thing is statically possible?).
 Then there are the problems with shared vs. thread-local RC (including
 casting between the two),

The problem is exactly the same as 'shared' exists already. What makes it any different? shared <-> not-shared requires blunt casting, and the same can apply to shared RC. 'shared' implies RC access must use atomics, and otherwise not, I don't imagine any distinction in data structure?
 and arrays/slices of RC objects.

Slices need to know their offset (or base pointer), or have an explicit RC pointer. Either way, I don't see how slices are a particularly troublesome case. 12byte slices on x32 - needs an extra field, 16 bytes should still be sufficient on x64 considering that 64bit pointers are only 40-48 bits, which means there are loads of spare bits in the pointer and in the slice length field; should be plenty to stash an offset.
 And, of course,
 Walter doesn't like it ;-)

True. But I'm still waiting to see another even theoretically workable solution.
May 05 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 3 May 2014 18:49, Benjamin Thaut via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Am 30.04.2014 22:21, schrieb Andrei Alexandrescu:
 Walter and I have had a long chat in which we figured our current
 offering of abstractions could be improved. Here are some thoughts.
 There's a lot of work ahead of us on that and I wanted to make sure
 we're getting full community buy-in and backup.

 First off, we're considering eliminating destructor calls from within
 the GC entirely. It makes for a faster and better GC, but the real
 reason here is that destructors are philosophically bankrupt in a GC
 environment. I think there's no need to argue that in this community.

 The GC never guarantees calling destructors even today, so this decision
 would be just a point in the definition space (albeit an extreme one).

 That means classes that need cleanup (either directly or by having
 fields that are structs with destructors) would need to garner that by
 other means, such as reference counting or manual. We're considering
 deprecating ~this() for classes in the future.

 Also, we're considering a revamp of built-in slices, as follows. Slices
 of types without destructors stay as they are.

 Slices T[] of structs with destructors shall be silently lowered into
 RCSlice!T, defined inside object.d. That type would occupy THREE words,
 one of which being a pointer to a reference count. That type would
 redefine all slice primitives to update the reference count accordingly.

 RCSlice!T will not convert implicitly to void[]. Explicit cast(void[])
 will be allowed, and will ignore the reference count (so if a void[]
 extracted from a T[] via a cast outlives all slices, dangling pointers
 will ensue).

 I foresee any number of theoretical and practical issues with this
 approach. Let's discuss some of them here.


 Thanks,

 Andrei

Honestly, that sounds like the entierly wrong apporach to me. Your approaching the problem in this way: "We can not implement a propper GC in D because the language design prevents us from doing so. So lets remove destructors to migate the issue of false pointers." While the approach should be. "The language does not allow to implement a propper GC (anything else then dirty mark & sweep), what needs to be changed to allow a implementation of a more sophisticated GC."

Couldn't agree more. Abandoning destructors is a disaster. Without destructors, you effectively have manual memory management, or rather, manual 'resource' management, which is basically the same thing, even if you have a GC. It totally undermines the point of memory management as a foundational element of the language if most things are to require manual release/finalisation/destruction or whatever you wanna call it.
 Also let me tell you that at work we have a large C# codebase which heavily
 relies on resource management. So basically every class in there inherits
 from C#'s IDisposable interface which is used to manually call the finalizer
 on the class (but the C# GC will also call that finalizer!). Basically the
 entire codebase feels like manual memory management. You have to think about
 manually destroying every class and the entire advantage of having a GC,
 e.g. not having to think about memory management and thus beeing more
 productive, vanished. It really feels like writing C++ with C# syntax. Do we
 really want that for D?

This is interesting to hear someone else say this. I have always found C# - an alleged GC language - to result in extensive manual memory management in practise too. I've ranted enough about it already, but I have come to the firm conclusion that the entire premise of a mark&sweep GC is practically corrupt. Especially in D. Given this example that you raise with C#, and my own experience that absolutely parallels your example, I realise that GC's failure extends into far more cases than just the ones I'm usually representing. I also maintain that GC isn't future-proof in essence. Computers grow exponentially, and GC performance inversely tracks the volume of memory in the system. Anything with an exponential growth curve is fundamentally not future-proof. I predict a 2025 Wikipedia entry: "GC was a cute idea that existed for a few years in the early 2000's while memory ranged in the 100's mb - few gb's, but quickly became unsustainable as computer technology advanced".
 And what if I want unsafe slices of structs with destructors, for
 performance? Maybe I perfectly know that the memory behind the slice will
 outlive the slice, and I don't want the overhead of all the reference
 counthing behind it?

 If you actually deprecate ~this, there would be two options for me.
 1) Migrate my entire codebase to some user defiend finalizer function (which
 doesn't have compiler support), which would be a lot of work.

Does ~this() actually work, or just usually work? Do you call your destructors manually like C#?
 2) Quit D. (which is becomeing more and more an option when reading the
 recent news group discussions.)

I'm starting to fear the same outcome for myself. I don't have any idea how to reconcile this problem in my working environment, and very little community sympathy. I'm not seeing real solutions emerging, and the only one I can imagine that's even theoretically possible is wildly unpopular (ARC). For years, I just (naively?) assumed that the GC was immature, and would improve with time. Never gave it much thought; assumed there were people much smarter than me with a plan... I can't imagine any way out of this without significant breaking changes to the type system. Probably a new pointer type at least. This thread is starting to convince me that the GC should probably be repositioned as a convenience library provided *beside* the language, rather than a foundation of the language. It should be exclusively opt-in, not opt-out, and upon opting-in, you accept the associated problems. The revelation that established GC languages like C# are actually broken too hadn't occurred to me until now, but if I had to nominate the single biggest disaster in C# from my experience, that's certainly it; it's built on a GC, but isn't really compatible with it either. I'm constantly cleaning up manually in C#, which leaves very little value in the feature, and a definite tendency to produce unreliability by users who aren't experts on garbage collection and presume it should 'just work'. I support the notion that if the GC isn't removed as a foundational feature of D, then destructors should probably be removed from D. That said, I really want my destructors, and would be very upset to see them go. So... ARC?
May 05 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 05 May 2014 17:24:38 +0000
schrieb "Dicebot" <public dicebot.lv>:

 That experimental package idea that was discussed months ago
 comes to my mind again. Add that thing as exp.rational and
 have people report bugs or shortcomings to the original
 author. When it seems to be usable by everyone interested it
 can move into Phobos proper after the formal review (that
 includes code style checks, unit tests etc. that mere users
 don't take as seriously).

And same objections still remain.

Sneaky didn't work this time. -- Marco
May 05 2014
prev sibling next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Sunday, 11 May 2014 at 12:52:29 UTC, Michel Fortin wrote:
 On 2014-05-11 08:29:13 +0000, Walter Bright 
 <newshound2 digitalmars.com> said:

 Again, O-C and C++/CX ARC are not memory safe because in order 
 to make it perform they provide unsafe escapes from it.

But D could provide memory-safe escapes. If we keep the current GC to collect cycles, we could also allow raw pointers managed by the GC alongside ARC. Let's say we have two kinds of pointers: rc+gc pointers (the default) and gc_only pointers (on demand). When assigning from a rc+gc pointer to a gc_only pointer, the compiler emits code that disables destruction via the reference counting. This makes the GC solely responsible for destructing and deallocating that memory block. You can still assign the pointer to a rc+gc pointer later on, but the reference count is no longer reliable which is why RC-based destruction has been disabled.

You know, this doesn't sound that far off from what Python does, unless I'm completely wrong about it. I believe that Python uses reference counting and uses GC to collect cycles in a way like you have described. I'm not sure how efficient it is. Python people don't tend to talk about speed that much.
May 11 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 07 May 2014 06:50:33 +0000
schrieb "Paulo Pinto" <pjmlp progtools.org>:

 A *nix package manager is brain dead idea for software
 development as it ties the language libraries to the specific OS
 one is using.

What do you have in mind here? I don't quite get the picture. Are you opposed to a developer writing a package only for his pet distribution of Linux and ignoring all others? The typical packages I know come with some configure script and offer enough hooks to custom tailer the installation, so the library/application can work on any distribution. Most of the Linux software is open source and and their packages are maintained by the community around the specific distribution. That doesn't preclude the use of package managers like Cabal, CPAN, Maven, you-name-it. But for a systems language integration with the existing C/C++ is of utmost importance. After all, D compiles to 'regular' native binaries. Typically when an application is useful, someone will add it to their favorite distribution as a package including all the libraries as dependencies.
 Good luck getting packages if the author did not consider your
 OS. Specially if they are only available in binary format, as it
 is standard in the enterprise world.

Then use dub. Oh wait... the packages on code.dlang.org are open source, too. And at least since the curl debacle we know that there is not one binary for all *nix systems. I don't know where you are trying to get with this argument. I think it has nothing to do with what dub strives for and is worth a separate topic "Binary D library distribution".
 With a language pakage manager I can produce package XPTO that
 will work on all OS, it won't conflict with the system packages,
 specially important on servers used for CI of multiple projects.
 
 --
 Paulo

What is this XPTO that will magically work on all OS? I never heard of it, but I'm almost certain it has to do with languages that compile at most to machine independent byte code. And why do you run CI on the live system instead of a chroot environment, if you are afraid of messing it up? :) I do trust my package manager to correctly install libraries into a chroot. It is as simple as prepending an environment variable override. As a bonus you can then cleanly uninstall/update libs in the chroot environment with all the sanity checks the package manager may offer. A language package manager is a good idea, but there are certain limits for it you leave the development stage. At that point the system package manager takes over. Both should be considered with equal care. -- Marco
May 11 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 11 May 2014 14:41:10 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 Your proposal still relies on a GC to provide the memory safety,
 [=E2=80=A6] it is a hybrid ARC/GC system.

But I thought ARC cannot be designed without GC to resolve cycles. Or is your comment pure rethoric? --=20 Marco
May 11 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 11 May 2014 17:50:25 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot sto=

 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do=E2=84=A2. Your general purpose function does not know how the memory was allocated, that it receives a pointer to. In particular it must not assume that it is safe to keep a reference to it as there are several memory management schemes that are incompatible with that, like reference counting or stack allocations. Expanding on these two, Rust can now safely use _more_ allocation schemes with functions that take borrowed pointers than is safely possible in D! RC pointers: You cannot pass them as raw pointers in D. In Rust they can be passed as borrowed. Stack pointers: Not allowed in D in safe code and inherently unsafe in system code. Again this is safe to do in Rust due to borrowing.
 The similarity is that there are=20
 one way conversions from one to the other, and one of the types is more g=

 I infer from your other statements about Rust that it doesn't actually ha=

 general pointer type.

Yes it does: http://static.rust-lang.org/doc/0.10/guide-unsafe.html#raw-pointers But the design principle in Rust is to only have them in system code (speaking in D terms), in particular to interface with C. Turning the argument back to D and assuming you wrote a function that takes a raw pointer because you plan to store it in a global variable. How do you make sure you get a pointer to something with infinite life-time? Let me answer this: You either use GC pointers exclusively or you rely on the convention that the function takes ownership of the memory. The former is impractical and the latter cannot be statically enforced. Borrowed pointers add an safe way to deal with the situation in all contexts where you don't need to store a reference. But if you _do_ need that capability: ask explicitly for GC pointers as they can guarantee unlimited life-time. If that's still too restrictive mark it system and use raw pointers (in Rust: unsafe keyword). Finally, this is not Rust vs. D, because D has had borrowed pointer function arguments since ages as well - maybe even longer than Rust. The semantics of "in/scope" were just never fully implemented. Once this is done we can also write: safe void main() { auto stack =3D 42; foo(&stack); } safe void foo(scope int*); --=20 Marco
May 11 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote:
 On 5/11/14, 2:49 PM, ponce wrote:
 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I
 didn't find any   while casually browsing the code. It's like 
 they are
 not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.

The right tally here would include bald pointers as well. Also, I'm quite surprised by the confidence of this assertion. -- Andrei

Well I wa'm probably much biased indeed. I have no data.
May 12 2014
prev sibling next sibling parent "ponce" <contact gam3sfrommars.fr> writes:
On Monday, 12 May 2014 at 00:44:54 UTC, Andrei Alexandrescu wrote:
 On 5/11/14, 2:49 PM, ponce wrote:
 On Sunday, 11 May 2014 at 21:43:06 UTC, sclytrack wrote:
 There is very little use of " ", it's mostly  "&" and "~". 
 Heck I
 didn't find any   while casually browsing the code. It's like 
 they are
 not using it at all.

Similarly in current C++ std::shared_ptr is barely there while std::unique_ptr is all over the place.

The right tally here would include bald pointers as well. Also, I'm quite surprised by the confidence of this assertion. -- Andrei

Well I was talking with no data indeed.
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 But I don't see how this could work in Rust.

Ask it to competent Rust developers/programmers. Bye, bearophile
May 12 2014
prev sibling next sibling parent "John Colvin" <john.loughran.colvin gmail.com> writes:
On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:
 What? You've never offered me a practical solution.

I have, you've just rejected them.
 What do I do?

1. you can simply do C++ style memory management. shared_ptr<>, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times.

The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from "well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler" to "it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause." And here we go again, "I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself..."
May 12 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:
 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 The only solutions to the libraries problem that I can see here 
 require drastic separation of calls to said libraries from any 
 even vaguely time critical code. This is quite restrictive.

I think this is more of library writing culture problem than engineering problem. High quality library shouldn't rely on any internal allocations at all, deferring this decision to user code. Otherwise you will eventually have problems, GC or not.
May 12 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:
 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:
 What? You've never offered me a practical solution.

I have, you've just rejected them.
 What do I do?

1. you can simply do C++ style memory management. shared_ptr<>, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times.

The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from "well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler" to "it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause." And here we go again, "I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself..."

A badly placed malloc() in library code can also trigger OS virtualization mechanisms and make processes being swapped out to disk, with the respective overhead in disk access and time spent on kernel code. So it is just not the "we can spare a few nano-seconds". -- Paulo
May 12 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sun, 11 May 2014 22:11:28 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 But I thought ARC cannot be designed without GC to resolve
 cycles.

It can be, there are various schemes to deal with that, including "don't =

 cycles". GC is just one of them.
=20
 http://en.wikipedia.org/wiki/Reference_counting#Dealing_with_reference_cy=

Yes that article mentions: a) "avoid creating them" b) "explicitly forbid reference cycles" c) "Judicious use of weak references" d) "manually track that data structure's lifetime" e) "tracing garbage collector" f) adding to a root list all objects whose reference count is decremented to a non-zero value and periodically searching all objects reachable from those roots. To pick up your statement again: =C2=BBYour proposal still relies on a GC to provide the memory safety, [=E2=80=A6] it is a hybrid ARC/GC system.=C2=AB a) and b) let's assume never creating cycles is not a feasible option in a systems programming language c) and d) don't provide said memory safety e) and f) ARE tracing garbage collectors ergo: =C2=BBBut I thought ARC cannot be designed without GC to resolve cycles.=C2=AB Your were arguing against Michel Fortin's proposal on the surface, when your requirement cannot even be fulfilled theoretically it seems. Which could mean that you don't like the idea of replacing D's GC with an ARC solution. =C2=BBThis is the best horse I could find for the price. It is pretty fast and ...=C2=AB =C2=BBNo, it still has four legs.=C2=AB --=20 Marco
May 12 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 May 2014 01:54:58 -0700
schrieb Walter Bright <newshound2 digitalmars.com>:

 On 5/11/2014 10:57 PM, Marco Leise wrote:
 Am Sun, 11 May 2014 17:50:25 -0700
 schrieb Walter Bright <newshound2 digitalmars.com>:

 As long as those pointers don't escape. Am I right in that one cannot =



 borrowed pointer into a global data structure?

Right, and that's the point and entirely positive-to-do=E2=84=A2.

This means that a global data structure in Rust has to decide what memory=

 allocation scheme its contents must use, and cannot (without tagging) mix=

 allocation schemes.
=20
 For example, let's say a compiler has internally a single hash table of s=

 With a GC, those strings can be statically allocated, or on the GC heap, =

 anything with a lifetime longer than the table's. But I don't see how thi=

 work in Rust.

:( Good question. I have no idea. --=20 Marco
May 12 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Mon, 12 May 2014 09:32:58 +0000
schrieb "Paulo Pinto" <pjmlp progtools.org>:

 On Monday, 12 May 2014 at 09:05:39 UTC, John Colvin wrote:
 On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:
 What? You've never offered me a practical solution.

I have, you've just rejected them.
 What do I do?

1. you can simply do C++ style memory management. shared_ptr<>, etc. 2. you can have the non-pausible code running in a thread that is not registered with the gc, so the gc won't pause it. This requires that this thread not allocate gc memory, but it can use gc memory allocated by other threads, as long as those other threads retain a root to it. 3. D allows you to create and use any memory management scheme you want. You are simply not locked into GC. For example, I rewrote my Empire game into D and it did not do any allocation at all - no GC, not even malloc. I know that you'll need to do allocation, I'm just pointing out that GC allocations and pauses are hardly inevitable. 4. for my part, I have implemented nogc so you can track down gc usage in code. I have also been working towards refactoring Phobos to eliminate unnecessary GC allocations and provide alternatives that do not allocate GC memory. Unfortunately, these PR's just sit there. 5. you can divide your app into multiple processes that communicate via interprocess communication. One of them pausing will not pause the others. You can even do things like turn off the GC collections in those processes, and when they run out of memory just kill them and restart them. (This is not an absurd idea, I've heard of people doing that effectively.) 6. If you call C++ libs, they won't be allocating memory with the D GC. D code can call C++ code. If you run those C++ libs in separate threads, they won't get paused, either (see (2)). 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory with malloc/free, and (ironically) only using GC for persistent data structures that should never be free'd. Then, I just turned off GC collections, because they'd never free anything anyway. 8. you can disable and enable collections, and you can cause collections to be run at times when nothing is happening (like when the user has not input anything for a while). The point is, the fact that D has 'new' that allocates GC memory simply does not mean you are obliged to use it. The GC is not going to pause your program if you don't allocate with it. Nor will it ever run a collection at uncontrollable, random, asynchronous times.

The only solutions to the libraries problem that I can see here require drastic separation of calls to said libraries from any even vaguely time critical code. This is quite restrictive. Yes, calling badly optimised libraries from a hot loop is a bad idea anyway, but the GC changes this from "well it might take a little more time than usual, but we can spare a few nano-seconds and it'll show up easily in the profiler" to "it might, sometimes, cause the GC to run a full collection on our 3.96 / 4.00 GB heap with an associated half-second pause." And here we go again, "I can't use that library, it's memory management scheme is incompatible with my needs, I'll have to rewrite it myself..."

A badly placed malloc() in library code can also trigger OS virtualization mechanisms and make processes being swapped out to disk, with the respective overhead in disk access and time spent on kernel code. So it is just not the "we can spare a few nano-seconds". -- Paulo

Yes, it could easily extend to a longer wait. I think we all know programs that hang while the system is swapping out. Don't let it get to that! A PC game would typically reduce caches or texture resolutions before running out of RAM. Linux has a threshold of free pages it tries to keep available at any time to satisfy occasional small allocations. http://www.science.unitn.it/~fiorella/guidelinux/tlk/node39.html All-in-all malloc is less likely to cause long pauses. It just allocates and doesn't ask itself if there might be dead memory to salvage to satisfy a request. Time will tell if all "well written" D libraries will be nogc to move the question of allocations to the user. -- Marco
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Timon Gehr:

 (Probably more, actually, because it does not provide 
 precision-unfriendly constructs such as undiscriminated unions.)

I suggested to add an optional method named "onGC" to unions that if present is called at run-time by the GC to know what's the real type of stored data, to make tracing more precise. Bye, bearophile
May 12 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 03:39:12 -0400, Manu via Digitalmars-d  
<digitalmars-d puremagic.com> wrote:

 On 12 May 2014 17:24, Dicebot via Digitalmars-d

 You will like Don's talk this year ;)

I'm super-disappointed I can't make it this year!

?!! http://dconf.org/2014/talks/evans.html
 We were evicted from
 our house, have to move, and I can't bail for a week and leave that
 all on my mrs while she kicks along the fulltime job :(

Oh that sucks... -Steve
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 12 May 2014 18:45, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/12/2014 12:12 AM, Manu via Digitalmars-d wrote:
 What? You've never offered me a practical solution.

I have, you've just rejected them.
 What do I do?

1. you can simply do C++ style memory management. shared_ptr<>, etc.

I already have C++. I don't want another one.
 2. you can have the non-pausible code running in a thread that is not
 registered with the gc, so the gc won't pause it. This requires that this
 thread not allocate gc memory, but it can use gc memory allocated by other
 threads, as long as those other threads retain a root to it.

It still sounds the same as manual memory management though in practise, like you say, the other thread must maintain a root to it, which means I need to manually retain it somehow, and when the worker thread finishes with it, it needs to send a signal or something back to say it's done so it can be released... it sounds more inconvenient than direct manual memory management in practise. Sounds slow too. Dec-ing a ref is certainly faster than inter-thread communication. This also makes library calls into effective RPC's if I can't call into them from the active threads. How long is a collect liable to take in the event the GC threads need to collect? Am I likely to lose my service threads for 100s of milliseconds at a time? I'll think on it, but I don't think there's anything practically applicable here, and it really sounds like it creates a lot more trouble and complexity than it addresses.
 3. D allows you to create and use any memory management scheme you want. You
 are simply not locked into GC. For example, I rewrote my Empire game into D
 and it did not do any allocation at all - no GC, not even malloc. I know
 that you'll need to do allocation, I'm just pointing out that GC allocations
 and pauses are hardly inevitable.

C++ lets me create any memory management scheme I like by the same argument. I lose all the parts of the language that implicitly depend on the GC, and 3rd party libs (that don't care about me and my project). Why isn't it a reasonable argument to say that not having access to libraries is completely unrealistic? You can't write modern software without extensive access to libraries. Period. I've said before, I don't want to be a second class citizen with access to only a subset of the language.
 4. for my part, I have implemented  nogc so you can track down gc usage in
 code. I have also been working towards refactoring Phobos to eliminate
 unnecessary GC allocations and provide alternatives that do not allocate GC
 memory. Unfortunately, these PR's just sit there.

The effort is appreciated, but it was never a solution. I said nogc was the exact wrong approach to my situation right from the start, and I predicted that would be used as an argument the moment it appeared. Tracking down GC usage isn't helpful when it leads you to a lib call that you can't change. And again, eliminating useful and productive parts of the language is not a goal we should be shooting for. I'll find it useful in the high-performance realtime bits; ie, the bits that I typically disassemble and scrutinise after every compile. But that's not what we're discussing here. I'm happy with D for my realtime code, I have the low-level tools I need to make the real-time code run fast. nogc is a little bonus that will allow to guarantee no sneaky allocations are finding their way into the fast code, and that might save a little time, but I never really saw that as a significant problem in the first place. What we're talking about is productivity, convenience and safety in the non-realtime code. The vast majority of code, that programmers spend most of their days working on. Consider it this way... why do you have all these features in D that cause implicit allocation if you don't feel they're useful and important parts of the language? Assuming you do feel they're important parts of the language, why do you feel it's okay to tell me I don't deserve access to them? Surely I'm *exactly* the target market for D...? High-pressure, intensive production environments, still depending exclusively on native code, with code teams often in the realm of 50-100, containing many juniors, aggressive schedules which can't afford to waste engineering hours... this is a code environment that's prone to MANY bugs, and countless wasted hours as a consequence. Convenience and safety are important to me... I don't know what you think I'm interested in D for if you think I should be happy to abandon a whole chunk of the language, just because I have a couple of realtime threads :/
 5. you can divide your app into multiple processes that communicate via
 interprocess communication. One of them pausing will not pause the others.
 You can even do things like turn off the GC collections in those processes,
 and when they run out of memory just kill them and restart them. (This is
 not an absurd idea, I've heard of people doing that effectively.)

Most of the platforms I work on barely have operating systems.
 6. If you call C++ libs, they won't be allocating memory with the D GC. D
 code can call C++ code. If you run those C++ libs in separate threads, they
 won't get paused, either (see (2)).

Whether this is practical or not thoroughly depends on the lib. Maybe this concept can be applicable in some small places, but it's not a salvation. I don't think this sufficient addresses the problems. None of the problems are actually going away, they're just moved somewhere else
 7. The Warp program I wrote avoids GC pauses by allocating ephemeral memory
 with malloc/free, and (ironically) only using GC for persistent data
 structures that should never be free'd. Then, I just turned off GC
 collections, because they'd never free anything anyway.

That idea is obviously not applicable in my environment. Resource usage is dynamic and fluid.
 8. you can disable and enable collections, and you can cause collections to
 be run at times when nothing is happening (like when the user has not input
 anything for a while).

If I disable collections, then I just crash when I receive that network packet? I'm back at manual memory management in practise. I also don't think it's reasonable to assume there will just be 'times when nothing is happening'. That's not how games work. Games are often really fast paced, and even if they're not, significant stuttering in the animation is usually considered a non-ship-able bug. https://www.youtube.com/watch?v=rqjOXR9QnMo https://www.youtube.com/watch?v=giiZMktZrNI https://www.youtube.com/watch?v=LoPC_ibBJiQ Where would you manually issue collects?
 The point is, the fact that D has 'new' that allocates GC memory simply does
 not mean you are obliged to use it.

D also has ~, closures, dynamic arrays, even array literals. There are various things that create implicit GC allocations. And library calls...
 The GC is not going to pause your
 program if you don't allocate with it. Nor will it ever run a collection at
 uncontrollable, random, asynchronous times.

Those claims come with massive dependency on very specific restrictions, like abandoning part of the language and moving library calls to separate threads and accessing them via RPC or something like that. None of your suggestions sound practical, or like they'd result in any less effort or complexity than manual management in the first place which everyone is already accustomed to. I'm almost certainly sacrificing safety in every case. You can't then go on to say you gave me plenty of options, but I rejected them, when none of them were really options. I wonder if you have a good conception of the scope/scale of the software we write. It's not comparable to Empire, or a linker, or a compiler, or a web server, or many things at all really. Games are some of the biggest, broadest software projects there are, very tightly integrated, with some of the most stringent operating requirements. They're also growing steadily... it's harder and harder to manage the scope without helpful language tools; this is why you see so many gamedevs in the independent space flirting with 'modern' languages like C#. For games with extremely small scope that don't push the platform (indy/casual games), this is sometimes okay, but there are plenty of cases where it has been a complete disaster as the scope has grown towards a more traditional 'big game'. My mates game I helped them with from last weekend is 'mid-scoped', but it's grown to saturate the PS4, and the GC is causing them a nightmare... right at the end of the project when trying to finalise the build for shipping, precisely as I've always predicted. C# is a better productivity experience than C++, and it allowed them to do a lot more work in a lot less time with a lot fewer people. But it's clearly not really compatible with the workload, and I think the future of the industry needs to do a lot better. I've said before, we are an industry in desperate need of salvation, it's LONG overdue, and I want something that actually works well for us, not a crappy set of compromises because the language has a fundamental incompatibility with my industry :/ ... It doesn't have to be that way.
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Manu:

 we are an industry in desperate need of salvation,
 it's LONG overdue, and I want something that actually works 
 well for us, not a crappy set of compromises because the
 language has a fundamental incompatibility with my industry :/

Perhaps the game industry has to start the creation of a language designed for its needs, like the scientific people have done (Julia), the browser ones (Rust), the Web ones have done, etc. With lot of work in less than ten years you can have an usable language. Bye, bearophile
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 08:45:56 UTC, Walter Bright wrote:
 2. you can have the non-pausible code running in a thread that 
 is not registered with the gc, so the gc won't pause it. This 
 requires that this thread not allocate gc memory, but it can 
 use gc memory allocated by other threads, as long as those 
 other threads retain a root to it.

This and nogc is a very promising trend, but you should still be able to partion the GC search space. The key to controlled real time performance is to partition the search space, that goes for anything algorithmic; memory management inclusive. That applies to any scheme like owned, ARC, GC etc. It makes little sense having to trace everything if only the physics engine is the one churning memory like crazy. And fork() is not a solution without extensive structuring of allocations. Stuff like physics touch all pages the physics objects are onto like 100+ times per second, so you need to group allocations to pages based on usage patterns. (No point in forking if you get write traps on 50.000 pages the next time the physics engine run :-).
 3. D allows you to create and use any memory management scheme 
 you want. You are simply not locked into GC. For example, I 
 rewrote my Empire game into D and it did not do any allocation 
 at all - no GC, not even malloc. I know that you'll need to do 
 allocation, I'm just pointing out that GC allocations and 
 pauses are hardly inevitable.

This is no doubt the best approach for a MMO client. You have a window on the world and cache as much as possible both to memory and disk. Basically get as much memory from the OS as you can hoard (with headroom set by heuristics) when your application has focus and release caches that are outside the window when you loose focus to another application. This means you need a dedicated runtime for games that can delay GC collection and eat into the caches when you are low on computational resources. You also need to distinguish between memory that is locked to RAM and memory that can swap. You should always lock memory for real time threads. So if you want to GC this, you need a GC that support multiple heaps. (Some hardware might also distinguish between RAM that is accessible by the GPU or that has different characteristics in areas such as persistence or performance.)
 5. you can divide your app into multiple processes that 
 communicate via interprocess communication. One of them pausing 
 will not pause the others. You can even do things like turn off

Why processes and not threads with their own local GC?
 6. If you call C++ libs, they won't be allocating memory with 
 the D GC. D code can call C++ code. If you run those C++ libs

But what happens if that C++ code does "new HeapStuff(D_allocated_memory)" and then calls back to D? You cannot presume that C++ coders have the discipline to always allocate local memory from the stack, so basically you cannot GC collect while there are C++ functions on the stack. In order to get there the GC collector needs to understand the malloc heap and trace that one too. Auditing all C++ libraries I want to use is too much work, and tracing the malloc heap is too time consuming, so at the end of the day you'll get a more robust environment by only scanning (tracing) the stacks when there is only D function calls on the stack, with a precise collector. That means you need to partition the search space otherwise the collector might not run in time. Freezing the world is really ugly. Most applications are actually soft real time. Games are part hard real time, part soft real time. The difference between games and other applications is that there is less headroom so you have to do more work to make the "glitches" and "stuttering" occur sufficiently seldom to be perceived as acceptable by the end user. But games are not special.
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 02:16, bearophile via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 Manu:


 we are an industry in desperate need of salvation,
 it's LONG overdue, and I want something that actually works well for us,
 not a crappy set of compromises because the
 language has a fundamental incompatibility with my industry :/

Perhaps the game industry has to start the creation of a language designed for its needs, like the scientific people have done (Julia), the browser ones (Rust), the Web ones have done, etc. With lot of work in less than ten years you can have an usable language.

But D is *so close*... and I like it! >_< I have to say that this discussion has certainly left me somewhat intrigued by Rust though. I've never given it a fair go because I find the syntax so distasteful and deterring. I wonder if there's a market for a rust fork that re-skin's the language ;)
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 16:16:06 UTC, bearophile wrote:
 Perhaps the game industry has to start the creation of a 
 language designed for its needs, like the scientific people 
 have done (Julia), the browser ones (Rust), the Web ones have 
 done, etc. With lot of work in less than ten years you can have 
 an usable language.

I don't think games are unique or special. Most games are even in the "easy" space by having mostly static data. Meaning the amount of unexpected dynamic data is pretty low. Games also have the luxury of redefining the requirements spec to match available technology. The games industry does however have its own culture and paradigms and fashions… With subcultures. However, most interactive applications will suffer from the same issues if you increase the load so that they run out of headroom. Even unix commands like find and grep have latency requirements if the interaction is to be pleasant. By good fortune "find" and "grep" haven't changed their interface for 40+ years, so they were designed for low performance CPUs. That does not mean that you cannot design a better "find"-like application today that will run into runtime related usability issues if you freeze the world. At the end of the day, a system level language should support key strategies used for writing performant system level code in a reliable manner. It should also not lock you to a specific runtime that you couldn't easily write yourself. It should also not lock you to a specific model of how to structure your code (like monitors). I am not even sure it should provide OS abstractions, because that is not really system level programming. That is unixy (Posix) programming. A system level programming language should be free of OS and modelling related legacy.
May 12 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d 
wrote:
 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me 
 somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so 
 distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the 
 language ;)

Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about "good coding practices". It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber.
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 Unions of pointers are so rare in actual code that treating 
 them conservatively is not a big problem.

std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough. Bye, bearophile
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 03:17, Walter Bright via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On 5/12/2014 4:35 AM, bearophile wrote:
 I suggested to add an optional method named "onGC" to unions that if
 present is
 called at run-time by the GC to know what's the real type of stored data,
 to
 make tracing more precise.

Unions of pointers are so rare in actual code that treating them conservatively is not a big problem.

I find it fairly common. I just searched through my code, and 7 out of 12 unions had pointers.
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 03:14, Dicebot via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 May 2014 at 17:03:41 UTC, Manu via Digitalmars-d wrote:
 But D is *so close*... and I like it! >_<

 I have to say that this discussion has certainly left me somewhat
 intrigued by Rust though.
 I've never given it a fair go because I find the syntax so distasteful
 and deterring.
 I wonder if there's a market for a rust fork that re-skin's the language
 ;)

Right now D has practical benefit of being more stable and library rich. But switching to Rust eventually does seem tempting as I find foundations of their type system much closer to my beliefs about "good coding practices". It lacks any good static reflection though. And this stuff is damn addictive when you try it of D caliber.

They have a lot more work to do. There doesn't seem to be a useful windows compiler for a start... >_<
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 BTW, the RTinfo can be used to discriminate unions.

I don't know if std.variant.VariantN is already using such RTinfo. I don't know much about RTinfo. Bye, bearophile
May 12 2014
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d
wrote:
 How long is a collect liable to take in the event the GC 
 threads need
 to collect? Am I likely to lose my service threads for 100s of
 milliseconds at a time?

 I'll think on it, but I don't think there's anything practically
 applicable here, and it really sounds like it creates a lot more
 trouble and complexity than it addresses.

Your concerns stem not as much from the speed concern of the GC, but from the freeze-the-world aspect of it. Would a concurrent collector not solve these issues? As http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection explains a little bit, the actual time your threads spend frozen should be little (but I admit I don't know exactly how little), and so long as you don't allocate too much during the collection itself (which you say you don't), you should be able to keep running your code during the collection. If it's not possible to implement concurrent collection in D (and it's already been shown it is possible), then I'd agree that ARC is very important. But depending on how little the stop-the-world time from a concurrent GC can get, perhaps this could work around some issues that you're desiring ARC for. A generational collector could help in theory with your high memory usage situations. I doubt you allocate a gigabyte each frame, so the actual generation 0 content should be fairly low. Much of your memory usage should be allocations that will not be freed for long periods of time, while the per-frame and other short allocations should be fast to collect as there aren't many of them. Depending on how tunable the GC is, I feel like it should be possible to get away with a GC even for soft real-time programs like games. The problem is it's hard to tell until we get a proper concurrent collector in D2, just like it's hard to tell how significant the impact of ARC is until we get an optimized implementation of it in the compiler. Neither of these is simple. I do quite like the idea of ARC, it's just something that someone would have to actually implement (well) in order to see how much of an impact it really has in D. For the truly low frequency situations, you could get away with a library type for ARC as well, and as you mentioned, for high frequency you would get around ARC regardless.
May 12 2014
prev sibling next sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 13 May 2014 04:07, Kapps via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 On Monday, 12 May 2014 at 16:03:28 UTC, Manu via Digitalmars-d
 wrote:
 How long is a collect liable to take in the event the GC threads need

 to collect? Am I likely to lose my service threads for 100s of
 milliseconds at a time?

 I'll think on it, but I don't think there's anything practically
 applicable here, and it really sounds like it creates a lot more
 trouble and complexity than it addresses.

Your concerns stem not as much from the speed concern of the GC, but from the freeze-the-world aspect of it. Would a concurrent collector not solve these issues?

I originally thought it would... but the more I think on it, I don't think it would make an awful lot of difference in practise. If the stalls were 'short' (like 1-5ms on the background threads, 500µs-1ms on the realtime threads), then maybe it would be workable, but I don't know that it would be even close to that? Also, I think it would be very difficult to implement on a machine without virtual memory, or much of an operating system in general? The problem remains that with no free memory, frequency of collection becomes so high, that it's extremely unlikely full collection so often would be better than ARC.
 As http://msdn.microsoft.com/en-us/library/ee787088%28v=vs.110%29.aspx#concurrent_garbage_collection
 explains a little bit, the actual time your threads spend frozen
 should be little (but I admit I don't know exactly how little),
 and so long as you don't allocate too much during the collection
 itself (which you say you don't), you should be able to keep
 running your code during the collection. If it's not possible to
 implement concurrent collection in D (and it's already been shown
 it is possible), then I'd agree that ARC is very important. But
 depending on how little the stop-the-world time from a concurrent
 GC can get, perhaps this could work around some issues that
 you're desiring ARC for. A generational collector could help in
 theory with your high memory usage situations. I doubt you
 allocate a gigabyte each frame, so the actual generation 0
 content should be fairly low. Much of your memory usage should be
 allocations that will not be freed for long periods of time,
 while the per-frame and other short allocations should be fast to
 collect as there aren't many of them.

Yeah, it would probably be better, if it's possible. Implementation needs to be considered from the perspective of embedded systems with no OS or MMU, and as little as 64mb of ram (the smallest modern systems). Mid-range systems are 512mb and no MMU. 'next-gen' systems are basically like little PC's with crappy OS's, so more likely a decent GC is possible on a ps4/xbone... but very few have the luxury of developing for just one system. It occurred to me earlier that things like strings might enjoy their own separate heap. And maybe some special logic for strings that outlived their scope to be actively returned to their heap rather than waiting for collection. If the heap were successfully broken down into a suite of sub-heaps, I have absolutely no idea how to make estimates about the performance of this system, and if it would approach an acceptable level. I'm skeptical it would, and it still won't decrease collection frequency. But I'd be happy to be surprised.
 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games. The problem is it's hard to tell until we get a
 proper concurrent collector in D2, just like it's hard to tell
 how significant the impact of ARC is until we get an optimized
 implementation of it in the compiler. Neither of these is simple.
 I do quite like the idea of ARC, it's just something that someone
 would have to actually implement (well) in order to see how much
 of an impact it really has in D.

I understand the problem. The first hurdle is overcoming the hostility against it though. There is a severe prejudice.
 For the truly low frequency
 situations, you could get away with a library type for ARC as
 well, and as you mentioned, for high frequency you would get
 around ARC regardless.

Yup.
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote:
 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games.

Even if you manage to make it work for game clients you also should make it work for low latency game servers, as code sharing is an important advantage. What a game/world server requires differs a lot, but highly dynamic and flexible worlds have to keep the physics to a single node (or tight cluster) for a region. That means you want to have as many players as possible tied to that node. In essence you want both performance, low latency, reliability, and little overhead in an evolutionary context (it has to support heavy modification over time). My gut feeling is that a runtime satisfying one game design will not satisfy another one as long as one insists on one global GC. In essence, it will never really work well. IMO, the same goes for ARC since RC does not perform well with multi-threading even when you use near optimal patterns and strategies. If ARC is only to be used where speed does not matter then you might as well use shared_ptr.
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 I, too, felt the need of onGC() - actually preGC() - in my 
 allocators implementation.
 ...
 A hook that nulls all freelist heads just as the collection 
 process starts would be helpful.

How is this going to help increase tracing precision of unions (and Algebraic built on top of unions)? Bye, bearophile
May 12 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 How did I give the impression it has anything to do with 
 unions? -- Andrei

OK, so yours is not an answer to my proposal, nor related to it. Bye, bearophile
May 12 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Mon, 12 May 2014 13:52:20 -0400, Walter Bright  
<newshound2 digitalmars.com> wrote:

 On 5/12/2014 7:46 AM, Steven Schveighoffer wrote:
 It doesn't matter where the false pointers are. The largest issue with  
 false
 pointers is not how many false pointers there are. It only matters how  
 large the
 block is that it "points" at. The larger your blocks get, the more  
 likely they
 are "pointed" at by the stack. On 32-bit systems, allocate 1/256th of  
 your
 memory space (i.e. 16.5MB), and the likelihood of random data on the  
 stack
 pointing at it is roughly 1/256. This problem is just about eliminated  
 with
 64-bit pointers.

Generally, it is a bad idea to allocate such large blocks on the GC heap. GC's work best when the size of the objects being allocated is very small relative to the size of the heap space. Fortunately, it's a mathematical inevitability that large allocations relative to the GC size are rare, and so it isn't much of a pain to handle them manually.

The issue arises when one allocates such a large block for temporary use repeatedly, but expects it to be collected between allocations. The consequences are extremely disastrous. The workaround is simply to keep it around, but that's not always a scalable solution.
 And in fact, even if it's forbidden, "requires" is too strong a word --  
 there is
 no static or runtime prevention of this.

It's still forbidden. Andrei wrote a template that will verify this at runtime, but I don't recall its name.

Can you cite the spec where it says it's forbidden? Forgotten templates are not a convincing argument. Regardless, Java can use a moving GC, and allows self references. The idea that self references prevent a moving GC is simply false. If you think about it a bit, you will understand why. -Steve
May 12 2014
prev sibling next sibling parent "Kapps" <opantm2+spam gmail.com> writes:
On Monday, 12 May 2014 at 19:13:50 UTC, Ola Fosheim Grøstad wrote:
 On Monday, 12 May 2014 at 18:07:51 UTC, Kapps wrote:
 Depending on how tunable the GC is, I feel like it should be
 possible to get away with a GC even for soft real-time programs
 like games.

Even if you manage to make it work for game clients you also should make it work for low latency game servers, as code sharing is an important advantage. What a game/world server requires differs a lot, but highly dynamic and flexible worlds have to keep the physics to a single node (or tight cluster) for a region. That means you want to have as many players as possible tied to that node. In essence you want both performance, low latency, reliability, and little overhead in an evolutionary context (it has to support heavy modification over time). My gut feeling is that a runtime satisfying one game design will not satisfy another one as long as one insists on one global GC. In essence, it will never really work well. IMO, the same goes for ARC since RC does not perform well with multi-threading even when you use near optimal patterns and strategies. If ARC is only to be used where speed does not matter then you might as well use shared_ptr.

.NET allows configuring the garbage collector by specifying workstation (concurrent, background [allow generation 0/1 collection while a generation 2 collection is going], one primary heap and a large object heap) or server (not certain if concurrent/background, but multiple heaps that get handled in parallel during collections). Or in situations where you have many processes running at once, disabling concurrent collection to reduce context switching overhead. In reality, most people leave the default concurrent collector, which is what I'd hope the default for D would be, but if it was sufficiently tunable something like vibe.d could decide to go with something more similar to what .NET uses for servers (which ASP.NET uses by default). I haven't been able to find good concrete numbers online, but the few sources I've found say that generation 0/1 collection tends to take <1 to 2-3 milliseconds and is not run concurrently because it's so short. This is quite sufficient for most projects, but perhaps could be tweaked a bit more for certain aspects like gaming, possibly even enabling concurrent collection for generation 0/1, but I'm not sure if this works well or is feasible. Still, the important thing is to get a good general one to use first, like the default one .NET uses for workstation applications.
May 12 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 12 May 2014 at 22:27:06 UTC, Kapps wrote:
 because it's so short. This is quite sufficient for most
 projects, but perhaps could be tweaked a bit more for certain
 aspects like gaming, possibly even enabling concurrent 
 collection
 for generation 0/1, but I'm not sure if this works well or is
 feasible. Still, the important thing is to get a good general 
 one
 to use first, like the default one .NET uses for workstation
 applications.

I agree that getting a good (100% precise) GC is an important first step. I am not so sure about generation based GC when you have a window on a world map that you move around which roughly is FIFO (first in, first out). But to get good speed I think you are better off having multiple pools that can be released with no collection when a network-connection drops (if you have one conceptual pool per connection), and optimized allocators that give you pre-initialized objects etc. In the ideal world all of this is transparent once you have specified your memory model (in detail), so you only have to issue a "new PlayerConnection" in the main logic of your program and can tweak the memory handling elsewhere. That is not the D way, from what I can tell from the forum posts so far, because "new" is going to stay tied to one global GC heap. So you have to write utility functions… which makes programs less legible.
May 12 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 12 May 2014 at 19:32:49 UTC, Jacob Carlborg wrote:
 On 2014-05-12 19:14, Dicebot wrote:

 It lacks any good static reflection though. And this stuff is 
 damn
 addictive when you try it of D caliber.

It has macros, that basically requires great support for static reflection to be usable.

Judging by http://static.rust-lang.org/doc/0.6/tutorial-macros.html those are not full-blown AST macros like ones you have been proposing, more like hygienic version of C macros.
May 13 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 12:14:06 -0400, Ola Fosheim Gr=C3=B8stad  =

<ola.fosheim.grostad+dlang gmail.com> wrote:

 On Tuesday, 13 May 2014 at 14:13:31 UTC, Steven Schveighoffer wrote:
 I know this is coming from a position of extreme ignorance, but why d=


 we have to do copy on write? What about pause on write?

Not sure how that will help? Pointers may still escape collection? (but you get that with transactional memory, on the cache level)

My understanding is that the way the fork collector works is that it mak= es = pages copy-on-write. Then if the original process writes to one of the = pages, the page is copied, which may be expensive in terms of total memo= ry = and time consumed. The idea I had was to make them pause-on-write. This means, when the = original process attempts to write to the page, it gets a page-fault, = which pauses the thread until the collector is done with it. This causes= = the same halting that normally happens with stop-the-world, but only = on-demand, instead of preemptively. If a thread is doing only reads, or = is = only touching non-Scanned memory, it continues. A collector may be able to take advantage of this knowledge to avoid as = = many pauses as possible, but I'm not sure. Just an idea, I make no claims to its actual benefits :) -Steve
May 13 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy  
<yebbliesnospam gmail.com> wrote:

 "Steven Schveighoffer"  wrote in message  
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...

 If a thread is doing only reads, or is  only touching non-Scanned  
 memory, it continues.

How long do threads usually go without touching the stack?

The stack would have to be COW. -Steve
May 13 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 13 May 2014 at 17:37:05 UTC, Steven Schveighoffer 
wrote:
 On Tue, 13 May 2014 13:36:17 -0400, Daniel Murphy 
 <yebbliesnospam gmail.com> wrote:

 "Steven Schveighoffer"  wrote in message 
 news:op.xfs6jhp3eav7ka stevens-macbook-pro-2.local...

 If a thread is doing only reads, or is  only touching 
 non-Scanned memory, it continues.

How long do threads usually go without touching the stack?

The stack would have to be COW.

When you fork a process, all of its pages are COW. What you propose is much more complicated. You would either first need to fork(), then somehow remap certain pages as shared. The remapping would have to happen in the parent process. I don't know whether there are even interfaces to do this.
May 13 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 13 May 2014 at 17:22:19 UTC, Steven Schveighoffer 
wrote:
 The idea I had was to make them pause-on-write. This means, 
 when the original process attempts to write to the page, it 
 gets a page-fault, which pauses the thread until the collector 
 is done with it. This causes the same halting that normally 
 happens with stop-the-world, but only on-demand, instead of 
 preemptively. If a thread is doing only reads, or is only 
 touching non-Scanned memory, it continues.

That would require a separate page table and would freeze threads real fast for global GC. It can work for local heaps like caches? I guess you could set up shared memory between two processes and use that as your heap.
 Just an idea, I make no claims to its actual benefits :)

Another idea is to use transactions and roll back the collector if someone does a write, for small heaps with few writes... Principle: run the collector when a core is idle and yield to cores that do real work. But I dont think current gen CPUs can handle big transactions...
May 13 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:
 On 13.05.2014 13:09, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Tuesday, 13 May 2014 at 06:06:40 UTC, Rainer Schuetze wrote:
 On 12.05.2014 13:53, "Marc Schütz" <schuetzm gmx.net>" wrote:
 I'm surprised that you didn't include:

 3. Thread-local GC, isolated zones (restricting where 
 references to
 objects of a particular heap can be placed), exempting 
 certain threads
 from GC completely, ...

This comes up from time to time, but to me it is very blurry how this can work in reality. Considering how "shared" is supposed to be used to be useful (do some locking, then cast away "shared") there is no guarantee by the language that any object is actually thread local (no references from other threads). Working with immutable (e.g. strings) is shared by design.

Yes, but only a part of the data is shared. I suspect the majority of the data in typical programs will be thread-local. If you use a message passing model, you can improve that even further (though it requires a way to move an object to another thread's heap). This way, you can - in the best case - avoid the shared heap completely.

Possible, but this is with a language without shared data (including immutable), not D. Safely transferring a large object tree from one thread to another will be very expensive. If you try to optimize that, the D compiler won't help you to guarantee memory safety.

It certainly requires a few modifications to the language. That's why I think the current discussions about uniqueness, scope, isolated are so important, because it would fit together very well with memory management concepts like ARC, isolated heaps, and safety. For example, if the uniqueness concept that Walter recently added to DMD were not only temporary (only used for implicit conversion to immutable, AFAIU), but a real, permanent type modifier, transferring such a tree can be made safe at practically no runtime costs, apart from notifying the GC about the changed ownership. (No copying needs to happen, because the important thing for thread-local heaps is actually just in which thread references to objects can be stored, not necessarily that the actual objects are placed into physically distinct heaps.)
 Actually I don't have much experience with "shared" (I usually 
 use __gshared if I need a shared global). Maybe I understand 
 the idea better if you show what happens with this code when 
 run with thread local GC:

 class C { C next; }

 shared(C) list;

 C newC()
 {
 	return new C; // allocated on the local heap?

Yes, because it doesn't say "new shared(C)".
 }	

 void prependToList(C c)
 {
 	synchronized(list)
 	{
 		C lst = cast(C) list;
 		c.next = lst;  // anything special happening here?
 		list = cast(shared(C)) c;  // and here?

The casts are a problem. You're basically lying to the compiler, so this cannot work. Instead, it should be something like: auto newC() { return new isolated(C); } void prependToList(isolated(C) c) { synchronized(list) { // transfer to shared heap // afterwards, c is no longer usable ("consumed") shared(C) tmp = c.makeShared(); tmp.next = list; list = tmp; } } This uses the "isolated" concept suggested by deadalnix here: http://forum.dlang.org/thread/yiwcgyfzfbkzcavuqdwz forum.dlang.org?page=1
 	}
 }

 void callMeFromMultipleThreads()
 {
 	prependToList(newC());
 }

May 14 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Walter Bright:

 On 5/12/2014 10:25 AM, bearophile wrote:
 Walter Bright:

 Unions of pointers are so rare in actual code that treating 
 them
 conservatively is not a big problem.

std.variant.Algebraic is based on on std.variant.VariantN, and on std.variant.VariantN is based on an union, and often you use algebraic data types to represent trees and similar data structures that contain many references/pointers. Adding Adding an onGC() method to std.variant.VariantN you allow the GC to manage Algebraic well enough.

BTW, the RTinfo can be used to discriminate unions.

Do I have to open a Bugzilla entry asking for VariantN to use such RTinfo to allow a more precise tracing of pointer-heavy Algebraics? Suggestions are welcome. Bye, bearophile
May 14 2014
prev sibling next sibling parent "Wyatt" <wyatt.epp gmail.com> writes:
On Tuesday, 13 May 2014 at 19:56:20 UTC, Rainer Schuetze wrote:
 I just read the first chapters, and according to that, existing 
 local gcs needs write barriers, so we are back to my second 
 proposal. The implementation in the paper even adds read 
 barriers.

for any modern scalable GC. At the very least, the algorithms they enable are important if D is really committed to the pluggable GC concept. (Given the ongoing discussion on performance tradeoffs of various GC methods and the needs of different groups, I'd even suggest it's a really important feature.)
 To me, shared is a type modifier that doesn't implicitely 
 convert to anything else without casting.

Interesting, maybe this should be clarified better. Having skimmed back over chapter 13 of TDPL, my understanding of its semantics are that it only really enforces atomicity and execution order. Also, this bit from near the beginning of 13.12 states: "For all numeric types and function pointers, shared-qualified values are convertible implicitly to and from unqualified values." That sounds kind of at-odds with your interpretation...? :/ -Wyatt
May 14 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Thursday, 15 May 2014 at 06:30:40 UTC, Jacob Carlborg wrote:
 On 14/05/14 23:47, Dmitry Olshansky wrote:

 This is curious:
 http://burntsushi.net/rustdoc/regex/

It seems they have compile time regular expressions too.

Yeah looks like my knowledge about their macro system is very outdated. Looks promising.
May 15 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 16 May 2014 at 16:58:38 UTC, Rainer Schuetze wrote:
 On 14.05.2014 12:56, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Tuesday, 13 May 2014 at 19:50:52 UTC, Rainer Schuetze wrote:
 class C { C next; }

 shared(C) list;

 C newC()
 {
    return new C; // allocated on the local heap?

Yes, because it doesn't say "new shared(C)".
 }

 void prependToList(C c)
 {
    synchronized(list)
    {
        C lst = cast(C) list;
        c.next = lst;  // anything special happening here?
        list = cast(shared(C)) c;  // and here?

The casts are a problem. You're basically lying to the compiler, so this cannot work. Instead, it should be something like: auto newC() { return new isolated(C); } void prependToList(isolated(C) c) { synchronized(list) { // transfer to shared heap // afterwards, c is no longer usable ("consumed") shared(C) tmp = c.makeShared(); tmp.next = list; list = tmp; } }

Sorry for the late reply. I currently don't know of a feasible way to replace casting away "shared" if you want to do more than just "tmp.next = list;", for example call "bool check(C c);" on tmp or list? I can't implement this with a shared object, because any operations on "shared" tend to not exist (ok, maybe I could for this very simple example, so consider C contains a container to operate on). Also, the object is already locked, I should not have to do it again in a function if it only accepts a shared C. How can I prepend something to the list that is not isolated? I guess I cannot, as it might create references to same object both shared and unshared. My gut feeling tells me that this ownership handling through the type system might work, but it is very restrictive and does not really contribute to the first goal of D listed on the front page: "convenience".

I haven't thought it through completely, of course, but I believe it doesn't have to be inconvenient. Maybe there is a nice solution. I already have something in mind, but I have to think it over first.
May 17 2014