www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Idea #1 on integrating RC with GC

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
Consider we add a library slice type called RCSlice!T. It would have the 
same primitives as T[] but would use reference counting through and 
through. When the last reference count is gone, the buffer underlying 
the slice is freed. The underlying allocator will be the GC allocator.

Now, what if someone doesn't care about the whole RC thing and aims at 
convenience? There would be a method .toGC that just detaches the slice 
and disables the reference counter (e.g. by setting it to uint.max/2 or 
whatever).

Then people who want reference counting say

auto x = fun();

and those who don't care say:

auto x = fun().toGC();


Destroy.

Andrei
Feb 04 2014
next sibling parent reply "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Feb 2014 18:51:35 -0500, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Consider we add a library slice type called RCSlice!T. It would have the  
 same primitives as T[] but would use reference counting through and  
 through. When the last reference count is gone, the buffer underlying  
 the slice is freed. The underlying allocator will be the GC allocator.

Doesn't that mean it lives in the GC heap and is scanned along with all the other data in the GC heap (and triggers GC cycles)? What is the benefit?
 Now, what if someone doesn't care about the whole RC thing and aims at  
 convenience? There would be a method .toGC that just detaches the slice  
 and disables the reference counter (e.g. by setting it to uint.max/2 or  
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();

Shouldn't the default be what is expected now? That is, I don't want to have to change all my code to return RCSlice!T instead of T[]. I admit, I don't know how that would work... -Steve
Feb 04 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 4:01 PM, Steven Schveighoffer wrote:
 On Tue, 04 Feb 2014 18:51:35 -0500, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> wrote:

 Consider we add a library slice type called RCSlice!T. It would have
 the same primitives as T[] but would use reference counting through
 and through. When the last reference count is gone, the buffer
 underlying the slice is freed. The underlying allocator will be the GC
 allocator.

Doesn't that mean it lives in the GC heap and is scanned along with all the other data in the GC heap (and triggers GC cycles)? What is the benefit?

GC.free is the benefit.
 Shouldn't the default be what is expected now? That is, I don't want to
 have to change all my code to return RCSlice!T instead of T[]. I admit,
 I don't know how that would work...

Me neither. Andrei
Feb 04 2014
prev sibling next sibling parent reply "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 15:51:35 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 Consider we add a library slice type called RCSlice!T. It would have the  
 same primitives as T[] but would use reference counting through and  
 through. When the last reference count is gone, the buffer underlying  
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at  
 convenience? There would be a method .toGC that just detaches the slice  
 and disables the reference counter (e.g. by setting it to uint.max/2 or  
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

I am assuming that you ignoring cyclic-refs for now. And dealing with them would mean that you would need something like: auto weak x = fun(); Since the compiler cannot figure out that this is supposed to be weak or not it would assume strong as that is the majority. Second thought. Why is their an extra step for those of us who don't want the GC. I know there has to be an extra step for some people, but I feel that we should make the simplest option the default and then open up avenues for more advanced control for people who want it. ARC is not as simple to understand as a GC for newbies, and ARC requires more careful control, so why not make getting into ARC a little harder, that way we prevent heartache for new people. So something more like: auto x = fun().toARC(); I can't imagine any situation where you could go from ARC to GC that could not also go from GC to ARC, they have to be equivalent operations to do that anyways. In the design world you make the simple option easy and more the more advanced options harder to get at to reduce the chance that the user accidentally shoot themselves in the foot. And the guys who want to the advanced options are power users anyways, they expect to have to work at it a little more. Related third thought. This goes against the current paradigm and will subtly break existing D code that has cyclic-refs in it. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 4:07 PM, Adam Wilson wrote:
 I am assuming that you ignoring cyclic-refs for now.

The GC will lift cycles.
 Second thought. Why is their an extra step for those of us who don't
 want the GC.

The flow goes from RC to losing RC and relying on GC. You can't add RC to some arbitrary allocation safely. Andrei
Feb 04 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 5:06 PM, Adam Wilson wrote:
 Ok, disregard my previous post. I think I understand better what Andrei
 is driving at, but it's not obvious, better examples are needed. But
 more to the point this is going to make us all equally miserable. The
 ARC guys don't get the compiler support needed to make ARC fast.

Of course they could. Compiler can have internal support for RC slices. Object will need some built-in support anyway.
 It
 completely ignores cyclic-refs inside the slice.

Wrong. It leaves cyclic references to the GC.
 It doesn't solve the
 primary ARC-camp complaint about the reliance on the GC.

Why not?
 The GC now guys
 have more hoops to jump-through when working with Phobos.

Why?
 And it
 increases complexity and cognitive load all around.

That comes with the territory. Have no illusion we can add RC support in any way at no cost. Andrei
Feb 04 2014
prev sibling next sibling parent "woh" <wojw yahoo.com> writes:
He said *new* library type, so obviously it would not break 
existing code since nothing uses it.

  Wednesday, 5 February 2014 at 00:07:30 UTC, Adam Wilson wrote:
 On Tue, 04 Feb 2014 15:51:35 -0800, Andrei Alexandrescu 
 <SeeWebsiteForEmail erdani.org> wrote:

 Consider we add a library slice type called RCSlice!T. It 
 would have the same primitives as T[] but would use reference 
 counting through and through. When the last reference count is 
 gone, the buffer underlying the slice is freed. The underlying 
 allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and 
 aims at convenience? There would be a method .toGC that just 
 detaches the slice and disables the reference counter (e.g. by 
 setting it to uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

I am assuming that you ignoring cyclic-refs for now. And dealing with them would mean that you would need something like: auto weak x = fun(); Since the compiler cannot figure out that this is supposed to be weak or not it would assume strong as that is the majority. Second thought. Why is their an extra step for those of us who don't want the GC. I know there has to be an extra step for some people, but I feel that we should make the simplest option the default and then open up avenues for more advanced control for people who want it. ARC is not as simple to understand as a GC for newbies, and ARC requires more careful control, so why not make getting into ARC a little harder, that way we prevent heartache for new people. So something more like: auto x = fun().toARC(); I can't imagine any situation where you could go from ARC to GC that could not also go from GC to ARC, they have to be equivalent operations to do that anyways. In the design world you make the simple option easy and more the more advanced options harder to get at to reduce the chance that the user accidentally shoot themselves in the foot. And the guys who want to the advanced options are power users anyways, they expect to have to work at it a little more. Related third thought. This goes against the current paradigm and will subtly break existing D code that has cyclic-refs in it.

Feb 04 2014
prev sibling next sibling parent "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 auto x = fun().toGC();

Recently I remember Kenji saying that we are now almost able to implement "dup" in library code. If that happens, why don't you call it "dup"? Bye, bearophile
Feb 04 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Feb 2014 19:15:24 -0500, woh <wojw yahoo.com> wrote:

 He said *new* library type, so obviously it would not break existing  
 code since nothing uses it.

I think the possible goal is that phobos functions that now return slices would return this (at some point in the future), to give the option of using RC or GC with any phobos calls. -Steve
Feb 04 2014
prev sibling next sibling parent "Steven Schveighoffer" <schveiguy yahoo.com> writes:
On Tue, 04 Feb 2014 19:42:55 -0500, bearophile <bearophileHUGS lycos.com>  
wrote:

 Andrei Alexandrescu:

 auto x = fun().toGC();

Recently I remember Kenji saying that we are now almost able to implement "dup" in library code. If that happens, why don't you call it "dup"?

No, toGC does not create a copy. It just detaches the RC component, and relies on GC to clean up the resulting slice (I assume typeof(fun().toGC()) == T[]) dup would imply that you wanted a copy in addition to the reference-counted version. In fact, you may want a copy that is also reference counted. -Steve
Feb 04 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 16:07:29 -0800, Adam Wilson <flyboynw gmail.com> wrote:

 On Tue, 04 Feb 2014 15:51:35 -0800, Andrei Alexandrescu  
 <SeeWebsiteForEmail erdani.org> wrote:

 Consider we add a library slice type called RCSlice!T. It would have  
 the same primitives as T[] but would use reference counting through and  
 through. When the last reference count is gone, the buffer underlying  
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at  
 convenience? There would be a method .toGC that just detaches the slice  
 and disables the reference counter (e.g. by setting it to uint.max/2 or  
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

I am assuming that you ignoring cyclic-refs for now. And dealing with them would mean that you would need something like: auto weak x = fun(); Since the compiler cannot figure out that this is supposed to be weak or not it would assume strong as that is the majority. Second thought. Why is their an extra step for those of us who don't want the GC. I know there has to be an extra step for some people, but I feel that we should make the simplest option the default and then open up avenues for more advanced control for people who want it. ARC is not as simple to understand as a GC for newbies, and ARC requires more careful control, so why not make getting into ARC a little harder, that way we prevent heartache for new people. So something more like: auto x = fun().toARC(); I can't imagine any situation where you could go from ARC to GC that could not also go from GC to ARC, they have to be equivalent operations to do that anyways. In the design world you make the simple option easy and more the more advanced options harder to get at to reduce the chance that the user accidentally shoot themselves in the foot. And the guys who want to the advanced options are power users anyways, they expect to have to work at it a little more. Related third thought. This goes against the current paradigm and will subtly break existing D code that has cyclic-refs in it.

Ok, disregard my previous post. I think I understand better what Andrei is driving at, but it's not obvious, better examples are needed. But more to the point this is going to make us all equally miserable. The ARC guys don't get the compiler support needed to make ARC fast. It completely ignores cyclic-refs inside the slice. It doesn't solve the primary ARC-camp complaint about the reliance on the GC. The GC now guys have more hoops to jump-through when working with Phobos. And it increases complexity and cognitive load all around. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu 
wrote:
 Consider we add a library slice type called RCSlice!T. It would 
 have the same primitives as T[] but would use reference 
 counting through and through. When the last reference count is 
 gone, the buffer underlying the slice is freed. The underlying 
 allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and 
 aims at convenience? There would be a method .toGC that just 
 detaches the slice and disables the reference counter (e.g. by 
 setting it to uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

RC need GC to collect loops. So you want to have the GC at the lowest level. That being understood, I'd rather connect things the other way around. auto x = foo(); auto x = foo().toRC();
Feb 04 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 5:12 PM, deadalnix wrote:
 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It would have
 the same primitives as T[] but would use reference counting through
 and through. When the last reference count is gone, the buffer
 underlying the slice is freed. The underlying allocator will be the GC
 allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the
 slice and disables the reference counter (e.g. by setting it to
 uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

RC need GC to collect loops. So you want to have the GC at the lowest level.

Correctamundo.
 That being understood, I'd rather connect things the other way around.

 auto x = foo();
 auto x = foo().toRC();

I don't know how to implement that. Andrei
Feb 04 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 17:12:37 -0800, deadalnix <deadalnix gmail.com> wrote:

 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It would have  
 the same primitives as T[] but would use reference counting through and  
 through. When the last reference count is gone, the buffer underlying  
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at  
 convenience? There would be a method .toGC that just detaches the slice  
 and disables the reference counter (e.g. by setting it to uint.max/2 or  
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

RC need GC to collect loops. So you want to have the GC at the lowest level. That being understood, I'd rather connect things the other way around. auto x = foo(); auto x = foo().toRC();

The ARC crowd is going to hate this because it's still a GC allocation then you hook it to RC. So they can still have random GC pauses. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Wednesday, 5 February 2014 at 01:52:48 UTC, Adam Wilson wrote:
 On Tue, 04 Feb 2014 17:12:37 -0800, deadalnix 
 <deadalnix gmail.com> wrote:

 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei 
 Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It 
 would have the same primitives as T[] but would use reference 
 counting through and through. When the last reference count 
 is gone, the buffer underlying the slice is freed. The 
 underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing 
 and aims at convenience? There would be a method .toGC that 
 just detaches the slice and disables the reference counter 
 (e.g. by setting it to uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

RC need GC to collect loops. So you want to have the GC at the lowest level. That being understood, I'd rather connect things the other way around. auto x = foo(); auto x = foo().toRC();

The ARC crowd is going to hate this because it's still a GC allocation then you hook it to RC. So they can still have random GC pauses.

GC.disable. Just saying.
Feb 04 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 18:08:32 -0800, Sean Kelly <sean invisibleduck.org>  
wrote:

 On Wednesday, 5 February 2014 at 01:52:48 UTC, Adam Wilson wrote:
 On Tue, 04 Feb 2014 17:12:37 -0800, deadalnix <deadalnix gmail.com>  
 wrote:

 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It would have  
 the same primitives as T[] but would use reference counting through  
 and through. When the last reference count is gone, the buffer  
 underlying the slice is freed. The underlying allocator will be the  
 GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims  
 at convenience? There would be a method .toGC that just detaches the  
 slice and disables the reference counter (e.g. by setting it to  
 uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

RC need GC to collect loops. So you want to have the GC at the lowest level. That being understood, I'd rather connect things the other way around. auto x = foo(); auto x = foo().toRC();

The ARC crowd is going to hate this because it's still a GC allocation then you hook it to RC. So they can still have random GC pauses.

GC.disable. Just saying.

Hmm, as long as it doesn't disable the allocation that might just work. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 20:21:17 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 2/4/14, 5:06 PM, Adam Wilson wrote:
 Ok, disregard my previous post. I think I understand better what Andrei
 is driving at, but it's not obvious, better examples are needed. But
 more to the point this is going to make us all equally miserable. The
 ARC guys don't get the compiler support needed to make ARC fast.

Of course they could. Compiler can have internal support for RC slices. Object will need some built-in support anyway.
 It
 completely ignores cyclic-refs inside the slice.

Wrong. It leaves cyclic references to the GC.

Ok, I can see that.
 It doesn't solve the
 primary ARC-camp complaint about the reliance on the GC.

Why not?

It still has to be loaded and running. It can still non-deterministically pause. Yes, the pause may happen less often but they will take a similar amount of time to run the mark phase as the current, which is statistically the longest phase. I already know that they are going to complain loudly because this does nothing to resolve their central complaint. Also Adam Ruppe brought up an interesting problem that I didn't understand with slicing in this configuration. If he could comment that would be appreciated.
 The GC now guys
 have more hoops to jump-through when working with Phobos.

Why?

New details to corral: Don't forget to add your .toGC() calls now kids!
 And it
 increases complexity and cognitive load all around.

That comes with the territory. Have no illusion we can add RC support in any way at no cost.

Yes, but why do it in such a way as to penalize existing practices. The penalty properly belongs on the new practices.
 Andrei

-- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
prev sibling next sibling parent reply "Jesse Phillips" <Jesse.K.Phillips+D gmail.com> writes:
On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu 
wrote:
 and those who don't care say:

 auto x = fun().toGC();

If I don't care, why would I place .toGC() at the end of my calls? What reason do I have to go out of my way to request this? What problems can I expect when I forget to add it?
Feb 04 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 11:20 PM, Jesse Phillips wrote:
 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 and those who don't care say:

 auto x = fun().toGC();

If I don't care, why would I place .toGC() at the end of my calls?

This is the way it all works: RC+GC is more structure than GC, so you start with more structure and then optionally "forget" it.
 What
 reason do I have to go out of my way to request this?

You use an API that uses e.g. string, not RCString.
 What problems can
 I expect when I forget to add it?

Passing x around won't compile. Andrei
Feb 04 2014
parent reply Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 2/5/2014 2:21 AM, Andrei Alexandrescu wrote:
 On 2/4/14, 11:20 PM, Jesse Phillips wrote:
 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 and those who don't care say:

 auto x = fun().toGC();

If I don't care, why would I place .toGC() at the end of my calls?

This is the way it all works: RC+GC is more structure than GC, so you start with more structure and then optionally "forget" it.
 What
 reason do I have to go out of my way to request this?

You use an API that uses e.g. string, not RCString.

IIUC, it sounds like it'd work like this: // Library author provides either one or both of these: T[] getFooGC() {...} RCSlice!T getFooARC() {...} And then if, for whatever reason, you have a RCSlice!T and need to pass it to something that expects a T[], then you can cancel the RC-ing via toGC. If that's so, then I'd think lib authors could easily provide APIs that offer GC by default and ARC as an opt-in choice with templating: enum WantARC { Yes, No } auto getFoo(WantARC arc = WantARC.No)() { static if(arc == WantARC.No) return getFoo().toGC(); else { RCSlice!T x = ...; return x; } } T[] fooGC = getFoo(); RCSlice!T = getFoo!(WantARC.Yes)(); And I imagine that boilerplate could be encapsulated in a utility template: private RCSlice!T getFooARC() { RCSlice!T x = ...; return x; } template makeGCDefault(...){...magic happens here...} alias getFoo = makeGCDefault!getFooARC;
Feb 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 12:03 AM, Nick Sabalausky wrote:
 IIUC, it sounds like it'd work like this:

 // Library author provides either one or both of these:
 T[] getFooGC() {...}
 RCSlice!T getFooARC() {...}

 And then if, for whatever reason, you have a RCSlice!T and need to pass
 it to something that expects a T[], then you can cancel the RC-ing via
 toGC.

Yah. We'd then need to do something similar on the parameter side, e.g. extend isSomeString to comprehend the RC variety as well.
 If that's so, then I'd think lib authors could easily provide APIs that
 offer GC by default and ARC as an opt-in choice with templating:

 enum WantARC { Yes, No }
 auto getFoo(WantARC arc = WantARC.No)() {
      static if(arc == WantARC.No)
          return getFoo().toGC();
      else {
      RCSlice!T x = ...;
          return x;
      }
 }

Nice. Though I'd say just return RC strings and let the client call toGC. They'd need to specify WantArc.No anyway so one way or another the user has to do something. Andrei
Feb 05 2014
parent Nick Sabalausky <SeeWebsiteToContactMe semitwist.com> writes:
On 2/5/2014 10:31 AM, Andrei Alexandrescu wrote:
 On 2/5/14, 12:03 AM, Nick Sabalausky wrote:
 If that's so, then I'd think lib authors could easily provide APIs that
 offer GC by default and ARC as an opt-in choice with templating:

 enum WantARC { Yes, No }
 auto getFoo(WantARC arc = WantARC.No)() {
      static if(arc == WantARC.No)
          return getFoo().toGC();
      else {
      RCSlice!T x = ...;
          return x;
      }
 }

Nice. Though I'd say just return RC strings and let the client call toGC. They'd need to specify WantArc.No anyway so one way or another the user has to do something.

Note the default parameter of WantARC.No. Therefore: // GC: T[] fooGC = getFoo(); // Nothing special specified // ARC: RCSlice!T = getFoo!(WantARC.Yes)(); Or am I missing something?
Feb 05 2014
prev sibling next sibling parent reply "Kagamin" <spam here.lot> writes:
My understanding was that ARC completely replaces GC and 
everything including slices becomes refcounted. Is having mixed 
incompatible GC and ARC code and trying to get them interoperate 
a good idea? Can you sell such mixed code to ARC guys?
Feb 04 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/4/14, 11:38 PM, Kagamin wrote:
 My understanding was that ARC completely replaces GC and everything
 including slices becomes refcounted. Is having mixed incompatible GC and
 ARC code and trying to get them interoperate a good idea? Can you sell
 such mixed code to ARC guys?

In an RC system you must collect cycles. ARC leaves that to the programmer in the form of weak pointers. This particular idea automates that. Andrei
Feb 04 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-05 12:25:48 +0000, "Francesco Cattoglio" 
<francesco.cattoglio gmail.com> said:

 On Wednesday, 5 February 2014 at 12:12:01 UTC, Paulo Pinto wrote:
 Yes, the GC just needs to check roots for already released blocks, if I 
 am not mistaken.

Perhaps I lost some explanation in the discussions, but does this improve performance in any way (other than allowing you to disable the GC)?

In general ARC+GC would release memory faster so you need less memory overall. Less garbage memory blocks floating around might make processor caches more efficients. And less memory pressure means the GC itself runs less often, thus less pauses, and shorter pauses since there is less memory to scan. On the other hand, if you allocate everything you need from the start and then just shuffle pointers around, ARC will make things slower. So there's some good and some bad. It's hard to know exactly how much of each in practice without an implementation to take some measurements with real world programs. But what's certain is that with ARC the overhead is more evenly distributed in time. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 9:40 AM, Manu wrote:
 On 6 February 2014 02:16, Dicebot <public dicebot.lv
 <mailto:public dicebot.lv>> wrote:

     On Wednesday, 5 February 2014 at 16:13:33 UTC, Manu wrote:

         Huh? Why would they? They don't create cycles, and would clean
         up reliably.


     Because they still return T* and not RC!T ? Andrei's post speaks
     purely about extra library type and does not mention about
     possibility to make it default allocation type for language.

     Or it is just silently assumed?


 Dunno, but I don't think any solution which uses RC!T will ever be
 acceptable.

Why not? What would be deemed acceptable? And who's the acceptor?
 It basically defeats the whole purpose.

What is the purpose? Andrei
Feb 05 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 9:51 AM, Dicebot wrote:
 On Wednesday, 5 February 2014 at 17:40:35 UTC, Manu wrote:
 Dunno, but I don't think any solution which uses RC!T will ever be
 acceptable. It basically defeats the whole purpose.

I feel terribly confused now :( Andrei, can you describe your intention with some big picture context?

Phobos needs to be able to return allocated memory without creating litter. With RCSlice!T the ownership is passed back to the user. The user can continue tracking it by using reference counting built into RCSlice, or make the object "immortal" by calling .toGC against it. It's simple, really. All about libraries returning (and sometimes accepting) slices. Andrei
Feb 05 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 10:38 AM, Namespace wrote:
 How would look the syntax for creating a RC slice?
 int[] arr = new int[100];
 or
 somewhat ugly as RCSlice!int arr = new int[100];
 ?
 Or (as I fear) RCSlice!int arr = RCSlice!int(100); ?

Just call a function! The "new" syntax for arrays is syntactically bankrupt anyway (can't distinguish between static and dynamic arrays). A historical mistake. Andrei
Feb 05 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 10:45 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Wednesday, 5 February 2014 at 18:33:54 UTC, Andrei Alexandrescu wrote:
 Phobos needs to be able to return allocated memory without creating
 litter. With RCSlice!T the ownership is passed back to the user. The
 user can continue tracking it by using reference counting built into
 RCSlice, or make the object "immortal" by calling .toGC against it.

Would it be possible to use RCSlice for all reference counting? I.e. that a reference counter for a struct would be a RCSlice of fixed length 1? If so then maybe it would be nice, but with a different name.

This is only focusing on slices. Andrei
Feb 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 11:45 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Wednesday, 5 February 2014 at 19:37:29 UTC, Andrei Alexandrescu wrote:
 This is only focusing on slices.

That is not going to work out well, because you should be able to decrease the ref count of a pointer to something arbitrary (void) without knowing the object type.

I said it a couple of times, and it seems it bears repeating: the charter of this is solely to create a slice type that takes care of itself. What this is not is a general solution for managing internal pointers or pointers to arbitrary objects. Andrei
Feb 05 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 12:20 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Wednesday, 5 February 2014 at 20:04:31 UTC, Andrei Alexandrescu wrote:
 I said it a couple of times, and it seems it bears repeating: the
 charter of this is solely to create a slice type that takes care of
 itself. What this is not is a general solution for managing internal
 pointers or pointers to arbitrary objects.

Then I think this is starting in the wrong end of the problem space. Slices are tiny dots in this picture.

I'm confused. Slices are the bulkiest allocation there is: chunks of contiguous data.
 It would be better to have a RC compiler switch and version{} statements
 in the libraries rather than having extensive special casing of RC/GC
 vs pure GC types in user code.

I don't see where the extensive casing comes from. User code decides which way they want and they go for it. Simple aliases make certain decisions swappable. Andrei
Feb 05 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 10:55 AM, Dicebot wrote:
 On Wednesday, 5 February 2014 at 18:33:54 UTC, Andrei Alexandrescu wrote:
 Phobos needs to be able to return allocated memory without creating
 litter. With RCSlice!T the ownership is passed back to the user. The
 user can continue tracking it by using reference counting built into
 RCSlice, or make the object "immortal" by calling .toGC against it.

Ok, clear. It will work then as far as I can see. But I am surprised this is considered a #1 issue to address.

"#1" as in "the first idea that came to mind". Just sorting by date :o).
 I have
 never been worried by lack of control over generated garbage in Phobos -
 it was the other way around, worries about control of how it is allocated.

 So yes, this is likely to solve issue you have mentioned but does not
 seem to be of any interest to me.

Noted. That part needs to be addressed by allocators. Andrei
Feb 05 2014
prev sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 08.02.2014 11:21, schrieb ixid:
 On Wednesday, 5 February 2014 at 16:50:40 UTC, Ola Fosheim Grøstad wrote:
 On Wednesday, 5 February 2014 at 16:04:06 UTC, Dicebot wrote:
 It is up to programmer to decide. Right now he does not have a choice
 and sometimes you can't afford to have GC in your program at all (as
 in can't have it linked to the binary), not just can't call
 collection cycles. Having sane fallback is very desired.

Yes, if D is going to be a system level programming language then there is no other option than.
 Proposed solution does not seem to save you from uncontrollably long
 collection cycles anyway as it still uses same memory pool so I don't
 see how it can help even games, not even speaking about more
 demanding applications.

Well, for games and game servers I think a 100ms delay once or twice per hour is inconsequential in terms of impact. If you can reduce the GC load by various means it might work out for most applications. 1. Reduce the set considered for GC by having the GC not scanning paths that are known to be covered by RC. 2. Improving speed of GC by avoiding interior pointers etc. 3. Reducing the number of calls to GC by having RC take care of the majority of the memory releases. 4. Have local GC by collecting roots of nodes that are known to create cycles. I don't think ARC is an option for OS-level development and critical applications anyway. :-)
 Different scenarios have different needs.

Haven't you just basically confirmed my opinion? :)

In a way. :-) But what if the question is this: How can you in a pragmatic way come up with a solution that cover most soft real time applications? A compiler switch that default to RC (i.e. turn standard GC features into standard RC features) could in theory get you pretty close, but I think clever RC/GC memory management requires whole program analysis…

For a competitive game 100ms delays during gameplay would be completely unacceptable.

In the stock exchange there are Java and .NET systems doing transactions at faster speed than that. 100ms is too much money to be lost if the transaction course changes. -- Paulo
Feb 08 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 1:05 AM, Adam Wilson wrote:
 On Tue, 04 Feb 2014 23:38:54 -0800, Kagamin <spam here.lot> wrote:

 My understanding was that ARC completely replaces GC and everything
 including slices becomes refcounted. Is having mixed incompatible GC
 and ARC code and trying to get them interoperate a good idea? Can you
 sell such mixed code to ARC guys?

I've been asking myself those questions a lot over the past couple days. If GC backed ARC is such a brilliant idea, how come nobody has done it yet?

It's a classic. Andrei
Feb 05 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 23:21:43 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 2/4/14, 11:20 PM, Jesse Phillips wrote:
 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 and those who don't care say:

 auto x = fun().toGC();

If I don't care, why would I place .toGC() at the end of my calls?

This is the way it all works: RC+GC is more structure than GC, so you start with more structure and then optionally "forget" it.
 What
 reason do I have to go out of my way to request this?

You use an API that uses e.g. string, not RCString.

The amount of existing-code-breakage here will be immense. Almost nothing, unless it uses Phobos only, will compile once this is released. It might even do more damage to D's still fragile public-image than the Phobos/Tango fiasco did.
 What problems can
 I expect when I forget to add it?

Passing x around won't compile.

And we're supposed to want that? (See Above)
 Andrei

-- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 04 2014
prev sibling next sibling parent "Kagamin" <spam here.lot> writes:
On Wednesday, 5 February 2014 at 07:45:26 UTC, Andrei 
Alexandrescu wrote:
 In an RC system you must collect cycles. ARC leaves that to the 
 programmer in the form of weak pointers. This particular idea 
 automates that.

Having GC as a backup for cycles doesn't prevent making everything transparently refcounted. As to allocation strategies, the code can just use compatible allocation strategy.
Feb 05 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 04 Feb 2014 23:38:54 -0800, Kagamin <spam here.lot> wrote:

 My understanding was that ARC completely replaces GC and everything  
 including slices becomes refcounted. Is having mixed incompatible GC and  
 ARC code and trying to get them interoperate a good idea? Can you sell  
 such mixed code to ARC guys?

I've been asking myself those questions a lot over the past couple days. If GC backed ARC is such a brilliant idea, how come nobody has done it yet? I mean, it is a rather obvious solution. What I am confident of is that it is going to create a metric-ton of implementation gotchas for the compiler to sort out (as if we don't have enough open trouble tickets already) and it is going to pretty steeply increase the complexity of language. I thought the whole point of D was to not be C++, particularly in terms of complexity? All for a potentially undeliverable promise of a little more speed and fewer (not none) collection pauses. I have a suspicion that the reason that it hasn't been done is that it doesn't actually improve the overall speed and quite possibly reduces it. It will take months of engineering effort just to ship the buggy initial functionality, and many more months and possibly years to sort out all the edge cases. This will in turn significantly reduce the bandwidth going towards fixing the features we already have that don't work right and improving the GC so that it isn't such an eyesore. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 05 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 5 February 2014 at 09:05:02 UTC, Adam Wilson wrote:
 On Tue, 04 Feb 2014 23:38:54 -0800, Kagamin <spam here.lot> 
 wrote:

 My understanding was that ARC completely replaces GC and 
 everything including slices becomes refcounted. Is having 
 mixed incompatible GC and ARC code and trying to get them 
 interoperate a good idea? Can you sell such mixed code to ARC 
 guys?

I've been asking myself those questions a lot over the past couple days. If GC backed ARC is such a brilliant idea, how come nobody has done it yet? I mean, it is a rather obvious solution. What I am confident of is that it is going to create a metric-ton of implementation gotchas for the compiler to sort out (as if we don't have enough open trouble tickets already) and it is going to pretty steeply increase the complexity of language. I thought the whole point of D was to not be C++, particularly in terms of complexity? All for a potentially undeliverable promise of a little more speed and fewer (not none) collection pauses. I have a suspicion that the reason that it hasn't been done is that it doesn't actually improve the overall speed and quite possibly reduces it. It will take months of engineering effort just to ship the buggy initial functionality, and many more months and possibly years to sort out all the edge cases. This will in turn significantly reduce the bandwidth going towards fixing the features we already have that don't work right and improving the GC so that it isn't such an eyesore.

They have done it. It is how the systems programming language Cedar, for the Mesa operating system at Xerox PARC used to work. There are papers about it, that I already posted multiple times. -- Paulo
Feb 05 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Wednesday, 5 February 2014 at 08:06:30 UTC, Kagamin wrote:
 On Wednesday, 5 February 2014 at 07:45:26 UTC, Andrei 
 Alexandrescu wrote:
 In an RC system you must collect cycles. ARC leaves that to 
 the programmer in the form of weak pointers. This particular 
 idea automates that.

Having GC as a backup for cycles doesn't prevent making everything transparently refcounted. As to allocation strategies, the code can just use compatible allocation strategy.

Yes, the GC just needs to check roots for already released blocks, if I am not mistaken.
Feb 05 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Wednesday, 5 February 2014 at 12:12:01 UTC, Paulo Pinto wrote:
 Yes, the GC just needs to check roots for already released 
 blocks, if I am not mistaken.

Perhaps I lost some explanation in the discussions, but does this improve performance in any way (other than allowing you to disable the GC)?
Feb 05 2014
prev sibling next sibling parent reply Manu <turkeyman gmail.com> writes:
--089e013a1822051cd004f1a80ed2
Content-Type: text/plain; charset=UTF-8

On 5 February 2014 09:51, Andrei Alexandrescu <SeeWebsiteForEmail erdani.org
 wrote:

 Consider we add a library slice type called RCSlice!T. It would have the
 same primitives as T[] but would use reference counting through and
 through. When the last reference count is gone, the buffer underlying the
 slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the slice and
 disables the reference counter (e.g. by setting it to uint.max/2 or
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

This doesn't excite me at all. What about all other types of allocations? I don't want to mangle my types. What about closures? What about allocations from phobos? What about allocations from 3rd party libs that I have no control over? I don't like that it requires additional specification, and special treatment to have it detach to the GC. There's nothing transparent about that. Another library solution like RefCounted doesn't address the problem. Counter question; why approach it this way? Is there a reason that it needs to be of one kind or the other? --089e013a1822051cd004f1a80ed2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 5= February 2014 09:51, Andrei Alexandrescu <span dir=3D"ltr">&lt;<a href=3D"= mailto:SeeWebsiteForEmail erdani.org" target=3D"_blank">SeeWebsiteForEmail = erdani.org</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">Consider we add a library slice type called = RCSlice!T. It would have the same primitives as T[] but would use reference= counting through and through. When the last reference count is gone, the b= uffer underlying the slice is freed. The underlying allocator will be the G= C allocator.<br> <br> Now, what if someone doesn&#39;t care about the whole RC thing and aims at = convenience? There would be a method .toGC that just detaches the slice and= disables the reference counter (e.g. by setting it to uint.max/2 or whatev= er).<br> <br> Then people who want reference counting say<br> <br> auto x =3D fun();<br> <br> and those who don&#39;t care say:<br> <br> auto x =3D fun().toGC();<br> <br> <br> Destroy.</blockquote><div><br></div><div>This doesn&#39;t excite me at all.= </div><div>What about all other types of allocations? I don&#39;t want to m= angle my types. What about closures? What about allocations from phobos? Wh= at about allocations from 3rd party libs that I have no control over?</div> <div>I don&#39;t like that it requires additional specification, and specia= l treatment to have it detach to the GC.</div><div>There&#39;s nothing tran= sparent about that. Another library solution like RefCounted doesn&#39;t ad= dress the problem.</div> <div><br></div><div>Counter question; why approach it this way?</div><div>I= s there a reason that it needs to be of one kind or the other?</div></div><= /div></div> --089e013a1822051cd004f1a80ed2--
Feb 05 2014
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-05 15:01:04 +0000, Manu <turkeyman gmail.com> said:

 Aligned data is a challenge. I have often wondered if it would be feasible
 to access the RC via a pointer hash or something, and keep it in a side
 table... sounds tricky, but I wonder if it's possible.

That's what Apple is doing (as seen in the CoreFoundation source code). They actually have eight such tables on OS X, each protected by a spin lock. The right table is chosen according to a few bits of the pointer. The obvious advantage is you can put immutable data in read-only memory without making the reference count immutable. The downside is that it's more complicated to access the counter.
 Fat pointers might be necessary. That's a bit annoying. Hmmm...

Since we're talking about adding reference counts to GC-allocated memory, you could use the GC to find the base address of the memory block. What is the cost of that?
 I'm sure there's a clever solution out there which would allow the ARC to
 detect if it's a raw C pointer or not...

Ask the GC for the base address of the memory block. If it does not come from a GC block, there's no reference counter to update. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-05 15:52:36 +0000, Manu <turkeyman gmail.com> said:

 On 6 February 2014 01:22, Michel Fortin <michel.fortin michelf.ca> wrote:
 
 On 2014-02-05 15:01:04 +0000, Manu <turkeyman gmail.com> said:
 
 Since we're talking about adding reference counts to GC-allocated memory,
 you could use the GC to find the base address of the memory block. What is
 the cost of that?

Can you elaborate? How would the GC know this?

How do you think the GC tracks internal pointers today? ;-) Just call addrOf if you need to know: http://dlang.org/phobos/core_memory.html#.GC.addrOf We'd have to call it too for incrementing/decrementing the reference count. It'd work, even though it seems rather heavyweight. The slow part of that function is the call to findPool, which does a binary search: https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L1581 That said, the GC is already doing that for every word of memory it scans, so it might not be as heavyweight as it seems (especially if the GC has less to scan later because of ARC). See the mark function: https://github.com/D-Programming-Language/druntime/blob/master/src/gc/gc.d#L2274 I'd tend to say that if you're in control of the reference count system, the code genereration, the allocation pools, as well as the GC algorithm you can probably do something that'll work well enough, by architecting things to work well together. But it requires an integrated approach. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 4:39 AM, Manu wrote:
 On 5 February 2014 09:51, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org <mailto:SeeWebsiteForEmail erdani.org>>
 wrote:

     Consider we add a library slice type called RCSlice!T. It would have
     the same primitives as T[] but would use reference counting through
     and through. When the last reference count is gone, the buffer
     underlying the slice is freed. The underlying allocator will be the
     GC allocator.

     Now, what if someone doesn't care about the whole RC thing and aims
     at convenience? There would be a method .toGC that just detaches the
     slice and disables the reference counter (e.g. by setting it to
     uint.max/2 or whatever).

     Then people who want reference counting say

     auto x = fun();

     and those who don't care say:

     auto x = fun().toGC();


     Destroy.


 This doesn't excite me at all.
 What about all other types of allocations? I don't want to mangle my
 types.

How do you mean that?
 What about closures?

This is "#1" and it only addresses slices.
 What about allocations from phobos?

Phobos would get added support for RC slices.
 What
 about allocations from 3rd party libs that I have no control over?

Outside the charter of this discussion.
 I don't like that it requires additional specification, and special
 treatment to have it detach to the GC.

But you didn't like that GC is implicit. Which way do you want it?
 There's nothing transparent about that.

I don't think we can attain 100% transparency.
 Another library solution like
 RefCounted doesn't address the problem.

 Counter question; why approach it this way?
 Is there a reason that it needs to be of one kind or the other?

This is technically possible today. Andrei
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 12:12:01 UTC, Paulo Pinto wrote:
 Yes, the GC just needs to check roots for already released 
 blocks, if I am not mistaken.

Yes, when they go to non-zero. This is the scheme used in PHPs ARC/GC solution, published in this paper: http://researcher.watson.ibm.com/researcher/files/us-bacon/Bacon01Concurrent.pdf You push roots that are candidates onto a queue when the counter is decreased to nonzero, then you do a concurrent scan when you have a set of roots to scan for cycles. So you probably need a clever ARC to reduce the scanning. I think however, that for most programs that use ARC you don't do the GC at all. For long lived processes such as servers you might run it using heuristics (based on memory headroom or during the night). According to the paper above the amount of cyclic garbage tends to be low, but varies a great deal. They site another paper relating to Inferno that supposedly claims that RC caught 98% of the garbage. So the need for cycle collection is rather application specific. Perl has traditionally not caught cycles at all, but then again perl programs tends to be short lived. ---- There are also some papers on near real time GC, such as Staccato and Metronome, on David F. Bacons list: http://researcher.watson.ibm.com/researcher/view_pubs.php?person=us-bacon&t=1
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 12:40:13 UTC, Manu wrote:
 Counter question; why approach it this way?
 Is there a reason that it needs to be of one kind or the other?

Sure, you could make all allocations with RC enabled make space for a counter at a negative offset (ptr-offset), but that would not work with C structs or internal pointers and aligned data might hog a bit of extra space. You also need to handle internal pointers to embedded objects and arrays of objects (not pointers to objects). How do you ref count those? I guess you could switch all pointers into (allocated_base,offset) tuples and special case (ptr,0). You could do like C++, have the ref counter be a separate object. Then record the properties of the pointer (such as offset), then have special magic checks for deallocation: testing for internal reference then decrease the real ref counter of the parent rather than deallocate. This is quite compatible with having a GC, you could free the object only when proven safe and just ignore it and leave it for GC when you cannot prove it. IMHO you proabably need to redesign the language in order to support transparent or efficient automatic memory handling. If you retain C-legacy, you also retain manual memory handling or a limited set of opportunites for automatic garbage collection.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e01183d64a1aac904f1aa0677
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 6 February 2014 00:04, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Wednesday, 5 February 2014 at 12:40:13 UTC, Manu wrote:

 Counter question; why approach it this way?
 Is there a reason that it needs to be of one kind or the other?

Sure, you could make all allocations with RC enabled make space for a counter at a negative offset (ptr-offset), but that would not work with C structs or internal pointers and aligned data might hog a bit of extra space.

Perhaps match a very particular and unlikely bit pattern in the negative offset to know if it is RC or not? Or I wonder if there's opportunity to pinch a single bit from pointers to mark that it is raw or RC allocated? Probably fine on 64bit. 32bit probably needs to match a bit pattern or something. Aligned data is a challenge. I have often wondered if it would be feasible to access the RC via a pointer hash or something, and keep it in a side table... sounds tricky, but I wonder if it's possible. You also need to handle internal pointers to embedded objects and arrays of
 objects (not pointers to objects). How do you ref count those? I guess yo=

 could switch all pointers into (allocated_base,offset) tuples and special
 case (ptr,0).

That's a good point. This is probably the trickiest detail. Maybe a clever way to make any pointer within the allocated range hash to the right index in the side table I referred to above. That sounds like a tricky hashing function, and probably slow. Fat pointers might be necessary. That's a bit annoying. Hmmm... Rather than (allocated_base, offset), I suspect (pointer, offset_from_base) would be better; typical dereferences would have no penalty. You could do like C++, have the ref counter be a separate object. Then
 record the properties of the pointer (such as offset), then have special
 magic checks for deallocation: testing for internal reference then decrea=

 the real ref counter of the parent rather than deallocate. This is quite
 compatible with having a GC, you could free the object only when proven
 safe and just ignore it and leave it for GC when you cannot prove it.

You mean like the smart pointer double indirection? I really don't like the double indirection, but it's a possibility. Or did I misunderstand? IMHO you proabably need to redesign the language in order to support
 transparent or efficient automatic memory handling. If you retain C-legac=

 you also retain manual memory handling or a limited set of opportunites f=

 automatic garbage collection.

I'm sure there's a clever solution out there which would allow the ARC to detect if it's a raw C pointer or not... --089e01183d64a1aac904f1aa0677 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 00:04, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola Fo= sheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot;= puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &lt= ;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"im">On Wednesday, 5 February 2014 at 12:40:1= 3 UTC, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Counter question; why approach it this way?<br> Is there a reason that it needs to be of one kind or the other?<br> </blockquote> <br></div> Sure, you could make all allocations with RC enabled make space for a count= er at a negative offset (ptr-offset), but that would not work with C struct= s or internal pointers and aligned data might hog a bit of extra space.<br> </blockquote><div><br></div><div>Perhaps match a very particular and unlike= ly bit pattern in the negative offset to know if it is RC or not?</div><div=
Or I wonder if there&#39;s opportunity to pinch a single bit from pointers=

ably needs to match a bit pattern or something.</div> <div><br></div><div>Aligned data is a challenge. I have often wondered if i= t would be feasible to access the RC via a pointer hash or something, and k= eep it in a side table... sounds tricky, but I wonder if it&#39;s possible.= <br> </div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);bord= er-left-style:solid;padding-left:1ex"> You also need to handle internal pointers to embedded objects and arrays of= objects (not pointers to objects). How do you ref count those? I guess you= could switch all pointers into (allocated_base,offset) tuples and special = case (ptr,0).<br> </blockquote><div><br></div><div>That&#39;s a good point. This is probably = the trickiest detail. Maybe a clever way to make any pointer within the all= ocated range hash to the right index in the side table I referred to above.= That sounds like a tricky hashing function, and probably slow.</div> <div>Fat pointers might be necessary. That&#39;s a bit annoying. Hmmm...</d= iv><div><br></div><div>Rather than (allocated_base, offset), I suspect (poi= nter, offset_from_base) would be better; typical dereferences would have no= penalty.</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-lef= t-style:solid;padding-left:1ex"> You could do like C++, have the ref counter be a separate object. Then reco= rd the properties of the pointer (such as offset), then have special magic = checks for deallocation: testing for internal reference then decrease the r= eal ref counter of the parent rather than deallocate. This is quite compati= ble with having a GC, you could free the object only when proven safe and j= ust ignore it and leave it for GC when you cannot prove it.<br> </blockquote><div><br></div><div>You mean like the smart pointer double ind= irection? I really don&#39;t like the double indirection, but it&#39;s a po= ssibility. Or did I misunderstand?</div><div><br></div><blockquote class=3D= "gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;borde= r-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"> IMHO you proabably need to redesign the language in order to support transp= arent or efficient automatic memory handling. If you retain C-legacy, you a= lso retain manual memory handling or a limited set of opportunites for auto= matic garbage collection.<br> </blockquote><div><br></div><div>I&#39;m sure there&#39;s a clever solution= out there which would allow the ARC to detect if it&#39;s a raw C pointer = or not...</div></div></div></div> --089e01183d64a1aac904f1aa0677--
Feb 05 2014
prev sibling next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu 
wrote:
 ...

 Destroy.

 Andrei

I simply don't understand what problem this is trying to solve. I can't remember anyone wanting RC just of the sake of RC - it is just a compromise to enable language features that require allocation with undeterministic lifetime in absense of GC. Only thing having RC on top of GC may help with is reducing overall memory footprint but it is not an issue anyway. I must be missing something, but this looks perfectly useless.
Feb 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 7:10 AM, Dicebot wrote:
 On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu wrote:
 ...

 Destroy.

 Andrei

I simply don't understand what problem this is trying to solve.

Allow people to clean up memory allocated by Phobos. Andrei
Feb 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 10:56 AM, Adam D. Ruppe wrote:
 On Wednesday, 5 February 2014 at 18:25:31 UTC, Andrei Alexandrescu wrote:
 Allow people to clean up memory allocated by Phobos.

A better solution would be for Phobos to allocate less memory. Instead of returning strings, accept output ranges that receive it.

You do figure that complicates usage considerably, right?
 Make slices
 work as output ranges well with control over growing.

 (We can still offer the simple string returning functions for convenience)

I was thinking RCSlice would be a better alternative. Andrei
Feb 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 1:10 PM, Dicebot wrote:
 tl; dr: it does not matter if this topic proposal works or not, it is
 simply an effort put into the wrong place

Noted. On the face of it it seems odd that reference counted chunks of typed memory are deemed useless, at the tail of a discussion discussing in great detail the important advantages of reference counting. I should also add that imparting useful semantics to scope is much more difficult than it might seem. In contrast, reference counted slices are realizable in the current language with a relatively small (scoped! :o)) library effort and can be put to use immediately. I see a lot of good reasons to add them to Phobos regardless of the larger memory allocation changes we plan to make. Andrei
Feb 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 3:16 PM, Adam D. Ruppe wrote:
 On Wednesday, 5 February 2014 at 22:32:52 UTC, Andrei Alexandrescu wrote:
 On the face of it it seems odd that reference counted chunks of typed
 memory are deemed useless

I don't think anybody has actually said that. They have their places, it is just useless to talk about throwing them in everywhere.

I think part of the problem is a disconnect in assumptions and expectations. My idea was to simply make a first simple and obvious step toward improving the situation. Apparently that wasn't quite understood because ten people have eleven notions about what's desirable and even possible with regard to alternate memory management schemes. One school of thought seems to be that D should be everything it is today, just with reference counting throughout instead of garbage collection. One build flag to rule them all would choose one or the other. One other school of thought (to which I subscribe) is that one should take advantage of reference counting where appropriate within a GC milieu, regardless of more radical RC approaches that may be available.
 I should also add that imparting useful semantics to scope is much
 more difficult than it might seem.

I'm not so sure about that*, but the fact is scope would be enormously useful if it was implemented. * Let's say it meant "assigning to any higher scope is prohibited". That should be trivially easy to check and ensures that variable itself doesn't escape. The tricky part would be preventing: int[] global; void foo(scope int[] a) { int[] b = a; global = b; } And that's easy to fix too: make ALL variables scope, unless specifically marked otherwise at the type declaration site (or if they are value types OR references to immutable data, which are very similar to value types in use).

Yah, that does break a bunch of code. Things like the type of "this" in class objects also comes to mind. Binding ref is also a related topic. All of these are complex matters, and I think a few simple sketches don't do them justice. Andrei
Feb 05 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 4:57 PM, Adam D. Ruppe wrote:
 On Thursday, 6 February 2014 at 00:42:20 UTC, Andrei Alexandrescu wrote:
 One other school of thought (to which I subscribe) is that one should
 take advantage of reference counting where appropriate within a GC
 milieu, regardless of more radical RC approaches that may be available.

I agree with that stance, but I don't think there's a blanket rule there. I think RC freeing small slices will waste more time than it saves. Large allocations, on the other hand, might be worth it. So std.file.read for example returns a large block - that's a good candidate for refcounting since it might be accidentally subject to false pointers, or sit around too long creating needless memory pressure, etc.

That sounds reasonable. One possibility would be to define FreshSlice!T to mean this is a freshly-allocated slice; then it can be converted to a refcounted one or just a GC one.
 Anywho, I'd just go through on a case-by-case basis and tackle the big
 fish. Of course, a user could just do scope(exit) GC.free(ret); too.

That won't work because user code can't always know whether something received from the library had been freshly allocated or not.
 Binding ref is also a related topic. All of these are complex matters,
 and I think a few simple sketches don't do them justice.

I'd rather discuss these details than adding RCSlice and toGC everywhere for more cost than benefit.

Have at it, of course. This is not constant sum, just don't hijack this discussion. I should warn you however we've been discussing this literally for years; your examples are just scratching the surface. Andrei
Feb 05 2014
prev sibling next sibling parent reply dennis luehring <dl.soluz gmx.net> writes:
Am 06.02.2014 11:21, schrieb "Ola Fosheim Grstad"
 Not having the source code to a library is rather risky in terms
 of having to work around bugs by trail and error, without even
 knowing what the bug actually is.

so you're not work in the normal software development business where non-source code third party dependicies are fully normal :)
Feb 06 2014
parent dennis luehring <dl.soluz gmx.net> writes:
Am 06.02.2014 11:43, schrieb "Ola Fosheim Grstad" 
<ola.fosheim.grostad+dlang gmail.com>":
 On Thursday, 6 February 2014 at 10:29:20 UTC, dennis luehring
 wrote:
 so you're not work in the normal software development business
 where
 non-source code third party dependicies are fully normal :)

Not in terms of libraries no, what libraries would that be?

all libraries that other deparments don't want you to see or you don't need to see - so unnormal in your environment?
Feb 06 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/9/14, 1:06 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
[snip]
 I think one needs profiled whole program analysis to get there. I think
 one needs to aim for upcoming hardware architectures. I think one need
 to back it up with programmer specified constraints and hints. I think
 one needs high level optimization in addition to low level. I think one
 needs a very light and transparent runtime with tuneable low cost
 allocation schemes.

 I also think one needs to reduce the focus on separate compilation with
 no knowledge about what goes on inside an object file and C++/C ABIs.

Your thoughts are appreciated (all 6 of them in as many sentences). There is something to be said, however, about armchair quarterbacking and holier-than-thou kibitzing on what others should be doing. This community is as close as it gets to a meritocracy, so if you think you know what's good, you do good. If you want your stupendously many "I think"s to carry weight, follow them with some "I do"s as well. Hop on github. This endless walk through your knowledge base just isn't useful. Andrei
Feb 09 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 12:48 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Monday, 10 February 2014 at 04:22:32 UTC, Andrei Alexandrescu wrote:
 Your thoughts are appreciated (all 6 of them in as many sentences).
 There is something to be said, however, about armchair quarterbacking
 and holier-than-thou kibitzing on what others should be doing. This
 community is as close as it gets to a meritocracy, so if you think you
 know what's good, you do good. If you want your stupendously many "I
 think"s to carry weight, follow them with some "I do"s as well. Hop on
 github. This endless walk through your knowledge base just isn't useful.

Good job, my initial response to Manu was a critique of going Ad Hominem and you as a person time and time again fail in that regard in many discussions.

Totally. For the most part I take posts one at a time and at face value only, it's just that sometimes larger patterns develop themselves. But as I told Walter, for better or (sometimes definitely) worse, our character flaws make history inside the D community.
 You do however deserve a round of ad hominem because you as
 one of the two people who are in a position to communicate the project
 vision and set forth MEASURABLE goals that can be tracked and evaluated,
 but you refuse to do so.

A fresh perspective is always good to take under consideration. It's also a good opportunity to bring more transparency to what we're doing, as I'll do below.
 All talk of meritocracy is essentially hypocrisy because all projects
 need to establish boundaries and a goal post, and you fail miserably in
 that regard. That's why D is a slow mover. "This endless walk through
 [my] knowledgebase" is of course not a walk through my knowledgebase, it
 is an assessment of the project that YOU FAIL to attempt to do. It is my
 attempt to try to figure out where this project is heading.

 You are right, I should not have to do it. YOU SHOULD DO IT. AND PRESENT
 IT. That way people won't be let down.

I have done so. Several times. Two very simple examples from recent history: 1. I stressed that good work on bugs with bounties is a gesture of good will with Facebook that will bring more support from the company. It's the trifecta: the bugs are not harder than those people work on anyway, it's good impact on the future of the language, and it's even non-negligible money. E.g. I wrote on 2014-01-11:
 My hope is to convince that the message Facebook is conveying here is
 much stronger than the actual sums involved; it's an initiation of
 cooperation and involvement with a community, and it would be awesome
 to respond in kind.

Taking a look at https://www.bountysource.com/trackers/383571-d-programming-language, however, reveals that there's little attention to those bugs, in SPITE of the fact that contributions on HARDER problems on the SAME project continued as furiously as ever, if not more. 2. I said many times our inability to review github contributions at the rate they arrive is an important problem we're facing. We currently have 216 open pull requests across our projects. I think this bottleneck very concretely limits the growth speed of D. This is a typical problem. Reviewing contributions is hard and thankless work. I know how we solved it at Facebook for our many open-sourced projects: we created a team for it, with a manager, tracking progress, the works. This is _exactly_ the kind of thing that can't be done in a volunteer community. The reality is that on a volunteer-driven project, it's not easy to tell people what to do. They're by definition in it for working what _they_ want to work on. Applying classic management techniques naively is unlikely to work because all management techniques are using resources toward goals and assume the appropriately qualified human resources will work on what the project requires be done. So I wasn't glib when I sent you to github. In a very concrete sense, you'd be helping there a ton more in a fraction of the time you spend posting.
 I like the initial vision Walter Bright put forth years ago, that is to
 make a better C++. That has somehow evolved into making a compiled C#.
 Can you please ASSESS that.

I think D must not define itself in relation to any other language.
 You and Walter Bright are leads.

 I expect any project and you to put forth:

 1. A clear vision that establish a firm boundary.
 2. A small set of clear measurable goals that give the project direction.
 3. A list of points stating what the project is not going to address in
 the immediate future.

Some of these are useful to put together at least as (a) thoughts on what I believe would be high-impact topics, (b) things that I plan personally to work on.
 This endless walk through what is wrong with D project management just
 isn't useful, because you don't want to listen.

Honestly, as one who's been at this for a long time and has done and witnessed a number of such attempts, I think you're exceedingly naive about what can be done with traditional project management approaches in this case. Three simple anecdotes out of many: 1. I've had a long chat with a Linux senior kernel guy who's been there since the very early days. Back then Linux did not have any form of project management, and nobody told people what to work on. People just worked on whatever itch they wanted to scratch. The way it succeeded is getting the attention of sufficiently many people that there was someone on each possible itch :o). Still, for a very long time (years after actual heavy corporate support emerged) many Linux tools looked like proofs of concept compared to the mature Windows equivalents. 2. A few months ago a prominent member of the community made (privately to Walter and myself) a strong argument along the same lines as yours: D could move much faster if some good management could be used with it, and offered to ask as a manager of the project. I explained him with (other) examples what I'm explaining you now, of which the most important point was that resource management can be done if there are resources to manage. He understood my point (and was gracious enough to continue work within the community). I don't think things have changed in that regard since he made is bid. 3. Only a few _days_ ago Walter and I were discussing with another prominent community member. He is the author of a project that beautifully plays into D's strengths to the end of really being the best in the world at a very measurable metric. Walter and I emphasized how finalizing and streamlining this project would both launch his career on a meteoric orbit, and have a strong impact on D. He, however, is busy with schoolwork and some other D projects that are comparatively just irrelevant - and that's where the discussion just kind of ended. Had he been a report of mine, I would have simple ensured that I assign all his other tasks to someone else, and discuss goals and milestones with him for the high-impact project. But he's not, so I can't. This alone would have been enough to disabuse me of any illusions I could do project management on Dlang. Ola, I'm sure you mean well. I trust you will find it within yourself the best way to contribute to this community. Andrei
Feb 10 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 3:15 PM, Andrei Alexandrescu wrote:
 2. A few months ago a prominent member of the community made (privately
 to Walter and myself) a strong argument along the same lines as yours: D
 could move much faster if some good management could be used with it,
 and offered to ask as a manager of the project.

s/ask/act/
Feb 10 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 3:24 PM, Dicebot wrote:
 On Monday, 10 February 2014 at 23:15:35 UTC, Andrei Alexandrescu wrote:
 Taking a look at
 https://www.bountysource.com/trackers/383571-d-programming-language,
 however, reveals that there's little attention to those bugs, in SPITE
 of the fact that contributions on HARDER problems on the SAME project
 continued as furiously as ever, if not more.

Interesting, I had directly the opposite impression when went through the bountysource list. Lot of issues have pull requests provided but are stalled because of slow feedback cycle.

Same difference. A bunch of people can pull, I'm hardly the bottleneck. It just means I failed to convince the community to get those bounties done. Andrei
Feb 10 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 6:34 AM, Dicebot wrote:
 On Tuesday, 11 February 2014 at 02:29:49 UTC, Andrei Alexandrescu wrote:
 Same difference. A bunch of people can pull, I'm hardly the
 bottleneck. It just means I failed to convince the community to get
 those bounties done.

 Andrei

Still that list (people with pull rights) is relatively short so it does not make much sense to appeal to wider auditory. Also I don't think volunteer effort can't be organized. It is a matter of people identifying themselves as part of well-defined organization as opposed to independent crowd of collaborators.

I think we could organize ourselves better, too. It's all a matter of finding the right angle. But I think management is not what we need. We need better leadership. Big difference.
 It is a common wisdom
 that if no one feels directly responsible for an issue, no one will ever
 pay attention to it.

Yah, on the trite side even I'd opine :o). I said this several times now, and I'll say it again: I have asked SEVERAL TIMES people INDIVIDUALLY to do things that are HIGH IMPACT for the D language instead of something else that was more important to them, to no avail. Of course, it would be learned helplessness to draw sweeping conclusions from the experience so far.
 Do you need any specific proposals?

Suggestions for doing things better are gladly considered. Andrei
Feb 11 2014
next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11-Feb-2014 21:12, Andrei Alexandrescu пишет:
 On 2/11/14, 6:34 AM, Dicebot wrote:
 Do you need any specific proposals?

Suggestions for doing things better are gladly considered.

I'd risk suggesting introducing something simple and self-organizable. To be concrete: define "interest groups" by major areas of D ecosystem (Fronted, one for each backend, druntime as a whole, GC alone, Phobos in bits and pieces ...) and let people join/leave/lead these. At the very least it would make it obvious who is into what at which point of time. Even more importantly - who you'd need to ask about what. The only question is where to track this stuff - maybe Wiki? P.S. Trello was a failed experiment, but IMHO it failed largely due to being behind the closed doors. -- Dmitry Olshansky
Feb 11 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 9:27 AM, Dmitry Olshansky wrote:
 11-Feb-2014 21:12, Andrei Alexandrescu пишет:
 On 2/11/14, 6:34 AM, Dicebot wrote:
 Do you need any specific proposals?

Suggestions for doing things better are gladly considered.

I'd risk suggesting introducing something simple and self-organizable. To be concrete: define "interest groups" by major areas of D ecosystem (Fronted, one for each backend, druntime as a whole, GC alone, Phobos in bits and pieces ...) and let people join/leave/lead these. At the very least it would make it obvious who is into what at which point of time. Even more importantly - who you'd need to ask about what. The only question is where to track this stuff - maybe Wiki?

Could you please set up a sample wiki page so we get a better feel. Thanks.
 P.S. Trello was a failed experiment, but IMHO it failed largely due to
 being behind the closed doors.

Yah, the whole Trello experiment has been on my tongue during this discussion. It's been made public long before its demise. Andrei
Feb 11 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 10:12 AM, Brad Anderson wrote:
 I think it got off to a bad start because it was private and even after
 it was public it wasn't talked about much within the community (it was
 hard to even find because it wouldn't turn up in Trello's search). I use
 and enjoy Trello at work so personally I think I'd try to get everyone
 to give it another shot if it were up to me.

I'm also in favor of a second go, particularly since I've initiated the first one :o). I'm still using Trello for tracking personal stuff, though not nearly as frequently as to make a big difference. Andrei
Feb 11 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 12:31 PM, Joseph Cassman wrote:
 Although I haven't yet contributed to the project by sending any pull
 requests I am interested in doing so. Perhaps the main thing that keeps
 holding me back is I feel it is hard to know what currently is being
 worked on with the current setup (i.e. who is doing what, will my
 efforts duplicate someone else's, will I be stepping on anyone's toes,
 will my solution idea be incompatible with the direction in which the
 language is moving). I frequent the forums and still find it hard to get
 an overall picture. This is probably my lack of not trying hard enough
 (e.g. I never have time to spend on IRC). I saw the Trello board before
 and I felt that it helped me get up to speed on who is doing what, what
 is being worked on, etc. If you brought it, or something like it, back
 that would be appreciated.

Thanks for the feedback. Hanging out on IRC should not be necessary.
 Sorry to change the subject but one thing that keeps discouraging me
 from trying to contribute code changes is the large number of unmerged
 pull requests in GitHub.

Yes. I think that's a disaster. We need to figure out the right approach to solving that.
 Since I am not a reviewer I am afraid that any
 effort I perform will get ignored and so I hold back since I have other
 stuff to do too. I know that this is something that concerns you a lot
 from reading other forums discussions so I apologize if this sounds like
 a complaint. I appreciate the work of all involved and know that no one
 has infinite time or energy. Not even sure if I could contribute to the
 code-base but would be nice to try.

 Managing a project is never easy. My hat's off to you and Walter for
 your efforts in that regard.

Thanks! We're doing best we can but I am convinced we can do a lot better, and am looking at some out-of-the-box things to try. Andrei
Feb 11 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 6:21 PM, deadalnix wrote:
 On Wednesday, 12 February 2014 at 01:54:15 UTC, Andrei
 Alexandrescu wrote:
 Yes. I think that's a disaster. We need to figure out the right
 approach to solving that.

For a while, we were doing the review sunday. That was fun, everybody was talking about the pull in IRC and we were going from one to the other. Maybe it is time to start this tradition again ?

Are you guys up for it? I would. Andrei
Feb 11 2014
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
11-Feb-2014 21:37, Andrei Alexandrescu пишет:
 On 2/11/14, 9:27 AM, Dmitry Olshansky wrote:
 11-Feb-2014 21:12, Andrei Alexandrescu пишет:
 On 2/11/14, 6:34 AM, Dicebot wrote:

I'd risk suggesting introducing something simple and self-organizable. To be concrete: define "interest groups" by major areas of D ecosystem (Fronted, one for each backend, druntime as a whole, GC alone, Phobos in bits and pieces ...) and let people join/leave/lead these. At the very least it would make it obvious who is into what at which point of time. Even more importantly - who you'd need to ask about what. The only question is where to track this stuff - maybe Wiki?

Could you please set up a sample wiki page so we get a better feel. Thanks.

Here is a sketch. Sorry couldn't make it more realistic - I'm packing my stuff for tomorrow's flight (about 6 hours to go): http://wiki.dlang.org/Groups The fact that I couldn't remember a hell lot of people in each area of expertise/interest proves the point - it has to be driven by individuals and generated automatically. Think of it as a "phone book". Or better a page "Who is who in D language" for dlang.org that lists key areas in D ecosystem and contacts of people working/interested in them. The thing is I want this kind of page to be dynamic and self-organizable by very simple rules: 1. A D-guy has github account and an e-mail. (We may go beyond that e.g. IRC nickname, Skype etc. but let it grow organically) 2. Joins and leaves groups. 3. May be designated a leader, the "goto guy" in some group. The list of who's where is constantly kept up to date from this information. The extras could be grown on top this as needed. Nothing too fancy, probably could be hacked in an evening with Vibe.d (or say even PHP, who cares as long as it works).
 P.S. Trello was a failed experiment, but IMHO it failed largely due to
 being behind the closed doors.

Yah, the whole Trello experiment has been on my tongue during this discussion. It's been made public long before its demise.

Thing is that I for one lost all the steam trying to get at it early on. By the moment it was public I couldn't care less, to make things work unlike github there simply was no way to contribute for an outsider. -- Dmitry Olshansky
Feb 11 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Dmitry Olshansky"  wrote in message news:lddunp$jp7$1 digitalmars.com...

 Here is a sketch. Sorry couldn't make it more realistic -  I'm packing my 
 stuff for tomorrow's flight (about 6 hours to go):
 http://wiki.dlang.org/Groups

This looks useful, I can't always remember who works on what. I would probably prefer compiler internals questions were posted to the dmd-internals mailing list though, so everyone can see them/subscribe/search them.
Feb 12 2014
parent reply "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Andrej Mitrovic"  wrote in message 
news:tandcufdqjfgwacvobpm forum.dlang.org...

 Personally I hate dmd-internals because it's riddled with merge 
 notification threads. Those github notifications really need to be in a 
 separate newsgroup or we need to move discussions elsewhere because (at 
 least for me) actual discussion threads get lost.

That's probably a good idea, it would be nice to have both dmd-internals and dmd-commits.
Feb 12 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/12/14, 4:19 AM, Daniel Murphy wrote:
 "Andrej Mitrovic"  wrote in message
 news:tandcufdqjfgwacvobpm forum.dlang.org...

 Personally I hate dmd-internals because it's riddled with merge
 notification threads. Those github notifications really need to be in
 a separate newsgroup or we need to move discussions elsewhere because
 (at least for me) actual discussion threads get lost.

That's probably a good idea, it would be nice to have both dmd-internals and dmd-commits.

Could filters help? Andrei
Feb 12 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 10:08 AM, Andrej Mitrovic wrote:
 On Tuesday, 11 February 2014 at 17:12:37 UTC, Andrei Alexandrescu wrote:
 I said this several times now, and I'll say it again: I have asked
 SEVERAL TIMES people INDIVIDUALLY to do things that are HIGH IMPACT
 for the D language instead of something else that was more important
 to them, to no avail.

Which things exactly?

One simple example: specific regressions and blockers in bugzilla. Andrei
Feb 11 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 10:55 AM, Andrej Mitrovic wrote:
 On Tuesday, 11 February 2014 at 18:50:56 UTC, Andrei Alexandrescu wrote:
 One simple example: specific regressions and blockers in bugzilla.

Are those the same as the one's which are marked as bounties?

No.
 It could
 be that those problems are simply hard to fix. Many of them seem to be
 backend or debugging-related bugs. I'm saying not many people know how
 to fix these.

1. Not at all. Many were trivial issues technically. Walter and I fixed a few ourselves in short time. Kenji also is very active on those. For other contributors, there is no clear message of how damaging regressions are and how they delay our releases. 2. People work on harder problems. As an example I learned about literally yesterday: Martin Nowak is working on a REPL for D. Was it on our roadmap? No, not on the short list, not even on a longer list if there was one. Would I tell Martin to do a REPL? No. Would I tell Martin to drop work on the REPL and work on higher-impact items such as shared libraries on Windows and OSX? No. Do I like the idea of D having a REPL? Sure, and I'm thankful for it. 3. In the mythical organization, barring exceptional circumstances, people don't tell their managers: "Well, I understand this task you assigned to me is important, but it's hard to fix. Tell you what. I'll work on something else instead." Andrei
Feb 11 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
I just want to chime in here with Andrei, and emphasize that we don't tell 
anyone what to do here.

We do try to lead, inspire, cajole, reward, acknowledge, promote, etc., with 
varying degrees of success.
Feb 11 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 4:25 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Monday, 10 February 2014 at 23:15:35 UTC, Andrei Alexandrescu wrote:
 This is a typical problem. Reviewing contributions is hard and
 thankless work. I know how we solved it at Facebook for our many
 open-sourced projects: we created a team for it, with a manager,
 tracking progress, the works. This is _exactly_ the kind of thing that
 can't be done in a volunteer community.

Maybe you can make some parts modular after you refactor into D. Then people can take ownership of modules and social recognition will encourage more commitment.

I think at this stage we need more people to start with. Someone pointed out recently we have 77 lifetime contributors to github, as opposed to e.g. Rust which has 292.
 I think D must not define itself in relation to any other language.

I respect that position. Of course, it does not help if outsiders have been told that D is a better C++. It kinda sticks. Because people really want that.

I don't think so, at all. Anyone working on D must drop the moniker "D is a better C++" like a bad habit, no two ways about that. Most of it does it sets C++ as the benchmark. People who already like C++ would be like "you wish" and people who hate C++ would be like "better crap is not what I need anyway".
 I am very hard trying to convince myself that D is more like compiled
 C#, which lowers my expectations, because that original vision of a
 "better C++" is very firmly stuck.

For someone who hasn't been around for a while, maybe. I fail to see C++ or C# mentioned anywhere on our home page. We want to make D a great language all around, with system-level access and also convenience features.
 But my point was more that you need to communicate a vision that is such
 that the people you want to attract don't sit on the fence. I am quite
 certain that more skilled C++ programmers would volunteer if they saw a
 vision they believed in.

It's there in <h2> at the top of our homepage: "Modern convenience. Modeling power. Native efficiency." By the way, this whole "plop a vision page" thing doesn't seem to be quite popular: https://www.google.com/search?q=rust%20language#q=vision+site:rust-lang.org&safe=off https://www.google.com/search?q=vision%20site%3Apython.org https://www.google.com/search?q=vision%20site%3Aisocpp.org https://www.google.com/search?q=scala#q=vision+site:scala-lang.org&safe=off https://www.google.com/search?q=vision%20site%3Agolang.org Andrei
Feb 11 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Joseph Cassman"  wrote in message 
news:yqtkzxlmxepvdxkghqzg forum.dlang.org...

 On Tuesday, 11 February 2014 at 20:02:45 UTC, Meta wrote:
 However, I think DDMD will bring about a small bump in contributors. 
 Personally, I haven't used C++ in awhile, and I've grown rusty. At this 
 point, D is probably the language I'm most comfortable with, especially 
 for hacking on something complex like a compiler. I imagine there are at 
 least a few people who feel the same way and are waiting for DDMD before 
 they take a crack at contributing.

I have to agree. It's been over ten years since I have done any C++ seriously and as a result feel a lot of friction when trying to get up to speed on working with the D compiler code-base. Truth be told, that is part of the reason I like D in the end, it's not C++. Being able to use D to hack on the compiler would make it much more feasible for me to contribute. Joseph

Seriously, don't wait. DMD uses a D-like subset of C++, and DDMD uses a C++-like subset of D. Working on DMD is unlike any other C++ codebase I have ever worked on because of this, and is remarkably pleasant.
Feb 12 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 09/02/2014 11:16, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" a écrit :
 On Sunday, 9 February 2014 at 10:06:12 UTC, Manu wrote:
 I don't think you've mage a game recently.

Pointless comment.
 Most big games are multi-year projects with teams numbering well in the

Most games are not big. Most games fail in the marketplace.
 I didn't say they should be a focus, I'm saying they must however be
 supported.

Must is a strong word, but since D is focusing on separate compilation it probably is a focus. Why are most comments about the application domain for D centered on "prestigious" projects such as AAA games, high volume trading system and safety critical appliations? The most likely application domain is a lot less "exciting": tools and simple servers. Get down to earth, plz.

Maybe if performances are not really critical, developers can already use Java or C# with a GC? IMO D have to target all applications have to be written in C/C++, where performances or portability, scalability,... are critical. Java and C# developers already have a lot of good tools that C/C++ developers doesn't have, we need love too :-) D claims to be a system language so it's normal to expect to be able to use it on critical ways easily, and maybe lesser for simple applications can already done with actual proven technologies. D is already more complicate to use than Java or C#, just take a look of keywords,... but it's not surprising for a system language with advanced features than other languages doesn't support. IMO D programmers want have a great control on memory even if it can be a pain. We can let D satisfy ego of few developers doesn't already have the need of a such language on some points since it is already much less error prone than C++. :-)
Feb 10 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 09/02/2014 21:15, francesco cattoglio a écrit :
 However, the last point was directed to the D community. The language
 needs to be more focused on being very good at some key areas, not
 cover everything.

there are game developers out there, willing to use D. I also see lots of movement from hobbysts. We can't ignore them completely. Undefinedly long pauses are really bad for them, and something needs to be done. Be it in user code, library solution or as core part of the language. I agree that AAA titles are not the main right now, but this doesn't mean indie projects shouldn't be doable. After all, 150 milliseconds pauses are really annoying for pretty much any first person game.

With a pause of 150ms, you just can't play an animation of any kind. A lot of sample application like a movie play will require to use threads just cause of a GC? I don't find that simple. It's the same for GUI application, having 150ms of pause after a button hit can be horrible, again I don't think it has to be necessary to thread all UI applications even if it's simpler with a language like D.
Feb 10 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 10.02.2014 22:23, schrieb Xavier Bigand:
 Le 09/02/2014 21:15, francesco cattoglio a écrit :
 However, the last point was directed to the D community. The language
 needs to be more focused on being very good at some key areas, not
 cover everything.

there are game developers out there, willing to use D. I also see lots of movement from hobbysts. We can't ignore them completely. Undefinedly long pauses are really bad for them, and something needs to be done. Be it in user code, library solution or as core part of the language. I agree that AAA titles are not the main right now, but this doesn't mean indie projects shouldn't be doable. After all, 150 milliseconds pauses are really annoying for pretty much any first person game.

With a pause of 150ms, you just can't play an animation of any kind. A lot of sample application like a movie play will require to use threads just cause of a GC? I don't find that simple. It's the same for GUI application, having 150ms of pause after a button hit can be horrible, again I don't think it has to be necessary to thread all UI applications even if it's simpler with a language like D.

A bit off topic, but can you still get new single core chips?
Feb 10 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 11.02.2014 00:11, schrieb deadalnix:
 On Monday, 10 February 2014 at 21:57:42 UTC, Paulo Pinto wrote:
 A bit off topic, but can you still get new single core chips?

Sure you can. But that is far from common, unless you have really strict constraints.

I know, I was just big a bit sarcastic about not using threads in this day and age.
Feb 10 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/02/2014 09:58, Manu a écrit :
 On 10 February 2014 17:58, francesco cattoglio
 <francesco.cattoglio gmail.com <mailto:francesco.cattoglio gmail.com>>
 wrote:

     On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:

         Sorry, I obviously mean, "the only *games* company..."

     That was a given. However I think AAA titles have the manpower to
     avoid those pauses, since the amount of work toward optimization is
     huge anyway, am I right? Ofc you still need minimal backend from the
     compiler and runtime support. If you lack control on internals,
     there's no way for you to optimize anything.


 If we wanted to spend that time+manpower (read, money & overtime/sanity)
 on bullshit like that, we have no reason to adopt D; we already have
 C/C++, and we already have decades of experience mitigating that nightmare.
 The point is, we are REALLY sick of it. Why would we sign up to replace
 it with more of the same thing.

         And people seem to forget promptly after every single time I
         repeat myself:
           * The GC frequency of execution is directly proportional to
         the amount of
         _free memory_. In console games; NONE.
           * The length of the associated pause is directly proportional
         to the
         amount of memory currently in use. In console games; all of it.

     For "simple" games, it would be nice to have a better GC and cut
     down allocations from the standard library. I guess that would
     suffice, no need to move to ARC.


 For 'simple' games, might as well write then in Java or C#, the tooling
 is much better, and support is offered by major multinational corporations.
 Not to say that they shouldn't be supported in D too, but that's not a
 target of interest to me, and I don't think it's an area which makes a
 particularly compelling argument for adoption of D.
 I've said before, console games is an industry desperate for salvation,
 and D makes a very strong case here in lieu of any other realistic
 alternatives... as long as this memory stuff is addressed acceptably.

 If there were to be some killer potential markets identified for D, I
 think this is definitely one of them.

Like you I don't see the interest for D to compete against C# or Java. IMO there is a big hole that C/C++ developers dream to see filled. We want : - a less error prone language : DONE - a better syntax : DONE - advanced meta-programming features : DONE - a fast build : DONE - a rich framework : Have some potential (miss a lot of QtCore equivalent, GUI libraries), progress really slowly - be multi-platform : almost DONE - be cross-platform (binary portable) : a potential way with LVM bytecode - no performance concessions : GC issues - better tools (IDE with refactor tools working, better debugger,...) - buildin tools (unittest, static analyser,...) : DONE For the moment D GC was a real pain for me on DQuick, I am not able to control when releasing my OpenGL resources easily. I can't wait a GC collect, cause handles are fewer than the central memory. The destruction order is also a real issue. I certainly have to learn new patterns, but I try to add a release method on Resource objects and add a check in debug to see if it was correctly called. I wasn't able to use Throwable.TraceInfo cause it's a class which means can't be printed in the destructor. So if a user forgive to call release method on a resource (leak) I just can't warm him with a great message... For me leaks aren't just unreferenced pointers, but also and mainly chunk of resources still retained when not necessary cause those are harder to track and are critical on small devices. A GC won't help you a lot here, because IMO it act like a pool on every objects. It seems a lot of developers aren't concern by memory usage when there is a GC. I am also concern of having all applications using a GC, cause it force user of multi-task OS to buy more memory or close few applications. I just buy 4Go more cause I can't let my smartgit,VS,Chrome,... when playing Battlefield 4???. Smartgit is in Java, VS in C# in a part and Chrome create a process per tab??? Please stop pushing us create applications memory consuming, it's not cheap. --- Maybe D is too ambitious, what we really need is a language that can be updated more easily than C++ to be as soon as possible usable in industry. Seriously I work for a little game company, we are few developers (about 9 developers on 3 different projects), we don't have means to use IncrediBuild or such tools to save times. Reducing the compile time and having a great framework are critical points for us. We have relatively few memory issues by managing it ourselves. We use C++ cause our targets are numerous : - Windows Pocket PC : not anymore - Symbian : not anymore - Nintendo DS : not anymore - Nintendo Wii : not anymore - iOS - Windows - MacOS X - Android - Xbox 360 - PS 3 Our applications/games can have some critical performances aspect, but we don't have to optimize all our code, cash misses are accepted :-) Our optimization are essentially high level or only concern the OpenGL render, maybe some few algorithm like simple occlusion, tiling,... Sadly for next games we didn't use our internal game engine but Unity 3D. We made some test on doing a scene like those of RTMI (http://www.youtube.com/watch?v=fGtfhKrg3l0) on Unity without success, cause of specific optimizations required for animations done with a lot of textures updates. Editors prefers Unity cause of C# that is synonymous of standard tools and easier to regain control by other developers.
Feb 10 2014
prev sibling next sibling parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/02/2014 09:59, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" a écrit :
 On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:
 The only company I know of that has made a commercial commitment to D
 is a
 AAA games company...


Unfortunately a AAA games company is not setting down the goal post for D. As long as the leads for the project have as their primary interests: non-real-time stuff and STL-like-libraries things won't develop in your (and mine and Fransescos) direction. It won't happen until the leads of the project COMMIT to a MEASURABLE goal and a major effort is made to meet that goal. That means putting other goals aside until that measurable goal has been met.
 Sorry, I obviously mean, "the only *games* company..."

Yeah, but that games company needs to commit to taking a lead role so that the goal post and vision changes in that direction.
 And people seem to forget promptly after every single time I repeat
 myself:
  * The GC frequency of execution is directly proportional to the
 amount of
 _free memory_. In console games; NONE.
  * The length of the associated pause is directly proportional to the
 amount of memory currently in use. In console games; all of it.

 This doesn't only describe games, it describes any embedded environment.

I've already stated that I don't believe in using D for anything multi-media.

???? So in these case I will forget D, and cry all the tears of my body. It will just a shame for a system language. And it's certainly kind of applications actually miss for D, to improve his visibility. Just take a look around you all applications are interactive, with more animations,...
 It is not part of the project vision to be good at that from what I am
 seeing, and I am not going to believe it is going to be good for that
 until the project leads commit to measurable goals.

 The leads believe in meritocracy, that means the project will flail
 around in any direction that is fun. That means there are no rails.
 There is no reason to pull or push a train that is not on rails. To get
 D to be a true better C++ you need a concerted effort.

Feb 10 2014
parent reply Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 11/02/2014 00:12, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" a écrit :
 On Monday, 10 February 2014 at 23:06:56 UTC, Xavier Bigand wrote:
 ????
 So in these case I will forget D, and cry all the tears of my body. It
 will just a shame for a system language.
 And it's certainly kind of applications actually miss for D, to
 improve his visibility.

Yes, but nobody that are in the "decision making body" of D has shown any resemblance of understanding or priority for real time applications. Seriously, the language is close to ten years in the making.
 Just take a look around you all applications are interactive, with
 more animations,...

Yeah, but at some point you just have to accept that people who don't have a need to write real time will avoid putting it on the road map.

Boostrapping D, will certainly reveal some issue with GC? I don't know much about compilers, but certainly the parser may see some performance gains with GC and the other parts? Maybe some major loose? I am curious to see that as a benchmark comparator. It's not real-time but it's a serious system application challenge.
 I would like to see a roadmap that says "real time" and "no gc" and
 "whole program optimization", "owned pointers", "shared pointers"++

 I see no road map.

Feb 10 2014
parent "Daniel Murphy" <yebbliesnospam gmail.com> writes:
"Xavier Bigand"  wrote in message news:ldbohi$1ohh$1 digitalmars.com...
 Boostrapping D, will certainly reveal some issue with GC? I don't know 
 much about compilers, but certainly the parser may see some performance 
 gains with GC and the other parts? Maybe some major loose?

The parser allocates lots of memory without freeing any (the entire parsed ast), so a GC cannot possibly be an improvement there over the current strategy of C++ new + never delete. DDMD has predictably shown that there is a performance hit, even with collections disabled, compared with the highly tuned allocator used in the C++ version. The big plus of a GC for the compiler is that now ctfe is much less likely to cause the compiler to run out of memory, as all the temporary objects generated while interpreting will be garbage collected.
Feb 10 2014
prev sibling next sibling parent Xavier Bigand <flamaros.xavier gmail.com> writes:
Le 10/02/2014 13:04, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" a écrit :
 On Monday, 10 February 2014 at 09:36:53 UTC, Manu wrote:
 I'm confused. A couple of posts ago, you seemed to be annoyed at me for
 consistently raising games as a target application space that was
 unrealistic, or not 'down to earth', or some fairly niche and irrelevant
 target workload.

Sorry about that. I have been following D since 2005, on and off, and kept waiting for the "better C++" to materialize so I can use it to do fun stuff with it (audio, 3D, raytracing etc). One hobby of mine is to read Intel/AMD CPU docs, raytracing papers and compiler stuff, and discussing those aspects and improving my understanding of those areas is fun. I am loosing hope in that direction for D, because I don't think D has anyone with a strong interest in project management that can drive it in that direction. The responses from the D leads shows signs, not of a lack of skills, but a lack of interest in project management "theory" (and unfortunately, that is an area where I know the theory quite well since I majored in that area). On the fun side I want what you want. I would love to see you be the third lead on D, to get a person that "falls to sleep thinking real time" into that position would make me believe in the project. On the "pay for bread" side I am looking at D from the perspective of having an alternative to Go on the server side. I guess that has made me "janus-faced" in this discussion. What would make me tilt in favour of Go instead of D, is that it has corporate backing and therefore give priority to production level stability. Even though I like the semantics of D better. Stability is important to me since I personally pay the price (literally) for technical flaws since I offer fixed priced solutions. Instead of A.A. and W.B. going defensive (and yes it is painful to see your child get a needle in the foot at the doctor to get that vaccine that will keep the child healthy in the long term) they should try to get someone into the team of leads that has an interest in software development process and software process improvement. Or at the very least, one person with real time focus. (Please note that I found it quite amusing that you claimed that I was ignorant of long running games, since I studied Anarchy Online from inception to end in a qualitative manner while trying to figure out the design properties of the design domain, from a system development perspective. You don't have to convince me, I do understand where you are coming from and enjoy reading about your perspective. ;^)
 Video games is a bigger industry than the movie industry. Casual/phones
 have captured a large slice in recent years, but the rest of the pie is
 almost entirely games consoles, which I don't think is a diminishing
 industry so much as the casual/phone space is rather growing the pie in
 overall volume. The industry is expanding as a whole.

Yes, unfortunately the revenue in the mobile app space is very low for the majority of developers which requires tools that make them very productive at the cost of technical quality. So lots of stuff is being done with cheap (and not really performant) tech to cut down on dev time.

Yes I can't tell you how hard it is. Plus editors that are much more concern historically by market than product quality, cause "mobile apps" are seen as cheap software. So few people take it seriously, maybe only Apple understand friendly applications have to be perfectly polished to be a commercial success.
 A more performant and productive language could certainly make a
 difference, but to get there you need to focus on that niche, otherwise
 it will take too many years to catch up with the alternatives (with
 their eco system). And the landscape keeps changing very quickly.
 Companies that offer 3rd party solutions fold all the time. So mobile
 devs are "jaded".

 I don't think anyone in the D community really has that power. If Walter
 were to dictate direction that was unpopular enough, the developer base
 would promptly dissolve.

Yes, some would leave, but others would join. Those who today look at D and say: - "This is kind of cool, but not quite there yet" - "when can I expect to see it land in the area where it makes me productive" - "is this cart worth pushing, can we actually make a significant improvement here or do I have to push this cart all by myself" I would imagine that there are more people sitting on the fence than not. What made Linux work out was that they were aiming for a well defined vision, Unix. Progress was easy to measure. What made Linux fail on the desktop that they did not have a well defined vision, so the community spread out on N alternatives and progress was hard to measure. This is a bit simplistic, but Open Source projects that does not have a strongly projected vision tends to wither and dissolve over time.
 the goal. Contributing to D is, in some way, a form of recreation for
 contributors.

But you still need a clear vision and well defined goals, because for every "fun" bit there is 2 "unfun" bits. For every "excellent feature", you have to axe "2 nice to haves". (kind of)
 Are you saying I don't complain enough? :) (at least, last year before
 I left)
 I would never want to assert authority on the language direction on
 behalf
 of a single company, like you say, it's a niche target, although a
 very big
 niche which I think will really benefit from D.

Actually, I think you have the passion to put forth a vision that could bring D to real time and thus make it a project that is making "fun" possible. With no "real time" person on the team I probably will take the "hobby focus" and enjoy discussing technological possibilites (such as the discussion we had about ref counting recently). If that makes A.A. upset. Great. He should be. I am implying that D needs leadership. He should take leadership. If he does not want to listen. Well, in that case I am not forcing him to read what I write. But pointing to github is pointing in the wrong direction. Github tracks missing bolts and nuts, not a skewed skeleton.
 I just make sure that people never forget that the niche exists, what the
 requirements are, and that tends to result in those targets being
 factored
 into conversations and designs.

I am perfectly cool with that. If AAA games is the vision. Good. My prime gripe is the lack of a clearly stated vision. I could go with any "system level" vision that is not covered by C++/C#.

Me too, and a lot of people in multimedia/game industry that follow D from far. I know few person interested that put their eyes back to C++ immediately cause of GC (it's maybe too soon).
 That's a shame, I see that as one of it's greatest (yet unrealised)
 potentials. What are some other reasons anyone would reach for a
 native language these days?

Scalable, low resource, servers. Servers that boot up real fast and handle many connections. I am currently musing at OSv. It is a kernel written in C++ that can run on top of KVM. Having something like Go or D on that platform could be interesting. Backing caches/databases/web services for low revenue mobile apps.
 If it's not an operating system, or some enterprising web service... what
 else commands native hardware access and performance than embedded
 development in a *highly* aggressive and competitive industry?

Again, I don't disagree. *smooch* ;)

Feb 10 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 4:04 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 One hobby of mine is to read Intel/AMD CPU docs, raytracing papers and
 compiler stuff, and discussing those aspects and improving my
 understanding of those areas is fun. I am loosing hope in that direction
 for D, because I don't think D has anyone with a strong interest in
 project management that can drive it in that direction. The responses
 from the D leads shows signs, not of a lack of skills, but a lack of
 interest in project management "theory" (and unfortunately, that is an
 area where I know the theory quite well since I majored in that area).

Terrific. The challenge here is to adapt project management theory to the realities of a volunteer project.
 Instead of A.A. and W.B. going defensive (and yes it is painful to see
 your child get a needle in the foot at the doctor to get that vaccine
 that will keep the child healthy in the long term) they should try to
 get someone into the team of leads that has an interest in software
 development process and software process improvement.

We're not getting defensive here. Clearly we could and should move faster than we do, and there must be ways to be better at that. I'm all ears on advice on how to do better. I just have difficulty seeing how more management would help. Andrei
Feb 10 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/10/14, 6:24 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 No, I don't think it is only a matter of resources. For instance, if I
 had the time

Oh, the unbelievable irony. Andrei
Feb 10 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/11/14, 1:42 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Tuesday, 11 February 2014 at 02:15:37 UTC, Andrei Alexandrescu wrote:
 On 2/10/14, 6:24 AM, "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 No, I don't think it is only a matter of resources. For instance, if I
 had the time

Oh, the unbelievable irony.

Not really. If you have too many outstanding issues it means you have added to many features. It means you failed to do feature freeze at an earlier stage. It could also mean that you don't give priority to mentoring. Sometimes it is better to let your best people do mentoring and help bringing "master level students" up to speed. People are not loyal to a project. People are loyal to other people. If a mentor invests time in you, you will feel a social debt. This is the principle of gifting. You can create a strategy for mentoring. One obvious one is to focus on making the code base suitable for academia. Then you can offer supervision of master students. Academics love to have good external supervisors taking some load off their backs. That means lowering the requirements for compilation speed in order to get in some high level optimization and other features that you cannot otherwise have. You can give priority to getting in support for more social bonding between developers, like give priority to an IDE that supports CSCW style collaboration (seeing the code view of others). With Skype that could make pair programming (from XP) possible. There are many options.

I confess I don't understand all of this (not sure whether the pointed-out irony has been acknowledged, not sure even whether it's subtle trolling), but upon reading it a couple of times I get the sense it's the exact management gobbledygook I'd like to protect this community from. Andrei
Feb 11 2014
prev sibling next sibling parent "Joseph Cassman" <jc7919 outlook.com> writes:
On Tuesday, 11 February 2014 at 18:52:40 UTC, Andrei Alexandrescu 
wrote:
 On 2/11/14, 10:12 AM, Brad Anderson wrote:
 I think it got off to a bad start because it was private and 
 even after
 it was public it wasn't talked about much within the community 
 (it was
 hard to even find because it wouldn't turn up in Trello's 
 search). I use
 and enjoy Trello at work so personally I think I'd try to get 
 everyone
 to give it another shot if it were up to me.

I'm also in favor of a second go, particularly since I've initiated the first one :o). I'm still using Trello for tracking personal stuff, though not nearly as frequently as to make a big difference. Andrei

Although I haven't yet contributed to the project by sending any pull requests I am interested in doing so. Perhaps the main thing that keeps holding me back is I feel it is hard to know what currently is being worked on with the current setup (i.e. who is doing what, will my efforts duplicate someone else's, will I be stepping on anyone's toes, will my solution idea be incompatible with the direction in which the language is moving). I frequent the forums and still find it hard to get an overall picture. This is probably my lack of not trying hard enough (e.g. I never have time to spend on IRC). I saw the Trello board before and I felt that it helped me get up to speed on who is doing what, what is being worked on, etc. If you brought it, or something like it, back that would be appreciated. Sorry to change the subject but one thing that keeps discouraging me from trying to contribute code changes is the large number of unmerged pull requests in GitHub. Since I am not a reviewer I am afraid that any effort I perform will get ignored and so I hold back since I have other stuff to do too. I know that this is something that concerns you a lot from reading other forums discussions so I apologize if this sounds like a complaint. I appreciate the work of all involved and know that no one has infinite time or energy. Not even sure if I could contribute to the code-base but would be nice to try. Managing a project is never easy. My hat's off to you and Walter for your efforts in that regard. Joseph
Feb 11 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Wednesday, 12 February 2014 at 11:01:52 UTC, Daniel Murphy 
wrote:
 "Dmitry Olshansky"  wrote in message 
 news:lddunp$jp7$1 digitalmars.com...

 Here is a sketch. Sorry couldn't make it more realistic -  I'm 
 packing my stuff for tomorrow's flight (about 6 hours to go):
 http://wiki.dlang.org/Groups

This looks useful, I can't always remember who works on what. I would probably prefer compiler internals questions were posted to the dmd-internals mailing list though, so everyone can see them/subscribe/search them.

Personally I hate dmd-internals because it's riddled with merge notification threads. Those github notifications really need to be in a separate newsgroup or we need to move discussions elsewhere because (at least for me) actual discussion threads get lost.
Feb 12 2014
prev sibling next sibling parent Andrej Mitrovic <andrej.mitrovich gmail.com> writes:
On 2/11/14, Joseph Cassman <jc7919 outlook.com> wrote:
 Sorry to change the subject but one thing that keeps discouraging
 me from trying to contribute code changes is the large number of
 unmerged pull requests in GitHub.

For what it's worth new Phobos pull requests get a review and are merged pretty quickly lately. The number of open pull requests is drastically falling as of late. Take a look at the blue line: http://i.imgur.com/ELfvwEO.png Generated from: http://d.puremagic.com/test-results/chart.ghtml?projectid=1 ----- Don't hesitate to contribute by submitting a pull request, there will be someone around that can review your work, and guide you in the process if you have any trouble along the way. Thanks!
Feb 16 2014
prev sibling parent "Joseph Cassman" <jc7919 outlook.com> writes:
On Sunday, 16 February 2014 at 20:48:50 UTC, Andrej Mitrovic 
wrote:
 On 2/11/14, Joseph Cassman <jc7919 outlook.com> wrote:
 Sorry to change the subject but one thing that keeps 
 discouraging
 me from trying to contribute code changes is the large number 
 of
 unmerged pull requests in GitHub.

For what it's worth new Phobos pull requests get a review and are merged pretty quickly lately. The number of open pull requests is drastically falling as of late. Take a look at the blue line: http://i.imgur.com/ELfvwEO.png Generated from: http://d.puremagic.com/test-results/chart.ghtml?projectid=1 ----- Don't hesitate to contribute by submitting a pull request, there will be someone around that can review your work, and guide you in the process if you have any trouble along the way. Thanks!

Appreciate the graphic. Looks encouraging. Joseph
Feb 16 2014
prev sibling next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 02/06/2014 12:16 AM, Adam D. Ruppe wrote:
 On Wednesday, 5 February 2014 at 22:32:52 UTC, Andrei Alexandrescu wrote:
 ...
 I should also add that imparting useful semantics to scope is much
 more difficult than it might seem.

I'm not so sure about that*, but the fact is scope would be enormously useful if it was implemented. * Let's say it meant "assigning to any higher scope is prohibited".

Then the type system should track scope depth (i.e. regions) in order to support abstraction properly. (This is Walter's usual identity function test, and it won't get less of a problem unless D gets a sane polymorphic type system.)
 That
 should be trivially easy to check and ensures that variable itself
 doesn't escape.

How would you express that a slice of a static stack array of GC-allocated references (and those may be escaped) shouldn't be escaped?
 The tricky part would be preventing:

 int[] global;
 void foo(scope int[] a) {
     int[] b = a;
     global = b;
 }

Well, to some extent this is a solved problem. Just decide which escape analysis strategy to use and maybe extend its precision with type information.
Feb 08 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Tuesday, 11 February 2014 at 17:37:36 UTC, Andrei Alexandrescu 
wrote:
 Yah, the whole Trello experiment has been on my tongue during 
 this discussion. It's been made public long before its demise.


 Andrei

I think it got off to a bad start because it was private and even after it was public it wasn't talked about much within the community (it was hard to even find because it wouldn't turn up in Trello's search). I use and enjoy Trello at work so personally I think I'd try to get everyone to give it another shot if it were up to me.
Feb 11 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 11 February 2014 at 18:12:41 UTC, Brad Anderson wrote:
 On Tuesday, 11 February 2014 at 17:37:36 UTC, Andrei 
 Alexandrescu wrote:
 Yah, the whole Trello experiment has been on my tongue during 
 this discussion. It's been made public long before its demise.


 Andrei

I think it got off to a bad start because it was private and even after it was public it wasn't talked about much within the community (it was hard to even find because it wouldn't turn up in Trello's search). I use and enjoy Trello at work so personally I think I'd try to get everyone to give it another shot if it were up to me.

I've been here on and off for lots of time, and it is the first time I hear about Trello at all :D
Feb 11 2014
prev sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Tuesday, 11 February 2014 at 18:50:56 UTC, Andrei Alexandrescu 
wrote:
 One simple example: specific regressions and blockers in 
 bugzilla.

Are those the same as the one's which are marked as bounties? It could be that those problems are simply hard to fix. Many of them seem to be backend or debugging-related bugs. I'm saying not many people know how to fix these.
Feb 11 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 10 February 2014 at 23:15:35 UTC, Andrei Alexandrescu 
wrote:
 Taking a look at 
 https://www.bountysource.com/trackers/383571-d-programming-language, 
 however, reveals that there's little attention to those bugs, 
 in SPITE of the fact that contributions on HARDER problems on 
 the SAME project continued as furiously as ever, if not more.

Interesting, I had directly the opposite impression when went through the bountysource list. Lot of issues have pull requests provided but are stalled because of slow feedback cycle.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:15:35 UTC, Andrei Alexandrescu 
wrote:
 develop themselves. But as I told Walter, for better or 
 (sometimes definitely) worse, our character flaws make history 
 inside the D community.

But I am also a hard core roleplayer… so you won't know when I am me, and when I am pulling your leg. I assume the same about you. ;-] The internet is a stage. What is real, what is not real? Hard to tell. What is a person, what is a character? Difficult question.
 This is a typical problem. Reviewing contributions is hard and 
 thankless work. I know how we solved it at Facebook for our 
 many open-sourced projects: we created a team for it, with a 
 manager, tracking progress, the works. This is _exactly_ the 
 kind of thing that can't be done in a volunteer community.

Maybe you can make some parts modular after you refactor into D. Then people can take ownership of modules and social recognition will encourage more commitment. I don't know the D social arena well enough to know if that works though.
 So I wasn't glib when I sent you to github. In a very concrete 
 sense, you'd be helping there a ton more in a fraction of the 
 time you spend posting.

But I don't want to do that when I am merely assessing D. I am not commited to D. Yet.
 I think D must not define itself in relation to any other 
 language.

I respect that position. Of course, it does not help if outsiders have been told that D is a better C++. It kinda sticks. Because people really want that. I am very hard trying to convince myself that D is more like compiled C#, which lowers my expectations, because that original vision of a "better C++" is very firmly stuck. Of course, the problem with C++ is that it is used very differently by different people. D is appealing more to the high-level version of C++. It probably depends on when you first used C++ or what parts of C++ you are interested in.
 done and witnessed a number of such attempts, I think you're 
 exceedingly naive about what can be done with traditional 
 project management approaches in this case.

There you go ad hominem again. I have studied online virtual worlds where people volunteer for worse… But my point was more that you need to communicate a vision that is such that the people you want to attract don't sit on the fence. I am quite certain that more skilled C++ programmers would volunteer if they saw a vision they believed in. So maybe they don't do more, but more hands available…
 Ola, I'm sure you mean well. I trust you will find it within 
 yourself the best way to contribute to this community.

You really need to avoid this ad hominem stuff… You see, as a hardcore roleplayer I could be tempted to switch over into a sarcastic mode. And that would not be fair to you. ;-)
Feb 10 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 11 February 2014 at 00:25:35 UTC, Ola Fosheim Grøstad 
wrote:
 There you go ad hominem again.

 You really need to avoid this ad hominem stuff… You see, as a 
 hardcore roleplayer I could be tempted to switch over into a 
 sarcastic mode. And that would not be fair to you. ;-)

Ok, I'm a bit fed up by your attitude so I'll express a personal opinion now: it seems to me that when people express any kind of judgment, it's ad hominem against you. When you express judgment, is not ad hominem, it's the fruit of some kind of analysis you made. You know that "ad hominem" doesn't mean "I disagree with you", right?
Feb 11 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 11 February 2014 at 09:55:09 UTC, Francesco Cattoglio 
wrote:
 Ok, I'm a bit fed up by your attitude so I'll express a 
 personal opinion now: it seems to me that when people express 
 any kind of judgment, it's ad hominem against you.

You either respond to assertions about technology or ignore it. The moment you address the person and not the argument you are going in the wrong direction. Making an analysis of project as an artifact is not "ad hominem", even if the creator of the artifact strongly disagrees with the analysis. If the creator dislike the analysis, ignore it. If the creator thinks it is interesting, respond to it. It is neither friendly or unfriendly. It is usually not necessary, because most projects are pretty clear on where they are heading. I the case of D, it is not clear. It is not clear if there is enough momentum in the D community to sustain a real-time D either. These "debates" make the fog less thick so one can see possible directions for D. Which is a combination of management and the willingness of the D community to "be vocal about real time issues". Being complacent and meek is not going to change the direction of D.
Feb 11 2014
prev sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Tuesday, 11 February 2014 at 12:47:01 UTC, Ola Fosheim Grøstad 
wrote:
 Making an analysis of project as an artifact is not "ad hominem"

hominem" either. A few posts ago, Manu wrote "I don't think you have made a game" = "I don't think you have the same amound of knowledge of the game industry". That is not ad hominem. You think that Andrei telling you to "do something" is "ad hominem". Again, it's not. He is just stating that sometimes talking is not enough. If you really care about knowing how many people are interested in getting a runtime D, open a voting pool somewhere on the internet, link it in a proper thread in the announce section, and try to see what results you get, try to interpret them and so on. If you believe the people here can achieve something great, you should try to help coordinating the community. It's true that we lack organization basics. Are you knowledgeable about the subject? Are you interested in helping the community? If your answer is "yes" to bot questions, than you should really do something, and I talking about non-coding stuff. Being a top contributor is not a requirement for helping in a tangible way.
 I the case of D, it is not clear.

 It is not clear if there is enough momentum in the D community 
 to sustain a real-time D either.

sense. As I already told you, if you want to contribute by helping coordinating efforts, I think all of us would be really happy about that.
Feb 11 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 04:22:32 UTC, Andrei Alexandrescu 
wrote:
 Your thoughts are appreciated (all 6 of them in as many 
 sentences). There is something to be said, however, about 
 armchair quarterbacking and holier-than-thou kibitzing on what 
 others should be doing. This community is as close as it gets 
 to a meritocracy, so if you think you know what's good, you do 
 good. If you want your stupendously many "I think"s to carry 
 weight, follow them with some "I do"s as well. Hop on github. 
 This endless walk through your knowledge base just isn't useful.

Good job, my initial response to Manu was a critique of going Ad Hominem and you as a person time and time again fail in that regard in many discussions. You do however deserve a round of ad hominem because you as one of the two people who are in a position to communicate the project vision and set forth MEASURABLE goals that can be tracked and evaluated, but you refuse to do so. All talk of meritocracy is essentially hypocrisy because all projects need to establish boundaries and a goal post, and you fail miserably in that regard. That's why D is a slow mover. "This endless walk through [my] knowledgebase" is of course not a walk through my knowledgebase, it is an assessment of the project that YOU FAIL to attempt to do. It is my attempt to try to figure out where this project is heading. You are right, I should not have to do it. YOU SHOULD DO IT. AND PRESENT IT. That way people won't be let down. I like the initial vision Walter Bright put forth years ago, that is to make a better C++. That has somehow evolved into making a compiled C#. Can you please ASSESS that. You and Walter Bright are leads. I expect any project and you to put forth: 1. A clear vision that establish a firm boundary. 2. A small set of clear measurable goals that give the project direction. 3. A list of points stating what the project is not going to address in the immediate future. This endless walk through what is wrong with D project management just isn't useful, because you don't want to listen.
Feb 10 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 10 February 2014 at 21:57:42 UTC, Paulo Pinto wrote:
 A bit off topic, but can you still get new single core chips?

Sure you can. But that is far from common, unless you have really strict constraints.
Feb 10 2014
prev sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 21:57:42 UTC, Paulo Pinto wrote:
 A bit off topic, but can you still get new single core chips?

Yes you can! http://en.wikipedia.org/wiki/Intel_Edison will be made available in 2014, single core. And since it will have very limited resources, C# and Java are probably off limits. I must say D would be a perfect language for it! :D
Feb 10 2014
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-04 23:51:35 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 Consider we add a library slice type called RCSlice!T. It would have 
 the same primitives as T[] but would use reference counting through and 
 through. When the last reference count is gone, the buffer underlying 
 the slice is freed. The underlying allocator will be the GC allocator.
 
 Now, what if someone doesn't care about the whole RC thing and aims at 
 convenience? There would be a method .toGC that just detaches the slice 
 and disables the reference counter (e.g. by setting it to uint.max/2 or 
 whatever).
 
 Then people who want reference counting say
 
 auto x = fun();
 
 and those who don't care say:
 
 auto x = fun().toGC();
 
 
 Destroy.

I don't think it makes much sense. ARC when used for D constructs should be treated an alternate GC algorithm, not a different kind of pointer. There's another possible use for ARC which is to manage reference-counted external objects not allocated by the D GC that are using reference counting (such as COM objects, or Objective-C objects). This could justify a different kind of pointer. But that's a separate issue from the GC algorithm used for D constructs. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 7:23 AM, Michel Fortin wrote:
 On 2014-02-04 23:51:35 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 Consider we add a library slice type called RCSlice!T. It would have
 the same primitives as T[] but would use reference counting through
 and through. When the last reference count is gone, the buffer
 underlying the slice is freed. The underlying allocator will be the GC
 allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the
 slice and disables the reference counter (e.g. by setting it to
 uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

I don't think it makes much sense. ARC when used for D constructs should be treated an alternate GC algorithm, not a different kind of pointer.

Why? The RC object has a different layout, so it may as well have a different type. Andrei
Feb 05 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-05 18:26:38 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 On 2/5/14, 7:23 AM, Michel Fortin wrote:
 I don't think it makes much sense. ARC when used for D constructs should
 be treated an alternate GC algorithm, not a different kind of pointer.

Why? The RC object has a different layout, so it may as well have a different type.

Well, it depends on your goal. If your goal is to avoid the garbage collector, you need all language constructs to use ARC. Having a single type in the language that relies on the GC defeats the purpose. What you want is simply to replace the current GC with another implantation, one that uses ARC. It shouldn't affect user code in any way, it's mostly an implementation detail (managed by the compiler and the runtime). If your goal is to have a deterministic lifetime for slices in some situations, then RCSlice as you proposes it is fine. That said, with a library type you'll have a hard time making the optimizer elide redundant increment/decrement pairs, so it'll never be optimal. I'm also not sure there's a lot of use cases for a deterministic slice lifetime working side by side with memory managed by the current GC. To me it seems you're trying to address a third problem here: that people have complained that Phobos relies on the GC too much. This comes from people who either don't want the GC to pause anything, or people who want to reduce memory allocations altogether. For the former group, replacing the current GC with an ARC+GC scheme at the language level, with the possibility to disable the GC, will fix most of Phobos (and most other libraries) with no code change required. For the later group, you need to make the API so that allocations are either not necessary, or when necessary provide a way to use a custom an allocator of some sort. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 12:23 PM, Michel Fortin wrote:
 On 2014-02-05 18:26:38 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 On 2/5/14, 7:23 AM, Michel Fortin wrote:
 I don't think it makes much sense. ARC when used for D constructs should
 be treated an alternate GC algorithm, not a different kind of pointer.

Why? The RC object has a different layout, so it may as well have a different type.

Well, it depends on your goal. If your goal is to avoid the garbage collector, you need all language constructs to use ARC. Having a single type in the language that relies on the GC defeats the purpose. What you want is simply to replace the current GC with another implantation, one that uses ARC. It shouldn't affect user code in any way, it's mostly an implementation detail (managed by the compiler and the runtime). If your goal is to have a deterministic lifetime for slices in some situations, then RCSlice as you proposes it is fine. That said, with a library type you'll have a hard time making the optimizer elide redundant increment/decrement pairs, so it'll never be optimal. I'm also not sure there's a lot of use cases for a deterministic slice lifetime working side by side with memory managed by the current GC. To me it seems you're trying to address a third problem here: that people have complained that Phobos relies on the GC too much. This comes from people who either don't want the GC to pause anything, or people who want to reduce memory allocations altogether. For the former group, replacing the current GC with an ARC+GC scheme at the language level, with the possibility to disable the GC, will fix most of Phobos (and most other libraries) with no code change required. For the later group, you need to make the API so that allocations are either not necessary, or when necessary provide a way to use a custom an allocator of some sort.

I want to make one positive step toward improving memory allocation in the D language. Andrei
Feb 05 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-05 22:19:27 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 I want to make one positive step toward improving memory allocation in 
 the D language.

I know. But I find your proposal confusing. Perhaps this is just one piece in your master plan where everything will make sense once we have all the pieces. But this piece by itself makes no sense to me; I have no idea where you're going with it. Is this the continuation of the old thread where you wanted ideas about how to eliminate hidden allocations in buildPath? Doesn't sound like it. Or is this about implementing ARC in the language for those who can't use the GC? The changes for this need to be done at a lower level (compiler, runtime), and no change would be required in Phobos. Or maybe this is to please the nogc crowd by making things reference-counted by default? While I'm not a fan of nogc, this will not work for them either as your proposal allocates from GC memory and this will sometime trigger a collect cycle. Or maybe you're trying to address the following issue: if we change D's GC to use the ARC+GC scheme, what if I don't want to increment/decrement at pointer assignment and instead rely purely on mark and sweep for certain pointers? I'm not sure if someone asked for that yet, but I guess it could be a valid concern. So, what problem are we trying to solve again? -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 05 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 4:53 PM, Michel Fortin wrote:
 On 2014-02-05 22:19:27 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 I want to make one positive step toward improving memory allocation in
 the D language.

I know. But I find your proposal confusing. Perhaps this is just one piece in your master plan where everything will make sense once we have all the pieces. But this piece by itself makes no sense to me; I have no idea where you're going with it. Is this the continuation of the old thread where you wanted ideas about how to eliminate hidden allocations in buildPath? Doesn't sound like it.

Actually buildPath is a good example because it concatenates strings. It should work transparently with RC and GC strings. Andrei
Feb 05 2014
parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2014-02-06 04:56:28 +0000, Andrei Alexandrescu 
<SeeWebsiteForEmail erdani.org> said:

 On 2/5/14, 4:53 PM, Michel Fortin wrote:
 On 2014-02-05 22:19:27 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:
 
 I want to make one positive step toward improving memory allocation in
 the D language.

I know. But I find your proposal confusing. Perhaps this is just one piece in your master plan where everything will make sense once we have all the pieces. But this piece by itself makes no sense to me; I have no idea where you're going with it. Is this the continuation of the old thread where you wanted ideas about how to eliminate hidden allocations in buildPath? Doesn't sound like it.

Actually buildPath is a good example because it concatenates strings. It should work transparently with RC and GC strings.

That thread about buildPath started like this: "Walter and I were talking about eliminating the surreptitious allocations in buildPath". But reference counting will do nothing to eliminate surreptitious allocations. It can't be that problem you're trying to address. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Feb 06 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 3:44 AM, Michel Fortin wrote:
 On 2014-02-06 04:56:28 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 On 2/5/14, 4:53 PM, Michel Fortin wrote:
 On 2014-02-05 22:19:27 +0000, Andrei Alexandrescu
 <SeeWebsiteForEmail erdani.org> said:

 I want to make one positive step toward improving memory allocation in
 the D language.

I know. But I find your proposal confusing. Perhaps this is just one piece in your master plan where everything will make sense once we have all the pieces. But this piece by itself makes no sense to me; I have no idea where you're going with it. Is this the continuation of the old thread where you wanted ideas about how to eliminate hidden allocations in buildPath? Doesn't sound like it.

Actually buildPath is a good example because it concatenates strings. It should work transparently with RC and GC strings.

That thread about buildPath started like this: "Walter and I were talking about eliminating the surreptitious allocations in buildPath". But reference counting will do nothing to eliminate surreptitious allocations.

That's exactly right. Currently buildPath uses ~= several times so it will produce allocations that the user is unable to free. If buildPath used reference counting through and through, temporary allocations would be freed eagerly inside buildPath, and the user will have a shot at freeing the end result.
 It can't be that problem you're trying to address.

Are you sure you want to debate with me what's in my mind? Andrei
Feb 06 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 4:29 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Thursday, 6 February 2014 at 11:44:34 UTC, Michel Fortin wrote:
 That thread about buildPath started like this: "Walter and I were
 talking about eliminating the surreptitious allocations in buildPath".
 But reference counting will do nothing to eliminate surreptitious
 allocations. It can't be that problem you're trying to address.

I think stuff like buildPath just shows how the language should be geared more towards compiler optimizations of allocation if performance is a real goal. The efficient thing to do is to optimize a string concat into a stack allocation (alloca) if it is a throw-away, and just change the stack frame upon return, with some heuristics and alternative execution paths in order to avoid running out of stack space.

This is a very incomplete, naive sketch. It could be made to work only in a language that has no backward compatibility to worry about. Andrei
Feb 06 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 6:14 PM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 7 February 2014 at 01:21:25 UTC, Andrei Alexandrescu wrote:
 This is a very incomplete, naive sketch. It could be made to work only
 in a language that has no backward compatibility to worry about.

Whaddya mean? On the object code level? The compiler should be conservative and generate the alternatives if needed. In this day and age a language should aim for whole program optimization and static analysis. Not necessarily the proof-of-concept compiler, but the language spec. A system level language should go out of its way to make stack and pool allocations likely/possible/probable as well as register based optimizations on calls etc.

I'm jaded - Walter and I bounced around for years ideas that vaguely prescribe a feature/optimization and gloss over many details and all difficulties... they're a dime a dozen. Andrei
Feb 07 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 18:26:38 UTC, Andrei 
Alexandrescu wrote:
 Why? The RC object has a different layout, so it may as well 
 have a different type.

It also has different usage requirements, so it should have a different type. BTW so should GC vs borrowed pointers.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11c1e67646958b04f1b1da5e
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 06:23, Michel Fortin <michel.fortin michelf.ca> wrote:

 On 2014-02-05 18:26:38 +0000, Andrei Alexandrescu <
 SeeWebsiteForEmail erdani.org> said:

  On 2/5/14, 7:23 AM, Michel Fortin wrote:
 I don't think it makes much sense. ARC when used for D constructs should
 be treated an alternate GC algorithm, not a different kind of pointer.

Why? The RC object has a different layout, so it may as well have a different type.

Well, it depends on your goal. If your goal is to avoid the garbage collector, you need all language constructs to use ARC. Having a single type in the language that relies on the GC defeats the purpose. What you want is simply to replace the current GC with another implantation, one that uses ARC. It shouldn't affect user code in any way, it's mostly an implementation detail (managed by the compiler and the runtime). If your goal is to have a deterministic lifetime for slices in some situations, then RCSlice as you proposes it is fine. That said, with a library type you'll have a hard time making the optimizer elide redundant increment/decrement pairs, so it'll never be optimal. I'm also not sure there's a lot of use cases for a deterministic slice lifetime working side by side with memory managed by the current GC. To me it seems you're trying to address a third problem here: that people have complained that Phobos relies on the GC too much. This comes from people who either don't want the GC to pause anything, or people who want to reduce memory allocations altogether. For the former group, replacing the current GC with an ARC+GC scheme at the language level, with the possibility to disable the GC, will fix most of Phobos (and most other libraries) with no code change required. For the later group, you need to make the API so that allocations are either not necessary, or when necessary provide a way to use a custom an allocator of some sort.

This. --001a11c1e67646958b04f1b1da5e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 06:23, Michel Fortin <span dir=3D"ltr">&lt;<a href=3D"mailto= :michel.fortin michelf.ca" target=3D"_blank">michel.fortin michelf.ca</a>&g= t;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im">On 2014-02-05 18:26:38 +00= 00, Andrei Alexandrescu &lt;<a href=3D"mailto:SeeWebsiteForEmail erdani.org= " target=3D"_blank">SeeWebsiteForEmail erdani.org</a><u></u>&gt; said:<br> <br> </div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-l= eft:1px #ccc solid;padding-left:1ex"><div class=3D"im"> On 2/5/14, 7:23 AM, Michel Fortin wrote:<br> </div><div class=3D"im"><blockquote class=3D"gmail_quote" style=3D"margin:0= 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I don&#39;t think it makes much sense. ARC when used for D constructs shoul= d<br> be treated an alternate GC algorithm, not a different kind of pointer.<br> </blockquote> <br> Why? The RC object has a different layout, so it may as well have a differe= nt type.<br> </div></blockquote> <br> Well, it depends on your goal. If your goal is to avoid the garbage collect= or, you need all language constructs to use ARC. Having a single type in th= e language that relies on the GC defeats the purpose. What you want is simp= ly to replace the current GC with another implantation, one that uses ARC. = It shouldn&#39;t affect user code in any way, it&#39;s mostly an implementa= tion detail (managed by the compiler and the runtime).<br> <br> If your goal is to have a deterministic lifetime for slices in some situati= ons, then RCSlice as you proposes it is fine. That said, with a library typ= e you&#39;ll have a hard time making the optimizer elide redundant incremen= t/decrement pairs, so it&#39;ll never be optimal. I&#39;m also not sure the= re&#39;s a lot of use cases for a deterministic slice lifetime working side= by side with memory managed by the current GC.<br> <br> To me it seems you&#39;re trying to address a third problem here: that peop= le have complained that Phobos relies on the GC too much. This comes from p= eople who either don&#39;t want the GC to pause anything, or people who wan= t to reduce memory allocations altogether. For the former group, replacing = the current GC with an ARC+GC scheme at the language level, with the possib= ility to disable the GC, will fix most of Phobos (and most other libraries)= with no code change required. For the later group, you need to make the AP= I so that allocations are either not necessary, or when necessary provide a= way to use a custom an allocator of some sort.</blockquote> <div><br></div><div>This.</div></div></div></div> --001a11c1e67646958b04f1b1da5e--
Feb 05 2014
prev sibling next sibling parent "Mike" <none none.com> writes:
On Wednesday, 5 February 2014 at 20:23:13 UTC, Michel Fortin 
wrote:

 What you want is simply to replace the current GC with another 
 implantation, one that uses ARC. It shouldn't affect user code 
 in any way, it's mostly an implementation detail (managed by 
 the compiler and the runtime).

Yes.
 To me it seems you're trying to address a third problem here: 
 that people have complained that Phobos relies on the GC too 
 much.

Yes.
 This comes from people who either don't want the GC to pause 
 anything, or people who want to reduce memory allocations 
 altogether. For the former group, replacing the current GC with 
 an ARC+GC scheme at the language level, with the possibility to 
 disable the GC, will fix most of Phobos (and most other 
 libraries) with no code change required.

Yes.
 For the later group, you need to make the API so that 
 allocations are either not necessary, or when necessary provide 
 a way to use a custom an allocator of some sort.

... and Yes
Feb 05 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
I know I'm coming into this a bit late, but in general I only 
feel like there's a problem with built-in dynamic types.  It 
generally isn't hard to manage the lifetime of classes manually, 
and I don't tend to churn through them.  Also, there are 
basically no routines in Phobos that operate on classes, so 
that's really entirely my problem as a user anyway.  What 
concerns me most is string processing, and in particular the 
routines in Phobos that do string processing.  And while I've 
always liked the idea of supplying a destination buffer to these 
routines, it doesn't help the case where the buffer is too small 
and allocation still needs to occur.

Instead of sorting out some crazy return type or supplying a raw 
destination buffer, what if we instead supply an appender?  Then 
the appender could grow the buffer in place or throw or whatever 
we want the behavior to be when out of space.  I think this would 
solve 90% of my concerns about unexpected GC pauses in my code.  
For the rest, any temporary allocations inside Phobos routines 
should either be eliminated, reused on each call per a static 
reference, or cleaned up.  I'm really okay with the occasional 
"new" inside a routine so long as repeated calls to that routine 
don't generate tons of garbage and thus trigger a collection.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 11:44:34 UTC, Michel Fortin wrote:
 That thread about buildPath started like this: "Walter and I 
 were talking about eliminating the surreptitious allocations in 
 buildPath". But reference counting will do nothing to eliminate 
 surreptitious allocations. It can't be that problem you're 
 trying to address.

I think stuff like buildPath just shows how the language should be geared more towards compiler optimizations of allocation if performance is a real goal. The efficient thing to do is to optimize a string concat into a stack allocation (alloca) if it is a throw-away, and just change the stack frame upon return, with some heuristics and alternative execution paths in order to avoid running out of stack space. E.g. a(){ return buildPath(...) ~ "!"; } Compiles to: a: call buildPath(...) alloca(size_to_endof_path_returned_by_buildPath+1) *someaddr = '!' return (stackbufferstart,length) buildPath: stackbuffer=alloca(sum_of_lengths) copy... return tuple(stackbufferstart,length)
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 00:30:00 UTC, Ola Fosheim Grøstad 
wrote:
    alloca(size_to_endof_path_returned_by_buildPath+1)

Of course, since the stack usually grows that should be size_to_start_of_path and you need to preallocate the necessary padding. A pity that stacks usually don't grow upwards, that would make appending much more flexible.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 01:21:25 UTC, Andrei Alexandrescu 
wrote:
 This is a very incomplete, naive sketch. It could be made to 
 work only in a language that has no backward compatibility to 
 worry about.

Whaddya mean? On the object code level? The compiler should be conservative and generate the alternatives if needed. In this day and age a language should aim for whole program optimization and static analysis. Not necessarily the proof-of-concept compiler, but the language spec. A system level language should go out of its way to make stack and pool allocations likely/possible/probable as well as register based optimizations on calls etc.
Feb 06 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 7 February 2014 at 16:25:01 UTC, Andrei Alexandrescu 
wrote:
 I'm jaded - Walter and I bounced around for years ideas that 
 vaguely prescribe a feature/optimization and gloss over many 
 details and all difficulties... they're a dime a dozen.

I see, in this particular case it will work if dynamic stack allocation works and if you have guards on the stack size (page traps at the end of the stack). The optimization is very limited, since it only works for returning one variable length structure under certain circumstances, but then again it is a common case for types where you want to do temporary allocation and prepending/appending something before making a system call or similar.
Feb 07 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 15:01:27 UTC, Manu wrote:
 Or I wonder if there's opportunity to pinch a single bit from 
 pointers to
 mark that it is raw or RC allocated? Probably fine on 64bit.

Yes, on 64 bit that is ok. I think current x86 map something like 53 bits, the top and bottom half of the 64 bit address space. The middle is unused. Anyway, you could have two heaps if the OS allows you to.
 32bit probably needs to match a bit pattern or something.

Or one could forget about 32 bit for ARC.
 Aligned data is a challenge. I have often wondered if it would 
 be feasible
 to access the RC via a pointer hash or something, and keep it 
 in a side
 table... sounds tricky, but I wonder if it's possible.

If you have your own allocator you probably could? Segment memory regions into allocations of a particular size and have a RC-count index at the start indexed by the masked MSBs of the pointer address and have smart pointers that know the object size. Kind of tricky to get acceptable speed, but possible to do. The problem is that you will get a latency of perhaps 200+ cycles on a cache miss. Then again, you could probably make do with 32 bit counters and they are probably accessed in proximity if they hold the same type of object. One cache line is 64 bytes, so you get 16 counters per cache line. With smart allocation you might get good cache locality of the counters (8MB of L3 cache is quite a bit). (I guess alignment is primarily a problem when you want 4KiB alignment (page size), maybe not worth worrying about.)
 Rather than (allocated_base, offset), I suspect (pointer, 
 offset_from_base)
 would be better; typical dereferences would have no penalty.

Yes, probably. I was thinking about avoiding GC of internal pointers too. I think scanning might be easier if all pointers point to the allocation base. That way the GC does not have to consider offsets.
 You mean like the smart pointer double indirection? I really 
 don't like the
 double indirection, but it's a possibility. Or did I 
 misunderstand?

Yes: struct { void* object_ptr; /* offset */ uint weakcounter; uint strongcounter; } The advantage is that it works well with RC ignorant collectors/allocators. "if in doubt just set the pointer to null and forget about freeing".
 I'm sure there's a clever solution out there which would allow 
 the ARC to
 detect if it's a raw C pointer or not...

Well, for a given OS/architecture you could probably always allocate your heap in a fixed memory range on 64 bit systems then test against that range.
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 15:25:27 UTC, Michel Fortin 
wrote:
 In general ARC+GC would release memory faster so you need less 
 memory overall. Less garbage memory blocks floating around 
 might make processor caches more efficients. And less memory 
 pressure means the GC itself runs less often, thus less pauses, 
 and shorter pauses since there is less memory to scan.

Which does not fix a single problem mentioned in embedded/gamedev threads. Difference between "somewhat less" and "reliably constrained" is beyond measure.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--047d7b5d5c72f116d604f1aabef6
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 01:22, Michel Fortin <michel.fortin michelf.ca> wrote:

 On 2014-02-05 15:01:04 +0000, Manu <turkeyman gmail.com> said:

  Aligned data is a challenge. I have often wondered if it would be feasible
 to access the RC via a pointer hash or something, and keep it in a side
 table... sounds tricky, but I wonder if it's possible.

That's what Apple is doing (as seen in the CoreFoundation source code). They actually have eight such tables on OS X, each protected by a spin lock. The right table is chosen according to a few bits of the pointer. The obvious advantage is you can put immutable data in read-only memory without making the reference count immutable. The downside is that it's more complicated to access the counter.

Indeed. Good to know someone else is doing it. Sounds like a realistic option then :) Fat pointers might be necessary. That's a bit annoying. Hmmm...

Since we're talking about adding reference counts to GC-allocated memory, you could use the GC to find the base address of the memory block. What is the cost of that?

Can you elaborate? How would the GC know this? I'm sure there's a clever solution out there which would allow the ARC to
 detect if it's a raw C pointer or not...

Ask the GC for the base address of the memory block. If it does not come from a GC block, there's no reference counter to update. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca

--047d7b5d5c72f116d604f1aabef6 Content-Type: text/html; charset=UTF-8 <div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On 6 February 2014 01:22, Michel Fortin <span dir="ltr">&lt;<a href="mailto:michel.fortin michelf.ca" target="_blank">michel.fortin michelf.ca</a>&gt;</span> wrote:<br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im">On 2014-02-05 15:01:04 +0000, Manu &lt;<a href="mailto:turkeyman gmail.com" target="_blank">turkeyman gmail.com</a>&gt; said:<br> <br> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Aligned data is a challenge. I have often wondered if it would be feasible<br> to access the RC via a pointer hash or something, and keep it in a side<br> table... sounds tricky, but I wonder if it&#39;s possible.<br> </blockquote> <br></div> That&#39;s what Apple is doing (as seen in the CoreFoundation source code). They actually have eight such tables on OS X, each protected by a spin lock. The right table is chosen according to a few bits of the pointer.<br> <br> The obvious advantage is you can put immutable data in read-only memory without making the reference count immutable. The downside is that it&#39;s more complicated to access the counter.</blockquote><div><br></div><div> Indeed. Good to know someone else is doing it. Sounds like a realistic option then :)</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class="im"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Fat pointers might be necessary. That&#39;s a bit annoying. Hmmm...<br> </blockquote> <br></div> Since we&#39;re talking about adding reference counts to GC-allocated memory, you could use the GC to find the base address of the memory block. What is the cost of that?</blockquote><div><br></div><div>Can you elaborate? How would the GC know this?</div> <div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="im"> <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> I&#39;m sure there&#39;s a clever solution out there which would allow the ARC to<br> detect if it&#39;s a raw C pointer or not...<br> </blockquote> <br></div> Ask the GC for the base address of the memory block. If it does not come from a GC block, there&#39;s no reference counter to update.<span class="HOEnZb"><font color="#888888"><br> <br> <br> -- <br> Michel Fortin<br> <a href="mailto:michel.fortin michelf.ca" target="_blank">michel.fortin michelf.ca</a><br> <a href="http://michelf.ca" target="_blank">http://michelf.ca</a><br> <br> </font></span></blockquote></div><br></div></div> --047d7b5d5c72f116d604f1aabef6--
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 15:46:21 UTC, Dicebot wrote:
 Which does not fix a single problem mentioned in 
 embedded/gamedev threads. Difference between "somewhat less" 
 and "reliably constrained" is beyond measure.

If you write code in a way that does not create much cycles you don't have to call the GC at all. So getting the GC out of the implicit allocations the language make might be the most important thing, but how much memory is wasted over an hour that way? A game should perhaps run for 1 hour without a hiccup, ARC might be good enough if RC collect 98% of all garbage. A real time audio application should run for 12 hours without a hiccup… you probably want a GC free audio callback. A real time server that monitors some vital resource should run for hours without a hiccup... You either want a real time GC or no GC. Different scenarios have different needs.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11c30ce2a24a6c04f1aadc85
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 6 February 2014 01:32, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Wednesday, 5 February 2014 at 15:01:27 UTC, Manu wrote:

 Or I wonder if there's opportunity to pinch a single bit from pointers t=


 mark that it is raw or RC allocated? Probably fine on 64bit.

Yes, on 64 bit that is ok. I think current x86 map something like 53 bits=

 the top and bottom half of the 64 bit address space. The middle is unused=

 Anyway, you could have two heaps if the OS allows you to.


  32bit probably needs to match a bit pattern or something.

Or one could forget about 32 bit for ARC.

The applications I describe where it is a necessity will often be 32bit systems. Aligned data is a challenge. I have often wondered if it would be feasible
 to access the RC via a pointer hash or something, and keep it in a side
 table... sounds tricky, but I wonder if it's possible.

If you have your own allocator you probably could? Segment memory regions into allocations of a particular size and have a RC-count index at the start indexed by the masked MSBs of the pointer address and have smart pointers that know the object size. Kind of tricky to get acceptable spee=

 but possible to do. The problem is that you will get a latency of perhaps
 200+ cycles on a cache miss. Then again, you could probably make do with =

 bit counters and they are probably accessed in proximity if they hold the
 same type of object. One cache line is 64 bytes, so you get 16 counters p=

 cache line. With smart allocation you might get good cache locality of th=

 counters (8MB of L3 cache is quite a bit).

Cache locality is a problem that can easily be refined. It would just need lots of experimental data. (I guess alignment is primarily a problem when you want 4KiB alignment
 (page size), maybe not worth worrying about.)


  Rather than (allocated_base, offset), I suspect (pointer,
 offset_from_base)
 would be better; typical dereferences would have no penalty.

Yes, probably. I was thinking about avoiding GC of internal pointers too. I think scanning might be easier if all pointers point to the allocation base. That way the GC does not have to consider offsets. You mean like the smart pointer double indirection? I really don't like
 the
 double indirection, but it's a possibility. Or did I misunderstand?

Yes: struct { void* object_ptr; /* offset */ uint weakcounter; uint strongcounter; } The advantage is that it works well with RC ignorant collectors/allocators. "if in doubt just set the pointer to null and forg=

 about freeing".

I see. Well, I don't like it :) ... but it's an option. I'm sure there's a clever solution out there which would allow the ARC to
 detect if it's a raw C pointer or not...

Well, for a given OS/architecture you could probably always allocate your heap in a fixed memory range on 64 bit systems then test against that ran=


--001a11c30ce2a24a6c04f1aadc85 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 01:32, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola Fo= sheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot;= puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &lt= ;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"im">On Wednesday, 5 February 2= 014 at 15:01:27 UTC, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Or I wonder if there&#39;s opportunity to pinch a single bit from pointers = to<br> mark that it is raw or RC allocated? Probably fine on 64bit.<br> </blockquote> <br></div> Yes, on 64 bit that is ok. I think current x86 map something like 53 bits, = the top and bottom half of the 64 bit address space. The middle is unused. = Anyway, you could have two heaps if the OS allows you to.<div class=3D"im"> <br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> 32bit probably needs to match a bit pattern or something.<br> </blockquote> <br></div> Or one could forget about 32 bit for ARC.</blockquote><div><br></div><div>T= he applications I describe where it is a necessity will often be 32bit syst= ems.</div><div><br></div><div><br></div><blockquote class=3D"gmail_quote" s= tyle=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> <div class=3D"im"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Aligned data is a challenge. I have often wondered if it would be feasible<= br> to access the RC via a pointer hash or something, and keep it in a side<br> table... sounds tricky, but I wonder if it&#39;s possible.<br> </blockquote> <br></div> If you have your own allocator you probably could? Segment memory regions i= nto allocations of a particular size and have a RC-count index at the start= indexed by the masked MSBs of the pointer address and have smart pointers = that know the object size. Kind of tricky to get acceptable speed, but poss= ible to do. The problem is that you will get a latency of perhaps 200+ cycl= es on a cache miss. Then again, you could probably make do with 32 bit coun= ters and they are probably accessed in proximity if they hold the same type= of object. One cache line is 64 bytes, so you get 16 counters per cache li= ne. With smart allocation you might get good cache locality of the counters= (8MB of L3 cache is quite a bit).<br> </blockquote><div><br></div><div>Cache locality is a problem that can easil= y be refined. It would just need lots of experimental data.</div><div><br><= /div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0= .8ex;border-left:1px #ccc solid;padding-left:1ex"> (I guess alignment is primarily a problem when you want 4KiB alignment (pag= e size), maybe not worth worrying about.)<div class=3D"im"><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Rather than (allocated_base, offset), I suspect (pointer, offset_from_base)= <br> would be better; typical dereferences would have no penalty.<br> </blockquote> <br></div> Yes, probably. I was thinking about avoiding GC of internal pointers too. I= think scanning might be easier if all pointers point to the allocation bas= e. That way the GC does not have to consider offsets.<div class=3D"im"><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> You mean like the smart pointer double indirection? I really don&#39;t like= the<br> double indirection, but it&#39;s a possibility. Or did I misunderstand?<br> </blockquote> <br></div> Yes:<br> <br> struct {<br> =C2=A0 =C2=A0 void* object_ptr;<br> =C2=A0 =C2=A0 /* offset */<br> =C2=A0 =C2=A0 uint weakcounter;<br> =C2=A0 =C2=A0 uint strongcounter;<br> }<br> <br> The advantage is that it works well with RC ignorant collectors/allocators.= &quot;if in doubt just set the pointer to null and forget about freeing&qu= ot;.</blockquote><div><br></div><div>I see. Well, I don&#39;t like it :) ..= . but it&#39;s an option.</div> <div><br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class=3D"= im"> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I&#39;m sure there&#39;s a clever solution out there which would allow the = ARC to<br> detect if it&#39;s a raw C pointer or not...<br> </blockquote> <br></div> Well, for a given OS/architecture you could probably always allocate your h= eap in a fixed memory range on 64 bit systems then test against that range.= <br> </blockquote></div><br></div></div> --001a11c30ce2a24a6c04f1aadc85--
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11330964b8b39504f1aae643
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 01:46, Dicebot <public dicebot.lv> wrote:

 On Wednesday, 5 February 2014 at 15:25:27 UTC, Michel Fortin wrote:

 In general ARC+GC would release memory faster so you need less memory
 overall. Less garbage memory blocks floating around might make processor
 caches more efficients. And less memory pressure means the GC itself runs
 less often, thus less pauses, and shorter pauses since there is less memory
 to scan.

Which does not fix a single problem mentioned in embedded/gamedev threads. Difference between "somewhat less" and "reliably constrained" is beyond measure.

The problem is completely solved; you turn the backing GC off. Devs are responsible for correct weak pointer attribution. --001a11330964b8b39504f1aae643 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 01:46, Dicebot <span dir=3D"ltr">&lt;<a href=3D"mailto:publi= c dicebot.lv" target=3D"_blank">public dicebot.lv</a>&gt;</span> wrote:<br>= <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div class=3D"im">On Wednesday, 5 February 2014 at 15:25:27 UTC, Michel For= tin wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> In general ARC+GC would release memory faster so you need less memory overa= ll. Less garbage memory blocks floating around might make processor caches = more efficients. And less memory pressure means the GC itself runs less oft= en, thus less pauses, and shorter pauses since there is less memory to scan= .<br> </blockquote> <br></div> Which does not fix a single problem mentioned in embedded/gamedev threads. = Difference between &quot;somewhat less&quot; and &quot;reliably constrained= &quot; is beyond measure.<br> </blockquote></div><br></div><div class=3D"gmail_extra">The problem is comp= letely solved; you turn the backing GC off. Devs are responsible for correc= t weak pointer attribution.</div></div> --001a11330964b8b39504f1aae643--
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 15:57:51 UTC, Ola Fosheim 
Grøstad wrote:
 If you write code in a way that does not create much cycles you 
 don't have to call the GC at all. So getting the GC out of the 
 implicit allocations the language make might be the most 
 important thing, but how much memory is wasted over an hour 
 that way?

It is up to programmer to decide. Right now he does not have a choice and sometimes you can't afford to have GC in your program at all (as in can't have it linked to the binary), not just can't call collection cycles. Having sane fallback is very desired. Proposed solution does not seem to save you from uncontrollably long collection cycles anyway as it still uses same memory pool so I don't see how it can help even games, not even speaking about more demanding applications.
 A game should perhaps run for 1 hour without a hiccup, ARC 
 might be good enough if RC collect 98% of all garbage.

 A real time audio application should run for 12 hours without a 
 hiccup… you probably want a GC free audio callback.

 A real time server that monitors some vital resource should run 
 for hours without a hiccup... You either want a real time GC or 
 no GC.

 Different scenarios have different needs.

Haven't you just basically confirmed my opinion? :)
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 16:03:56 UTC, Manu wrote:
 The problem is completely solved; you turn the backing GC off. 
 Devs are
 responsible for correct weak pointer attribution.

What does it give you over current situation with GC switched off and RefCounted used everywhere? Language features will still leak GC memory.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11330964395f4f04f1ab09e6
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 02:05, Dicebot <public dicebot.lv> wrote:

 On Wednesday, 5 February 2014 at 16:03:56 UTC, Manu wrote:

 The problem is completely solved; you turn the backing GC off. Devs are
 responsible for correct weak pointer attribution.

What does it give you over current situation with GC switched off and RefCounted used everywhere? Language features will still leak GC memory.

Huh? Why would they? They don't create cycles, and would clean up reliably. --001a11330964395f4f04f1ab09e6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 02:05, Dicebot <span dir=3D"ltr">&lt;<a href=3D"mailto:publi= c dicebot.lv" target=3D"_blank">public dicebot.lv</a>&gt;</span> wrote:<br>= <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div class=3D"im">On Wednesday, 5 February 2014 at 16:03:56 UTC, Manu wrote= :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> The problem is completely solved; you turn the backing GC off. Devs are<br> responsible for correct weak pointer attribution.<br> </blockquote> <br></div> What does it give you over current situation with GC switched off and RefCo= unted used everywhere? Language features will still leak GC memory.<br> </blockquote></div><br></div><div class=3D"gmail_extra">Huh? Why would they= ? They don&#39;t create cycles, and would clean up reliably.</div></div> --001a11330964395f4f04f1ab09e6--
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 16:13:33 UTC, Manu wrote:
 Huh? Why would they? They don't create cycles, and would clean 
 up reliably.

Because they still return T* and not RC!T ? Andrei's post speaks purely about extra library type and does not mention about possibility to make it default allocation type for language. Or it is just silently assumed?
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 16:04:06 UTC, Dicebot wrote:
 It is up to programmer to decide. Right now he does not have a 
 choice and sometimes you can't afford to have GC in your 
 program at all (as in can't have it linked to the binary), not 
 just can't call collection cycles. Having sane fallback is very 
 desired.

Yes, if D is going to be a system level programming language then there is no other option than.
 Proposed solution does not seem to save you from uncontrollably 
 long collection cycles anyway as it still uses same memory pool 
 so I don't see how it can help even games, not even speaking 
 about more demanding applications.

Well, for games and game servers I think a 100ms delay once or twice per hour is inconsequential in terms of impact. If you can reduce the GC load by various means it might work out for most applications. 1. Reduce the set considered for GC by having the GC not scanning paths that are known to be covered by RC. 2. Improving speed of GC by avoiding interior pointers etc. 3. Reducing the number of calls to GC by having RC take care of the majority of the memory releases. 4. Have local GC by collecting roots of nodes that are known to create cycles. I don't think ARC is an option for OS-level development and critical applications anyway. :-)
 Different scenarios have different needs.

Haven't you just basically confirmed my opinion? :)

In a way. :-) But what if the question is this: How can you in a pragmatic way come up with a solution that cover most soft real time applications? A compiler switch that default to RC (i.e. turn standard GC features into standard RC features) could in theory get you pretty close, but I think clever RC/GC memory management requires whole program analysis…
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 16:01:09 UTC, Manu wrote:
 On 6 February 2014 01:32, <"Ola Fosheim Grøstad\"
 <ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:
 The applications I describe where it is a necessity will often 
 be 32bit
 systems.

You mean for embedded? Mobile CPUs are going 64 bit…
 Cache locality is a problem that can easily be refined. It 
 would just need
 lots of experimental data.

Well, in that case the math is not so difficult. You could have 1 index every 4MiB, and if your smallest allocation unit is 256 bytes then you get a counter index of 16384 uint32 (64Kib) The access would be easy and something like (probably not 100% correct): counter_addr = (ptr&~0xffff) + ( (ptr>>12)&0xfffc )
 Yes:

 struct {
     void* object_ptr;
     /* offset */
     uint weakcounter;
     uint strongcounter;
 }

 The advantage is that it works well with RC ignorant
 collectors/allocators. "if in doubt just set the pointer to 
 null and forget
 about freeing".

I see. Well, I don't like it :) ... but it's an option.

The aesthetics isn't great, it is not a minimalist approach, but consider the versatility: You could put in a function pointer to a deallocator (c-malloc, QT, GTK or some other library deallocator etc) and other kind of meta information that makes you able to treat reference counting in a uniform manner even for external resources. With the right semantics you can have pointers to cached objects that suddenly disappear etc (by using weak counter in a clever way).
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11c2c4840ddb4d04f1ac400d
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 02:16, Dicebot <public dicebot.lv> wrote:

 On Wednesday, 5 February 2014 at 16:13:33 UTC, Manu wrote:

 Huh? Why would they? They don't create cycles, and would clean up
 reliably.

Because they still return T* and not RC!T ? Andrei's post speaks purely about extra library type and does not mention about possibility to make it default allocation type for language. Or it is just silently assumed?

Dunno, but I don't think any solution which uses RC!T will ever be acceptable. It basically defeats the whole purpose. --001a11c2c4840ddb4d04f1ac400d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 02:16, Dicebot <span dir=3D"ltr">&lt;<a href=3D"mailto:publi= c dicebot.lv" target=3D"_blank">public dicebot.lv</a>&gt;</span> wrote:<br>= <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> <div class=3D"im">On Wednesday, 5 February 2014 at 16:13:33 UTC, Manu wrote= :<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Huh? Why would they? They don&#39;t create cycles, and would clean up relia= bly.<br> </blockquote> <br></div> Because they still return T* and not RC!T ? Andrei&#39;s post speaks purely= about extra library type and does not mention about possibility to make it= default allocation type for language.<br> <br> Or it is just silently assumed?<br> </blockquote></div><br></div><div class=3D"gmail_extra">Dunno, but I don&#3= 9;t think any solution which uses RC!T will ever be acceptable. It basicall= y defeats the whole purpose.</div></div> --001a11c2c4840ddb4d04f1ac400d--
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 17:40:35 UTC, Manu wrote:
 Dunno, but I don't think any solution which uses RC!T will ever 
 be
 acceptable. It basically defeats the whole purpose.

I feel terribly confused now :( Andrei, can you describe your intention with some big picture context?
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 17:24:46 UTC, Ola Fosheim 
Grøstad wrote:
 The access would be easy and something like (probably not 100% 
 correct):

 counter_addr = (ptr&~0xffff) + ( (ptr>>12)&0xfffc )

It was of course wrong, that would make the smallest allocation unit 16KiB. Anyway, if tuned to the indexed loads of the CPU then it would not be all the slow. On the x86 you should be able to do something like (pseudo): uint64 reg1 = ptr & 0xfff....f0000 uint32 reg2 = ptr >> 8 uint64 reg3 = load_effective_address( reg1 + 4*reg2 ) increment( *reg3 ) So only 4-5 cheap instructions for single threaded counting. You could also use the most significant bit for bookkeeping of single-threaded vs multi-threaded ref counting: test (*reg3) if (positive) goto nolock: lockprefix nolock: increment( *reg3 )
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
How would look the syntax for creating a RC slice?
int[] arr = new int[100];
or
somewhat ugly as RCSlice!int arr = new int[100];
?
Or (as I fear) RCSlice!int arr = RCSlice!int(100); ?
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Wednesday, 5 February 2014 at 18:42:55 UTC, Andrei 
Alexandrescu wrote:
 On 2/5/14, 10:38 AM, Namespace wrote:
 How would look the syntax for creating a RC slice?
 int[] arr = new int[100];
 or
 somewhat ugly as RCSlice!int arr = new int[100];
 ?
 Or (as I fear) RCSlice!int arr = RCSlice!int(100); ?

Just call a function! The "new" syntax for arrays is syntactically bankrupt anyway (can't distinguish between static and dynamic arrays). A historical mistake. Andrei

Thanks. So I can still rely on my own code. :)
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 18:33:54 UTC, Andrei 
Alexandrescu wrote:
 Phobos needs to be able to return allocated memory without 
 creating litter. With RCSlice!T the ownership is passed back to 
 the user. The user can continue tracking it by using reference 
 counting built into RCSlice, or make the object "immortal" by 
 calling .toGC against it.

Would it be possible to use RCSlice for all reference counting? I.e. that a reference counter for a struct would be a RCSlice of fixed length 1? If so then maybe it would be nice, but with a different name.
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Wednesday, 5 February 2014 at 18:45:28 UTC, Ola Fosheim 
Grøstad wrote:
 On Wednesday, 5 February 2014 at 18:33:54 UTC, Andrei 
 Alexandrescu wrote:
 Phobos needs to be able to return allocated memory without 
 creating litter. With RCSlice!T the ownership is passed back 
 to the user. The user can continue tracking it by using 
 reference counting built into RCSlice, or make the object 
 "immortal" by calling .toGC against it.

Would it be possible to use RCSlice for all reference counting? I.e. that a reference counter for a struct would be a RCSlice of fixed length 1? If so then maybe it would be nice, but with a different name.

shared_ptr? ;) Welcome to C++.
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 18:51:05 UTC, Namespace wrote:
 On Wednesday, 5 February 2014 at 18:45:28 UTC, Ola Fosheim 
 Grøstad wrote:

 If so then maybe it would be nice, but with a different name.

shared_ptr? ;) Welcome to C++.

Yes! :-) Even better: Let the length be 0 for weak pointers after the resource is released. Not so bad, actually.
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 18:33:54 UTC, Andrei 
Alexandrescu wrote:
 Phobos needs to be able to return allocated memory without 
 creating litter. With RCSlice!T the ownership is passed back to 
 the user. The user can continue tracking it by using reference 
 counting built into RCSlice, or make the object "immortal" by 
 calling .toGC against it.

Ok, clear. It will work then as far as I can see. But I am surprised this is considered a #1 issue to address. I have never been worried by lack of control over generated garbage in Phobos - it was the other way around, worries about control of how it is allocated. So yes, this is likely to solve issue you have mentioned but does not seem to be of any interest to me.
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 18:25:31 UTC, Andrei 
Alexandrescu wrote:
 Allow people to clean up memory allocated by Phobos.

A better solution would be for Phobos to allocate less memory. Instead of returning strings, accept output ranges that receive it. Make slices work as output ranges well with control over growing. (We can still offer the simple string returning functions for convenience)
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
Would it not be possible to add an "int rc" to the internal Array 
struct? So that int[] arr = [1, 2, 3]; is ref counted per 
default? If you want it to be immortal, call toGC(arr) or with 
UFCS arr.toGC() which set the rc to uint.max / 2?
That would keep the nice syntax. :)
Feb 05 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Feb 2014 17:24:44 +0000
schrieb "Ola Fosheim Gr=C3=B8stad"
<ola.fosheim.grostad+dlang gmail.com>:

 You could put in a function pointer to a deallocator (c-malloc,=20
 QT, GTK or some other library deallocator etc) and other kind of=20
 meta information that makes you able to treat reference counting=20
 in a uniform manner even for external resources.

Now that sounds interesting. That's an area where AGC typically doesn't kick in, because it can only see the small amount of memory used on the D wrapper side. --=20 Marco
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 19:09:57 UTC, Marco Leise wrote:
 You could put in a function pointer to a deallocator 
 (c-malloc, QT, GTK or some other library deallocator etc) and 
 other kind of meta information that makes you able to treat 
 reference counting in a uniform manner even for external 
 resources.

Now that sounds interesting.

It sounds to me like... a separate type. The deallocator is the destructor!
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 19:04:10 UTC, Namespace wrote:
 Would it not be possible to add an "int rc" to the internal 
 Array struct? So that int[] arr = [1, 2, 3]; is ref counted per 
 default?

Please no (it'd also have to be int* rc btw). This makes slices fatter than they need to be. We should be looking at reducing costs, not shuffling them around or adding to them.
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 19:13:01 UTC, Adam D. Ruppe 
wrote:
 It sounds to me like... a separate type. The deallocator is the 
 destructor!

Yes, but "destructor" of the reference counter object is "decrease count by one", when it goes to zero. Not when it is destroyed. Sure, you can use classes for it, but I don't think you want to. It is better to have a uniform reference counter object that can be fully inlined with no function calls as the general case and then only call the external "deallocator" with itself as parameter when it is present.
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Wednesday, 5 February 2014 at 19:15:41 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 5 February 2014 at 19:04:10 UTC, Namespace wrote:
 Would it not be possible to add an "int rc" to the internal 
 Array struct? So that int[] arr = [1, 2, 3]; is ref counted 
 per default?

Please no (it'd also have to be int* rc btw). This makes slices fatter than they need to be. We should be looking at reducing costs, not shuffling them around or adding to them.

Only if you change the current implementation from this: ---- struct Array { void* ptr; size_t length; } ---- to this: ---- struct Array { void* ptr; int* rc; size_t length; } ---- But we could also implement it like this: ---- struct Array { static struct Payload { void* ptr; int* rc; } Payload* p; size_t length; } ---- with the same costs.
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 19:36:00 UTC, Namespace wrote:
 	static struct Payload {
 		void* ptr;
 		int* rc;
 	}

 	Payload* p;

Now there's double indirection to get to the data... and you also forgot the necessary postblits and dtors to maintain the reference count. (these could be inlined or elided in some cases, but not all) You can't put the rc at the beginning of the pointer either, since then a = a[1..$] won't work. So the added pointer is unavoidable, your way or my way, both have a cost.
Feb 05 2014
prev sibling next sibling parent "Jameson Ernst" <jameson example.com> writes:
I'm just a casual end-user of D, but have been following this and 
related discussions with great interest. Just yesterday I was 
trying to "sell" D to a friend, and he basically told me that 
he'd be interested once the memory management situation gets 
resolved. I've been thinking about this a lot lately, even though 
I'm probably way out of my depth given the experts that frequent 
this forum. Still, I had an idea about this and thought I'd throw 
it out there.

On Wednesday, 5 February 2014 at 04:18:49 UTC, Andrei 
Alexandrescu wrote:
 On 2/4/14, 5:12 PM, deadalnix wrote:
 That being understood, I'd rather connect things the other way 
 around.

 auto x = foo();
 auto x = foo().toRC();

I don't know how to implement that. Andrei

From the discussion currently going on about postblits, it seems like there's a new emerging concept of a "unique expression," the value of which is guaranteed not to be referenced elsewhere. Could this concept perhaps be leveraged to go backwards from GC to RC? If you perform a GC allocation in the context of a unique expression, would it then be safe to force it back into an RC context, knowing that there are no outstanding references to it? What's more, it would allow library writes to mostly perform allocations a single way, giving the choice to the caller how they'd like to manage the lifetime of the newly returned unique object. I could be completely misunderstanding the unique-expression concept though.
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 19:37:29 UTC, Andrei 
Alexandrescu wrote:
 This is only focusing on slices.

That is not going to work out well, because you should be able to decrease the ref count of a pointer to something arbitrary (void) without knowing the object type. And the ref count object should persist after destruction of the object in the case of weak pointers.
Feb 05 2014
prev sibling next sibling parent "Namespace" <rswhite4 googlemail.com> writes:
On Wednesday, 5 February 2014 at 19:42:26 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 5 February 2014 at 19:36:00 UTC, Namespace wrote:
 	static struct Payload {
 		void* ptr;
 		int* rc;
 	}

 	Payload* p;

Now there's double indirection to get to the data... and you also forgot the necessary postblits and dtors to maintain the reference count. (these could be inlined or elided in some cases, but not all) You can't put the rc at the beginning of the pointer either, since then a = a[1..$] won't work. So the added pointer is unavoidable, your way or my way, both have a cost.

Hm, you're right. Would have been nice if the nice syntax could be retained, instead of further unsightly library solution.
Feb 05 2014
prev sibling next sibling parent "Graham Fawcett" <fawcett uwindsor.ca> writes:
On Tuesday, 4 February 2014 at 23:51:35 UTC, Andrei Alexandrescu
wrote:
 Consider we add a library slice type called RCSlice!T. It would 
 have the same primitives as T[] but would use reference 
 counting through and through. When the last reference count is 
 gone, the buffer underlying the slice is freed. The underlying 
 allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and 
 aims at convenience? There would be a method .toGC that just 
 detaches the slice and disables the reference counter (e.g. by 
 setting it to uint.max/2 or whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

I think I'd rather have a higher-level solution than this. I would worry about having to reason about code that's littered with toGC() or toARC() calls: my peanut butter (application logic) is now mixed in with my chocolate (allocation logic), making both harder to understand. I thought there was some discussion about arenas and custom allocators a while back, and that seemed like a sensible way to address these issues. I had imagined (naively?) that I would have been able to write something like: auto someFunction(T)(T params) { ComplicatedResult result; { // set up allocator at start of scope auto arena = GrowableArena(); push_allocator(arena); // thread-local allocator scope(exit) pop_allocator(); // call as many expensive functions as you like... auto tmp = allocation_intensive_computation(params); // transitive-move result from arena, at the scope's tail result = tmp.toGC(); } // arena goes out of scope and is deallocated return result; } So, allocation and lifetime issues are handled at the boundaries of scopes that have custom allocators associated with them. Um... destroy? Graham
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 19:39:43 UTC, Andrei 
Alexandrescu wrote:
 You do figure that complicates usage considerably, right?

I don't see much evidence for that. Many, many newer modules in Phobos are currently allocation free yet still pretty easy to use. A major source of little allocations in my code is std.conv and std.string. But these aren't difficult to change to external allocation, in theory at least: string s = to!string(50); // GC allocates (I'd keep this for convenience and compatibility) char[16] buffer; char[] s = toBuffer(buffer[], 50); // same thing, using a buffer char[] s = toLowerBuffer(buffer[], "FOO"); assert(buffer.ptr is s); assert(s == "foo"); That's not hard to use (though remembering that s is a borrowed reference to a stack buffer might be - escape analysis is something we should really have). And it gives full control over both allocation and deallocation. It'd take some changes in phobos, but so does the RCSlice sooo yeah, and this actually decouples it from the GC. The tricky part might be making it work with buffers, growable buffers, sink functions, etc., but we've solved similar problems with input ranges.
 I was thinking RCSlice would be a better alternative.

I very rarely care about when little slices are freed. Large blocks of memory might be another story (I've used malloc+free for a big internal buffer in my png.d after getting memory leaks from false poitners with teh gc) but those can be handled on a case by case basis. std.base64 for example might make sense to return one of these animals. I don't have a problem with refcounting on principle but most the time, it just doesn't matter.
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 20:04:31 UTC, Andrei 
Alexandrescu wrote:
 I said it a couple of times, and it seems it bears repeating: 
 the charter of this is solely to create a slice type that takes 
 care of itself. What this is not is a general solution for 
 managing internal pointers or pointers to arbitrary objects.

Then I think this is starting in the wrong end of the problem space. Slices are tiny dots in this picture. It would be better to have a RC compiler switch and version{} statements in the libraries rather than having extensive special casing of RC/GC vs pure GC types in user code.
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 19:48:58 UTC, Namespace wrote:
 Hm, you're right. Would have been nice if the nice syntax could 
 be retained, instead of further unsightly library solution.

The thing is most slices don't need anything special - they are inspected, but not stored. Since they aren't stored, the allocation isn't this function's problem. Yesterday, I wrote a post with a function int average(int[] numbers) as an illustration here. numbers might be on the stack, the Gc heap, or a refcounted array, and none of that matters. Average just looks at it. As long as there isn't something like another thread that frees the memory in the middle of average's execution, it will be fine. (And if there is a magic thread freeing things willy nilly, now that's a real WTF!) This is why the borrowed idea (implemented by escape analysis in my mind, i think that would work and get us most the benefits without all of Rust's complexity) is nice: with a borrowed pointer, you explicitly know freeing it isn't your problem. You don't have to count or carry a refcount, you don't have to run the gc, you don't have to call free. You can take a lightweight slice and use it with confidence... as long as it doesn't escape the scope and thus accidentally stick around after the function returns.
Feb 05 2014
prev sibling next sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
05-Feb-2014 03:51, Andrei Alexandrescu пишет:
 Consider we add a library slice type called RCSlice!T. It would have the
 same primitives as T[] but would use reference counting through and
 through. When the last reference count is gone, the buffer underlying
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the slice
 and disables the reference counter (e.g. by setting it to uint.max/2 or
 whatever).

 Then people who want reference counting say

 auto x = fun();

How abut just adding a template argument that indicates which container type to use for internal allocation? Array!T a = fun!(Array)(); //Ref-counted T[] a = fun(); //default args - GC IMHO solves Phobos side of equation. -- Dmitry Olshansky
Feb 05 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/5/14, 12:35 PM, Dmitry Olshansky wrote:
 05-Feb-2014 03:51, Andrei Alexandrescu пишет:
 Consider we add a library slice type called RCSlice!T. It would have the
 same primitives as T[] but would use reference counting through and
 through. When the last reference count is gone, the buffer underlying
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the slice
 and disables the reference counter (e.g. by setting it to uint.max/2 or
 whatever).

 Then people who want reference counting say

 auto x = fun();

How abut just adding a template argument that indicates which container type to use for internal allocation? Array!T a = fun!(Array)(); //Ref-counted T[] a = fun(); //default args - GC IMHO solves Phobos side of equation.

Good idea. Andrei
Feb 05 2014
prev sibling next sibling parent reply Johannes Pfau <nospam example.com> writes:
Am Wed, 05 Feb 2014 20:18:33 +0000
schrieb "Adam D. Ruppe" <destructionator gmail.com>:

 On Wednesday, 5 February 2014 at 19:39:43 UTC, Andrei 
 Alexandrescu wrote:
 You do figure that complicates usage considerably, right?

I don't see much evidence for that. Many, many newer modules in Phobos are currently allocation free yet still pretty easy to use. A major source of little allocations in my code is std.conv and std.string. But these aren't difficult to change to external allocation, in theory at least: string s = to!string(50); // GC allocates (I'd keep this for convenience and compatibility) char[16] buffer; char[] s = toBuffer(buffer[], 50); // same thing, using a buffer char[] s = toLowerBuffer(buffer[], "FOO"); assert(buffer.ptr is s); assert(s == "foo");

I think using a template parameter to allow for all kinds of allocators (std.allocator) is better. But of course it should have zero-overhead for a static-buffer allocator. Or we just special case static buffers _and_ add allocators ;-) But as Andrei said that discussion is not part of this thread. The main reason for RCSlice is not returning from Phobos functions, it's passing slices to Phobos functions. You can always create a RCSlice!string with any kind of string and as Andrei wants them GC-backed anyway, you can just create the RCSlice!string after the function returned a GC allocated string. (As long as the function doesn't keep a reference) But if we pass a RCd slice to a phobos function as a normal string, we'd have to stop reference counting. While this doesn't matter for GC-backed slices it basically means that manually allocated slices would never be freed. However, I wonder if that's really a problem in phobos. I'd guess most functions accepting slice input don't store a reference. We should probably start documenting that. (Or finish 'scope' as you already said implicitly ;-).
Feb 05 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:06:56 UTC, Xavier Bigand wrote:
 ????
 So in these case I will forget D, and cry all the tears of my 
 body. It will just a shame for a system language.
 And it's certainly kind of applications actually miss for D, to 
 improve his visibility.

Yes, but nobody that are in the "decision making body" of D has shown any resemblance of understanding or priority for real time applications. Seriously, the language is close to ten years in the making.
 Just take a look around you all applications are interactive, 
 with more animations,...

Yeah, but at some point you just have to accept that people who don't have a need to write real time will avoid putting it on the road map. I would like to see a roadmap that says "real time" and "no gc" and "whole program optimization", "owned pointers", "shared pointers"++ I see no road map.
Feb 10 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 11 February 2014 at 02:29:49 UTC, Andrei Alexandrescu 
wrote:
 Same difference. A bunch of people can pull, I'm hardly the 
 bottleneck. It just means I failed to convince the community to 
 get those bounties done.

 Andrei

Still that list (people with pull rights) is relatively short so it does not make much sense to appeal to wider auditory. Also I don't think volunteer effort can't be organized. It is a matter of people identifying themselves as part of well-defined organization as opposed to independent crowd of collaborators. It is a common wisdom that if no one feels directly responsible for an issue, no one will ever pay attention to it. Do you need any specific proposals?
Feb 11 2014
prev sibling next sibling parent "Meta" <jared771 gmail.com> writes:
On Tuesday, 11 February 2014 at 17:55:36 UTC, Andrei Alexandrescu 
wrote:
 I think at this stage we need more people to start with. 
 Someone pointed out recently we have 77 lifetime contributors 
 to github, as opposed to e.g. Rust which has 292.

It's unfortunate that D's biggest competitor is Rust. Being pushed by Mozilla means they automatically have access to a larger pool of contributors that contribute to other Mozilla stuff, who probably wouldn't have contributed to Rust otherwise. However, I think DDMD will bring about a small bump in contributors. Personally, I haven't used C++ in awhile, and I've grown rusty. At this point, D is probably the language I'm most comfortable with, especially for hacking on something complex like a compiler. I imagine there are at least a few people who feel the same way and are waiting for DDMD before they take a crack at contributing.
Feb 11 2014
prev sibling next sibling parent "Joseph Cassman" <jc7919 outlook.com> writes:
On Tuesday, 11 February 2014 at 20:02:45 UTC, Meta wrote:
 On Tuesday, 11 February 2014 at 17:55:36 UTC, Andrei 
 Alexandrescu wrote:
 I think at this stage we need more people to start with. 
 Someone pointed out recently we have 77 lifetime contributors 
 to github, as opposed to e.g. Rust which has 292.

It's unfortunate that D's biggest competitor is Rust. Being pushed by Mozilla means they automatically have access to a larger pool of contributors that contribute to other Mozilla stuff, who probably wouldn't have contributed to Rust otherwise. However, I think DDMD will bring about a small bump in contributors. Personally, I haven't used C++ in awhile, and I've grown rusty. At this point, D is probably the language I'm most comfortable with, especially for hacking on something complex like a compiler. I imagine there are at least a few people who feel the same way and are waiting for DDMD before they take a crack at contributing.

I have to agree. It's been over ten years since I have done any C++ seriously and as a result feel a lot of friction when trying to get up to speed on working with the D compiler code-base. Truth be told, that is part of the reason I like D in the end, it's not C++. Being able to use D to hack on the compiler would make it much more feasible for me to contribute. Joseph
Feb 11 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 11 February 2014 at 17:55:36 UTC, Andrei Alexandrescu 
wrote:
 better C++. It kinda sticks. Because people really want that.

I don't think so, at all. Anyone working on D must drop the moniker "D is a better C++" like a bad habit, no two ways about that. Most of it does it sets C++ as the benchmark. People who already like C++ would be like "you wish" and people who hate C++ would be like "better crap is not what I need anyway".

(I don't really agree, because whenever I use C++ I feel like implementing my own language or recently: take another look at D.) But for people looking for a system level programming language, C++ is a baseline. So you will invariably be measured against it. If you think in terms of "lost opportunities", I believe D looses out by not providing a compiler switch that turns off the GC and issues errors whenever GC-dependent constructs are being used. Not elegant, but simple and makes it easy to start doing real time programming because you then have achieved near parity with C++ baseline requirements. If you have limited resources you need to switch to thinking about baselines, key requirements and where you loose your audience. D looses opportunities for growth by not having a stable release that is perceived as current. You could (in theory) make a base line release, by focusing on the compiler and the runtime, pushing features that are not essential for forward compatibility into the next release and only release the aspect of phobos you are happy with. A known current, stable and supported release makes it possible to plan for others wanting to use D for production and would open up for branches for their own projects, like real-time features. I believe that by delaying a stable release in order to make it feature complete, you loose out, because you move out of the planning window (let's say 6-12 months) for those assessing possible technologies.
 We want to make D a great language all around, with 
 system-level access and also convenience features.

If you go too hard in the Application Language direction the System Level will be less visible. And there is more of a future for a system level compiled language with limited tool support.
 It's there in <h2> at the top of our homepage: "Modern 
 convenience. Modeling power. Native efficiency."

Slogans doesn't tell me much, I think we talk past each other here.
 By the way, this whole "plop a vision page" thing doesn't seem 
 to be quite popular:

I think Rust's homepage and documentation is clear. By "vision" I mean that future which the tool will bring with it as it appears to the reader. That which brings excitement and engagement: "I want that future". Basically the long term goals. Those that you might not achieve in generation 1, 2, 3… but which you are approaching. What I found exciting about W.B.'s original D1 was those aspects where D also would bring along semantics that make it possible to optimize better than C/C++, such as whole program optimization. It does not have to be in this generation of compilers to be communicated as language goals. Immutability, pure, templates etc are useful, but not exciting. And not unique for D. Anyway, I wish you the best of luck. I look forward to your next stable release and hope to be able to adapt the stable runtime to real time uses and make it available as patches for those interested. A long running development branch is outside my time window (e.g. I might have moved on to other things by the time it can be expected to reach maturity). Cheers, Ola.
Feb 14 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 20:46:32 UTC, Johannes Pfau 
wrote:
 However, I wonder if that's really a problem in phobos. I'd 
 guess most functions accepting slice input don't store a 
 reference.
 We should probably start documenting that. (Or finish 'scope' 
 as you already said implicitly ;-).

Aye. If the reference never escapes, it doesn't need to be counted or freed (indeed, it really MUST never be freed, since whomever passed you that reference may still be using it and is responsible for freeing it (or passing the buck to the GC))
Feb 05 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 5 February 2014 at 20:18:33 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 5 February 2014 at 19:39:43 UTC, Andrei 
 Alexandrescu wrote:
 You do figure that complicates usage considerably, right?

I don't see much evidence for that. Many, many newer modules in Phobos are currently allocation free yet still pretty easy to use. A major source of little allocations in my code is std.conv and std.string. But these aren't difficult to change to external allocation, in theory at least: string s = to!string(50); // GC allocates (I'd keep this for convenience and compatibility) char[16] buffer; char[] s = toBuffer(buffer[], 50); // same thing, using a buffer char[] s = toLowerBuffer(buffer[], "FOO"); assert(buffer.ptr is s); assert(s == "foo"); That's not hard to use (though remembering that s is a borrowed reference to a stack buffer might be - escape analysis is something we should really have). And it gives full control over both allocation and deallocation. It'd take some changes in phobos, but so does the RCSlice sooo yeah, and this actually decouples it from the GC.

Yeah, because RCSlice would require changes to Phobos too I'd much rather have this approach because it is just so much more flexible and hardly adds any inconvenience. Combined with the upcoming allocators it would be incredibly powerful. You could have an output range that uses an allocator which stores on the stack unless it grows too big (and the stack size could be completely customizable by the user who knows best). Or you could pass in an output range that reference counts its memory. Or an output range that must remain unique and frees its contents when it goes out of scope. I think three things would work together really well for addressing users that want to avoid the GC while making use of Phobos. 1) Increasing the support for output ranges, 2) Andrei's slick allocator design, and 3) nogc. With those three I really think managing memory and avoiding the GC will be rather pleasant. nogc would enable people trying to avoid all the tough to spot implicit GC allocations to identify them easily. Once uncovered, they just switch to the output range version of a function in Phobos and they then use std.allocator with the output range they feed in to create an ideal allocation strategy for their use case (whether it stack, GC, scope freed heap, reference counted, a memory pool, or some hybrid of those).
 The tricky part might be making it work with buffers, growable 
 buffers, sink functions, etc., but we've solved similar 
 problems with input ranges.


 I was thinking RCSlice would be a better alternative.

I very rarely care about when little slices are freed. Large blocks of memory might be another story (I've used malloc+free for a big internal buffer in my png.d after getting memory leaks from false poitners with teh gc) but those can be handled on a case by case basis. std.base64 for example might make sense to return one of these animals. I don't have a problem with refcounting on principle but most the time, it just doesn't matter.

Feb 05 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Wednesday, 5 February 2014 at 21:00:32 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 5 February 2014 at 20:46:32 UTC, Johannes Pfau 
 wrote:
 However, I wonder if that's really a problem in phobos. I'd 
 guess most functions accepting slice input don't store a 
 reference.
 We should probably start documenting that. (Or finish 'scope' 
 as you already said implicitly ;-).

Aye. If the reference never escapes, it doesn't need to be counted or freed (indeed, it really MUST never be freed, since whomever passed you that reference may still be using it and is responsible for freeing it (or passing the buck to the GC))

I wonder if passing it in as "scope" could be utilized in some way to forgo reference count increments/decrements.
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Wednesday, 5 February 2014 at 21:00:32 UTC, Adam D. Ruppe 
wrote:
 On Wednesday, 5 February 2014 at 20:46:32 UTC, Johannes Pfau 
 wrote:
 However, I wonder if that's really a problem in phobos. I'd 
 guess most functions accepting slice input don't store a 
 reference.
 We should probably start documenting that. (Or finish 'scope' 
 as you already said implicitly ;-).

Aye. If the reference never escapes, it doesn't need to be counted or freed (indeed, it really MUST never be freed, since whomever passed you that reference may still be using it and is responsible for freeing it (or passing the buck to the GC))

This. I think simply implementing scope will be much more important and effective step in optimizing D memory management model than current roadmap Andrei has posted recently. Simply because it enables non-breaking enhancements to Phobos that provide both allocation-free and safe functionality at the same time and is completely orthogonal to underlying allocation model.
Feb 05 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
tl; dr: it does not matter if this topic proposal works or not, 
it is simply an effort put into the wrong place
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 5 February 2014 at 22:18:33 UTC, Andrei 
Alexandrescu wrote:
 I'm confused. Slices are the bulkiest allocation there is: 
 chunks of contiguous data.

IMHO there should be one highly optimized low level ref-count structure that works everywhere, has compiler support and has an attractive syntax. That seems to be the most obvious solution. If not you might as well write your own array implementation, with better performance for your domain, or use an existing C++ framework that is geared towards your application domain. If we are talking games and simulation then you probably have your own allocator for search data structures (engine level) which are kind of fixed, predictable, tuneable, highly optimized, and uniquely owned. You allocate more, but probably seldom release. What is allocated/deallocated is "scripty" evolutionary stuff (content level). Dynamic game-world stuff that come in all kind of shapes and sizes: enemies, weapons, cars, events etc. Objects and their shifty relationships. 100.000+ objects for a game server. Stuff that is heterogeneous and is constantly modified in order to improve game play.
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Wednesday, 5 February 2014 at 22:32:52 UTC, Andrei 
Alexandrescu wrote:
 On the face of it it seems odd that reference counted chunks of 
 typed memory are deemed useless

I don't think anybody has actually said that. They have their places, it is just useless to talk about throwing them in everywhere.
 I should also add that imparting useful semantics to scope is 
 much more difficult than it might seem.

I'm not so sure about that*, but the fact is scope would be enormously useful if it was implemented. * Let's say it meant "assigning to any higher scope is prohibited". That should be trivially easy to check and ensures that variable itself doesn't escape. The tricky part would be preventing: int[] global; void foo(scope int[] a) { int[] b = a; global = b; } And that's easy to fix too: make ALL variables scope, unless specifically marked otherwise at the type declaration site (or if they are value types OR references to immutable data, which are very similar to value types in use). The type declaration can be marked as a reference encapsulation and those are allowed to be passed up (if the type otherwise allows; e.g. postblit is not disabled). That would break a fair chunk of existing code**, but it'd make memory management explicit, correct, and user extensible. ** I think moving to not null by default at the same time would be good, just rip off teh whole band aid.
Feb 05 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e01494a16f4167104f1b1ff06
Content-Type: text/plain; charset=UTF-8

On 6 February 2014 09:16, Adam D. Ruppe <destructionator gmail.com> wrote:

 On Wednesday, 5 February 2014 at 22:32:52 UTC, Andrei Alexandrescu wrote:

 On the face of it it seems odd that reference counted chunks of typed
 memory are deemed useless

I don't think anybody has actually said that. They have their places, it is just useless to talk about throwing them in everywhere. I should also add that imparting useful semantics to scope is much more
 difficult than it might seem.

I'm not so sure about that*, but the fact is scope would be enormously useful if it was implemented. * Let's say it meant "assigning to any higher scope is prohibited". That should be trivially easy to check and ensures that variable itself doesn't escape. The tricky part would be preventing: int[] global; void foo(scope int[] a) { int[] b = a; global = b; } And that's easy to fix too: make ALL variables scope, unless specifically marked otherwise at the type declaration site (or if they are value types OR references to immutable data, which are very similar to value types in use).

Surely a simpler solution is to mark b scope too? Does that break-down at some point? The type declaration can be marked as a reference encapsulation and those
 are allowed to be passed up (if the type otherwise allows; e.g. postblit is
 not disabled).

 That would break a fair chunk of existing code**, but it'd make memory
 management explicit, correct, and user extensible.

 ** I think moving to not null by default at the same time would be good,
 just rip off teh whole band aid.

--089e01494a16f4167104f1b1ff06 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 09:16, Adam D. Ruppe <span dir=3D"ltr">&lt;<a href=3D"mailto= :destructionator gmail.com" target=3D"_blank">destructionator gmail.com</a>= &gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex">On Wednesday, 5 February 2014 at 22:32:52 UT= C, Andrei Alexandrescu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> On the face of it it seems odd that reference counted chunks of typed memor= y are deemed useless<br> </blockquote> <br> I don&#39;t think anybody has actually said that. They have their places, i= t is just useless to talk about throwing them in everywhere.<br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I should also add that imparting useful semantics to scope is much more dif= ficult than it might seem.<br> </blockquote> <br> I&#39;m not so sure about that*, but the fact is scope would be enormously = useful if it was implemented.<br> <br> * Let&#39;s say it meant &quot;assigning to any higher scope is prohibited&= quot;. That should be trivially easy to check and ensures that variable its= elf doesn&#39;t escape. The tricky part would be preventing:<br> <br> int[] global;<br> void foo(scope int[] a) {<br> =C2=A0 =C2=A0int[] b =3D a;<br> =C2=A0 =C2=A0global =3D b;<br> }<br> <br> <br> And that&#39;s easy to fix too: make ALL variables scope, unless specifical= ly marked otherwise at the type declaration site (or if they are value type= s OR references to immutable data, which are very similar to value types in= use).<br> </blockquote><div><br></div><div>Surely a simpler solution is to mark b sco= pe too? Does that break-down at some point?</div><div><br></div><blockquote= class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc soli= d;padding-left:1ex"> The type declaration can be marked as a reference encapsulation and those a= re allowed to be passed up (if the type otherwise allows; e.g. postblit is = not disabled).<br> <br> That would break a fair chunk of existing code**, but it&#39;d make memory = management explicit, correct, and user extensible.<br> <br> ** I think moving to not null by default at the same time would be good, ju= st rip off teh whole band aid.<br> </blockquote></div><br></div></div> --089e01494a16f4167104f1b1ff06--
Feb 05 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Thursday, 6 February 2014 at 00:42:20 UTC, Andrei Alexandrescu 
wrote:
 One other school of thought (to which I subscribe) is that one 
 should take advantage of reference counting where appropriate 
 within a GC milieu, regardless of more radical RC approaches 
 that may be available.

I agree with that stance, but I don't think there's a blanket rule there. I think RC freeing small slices will waste more time than it saves. Large allocations, on the other hand, might be worth it. So std.file.read for example returns a large block - that's a good candidate for refcounting since it might be accidentally subject to false pointers, or sit around too long creating needless memory pressure, etc. (My png.d used to use large GC allocations internally and it ended up being problematic. I switched to malloc/free for this specific task and took care of that problem. But the little garbage created by stuff like toLower has never been a problem to me. (Well, except in a tight loop, but I wouldn't want to refcount in a tight loop either, reusing a static buffer is better tere.)) Anywho, I'd just go through on a case-by-case basis and tackle the big fish. Of course, a user could just do scope(exit) GC.free(ret); too.
 Yah, that does break a bunch of code. Things like the type of 
 "this" in class objects also comes to mind.

I talked about this yesterday: this should be scope too, since an object doesn't know its own allocation method. If the class object is on the stack, escaping this is wrong.. and thanks to emplace, it might be on the stack without the object ever knowing. Thus it must be conservative.
 Binding ref is also a related topic. All of these are complex 
 matters, and I think a few simple sketches don't do them 
 justice.

I'd rather discuss these details than adding RCSlice and toGC everywhere for more cost than benefit. Note that working scope would also help with library RC, in efficiency, correctness, and ease of use.
Feb 05 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 00:42:20 UTC, Andrei Alexandrescu
wrote:
 One school of thought seems to be that D should be everything 
 it is today, just with reference counting throughout instead of 
 garbage collection. One build flag to rule them all would 
 choose one or the other.

 One other school of thought (to which I subscribe) is that one 
 should take advantage of reference counting where appropriate 
 within a GC milieu, regardless of more radical RC approaches 
 that may be available.

The third school of thought is that one should be able to have different types of allocation schemes without changing the object type, but somehow tie it to the allocator and if needed to the pointer type/storage class. If you allocate as fully owned, it stays owned. If you allocate as shared with immediate release (RC) it stays shared. If you allocate as shared with delayed collection (GC) it stays that way. The RC/GC metadata is a hidden feature and allocator/runtime/compiler dependent component. Possibly you won't have GC or RC, but one pure GC runtime, one integrated RC/GC runtime, one pure ARC runtime, one integrated ARC/GC runtime etc. That's probably most realistic since the allocation metadata might be in conflict. You should be able to switch to the runtime you care about if needed as a compile time switch: 1. Pure Owned/borrowed: hard core performance, OS level development 2. Manual RC (+GC): high throughput, low latency 3. ARC (+GC): ease of use, low throughput, low latency 4. GC: ease of use, high throughput, higher latency, long lived 5. Realtime GC 6. ?? I see no reason for having objects treated differently if they are "owned", just because they have a different type of ownership. If the function dealing with it does not own it, but borrows it, then it should not matter. The object should have the same layout, the ownership/allocation metadata should be encapsulated and hidden. It is only when you transfer ownership that you need to know if the object is under RC or not. You might not even want to use counters in a particular implementation, maybe it is better to use a linked list in some scenarios. "reference counting" is a misnomer, it should be called "ownership tracker". The current default is that all pointers are shared. What D needs is defined semantics for ownership. Then you can start switching one runtime for another one and have the compiler/runtime act as an efficient unit.
Feb 05 2014
prev sibling next sibling parent "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 6 February 2014 at 00:32:05 UTC, Manu wrote:
 On 6 February 2014 09:16, Adam D. Ruppe 
 <destructionator gmail.com> wrote:
 And that's easy to fix too: make ALL variables scope, unless 
 specifically
 marked otherwise at the type declaration site (or if they are 
 value types
 OR references to immutable data, which are very similar to 
 value types in
 use).

Surely a simpler solution is to mark b scope too? Does that break-down at some point?

scope in declarations is currently used as a storage class for classes on the stack (a deprecated feature) so it couldn't be used for class references until it's been removed from the language for awhile. It does seem like it'd help the compiler a lot if you disallowed assigning references to variables not also marked as scope though. It's probably hard to evaluate how big of a pain it'd be without using it in real world code. It'd technically be a breaking change but scope isn't implemented at all anyway so I think this current users of scope would probably welcome a change that would make it actually start working.
Feb 05 2014
prev sibling next sibling parent "Joseph Cassman" <jc7919 outlook.com> writes:
On Wednesday, 5 February 2014 at 21:03:25 UTC, Brad Anderson 
wrote:
 On Wednesday, 5 February 2014 at 20:18:33 UTC, Adam D. Ruppe 
 wrote:
 [...]
 A major source of little allocations in my code is std.conv 
 and std.string. But these aren't difficult to change to 
 external allocation, in theory at least:

 string s = to!string(50); // GC allocates (I'd keep this for 
 convenience and compatibility)

 char[16] buffer;
 char[] s = toBuffer(buffer[], 50); // same thing, using a 
 buffer

 char[] s = toLowerBuffer(buffer[], "FOO");
 assert(buffer.ptr is s);
 assert(s == "foo");


 That's not hard to use (though remembering that s is a 
 borrowed reference to a stack buffer might be - escape 
 analysis is something we should really have).

 And it gives full control over both allocation and 
 deallocation. It'd take some changes in phobos, but so does 
 the RCSlice sooo yeah, and this actually decouples it from the 
 GC.

Yeah, because RCSlice would require changes to Phobos too I'd much rather have this approach because it is just so much more flexible and hardly adds any inconvenience. Combined with the upcoming allocators it would be incredibly powerful. You could have an output range that uses an allocator which stores on the stack unless it grows too big (and the stack size could be completely customizable by the user who knows best). Or you could pass in an output range that reference counts its memory. Or an output range that must remain unique and frees its contents when it goes out of scope. I think three things would work together really well for addressing users that want to avoid the GC while making use of Phobos. 1) Increasing the support for output ranges, 2) Andrei's slick allocator design, and 3) nogc. With those three I really think managing memory and avoiding the GC will be rather pleasant. nogc would enable people trying to avoid all the tough to spot implicit GC allocations to identify them easily. Once uncovered, they just switch to the output range version of a function in Phobos and they then use std.allocator with the output range they feed in to create an ideal allocation strategy for their use case (whether it stack, GC, scope freed heap, reference counted, a memory pool, or some hybrid of those). [...]

My thinking as well. That combination of functionality looks very advantageous to me. It is more flexible than just providing two choices to the programmer: GC and RC. To me both GC and RC are useful, depending on the type of program being written. However, why limit to just the two? There are other styles of memory allocation/management I might need to make use of, perhaps even in the same program. I really like the new allocator module. I had been thinking that a goal for its use was to allow replacing the compiler-supported allocation style with a custom one, either at the module level or on a function-by-function basis, as shown in the code above. In my opinion, this would give the necessary flexibility over memory allocation by giving final control to the programmer (i.e. control over external and internal allocation style). Doing so seems good to me as the programmer knows a priori the type of allocation pattern to support based on the type of program being produced (e.g. real-time, long-running process, batch system). Of course minimizing memory allocation in Phobos is an excellent goal and that work will proceed orthogonal to this effort. However, in the end, some memory will have to be allocated. Letting the programmer choose how that memory is to be allocated by giving full access to std.allocator seems the way to go. Joseph
Feb 05 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 6 February 2014 at 02:31:19 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 6 February 2014 at 00:42:20 UTC, Andrei 
 Alexandrescu
 wrote:
 One school of thought seems to be that D should be everything 
 it is today, just with reference counting throughout instead 
 of garbage collection. One build flag to rule them all would 
 choose one or the other.

 One other school of thought (to which I subscribe) is that one 
 should take advantage of reference counting where appropriate 
 within a GC milieu, regardless of more radical RC approaches 
 that may be available.

The third school of thought is that one should be able to have different types of allocation schemes without changing the object type, but somehow tie it to the allocator and if needed to the pointer type/storage class. If you allocate as fully owned, it stays owned. If you allocate as shared with immediate release (RC) it stays shared. If you allocate as shared with delayed collection (GC) it stays that way. The RC/GC metadata is a hidden feature and allocator/runtime/compiler dependent component. Possibly you won't have GC or RC, but one pure GC runtime, one integrated RC/GC runtime, one pure ARC runtime, one integrated ARC/GC runtime etc. That's probably most realistic since the allocation metadata might be in conflict. You should be able to switch to the runtime you care about if needed as a compile time switch: 1. Pure Owned/borrowed: hard core performance, OS level development 2. Manual RC (+GC): high throughput, low latency 3. ARC (+GC): ease of use, low throughput, low latency 4. GC: ease of use, high throughput, higher latency, long lived 5. Realtime GC 6. ?? I see no reason for having objects treated differently if they are "owned", just because they have a different type of ownership. If the function dealing with it does not own it, but borrows it, then it should not matter. The object should have the same layout, the ownership/allocation metadata should be encapsulated and hidden. It is only when you transfer ownership that you need to know if the object is under RC or not. You might not even want to use counters in a particular implementation, maybe it is better to use a linked list in some scenarios. "reference counting" is a misnomer, it should be called "ownership tracker". The current default is that all pointers are shared. What D needs is defined semantics for ownership. Then you can start switching one runtime for another one and have the compiler/runtime act as an efficient unit.

That won't play ball with third party libraries distributed in binary form. This is one of the reasons why Apple's Objective-C GC failed. -- Paulo
Feb 06 2014
prev sibling next sibling parent reply luka8088 <luka8088 owave.net> writes:
On 5.2.2014. 0:51, Andrei Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It would have the
 same primitives as T[] but would use reference counting through and
 through. When the last reference count is gone, the buffer underlying
 the slice is freed. The underlying allocator will be the GC allocator.
 
 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the slice
 and disables the reference counter (e.g. by setting it to uint.max/2 or
 whatever).
 
 Then people who want reference counting say
 
 auto x = fun();
 
 and those who don't care say:
 
 auto x = fun().toGC();
 
 
 Destroy.
 
 Andrei

Here is a thought: Let's say we have class A and class B, and class A accepts references to B as children: class A { B child1; B child2; B child3; } I think that the ultimate goal is to allow the user to choose between kinds of memory management, especially between automatic and manual. The problem here is that class A needs to be aware whether memory management is manual or automatic. And it seems to me that a new type qualifier is a way to go: class A { garbageCollected(B) child1; referenceCounted(B) child2; manualMemory(B) child3; } Now suppose we want to have only one child but we want to support compatibility with other kinds of memory management: class A { manualMemory(B) child; this (B newChild) { child = newChild.toManualMemory(); } this (referenceCounted(B) newChild) { child = newChild.toManualMemory(); } this (manualMemory(B) newChild) { child = newChild; } ~this () { delete child; } } This way we could write code that supports multiple models, and let the user choose which one to use. The this that I would like to point out is that this suggestion would work with existing code as garbageCollected memory management model would be a default: auto b = new B(); auto a = new A(b); Another thing to note is that in this way a garbage collector would know that we now have two references to one object (instance of class B). One is variable b and another is child in object a. And because of the notation garbage collector is aware that if could free this object when variable b goes out of scope but it should not do it because there is a still a manually managed reference to that object. I am sure that there are many more possible loopholes but maybe it will give someone a better idea :)
Feb 06 2014
next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/6/14, 12:28 AM, luka8088 wrote:
 On 5.2.2014. 0:51, Andrei Alexandrescu wrote:
 Consider we add a library slice type called RCSlice!T. It would have the
 same primitives as T[] but would use reference counting through and
 through. When the last reference count is gone, the buffer underlying
 the slice is freed. The underlying allocator will be the GC allocator.

 Now, what if someone doesn't care about the whole RC thing and aims at
 convenience? There would be a method .toGC that just detaches the slice
 and disables the reference counter (e.g. by setting it to uint.max/2 or
 whatever).

 Then people who want reference counting say

 auto x = fun();

 and those who don't care say:

 auto x = fun().toGC();


 Destroy.

 Andrei

Here is a thought: Let's say we have class A and class B, and class A accepts references to B as children: class A { B child1; B child2; B child3; } I think that the ultimate goal is to allow the user to choose between kinds of memory management, especially between automatic and manual. The problem here is that class A needs to be aware whether memory management is manual or automatic. And it seems to me that a new type qualifier is a way to go: class A { garbageCollected(B) child1; referenceCounted(B) child2; manualMemory(B) child3; }

There common theme here is that the original post introduces two distinct types of slices, depending on how they are to be freed (by refcounting or tracing). Andrei
Feb 06 2014
prev sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 08:28:34 UTC, luka8088 wrote:
 is manual or automatic. And it seems to me that a new type 
 qualifier is
 a way to go:

 class A {
   garbageCollected(B) child1;
   referenceCounted(B) child2;
   manualMemory(B) child3;
 }

class A { shared delayedrelease nodestructor cycles B child1; shared immediaterelease nocycles B child2; owned nocycles B child3; } Based on the required qualities, static analysis and profiling the compiler choose the most efficient storage that meet the constraints and match it up to the available runtime.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 08:06:54 UTC, Paulo Pinto wrote:
 That won't play ball with third party libraries distributed in 
 binary form.

That is not obvious, you specify the runtime. Anyway, whole program analysis also does not play well with binary libraries without detailed semantic metadata. Does shared_ptr in C++11 work with binary libraries that use it, if the it is compiled with a compiler from another vendor?
Feb 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 6 February 2014 at 08:38:57 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 6 February 2014 at 08:06:54 UTC, Paulo Pinto wrote:
 That won't play ball with third party libraries distributed in 
 binary form.

That is not obvious, you specify the runtime. Anyway, whole program analysis also does not play well with binary libraries without detailed semantic metadata.

So what do you do when different libraries require different runtimes? To be more specific to my previous comment. Objective-C GC required special compilation flags and care needed to be taken in GC enabled code, like in C GCs. This did not played well when mixing code that used the GC enabled runtime, with code that did not. Thus the endless causes of core dumps in Objective-C code that made use of the GC. The Apple decision to create ARC and dump the GC wasn't because it is better as they later sold it, but because the compiler inserts for the developer the usual [... retain]/[... release] calls that they were already writing since the NeXT days. So no distinct runtimes were required as the generated code is no different than an Objective-C developer would have written by hand. This was the best way to achieve some form of automatic memory management, while preserving compatibility across libraries delivered in binary form.
 Does shared_ptr in C++11 work with binary libraries that use 
 it, if the it is compiled with a compiler from another vendor?

As far as I am aware, no. In any case there isn't a standard C++ ABI defined. Well, there are a few, but vendors don't use them.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 09:27:19 UTC, Paulo Pinto wrote:
 So what do you do when different libraries require different 
 runtimes?

I guess a given compiler could have a "cross compiler option" that generates libraries for all the available runtimes the compiler supports?
 To be more specific to my previous comment. Objective-C GC 
 required special compilation flags and care needed to be taken 
 in GC enabled code, like in C GCs.

I understand. I am not sure if having multiple flags that creates a combinatorial explosion would be a good idea. I think you should have a set of invidual runtimes targeting typical scenarios, supporting different sets of functionality. (embedded, kernel, multimedia, server, batch, hpc…) However, I think it would only work for the same compiler, because you really don't want to prevent innovation…
 So no distinct runtimes were required as the generated code is 
 no different than an Objective-C developer would have written 
 by hand.

You might be able to design the runtime/object code in such a way that you get link errors.
 In any case there isn't a standard C++ ABI defined. Well, there 
 are a few, but vendors don't use them.

Yeah, well I am not personally bothered by it. The only time I consider using binary-only libraries is for graphics and OS level stuff that is heavily used by others so that it is both well tested and workaround is available on the net. (OpenGL, Direct-X, Cocoa etc) Not having the source code to a library is rather risky in terms of having to work around bugs by trail and error, without even knowing what the bug actually is. Thankfully, most useful libraries are open source.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 10:29:20 UTC, dennis luehring 
wrote:
 so you're not work in the normal software development business 
 where
 non-source code third party dependicies are fully normal :)

Not in terms of libraries no, what libraries would that be? In terms of infrastructure, yes, but I try to avoid using features I cannot replace with alternatives and strongly prefer open source alternatives.
Feb 06 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a11c1e676d78e3204f1bb2eba
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 6 February 2014 20:21, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Thursday, 6 February 2014 at 09:27:19 UTC, Paulo Pinto wrote:

 So what do you do when different libraries require different runtimes?

I guess a given compiler could have a "cross compiler option" that generates libraries for all the available runtimes the compiler supports? To be more specific to my previous comment. Objective-C GC required
 special compilation flags and care needed to be taken in GC enabled code=


 like in C GCs.

I understand. I am not sure if having multiple flags that creates a combinatorial explosion would be a good idea. I think you should have a s=

 of invidual runtimes targeting typical scenarios, supporting different se=

 of functionality. (embedded, kernel, multimedia, server, batch, hpc=E2=80=

 However, I think it would only work for the same compiler, because you
 really don't want to prevent innovation=E2=80=A6


  So no distinct runtimes were required as the generated code is no
 different than an Objective-C developer would have written by hand.

You might be able to design the runtime/object code in such a way that yo=

 get link errors.


  In any case there isn't a standard C++ ABI defined. Well, there are a
 few, but vendors don't use them.

Yeah, well I am not personally bothered by it. The only time I consider using binary-only libraries is for graphics and OS level stuff that is heavily used by others so that it is both well tested and workaround is available on the net. (OpenGL, Direct-X, Cocoa etc) Not having the source code to a library is rather risky in terms of havin=

 to work around bugs by trail and error, without even knowing what the bug
 actually is.

 Thankfully, most useful libraries are open source.

Some that I regularly encounter: system libs, opengl, directx, fmod, physics (havok, phyzx, etc), animation (euphoria, natural motion), bink, lip-sync libraries, proprietary engine libraries, art-package integration libraries (3ds max, maya, photoshop), fbx, and many, many more. Yes these are C libs, but the idea that people don't regularly use proprietary libs is fantasy. --001a11c1e676d78e3204f1bb2eba Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 20:21, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola Fo= sheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot;= puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &lt= ;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Thursday, 6 February 2014= at 09:27:19 UTC, Paulo Pinto wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> So what do you do when different libraries require different runtimes?<br> </blockquote> <br></div> I guess a given compiler could have a &quot;cross compiler option&quot; tha= t generates libraries for all the available runtimes the compiler supports?= <div class=3D""><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> To be more specific to my previous comment. Objective-C GC required special= compilation flags and care needed to be taken in GC enabled code, like in = C GCs.<br> </blockquote> <br></div> I understand. I am not sure if having multiple flags that creates a combina= torial explosion would be a good idea. I think you should have a set of inv= idual runtimes targeting typical scenarios, supporting different sets of fu= nctionality. (embedded, kernel, multimedia, server, batch, hpc=E2=80=A6)<br=

<br> However, I think it would only work for the same compiler, because you real= ly don&#39;t want to prevent innovation=E2=80=A6<div class=3D""><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> So no distinct runtimes were required as the generated code is no different= than an Objective-C developer would have written by hand.<br> </blockquote> <br></div> You might be able to design the runtime/object code in such a way that you = get link errors.<div class=3D""><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> In any case there isn&#39;t a standard C++ ABI defined. Well, there are a f= ew, but vendors don&#39;t use them.<br> </blockquote> <br></div> Yeah, well I am not personally bothered by it. The only time I consider usi= ng binary-only libraries is for graphics and OS level stuff that is heavily= used by others so that it is both well tested and workaround is available = on the net. (OpenGL, Direct-X, Cocoa etc)<br> <br> Not having the source code to a library is rather risky in terms of having = to work around bugs by trail and error, without even knowing what the bug a= ctually is.<br> <br> Thankfully, most useful libraries are open source.<br> </blockquote></div><br></div><div class=3D"gmail_extra">Some that I regular= ly encounter: system libs, opengl, directx, fmod, physics (havok, phyzx, et= c), animation (euphoria, natural motion), bink, lip-sync libraries, proprie= tary engine libraries, art-package integration libraries (3ds max, maya, ph= otoshop), fbx, and many, many more.</div> <div class=3D"gmail_extra">Yes these are C libs, but the idea that people d= on&#39;t regularly use proprietary libs is fantasy.</div></div> --001a11c1e676d78e3204f1bb2eba--
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 11:08:51 UTC, dennis luehring 
wrote:
 all libraries that other deparments don't want you to see or 
 you don't
 need to see - so unnormal in your environment?

My environment is kind of not the point (but yes that would have been un-normal :-). If it is internal then you surely have selected the compile options as a project policy and that makes this a non-issue. It is external libraries that are an issue, and depending on closed source external libraries is risky if they cannot be replaced. Depending on your contract you might have to reimplement the entire library for scratch for free if you cannot fix a problem. And unfortunately, small players file for bankruptcy all the time, so open source is the best insurance you can get.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 11:29:35 UTC, Manu wrote:
 Yes these are C libs, but the idea that people don't regularly 
 use
 proprietary libs is fantasy.

I never claimed that people don't do it. I claimed that it is risky if you don't have a replacement for it. And I don't think one should lobotomise a development environment to support it. C++ gets traction without being very good at that. Plenty of languages that are source-distributed are getting traction too.
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 11:29:35 UTC, Manu wrote:
 Some that I regularly encounter: system libs, opengl, directx, 
 fmod,
 physics (havok, phyzx, etc), animation (euphoria, natural

And just to nitpick: 1. Games are hit or miss with a very short life cycle. This is not typical. Most software have a life cycle counted in years with contractual support requirements that can be harsh, not months with very little possibility of damage claims for the end user. 2. The life cycle of games is porting when your product succeeds. You are F*CK*D if you don't have source code access and want to port to an emerging platform, so I believe you can obtain source code for libraries like Havoc and FMOD for that reason alone. I really don't think closed source libraries should be the focus of D if it prevents having a good infrastructure.
Feb 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 6 February 2014 at 11:59:49 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 6 February 2014 at 11:29:35 UTC, Manu wrote:
 Some that I regularly encounter: system libs, opengl, directx, 
 fmod,
 physics (havok, phyzx, etc), animation (euphoria, natural

And just to nitpick: 1. Games are hit or miss with a very short life cycle. This is not typical. Most software have a life cycle counted in years with contractual support requirements that can be harsh, not months with very little possibility of damage claims for the end user. 2. The life cycle of games is porting when your product succeeds. You are F*CK*D if you don't have source code access and want to port to an emerging platform, so I believe you can obtain source code for libraries like Havoc and FMOD for that reason alone. I really don't think closed source libraries should be the focus of D if it prevents having a good infrastructure.

D will never be taken serious by its target audience if no proper support is available. In the enterprise world I work on, very few projects have 100% source code available. -- Paulo
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 12:12:31 UTC, Paulo Pinto wrote:
 D will never be taken serious by its target audience if no 
 proper support is available.

What is the target audience? Is it clearly defined?
 In the enterprise world I work on, very few projects have 100% 
 source code available.

In my view discouraging libraries as binary blobs is a net positive, if that means you loose a specific audience I still think it is a win. Because I don't think binary blobs are positive for the eco system. I am quite certain that the plethora of libraries that you find for Python, Ruby and Perl exist due to the encouragement of source distribution and ease of library modification. (e.g. you cannot use the library without source access). Binary blobs in C are less problematic (but still problematic) because of the language stability and compiler maturity. Binary blobs that are DMD2 means you cannot move to DMD3 or recompile it to fix a compiler induced bug among other things. Maybe some commercial players will make DMD2 blobs available and then pull out due to lack of profit, that's not unlikely, and it sucks more than not having the libraries in the first place.
Feb 06 2014
prev sibling next sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Thursday, 6 February 2014 at 12:43:19 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 6 February 2014 at 12:12:31 UTC, Paulo Pinto wrote:
 D will never be taken serious by its target audience if no 
 proper support is available.

What is the target audience? Is it clearly defined?
 In the enterprise world I work on, very few projects have 100% 
 source code available.

In my view discouraging libraries as binary blobs is a net positive, if that means you loose a specific audience I still think it is a win. Because I don't think binary blobs are positive for the eco system. I am quite certain that the plethora of libraries that you find for Python, Ruby and Perl exist due to the encouragement of source distribution and ease of library modification. (e.g. you cannot use the library without source access). Binary blobs in C are less problematic (but still problematic) because of the language stability and compiler maturity. Binary blobs that are DMD2 means you cannot move to DMD3 or recompile it to fix a compiler induced bug among other things. Maybe some commercial players will make DMD2 blobs available and then pull out due to lack of profit, that's not unlikely, and it sucks more than not having the libraries in the first place.

You would be amazed how many times I have written FFI code that decrypts source code on load. -- Paulo
Feb 06 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 6 February 2014 at 13:03:43 UTC, Paulo Pinto wrote:
 You would be amazed how many times I have written FFI code that 
 decrypts source code on load.

You can probably create a decent decompiler for D-code given the reliance on GC/stack frames etc, so I am not sure if that is a rational point (perhaps a political one). Let me put it this way then: If havoc is available as a poorly adapted blob it will discourage development of a native idiomatic open source D physics engine, because it pays off more to spend time working around blob-related issues and get stellar performance. With no physics engine you will have something primitive in D instead, but the moment it is good enough for creating simple apps people that are interested will improve on that rather than working around Havoc issues. So, slower start, but better for the eco system in the long term.
Feb 06 2014
prev sibling next sibling parent Marco Leise <Marco.Leise gmx.de> writes:
Am Wed, 05 Feb 2014 19:44:50 +0000
schrieb "Jameson Ernst" <jameson example.com>:

 I'm just a casual end-user of D, but have been following this and 
 related discussions with great interest. Just yesterday I was 
 trying to "sell" D to a friend, and he basically told me that 
 he'd be interested once the memory management situation gets 
 resolved. I've been thinking about this a lot lately, even though 
 I'm probably way out of my depth given the experts that frequent 
 this forum. Still, I had an idea about this and thought I'd throw 
 it out there.
 
 On Wednesday, 5 February 2014 at 04:18:49 UTC, Andrei 
 Alexandrescu wrote:
 On 2/4/14, 5:12 PM, deadalnix wrote:
 That being understood, I'd rather connect things the other way 
 around.

 auto x = foo();
 auto x = foo().toRC();

I don't know how to implement that. Andrei

From the discussion currently going on about postblits, it seems like there's a new emerging concept of a "unique expression," the value of which is guaranteed not to be referenced elsewhere. Could this concept perhaps be leveraged to go backwards from GC to RC? If you perform a GC allocation in the context of a unique expression, would it then be safe to force it back into an RC context, knowing that there are no outstanding references to it? What's more, it would allow library writes to mostly perform allocations a single way, giving the choice to the caller how they'd like to manage the lifetime of the newly returned unique object. I could be completely misunderstanding the unique-expression concept though.

The intent is to make it possible to avoid the GC, but if I understand you correctly you talk about always allocating through the GC first. -- Marco
Feb 06 2014
prev sibling next sibling parent "Jameson Ernst" <jameson example.com> writes:
On Thursday, 6 February 2014 at 23:13:11 UTC, Marco Leise wrote:
 The intent is to make it possible to avoid the GC, but if I
 understand you correctly you talk about always allocating
 through the GC first.

One of the ideas on the table is to use ARC backed by the GC to clear cycles, which would mean the GC needs to be aware of all allocations anyway so it can scan and clear them if a cycle is formed. It's not safe to start reference counting GC memory at an arbitrary time after allocation, since references to it could be anywhere, but if you have a unique expression, then you know there are no other references to the new data, so it's safe to begin reference counting it at that time before it propagates out into the program at large.
Feb 06 2014
prev sibling next sibling parent "ixid" <nuaccount gmail.com> writes:
On Wednesday, 5 February 2014 at 16:50:40 UTC, Ola Fosheim 
Grøstad wrote:
 On Wednesday, 5 February 2014 at 16:04:06 UTC, Dicebot wrote:
 It is up to programmer to decide. Right now he does not have a 
 choice and sometimes you can't afford to have GC in your 
 program at all (as in can't have it linked to the binary), not 
 just can't call collection cycles. Having sane fallback is 
 very desired.

Yes, if D is going to be a system level programming language then there is no other option than.
 Proposed solution does not seem to save you from 
 uncontrollably long collection cycles anyway as it still uses 
 same memory pool so I don't see how it can help even games, 
 not even speaking about more demanding applications.

Well, for games and game servers I think a 100ms delay once or twice per hour is inconsequential in terms of impact. If you can reduce the GC load by various means it might work out for most applications. 1. Reduce the set considered for GC by having the GC not scanning paths that are known to be covered by RC. 2. Improving speed of GC by avoiding interior pointers etc. 3. Reducing the number of calls to GC by having RC take care of the majority of the memory releases. 4. Have local GC by collecting roots of nodes that are known to create cycles. I don't think ARC is an option for OS-level development and critical applications anyway. :-)
 Different scenarios have different needs.

Haven't you just basically confirmed my opinion? :)

In a way. :-) But what if the question is this: How can you in a pragmatic way come up with a solution that cover most soft real time applications? A compiler switch that default to RC (i.e. turn standard GC features into standard RC features) could in theory get you pretty close, but I think clever RC/GC memory management requires whole program analysis…

For a competitive game 100ms delays during gameplay would be completely unacceptable.
Feb 08 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 8 February 2014 at 10:21:27 UTC, ixid wrote:
 For a competitive game 100ms delays during gameplay would be 
 completely unacceptable.

But inconsequential, in the economic sense: you cannot demand a refund for such a tiny deficiency.
Feb 08 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e0112cd327c7d5604f1f65c0a
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 6 February 2014 21:59, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Thursday, 6 February 2014 at 11:29:35 UTC, Manu wrote:

 Some that I regularly encounter: system libs, opengl, directx, fmod,
 physics (havok, phyzx, etc), animation (euphoria, natural

And just to nitpick: 1. Games are hit or miss with a very short life cycle. This is not typical. Most software have a life cycle counted in years with contractua=

 support requirements that can be harsh, not months with very little
 possibility of damage claims for the end user.

I don't think you've mage a game recently. Most big games are multi-year projects with teams numbering well in the hundreds, and then downloadable-content (ie, after-market content) is basically a given these days, and often supported by a different team than initially wrote the code. 2. The life cycle of games is porting when your product succeeds. You are
 F*CK*D if you don't have source code access and want to port to an emergi=

 platform, so I believe you can obtain source code for libraries like Havo=

 and FMOD for that reason alone.

 I really don't think closed source libraries should be the focus of D if
 it prevents having a good infrastructure.

I didn't say they should be a focus, I'm saying they must however be supported. --089e0112cd327c7d5604f1f65c0a Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 6= February 2014 21:59, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola Fo= sheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot;= puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &lt= ;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Thursday, 6 February 2014= at 11:29:35 UTC, Manu wrote:<br> </div><div class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0 0= 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Some that I regularly encounter: system libs, opengl, directx, fmod,<br> physics (havok, phyzx, etc), animation (euphoria, natural<br> </blockquote> <br></div> And just to nitpick:<br> <br> 1. Games are hit or miss with a very short life cycle. This is not typical.= Most software have a life cycle counted in years with contractual support = requirements that can be harsh, not months with very little possibility of = damage claims for the end user.<br> </blockquote><div><br></div><div>I don&#39;t think you&#39;ve mage a game r= ecently.</div><div>Most big games are multi-year projects with teams number= ing well in the hundreds, and then downloadable-content (ie, after-market c= ontent) is basically a given these days, and often supported by a different= team than initially wrote the code.</div> <div><br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D"ma= rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> 2. The life cycle of games is porting when your product succeeds. You are F= *CK*D if you don&#39;t have source code access and want to port to an emerg= ing platform, so I believe you can obtain source code for libraries like Ha= voc and FMOD for that reason alone.<br> <br> I really don&#39;t think closed source libraries should be the focus of D i= f it prevents having a good infrastructure.<br> </blockquote></div><br></div><div class=3D"gmail_extra">I didn&#39;t say th= ey should be a focus, I&#39;m saying they must however be supported.</div><= /div> --089e0112cd327c7d5604f1f65c0a--
Feb 09 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 9 February 2014 at 10:06:12 UTC, Manu wrote:
 I don't think you've mage a game recently.

Pointless comment.
 Most big games are multi-year projects with teams numbering 
 well in the

Most games are not big. Most games fail in the marketplace.
 I didn't say they should be a focus, I'm saying they must 
 however be supported.

Must is a strong word, but since D is focusing on separate compilation it probably is a focus. Why are most comments about the application domain for D centered on "prestigious" projects such as AAA games, high volume trading system and safety critical appliations? The most likely application domain is a lot less "exciting": tools and simple servers. Get down to earth, plz.
Feb 09 2014
prev sibling next sibling parent "Sean Kelly" <sean invisibleduck.org> writes:
On Saturday, 8 February 2014 at 16:41:42 UTC, Paulo Pinto wrote:
 In the stock exchange there are Java and .NET systems doing 
 transactions at faster speed than that.

 100ms is too much money to be lost if the transaction course 
 changes.

It might be worth referring back to Don's talk at least year's D conference. His company does targeted advertising, and if I remember correctly, their entire transaction response time limit including time on the wire is around 40ms. It's small amounts rather than potentially millions per transaction, but either way, for many network services these days, profit is directly tied to exceedingly rapid response time. Even for cloud services, let's say a user is willing to accept a 500ms transaction time, including transport. Now let's say the could service has to talk to 5 other services to process the request, each which has a normal transaction time of 50ms. If one occasionally spikes to 250ms, that's a problem for that one request, but it's an even bigger problem for all the requests piling up behind it, since in this case the entire process is blocking on a garbage collection. Under very heavy load, the process may not even be able to recover. You can see this in Netty for Java, as it has a fixed request queue size (defaults to like 5) and once this limit is reached it just starts rejecting requests. A long collection means request failures.
Feb 09 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Sunday, 9 February 2014 at 10:16:20 UTC, Ola Fosheim Grøstad 
wrote:
 On Sunday, 9 February 2014 at 10:06:12 UTC, Manu wrote:
 I don't think you've mage a game recently.

Pointless comment. [snip] Get down to earth, plz.

I love how you wrote that "Pointless comment" then added that "Get down to earth, plz" at the end. :P [ok, sorry for metaposting, but I couldn't resist]
Feb 09 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 9 February 2014 at 17:48:21 UTC, Francesco Cattoglio 
wrote:
 I love how you wrote that "Pointless comment" then added that 
 "Get down to earth, plz" at the end. :P

Yes, you did love that, didn't you? It is generally a good idea to set down the foot firmly when people go ad hominem. However, the last point was directed to the D community. The language needs to be more focused on being very good at some key areas, not cover everything.
Feb 09 2014
prev sibling next sibling parent "francesco cattoglio" <francesco.cattoglio gmail.com> writes:
 However, the last point was directed to the D community. The 
 language needs to be more focused on being very good at some 
 key areas, not cover everything.

there are game developers out there, willing to use D. I also see lots of movement from hobbysts. We can't ignore them completely. Undefinedly long pauses are really bad for them, and something needs to be done. Be it in user code, library solution or as core part of the language. I agree that AAA titles are not the main right now, but this doesn't mean indie projects shouldn't be doable. After all, 150 milliseconds pauses are really annoying for pretty much any first person game.
Feb 09 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Sunday, 9 February 2014 at 20:15:51 UTC, francesco cattoglio 
wrote:
 I totally agree on this, but the problem here is that
 there are game developers out there, willing to use D. I also 
 see lots of movement from hobbysts. We can't ignore them 
 completely. Undefinedly long pauses are really bad for them, 
 and something needs to be done. Be it in user code, library 
 solution or as core part of the language.

I totally agree with you. I totally agree that the best focus for D would be to aim for towards being a kick-ass low level system language with programmer directed optimization (compiler hints), good memory layout control, and low latency, for medium sized code bases, and without historical c/c++ cruft. Because that spot is open and it rings a bell... For some reason the D community does not pursue it… as a shared goal. And that means D will never get there. I think one needs profiled whole program analysis to get there. I think one needs to aim for upcoming hardware architectures. I think one need to back it up with programmer specified constraints and hints. I think one needs high level optimization in addition to low level. I think one needs a very light and transparent runtime with tuneable low cost allocation schemes. I also think one needs to reduce the focus on separate compilation with no knowledge about what goes on inside an object file and C++/C ABIs. The compiler should have as much information available as possible. Then you also need to slack on C#/Java features, reflection, programming in the large, garbage collection and enforced correctness. To stay focused. However the goal of D appears to be to blend all these thing in one big melting pot. That makes things move slowly in a manner where I have trouble seeing the vision. So the outlook today appears to be that D is primarily useful for the in-between-spot which is really tools and perhaps some server stuff.
 I agree that AAA titles are not the main right now, but this 
 doesn't mean indie projects shouldn't be doable. After all, 150 
 milliseconds pauses are really annoying for pretty much any 
 first person game.

Yeah, it is indeed both uneccessary and undesirable with that kind of latency, but if it happens once per hour it has no real consequences in most scenarios. For some reason the GC is the holy grail of D, so I am just kind of figuring out what the benefits/damage is. Personally I have given up all hope of D becoming useful for client side stuff: 1. because of all the resistance in the forum. 2. because C++ portability is hard to beat (android, ios, win, pnacl etc) 3. because C++ provides low latency 4. because the capabilities of HTML5 browsers are eating heavily into the client space and might change the landscape in a few years I think the perceived D-vision-of-2013 might be useful on the efficient-server-side and there the GC actually makes sense every once in a while to clean up leaks. And that is OK too, even thought that space is a bit more crowded than the low level space. Just let's avoid the high profile argument. Be exceptionally good at some modest stuff first then expand into other areas next iteration. I think the fallacy is the notion of General Purpose Programming Language, as if those actually exist. :-) They don't, because human beings are following trends and you need a focused culture to sustain the eco system. So yeah, I agree with you, but I don't see the clearly communicated vision and therefore don't believe in the growth of a eco system around D for client side programming. :-/
Feb 09 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a1136865891838f04f2057500
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 9 February 2014 20:16, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Sunday, 9 February 2014 at 10:06:12 UTC, Manu wrote:

 I don't think you've mage a game recently.

Pointless comment. Most big games are multi-year projects with teams numbering well in the

Most games are not big. Most games fail in the marketplace.

So D is an indy/mobile games language? Well, it certainly won't be that until Android and iPhone is well supported. AAA games however is technically possible right now, but needs more work before it will appeal to the industry without objection. I didn't say they should be a focus, I'm saying they must however be
 supported.

Must is a strong word, but since D is focusing on separate compilation it probably is a focus. Why are most comments about the application domain for D centered on "prestigious" projects such as AAA games, high volume trading system and safety critical appliations?

Perhaps it's because these are precisely the first few major businesses who have made a commercial commitment to D? If the language fails to satisfy ambitious early adopters, why should others follow? The most likely application domain is a lot less "exciting": tools and
 simple servers.

 Get down to earth, plz.

I don't write 'unexciting' tools and simple servers, so they are not my focal points. There are plenty of other people here that keep those usage targets in check. There are relatively few (although numbers are growing surprisingly fast) who keep the big-games, realtime, or embedded/resource-limited usage targets in check. I'm one of them, and I want a future for D in my industry. If it's declared that that's not a goal for D, then I will leave the community on that day. I am standing firmly on planet earth, and those jobs are what pay my bills. I know the requirements of my industry. --001a1136865891838f04f2057500 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 9= February 2014 20:16, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola Fo= sheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot;= puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &lt= ;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span>= wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Sunday, 9 February 2014 a= t 10:06:12 UTC, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I don&#39;t think you&#39;ve mage a game recently.<br> </blockquote> <br></div> Pointless comment.<div class=3D""><br> <br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Most big games are multi-year projects with teams numbering well in the<br> </blockquote> <br></div> Most games are not big.<br> Most games fail in the marketplace.</blockquote><div><br></div><div>So D is= an indy/mobile games language? Well, it certainly won&#39;t be that until = Android and iPhone is well supported.</div><div>AAA games however is techni= cally possible right now, but needs more work before it will appeal to the = industry without objection.</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex= ;border-left:1px #ccc solid;padding-left:1ex"><div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> I didn&#39;t say they should be a focus, I&#39;m saying they must however b= e supported.<br> </blockquote> <br></div> Must is a strong word, but since D is focusing on separate compilation it p= robably is a focus.<br> <br> Why are most comments about the application domain for D centered on &quot;= prestigious&quot; projects such as AAA games, high volume trading system an= d safety critical appliations?<br></blockquote><div><br></div><div>Perhaps = it&#39;s because these are precisely the first few major businesses who hav= e made a commercial commitment to D?</div> <div>If the language fails to satisfy ambitious early adopters, why should = others follow?</div><div><br></div><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> The most likely application domain is a lot less &quot;exciting&quot;: tool= s and simple servers.<br> <br> Get down to earth, plz.<br> </blockquote></div></div><div class=3D"gmail_extra"><br></div><div class=3D= "gmail_extra">I don&#39;t write &#39;unexciting&#39; tools and simple serve= rs, so they are not my focal points. There are plenty of other people here = that keep those usage targets in check.</div> <div class=3D"gmail_extra">There are relatively few (although numbers are g= rowing surprisingly fast) who keep the big-games, realtime, or embedded/res= ource-limited usage targets in check. I&#39;m one of them, and I want a fut= ure for D in my industry. If it&#39;s declared that that&#39;s not a goal f= or D, then I will leave the community on that day.</div> <div class=3D"gmail_extra">I am standing firmly on planet earth, and those = jobs are what pay my bills. I know the requirements of my industry.</div></= div> --001a1136865891838f04f2057500--
Feb 09 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a1134a45c7b6dbe04f2059679
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 06:15, francesco cattoglio <
francesco.cattoglio gmail.com> wrote:

 However, the last point was directed to the D community. The language
 needs to be more focused on being very good at some key areas, not cover
 everything.

there are game developers out there, willing to use D. I also see lots of movement from hobbysts. We can't ignore them completely. Undefinedly long pauses are really bad for them, and something needs to be done. Be it in user code, library solution or as core part of the language. I agree that AAA titles are not the main right now, but this doesn't mean indie projects shouldn't be doable. After all, 150 milliseconds pauses are really annoying for pretty much any first person game.

The only company I know of that has made a commercial commitment to D is a AAA games company... --001a1134a45c7b6dbe04f2059679 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 06:15, francesco cattoglio <span dir=3D"ltr">&lt;<a href=3D= "mailto:francesco.cattoglio gmail.com" target=3D"_blank">francesco.cattogli= o gmail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D""><blockquote class=3D"gmail_q= uote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1e= x"> However, the last point was directed to the D community. The language needs= to be more focused on being very good at some key areas, not cover everyth= ing.<br> </blockquote></div> I totally agree on this, but the problem here is that<br> there are game developers out there, willing to use D. I also see lots of m= ovement from hobbysts. We can&#39;t ignore them completely. Undefinedly lon= g pauses are really bad for them, and something needs to be done. Be it in = user code, library solution or as core part of the language.<br> <br> I agree that AAA titles are not the main right now, but this doesn&#39;t me= an indie projects shouldn&#39;t be doable. After all, 150 milliseconds paus= es are really annoying for pretty much any first person game.<br> </blockquote></div><br></div><div class=3D"gmail_extra">The only company I = know of that has made a commercial commitment to D is a AAA games company..= .</div></div> --001a1134a45c7b6dbe04f2059679--
Feb 09 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a1134a45c9fec8804f205bce8
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 14:15, Manu <turkeyman gmail.com> wrote:

 On 10 February 2014 06:15, francesco cattoglio <
 francesco.cattoglio gmail.com> wrote:

 However, the last point was directed to the D community. The language
 needs to be more focused on being very good at some key areas, not cover
 everything.

there are game developers out there, willing to use D. I also see lots of movement from hobbysts. We can't ignore them completely. Undefinedly long pauses are really bad for them, and something needs to be done. Be it in user code, library solution or as core part of the language. I agree that AAA titles are not the main right now, but this doesn't mean indie projects shouldn't be doable. After all, 150 milliseconds pauses are really annoying for pretty much any first person game.

The only company I know of that has made a commercial commitment to D is a AAA games company...

Sorry, I obviously mean, "the only *games* company..." 150ms pause isn't only annoying, it's the sort of bug that might have your title refused for release by the platform vendors. And people seem to forget promptly after every single time I repeat myself: * The GC frequency of execution is directly proportional to the amount of _free memory_. In console games; NONE. * The length of the associated pause is directly proportional to the amount of memory currently in use. In console games; all of it. This doesn't only describe games, it describes any embedded environment. --001a1134a45c9fec8804f205bce8 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 14:15, Manu <span dir=3D"ltr">&lt;<a href=3D"mailto:turkeym= an gmail.com" target=3D"_blank">turkeyman gmail.com</a>&gt;</span> wrote:<b= r><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:= 1px #ccc solid;padding-left:1ex"> <div dir=3D"ltr"><div><div class=3D"h5"><div class=3D"gmail_extra"><div cla= ss=3D"gmail_quote">On 10 February 2014 06:15, francesco cattoglio <span dir= =3D"ltr">&lt;<a href=3D"mailto:francesco.cattoglio gmail.com" target=3D"_bl= ank">francesco.cattoglio gmail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div><blockquote class=3D"gmail_quote" style= =3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> However, the last point was directed to the D community. The language needs= to be more focused on being very good at some key areas, not cover everyth= ing.<br> </blockquote></div> I totally agree on this, but the problem here is that<br> there are game developers out there, willing to use D. I also see lots of m= ovement from hobbysts. We can&#39;t ignore them completely. Undefinedly lon= g pauses are really bad for them, and something needs to be done. Be it in = user code, library solution or as core part of the language.<br> <br> I agree that AAA titles are not the main right now, but this doesn&#39;t me= an indie projects shouldn&#39;t be doable. After all, 150 milliseconds paus= es are really annoying for pretty much any first person game.<br> </blockquote></div><br></div></div></div><div class=3D"gmail_extra">The onl= y company I know of that has made a commercial commitment to D is a AAA gam= es company...</div></div> </blockquote></div><br></div><div class=3D"gmail_extra">Sorry, I obviously = mean, &quot;the only *games* company...&quot;</div><div class=3D"gmail_extr= a"><br></div><div class=3D"gmail_extra">150ms pause isn&#39;t only annoying= , it&#39;s the sort of bug that might have your title refused for release b= y the platform vendors.</div> <div class=3D"gmail_extra">And people seem to forget promptly after every s= ingle time I repeat myself:</div><div class=3D"gmail_extra">=C2=A0* The GC = frequency of execution is directly proportional to the amount of _free memo= ry_. In console games; NONE.</div> <div class=3D"gmail_extra">=C2=A0* The length of the associated pause is di= rectly proportional to the amount of memory currently in use. In console ga= mes; all of it.</div><div class=3D"gmail_extra"><br></div><div class=3D"gma= il_extra"> This doesn&#39;t only describe games, it describes any embedded environment= .</div></div> --001a1134a45c9fec8804f205bce8--
Feb 09 2014
prev sibling next sibling parent "francesco cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:
 Sorry, I obviously mean, "the only *games* company..."

avoid those pauses, since the amount of work toward optimization is huge anyway, am I right? Ofc you still need minimal backend from the compiler and runtime support. If you lack control on internals, there's no way for you to optimize anything.
 And people seem to forget promptly after every single time I 
 repeat myself:
  * The GC frequency of execution is directly proportional to 
 the amount of
 _free memory_. In console games; NONE.
  * The length of the associated pause is directly proportional 
 to the
 amount of memory currently in use. In console games; all of it.

down allocations from the standard library. I guess that would suffice, no need to move to ARC.
Feb 09 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e01184bae3560ef04f2098af2
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 17:58, francesco cattoglio <
francesco.cattoglio gmail.com> wrote:

 On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:

 Sorry, I obviously mean, "the only *games* company..."

those pauses, since the amount of work toward optimization is huge anyway, am I right? Ofc you still need minimal backend from the compiler and runtime support. If you lack control on internals, there's no way for you to optimize anything.

If we wanted to spend that time+manpower (read, money & overtime/sanity) on bullshit like that, we have no reason to adopt D; we already have C/C++, and we already have decades of experience mitigating that nightmare. The point is, we are REALLY sick of it. Why would we sign up to replace it with more of the same thing. And people seem to forget promptly after every single time I repeat myself:
  * The GC frequency of execution is directly proportional to the amount of
 _free memory_. In console games; NONE.
  * The length of the associated pause is directly proportional to the
 amount of memory currently in use. In console games; all of it.

allocations from the standard library. I guess that would suffice, no need to move to ARC.

For 'simple' games, might as well write then in Java or C#, the tooling is much better, and support is offered by major multinational corporations. Not to say that they shouldn't be supported in D too, but that's not a target of interest to me, and I don't think it's an area which makes a particularly compelling argument for adoption of D. I've said before, console games is an industry desperate for salvation, and D makes a very strong case here in lieu of any other realistic alternatives... as long as this memory stuff is addressed acceptably. If there were to be some killer potential markets identified for D, I think this is definitely one of them. --089e01184bae3560ef04f2098af2 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 17:58, francesco cattoglio <span dir=3D"ltr">&lt;<a href=3D= "mailto:francesco.cattoglio gmail.com" target=3D"_blank">francesco.cattogli= o gmail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"><div class=3D"">On Monday, 10 February 2014 = at 04:26:10 UTC, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> Sorry, I obviously mean, &quot;the only *games* company...&quot;<br> </blockquote></div> That was a given. However I think AAA titles have the manpower to avoid tho= se pauses, since the amount of work toward optimization is huge anyway, am = I right? Ofc you still need minimal backend from the compiler and runtime s= upport. If you lack control on internals, there&#39;s no way for you to opt= imize anything.</blockquote> <div><br></div><div>If we wanted to spend that time+manpower (read, money &= amp; overtime/sanity) on bullshit like that, we have no reason to adopt D; = we already have C/C++, and we already have decades of experience mitigating= that nightmare.</div> <div>The point is, we are REALLY sick of it. Why would we sign up to replac= e it with more of the same thing.</div><div><br></div><blockquote class=3D"= gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-= left:1ex"> <div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p= x #ccc solid;padding-left:1ex"> And people seem to forget promptly after every single time I repeat myself:= <br> =C2=A0* The GC frequency of execution is directly proportional to the amoun= t of<br> _free memory_. In console games; NONE.<br> =C2=A0* The length of the associated pause is directly proportional to the<= br> amount of memory currently in use. In console games; all of it.<br> </blockquote></div> For &quot;simple&quot; games, it would be nice to have a better GC and cut = down allocations from the standard library. I guess that would suffice, no = need to move to ARC.<br> </blockquote></div></div><div class=3D"gmail_extra"><br></div><div class=3D= "gmail_extra">For &#39;simple&#39; games, might as well write then in Java = or C#, the tooling is much better, and support is offered by major multinat= ional corporations.</div> <div class=3D"gmail_extra">Not to say that they shouldn&#39;t be supported = in D too, but that&#39;s not a target of interest to me, and I don&#39;t th= ink it&#39;s an area which makes a particularly compelling argument for ado= ption of D.</div> <div class=3D"gmail_extra">I&#39;ve said before, console games is an indust= ry desperate for salvation, and D makes a very strong case here in lieu of = any other realistic alternatives... as long as this memory stuff is address= ed acceptably.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">If there we= re to be some killer potential markets identified for D, I think this is de= finitely one of them.</div></div> --089e01184bae3560ef04f2098af2--
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:
 The only company I know of that has made a commercial 
 commitment to D is a
 AAA games company...


Unfortunately a AAA games company is not setting down the goal post for D. As long as the leads for the project have as their primary interests: non-real-time stuff and STL-like-libraries things won't develop in your (and mine and Fransescos) direction. It won't happen until the leads of the project COMMIT to a MEASURABLE goal and a major effort is made to meet that goal. That means putting other goals aside until that measurable goal has been met.
 Sorry, I obviously mean, "the only *games* company..."

Yeah, but that games company needs to commit to taking a lead role so that the goal post and vision changes in that direction.
 And people seem to forget promptly after every single time I 
 repeat myself:
  * The GC frequency of execution is directly proportional to 
 the amount of
 _free memory_. In console games; NONE.
  * The length of the associated pause is directly proportional 
 to the
 amount of memory currently in use. In console games; all of it.

 This doesn't only describe games, it describes any embedded 
 environment.

I've already stated that I don't believe in using D for anything multi-media. It is not part of the project vision to be good at that from what I am seeing, and I am not going to believe it is going to be good for that until the project leads commit to measurable goals. The leads believe in meritocracy, that means the project will flail around in any direction that is fun. That means there are no rails. There is no reason to pull or push a train that is not on rails. To get D to be a true better C++ you need a concerted effort.
Feb 10 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 08:58:29 UTC, Manu wrote:
 If we wanted to spend that time+manpower (read, money & 
 overtime/sanity) on
 bullshit like that, we have no reason to adopt D; we already 
 have C/C++,
 and we already have decades of experience mitigating that 
 nightmare.
 The point is, we are REALLY sick of it. Why would we sign up to 
 replace it
 with more of the same thing.

What I meant: you can't get a language that performs at its best without doing your homework. Ideal performance requires time and money, no matter the tools. This doesn't mean I expect you to do the same amount of work you have to do in C++. Perhaps I expressed myself in a bad way, so here is a second try at that: I expect only simpler games to run at ideal speed without efforts. "Simple" here doesn't mean "generic 2D platformer", it should be interpreted as "not cutting edge". If you really need to squeeze every bit of performance, you can't only rely on automation.
Feb 10 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--001a1133177cb180b004f20a12b0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 10 February 2014 18:59, <"Ola Fosheim Gr=C3=B8stad\"
<ola.fosheim.grostad+dlang gmail.com>" puremagic.com> wrote:

 On Monday, 10 February 2014 at 04:26:10 UTC, Manu wrote:

 The only company I know of that has made a commercial commitment to D is=


 AAA games company...



 As long as the leads for the project have as their primary interests:
 non-real-time stuff and STL-like-libraries things won't develop in your
 (and mine and Fransescos) direction.

I'm confused. A couple of posts ago, you seemed to be annoyed at me for consistently raising games as a target application space that was unrealistic, or not 'down to earth', or some fairly niche and irrelevant target workload. Video games is a bigger industry than the movie industry. Casual/phones have captured a large slice in recent years, but the rest of the pie is almost entirely games consoles, which I don't think is a diminishing industry so much as the casual/phone space is rather growing the pie in overall volume. The industry is expanding as a whole. Yes both subsets of the industry are important, but in the casual space there are already other realistic and established languages as options, and D is much further away from use on those industries; mature cross-compilers, OS support, cross-language support, etc are all requirements for casual/phone games in D. It won't happen until the leads of the project COMMIT to a MEASURABLE goal
 and a major effort is made to meet that goal. That means putting other
 goals aside until that measurable goal has been met.

I don't think anyone in the D community really has that power. If Walter were to dictate direction that was unpopular enough, the developer base would promptly dissolve. I agree in some sense, that I would like to see some set of specific goals agreed and targeted with each release cycle. Ideally, with a roadmap towards particular goalposts which may enable new usage spaces. It may be possible that Walter and Andrei might have that sort of rallying power, but if the goal is not of interest to the majority of contributors, it just won't happen regardless how many people post happy thoughts about the goal. Contributing to D is, in some way, a form of recreation for contributors. Sorry, I obviously mean, "the only *games* company..."

Yeah, but that games company needs to commit to taking a lead role so tha=

 the goal post and vision changes in that direction.

Are you saying I don't complain enough? :) (at least, last year before I left) I would never want to assert authority on the language direction on behalf of a single company, like you say, it's a niche target, although a very big niche which I think will really benefit from D. I just make sure that people never forget that the niche exists, what the requirements are, and that tends to result in those targets being factored into conversations and designs. And people seem to forget promptly after every single time I repeat myself= :
  * The GC frequency of execution is directly proportional to the amount =


 _free memory_. In console games; NONE.
  * The length of the associated pause is directly proportional to the
 amount of memory currently in use. In console games; all of it.

 This doesn't only describe games, it describes any embedded environment.

I've already stated that I don't believe in using D for anything multi-media.

That's a shame, I see that as one of it's greatest (yet unrealised) potentials. What are some other reasons anyone would reach for a native language these days? If it's not an operating system, or some enterprising web service... what else commands native hardware access and performance than embedded development in a *highly* aggressive and competitive industry? It is not part of the project vision to be good at that from what I am
 seeing, and I am not going to believe it is going to be good for that unt=

 the project leads commit to measurable goals.

 The leads believe in meritocracy, that means the project will flail aroun=

 in any direction that is fun. That means there are no rails. There is no
 reason to pull or push a train that is not on rails. To get D to be a tru=

 better C++ you need a concerted effort.

Yeah, I agree in theory... I think the short-term goals need to be set by the target requirements of the people actually using it, then they can produce stories about how it went well for them. --001a1133177cb180b004f20a12b0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 18:59, <span dir=3D"ltr">&lt;<a href=3D"mailto:&quot;Ola F= osheim Gr=C3=B8stad\&quot; &lt;ola.fosheim.grostad+dlang gmail.com&gt;&quot= ; puremagic.com" target=3D"_blank">&quot;Ola Fosheim Gr=C3=B8stad\&quot; &l= t;ola.fosheim.grostad+dlang gmail.com&gt;&quot; puremagic.com</a>&gt;</span=
 wrote:<br>

left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"">On Monday, 10 February 2014 at 04:26:10 UT= C, Manu wrote:<br> </div><div class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);bor= der-left-style:solid;padding-left:1ex"><blockquote class=3D"gmail_quote" st= yle=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb= (204,204,204);border-left-style:solid;padding-left:1ex"> The only company I know of that has made a commercial commitment to D is a<= br> AAA games company...<br> </blockquote></blockquote> <br></div> Unfortunately a AAA games company is not setting down the goal post for D. = As long as the leads for the project have as their primary interests: non-r= eal-time stuff and STL-like-libraries things won&#39;t develop in your (and= mine and Fransescos) direction.<br> </blockquote><div><br></div><div>I&#39;m confused. A couple of posts ago, y= ou seemed to be annoyed at me for consistently raising games as a target ap= plication space that was unrealistic, or not &#39;down to earth&#39;, or so= me fairly niche and irrelevant target workload.</div> <div>Video games is a bigger industry than the movie industry. Casual/phone= s have captured a large slice in recent years, but the rest of the pie is a= lmost entirely games consoles, which I don&#39;t think is a diminishing ind= ustry so much as the casual/phone space is rather growing the pie in overal= l volume. The industry is expanding as a whole.</div> <div><br></div><div>Yes both subsets of the industry are important, but in = the casual space there are already other realistic and established language= s as options, and D is much further away from use on those industries; matu= re cross-compilers, OS support, cross-language support, etc are all require= ments for casual/phone games in D.</div> <div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0p= x 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-lef= t-style:solid;padding-left:1ex"> It won&#39;t happen until the leads of the project COMMIT to a MEASURABLE g= oal and a major effort is made to meet that goal. That means putting other = goals aside until that measurable goal has been met.</blockquote><div> <br></div><div>I don&#39;t think anyone in the D community really has that = power. If Walter were to dictate direction that was unpopular enough, the d= eveloper base would promptly dissolve.</div><div><br></div><div>I agree in = some sense, that I would like to see some set of specific goals agreed and = targeted with each release cycle. Ideally, with a roadmap towards particula= r goalposts which may enable new usage spaces.</div> <div>It may be possible that Walter and Andrei might have that sort of rall= ying power, but if the goal is not of interest to the majority of contribut= ors, it just won&#39;t happen regardless how many people post happy thought= s about the goal. Contributing to D is, in some way, a form of recreation f= or contributors.<br> </div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px = 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);bord= er-left-style:solid;padding-left:1ex"><div class=3D""> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Sorry, I obviously mean, &quot;the only *games* company...&quot;<br> </blockquote> <br></div> Yeah, but that games company needs to commit to taking a lead role so that = the goal post and vision changes in that direction.<br></blockquote><div><b= r></div><div>Are you saying I don&#39;t complain enough? :) (at least, last= year before I left)</div> <div><div>I would never want to assert authority on the language direction = on behalf of a single company, like you say, it&#39;s a niche target, altho= ugh a very big niche which I think will really benefit from D.</div></div> <div>I just make sure that people never forget that the niche exists, what = the requirements are, and that tends to result in those targets being facto= red into conversations and designs.<br></div><div><br></div><blockquote cla= ss=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left-width:1px;= border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex= "> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D""> And people seem to forget promptly after every single time I repeat myself:= <br> =C2=A0* The GC frequency of execution is directly proportional to the amoun= t of<br> _free memory_. In console games; NONE.<br> =C2=A0* The length of the associated pause is directly proportional to the<= br> amount of memory currently in use. In console games; all of it.<br> <br></div><div class=3D""> This doesn&#39;t only describe games, it describes any embedded environment= .<br> </div></blockquote> <br> I&#39;ve already stated that I don&#39;t believe in using D for anything mu= lti-media.<br></blockquote><div><br></div><div>That&#39;s a shame, I see th= at as one of it&#39;s greatest (yet unrealised) potentials. What are some o= ther reasons anyone would reach for a native language these days?</div> <div>If it&#39;s not an operating system, or some enterprising web service.= .. what else commands native hardware access and performance than embedded = development in a *highly* aggressive and competitive industry?</div><div> <br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8= ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-sty= le:solid;padding-left:1ex"> It is not part of the project vision to be good at that from what I am seei= ng, and I am not going to believe it is going to be good for that until the= project leads commit to measurable goals.<br> <br> The leads believe in meritocracy, that means the project will flail around = in any direction that is fun. That means there are no rails. There is no re= ason to pull or push a train that is not on rails. To get D to be a true be= tter C++ you need a concerted effort.<br> </blockquote><div><br></div><div>Yeah, I agree in theory... I think the sho= rt-term goals need to be set by the target requirements of the people actua= lly using it, then they can produce stories about how it went well for them= .</div> </div></div></div> --001a1133177cb180b004f20a12b0--
Feb 10 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--047d7bd6c6205b6f3904f20a42d6
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 19:17, Francesco Cattoglio <
francesco.cattoglio gmail.com> wrote:

 On Monday, 10 February 2014 at 08:58:29 UTC, Manu wrote:

 If we wanted to spend that time+manpower (read, money & overtime/sanity)
 on
 bullshit like that, we have no reason to adopt D; we already have C/C++,
 and we already have decades of experience mitigating that nightmare.
 The point is, we are REALLY sick of it. Why would we sign up to replace it
 with more of the same thing.

What I meant: you can't get a language that performs at its best without doing your homework. Ideal performance requires time and money, no matter the tools. This doesn't mean I expect you to do the same amount of work you have to do in C++. Perhaps I expressed myself in a bad way, so here is a second try at that: I expect only simpler games to run at ideal speed without efforts. "Simple" here doesn't mean "generic 2D platformer", it should be interpreted as "not cutting edge". If you really need to squeeze every bit of performance, you can't only rely on automation.

Fair enough, and I'd say you're right. D will not change the careful application of region allocators or pools or any number of tightly controlled, context-specific allocation patterns. But D has a lot of language features that allocate; closures, strings, concatenation, AAs, 3rd party libraries that also allocate on your behalf, and not ALL your code should have to apply carefully controlled allocation patterns. The existing GC is plainly unacceptable, but some form of GC must be present, otherwise D doesn't work. GC backed ARC is a realistic option as far as I can tell, and it's the only one I've ever heard that ticks all the boxes, and would seem to satisfy everyone in an acceptable way (ie, no additional complexity for 'don't care' users). I'm not married to it by any means, but it seems to be the only horse in the race. I haven't heard any other options that tick all the boxes. --047d7bd6c6205b6f3904f20a42d6 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 19:17, Francesco Cattoglio <span dir=3D"ltr">&lt;<a href=3D= "mailto:francesco.cattoglio gmail.com" target=3D"_blank">francesco.cattogli= o gmail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"">On Monday, 10 February 2014 at 08:58:29 UT= C, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> If we wanted to spend that time+manpower (read, money &amp; overtime/sanity= ) on<br> bullshit like that, we have no reason to adopt D; we already have C/C++,<br=

The point is, we are REALLY sick of it. Why would we sign up to replace it<= br> with more of the same thing.<br> </blockquote> <br></div> What I meant: you can&#39;t get a language that performs at its best withou= t doing your homework. Ideal performance requires time and money, no matter= the tools. This doesn&#39;t mean I expect you to do the same amount of wor= k you have to do in C++.<br> Perhaps I expressed myself in a bad way, so here is a second try at that: I= expect only simpler games to run at ideal speed without efforts. &quot;Sim= ple&quot; here doesn&#39;t mean &quot;generic 2D platformer&quot;, it shoul= d be interpreted as &quot;not cutting edge&quot;. If you really need to squ= eeze every bit of performance, you can&#39;t only rely on automation.<br> </blockquote></div><br></div><div class=3D"gmail_extra">Fair enough, and I&= #39;d say you&#39;re right. D will not change the careful application of re= gion allocators or pools or any number of tightly controlled, context-speci= fic allocation patterns.</div> <div class=3D"gmail_extra">But D has a lot of language features that alloca= te; closures, strings, concatenation, AAs, 3rd party libraries that also al= locate on your behalf, and not ALL your code should have to apply carefully= controlled allocation patterns. The existing GC is plainly unacceptable, b= ut some form of GC must be present, otherwise D doesn&#39;t work.</div> <div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">GC backed A= RC is a realistic option as far as I can tell, and it&#39;s the only one I&= #39;ve ever heard that ticks all the boxes, and would seem to satisfy every= one in an acceptable way (ie, no additional complexity for &#39;don&#39;t c= are&#39; users).</div> <div class=3D"gmail_extra">I&#39;m not married to it by any means, but it s= eems to be the only horse in the race. I haven&#39;t heard any other option= s that tick all the boxes.</div></div> --047d7bd6c6205b6f3904f20a42d6--
Feb 10 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 08:59:28 UTC, Ola Fosheim Grøstad 
wrote:
 It won't happen until the leads of the project COMMIT to a 
 MEASURABLE goal and a major effort is made to meet that goal. 
 That means putting other goals aside until that measurable goal 
 has been met.

works. Don't get me wrong, I've only be here for 2 years and I'm not a guru of social interactions, but to me it's clear that the D "project" is a bit different from your usual "leaders decide, everyone else follows". The "heads" (read Andrei and Walter) surely have some "powers". Perhaps they have abused those in the past (introducing features without general consensus), they might be able to impose a veto on a feature, but they won't prevent you from contributing, if that contribution is approved by the rest of the user base.
 Yeah, but that games company needs to commit to taking a lead 
 role so that the goal post and vision changes in that direction.

"leader" is more or less someone that you listen to, someone you trust because you think what they are asking and proposing rational stuff. Personally, as an example, I listen to Manu and I listen to Daniel Murphy, because they appear to have a nice project that can give some great visibility to D. There is a problem: most of the times the user of a tool has no time to work on the tool itself. What should Manu do other than going to Dconf, presenting their hard work, and convincing his coworkers? The "project leaders" have really limited resources. Stating a vision for the D as a language is useless if you don't have the resources to achieve it. Sure, if you have your voice heard (like Andrei) it's easier to convince other people to share your vision, but this doesn't mean you can force people on working on something extremely specific. This nets to having zero "traditional" leader power. What could be done is doing a massive crowdfunding campaign, get a few full-time hired developers, and change that.
 The leads believe in meritocracy, that means the project will 
 flail around in any direction that is fun. That means there are 
 no rails. There is no reason to pull or push a train that is 
 not on rails. To get D to be a true better C++ you need a 
 concerted effort.

The one backing the project will shape it the most. True democracy is pure utopia. People have different interests and different ideas, which are often conflicting. In the end it's all about resources.
Feb 10 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 09:36:53 UTC, Manu wrote:
 Are you saying I don't complain enough? :) (at least, last year 
 before I left)

Just out of curiosity: what do you mean exactly?
Feb 10 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--089e01229a060a7c9804f20b24ec
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 20:12, Francesco Cattoglio <
francesco.cattoglio gmail.com> wrote:

 On Monday, 10 February 2014 at 09:36:53 UTC, Manu wrote:

 Are you saying I don't complain enough? :) (at least, last year before I
 left)

Just out of curiosity: what do you mean exactly?

I left Remedy a year back, so I don't speak on their behalf anymore. Is that what you mean? He said "that games company needs to commit to taking a lead role so that the goal post and vision changes in that direction", and I'm not sure what that could mean in terms of tangible action, other than my advocating the things that were of critical importance to our project at the time. --089e01229a060a7c9804f20b24ec Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 20:12, Francesco Cattoglio <span dir=3D"ltr">&lt;<a href=3D= "mailto:francesco.cattoglio gmail.com" target=3D"_blank">francesco.cattogli= o gmail.com</a>&gt;</span> wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"><div class=3D"">On Monday, 10 February 2014 at 09:36:53 UT= C, Manu wrote:<br> <blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-= left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;p= adding-left:1ex"> Are you saying I don&#39;t complain enough? :) (at least, last year before = I left)<br> </blockquote> <br></div> Just out of curiosity: what do you mean exactly?<br> </blockquote></div><br></div><div class=3D"gmail_extra">I left Remedy a yea= r back, so I don&#39;t speak on their behalf anymore. Is that what you mean= ?</div><div class=3D"gmail_extra"><br></div><div class=3D"gmail_extra">He s= aid &quot;<span style=3D"font-family:arial,sans-serif;font-size:13px">that = games company needs to commit to taking a lead role so that the goal post a= nd vision changes in that direction&quot;, and I&#39;m not sure what that c= ould mean in terms of tangible action, other than my advocating the things = that were of critical importance to our project at the time.</span></div> </div> --089e01229a060a7c9804f20b24ec--
Feb 10 2014
prev sibling next sibling parent "Francesco Cattoglio" <francesco.cattoglio gmail.com> writes:
On Monday, 10 February 2014 at 10:53:05 UTC, Manu wrote:
 On 10 February 2014 20:12, Francesco Cattoglio <
 francesco.cattoglio gmail.com> wrote:
 I left Remedy a year back, so I don't speak on their behalf 
 anymore. Is
 that what you mean?

left the forums" and I was like "wait, what?".
 He said "that games company needs to commit to taking a lead 
 role so that
 the goal post and vision changes in that direction", and I'm 
 not sure what
 that could mean in terms of tangible action, other than my 
 advocating the
 things that were of critical importance to our project at the 
 time.

Reports on previous experience is indeed invaluable for future progression.
Feb 10 2014
prev sibling next sibling parent "Szymon Gatner" <noemail gmail.com> writes:
On Monday, 10 February 2014 at 10:53:05 UTC, Manu wrote:

 I left Remedy a year back, so I don't speak on their behalf 
 anymore. Is
 that what you mean?

Sorry for being OT, but where do you work now?
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 09:36:53 UTC, Manu wrote:
 I'm confused. A couple of posts ago, you seemed to be annoyed 
 at me for
 consistently raising games as a target application space that 
 was
 unrealistic, or not 'down to earth', or some fairly niche and 
 irrelevant
 target workload.

Sorry about that. I have been following D since 2005, on and off, and kept waiting for the "better C++" to materialize so I can use it to do fun stuff with it (audio, 3D, raytracing etc). One hobby of mine is to read Intel/AMD CPU docs, raytracing papers and compiler stuff, and discussing those aspects and improving my understanding of those areas is fun. I am loosing hope in that direction for D, because I don't think D has anyone with a strong interest in project management that can drive it in that direction. The responses from the D leads shows signs, not of a lack of skills, but a lack of interest in project management "theory" (and unfortunately, that is an area where I know the theory quite well since I majored in that area). On the fun side I want what you want. I would love to see you be the third lead on D, to get a person that "falls to sleep thinking real time" into that position would make me believe in the project. On the "pay for bread" side I am looking at D from the perspective of having an alternative to Go on the server side. I guess that has made me "janus-faced" in this discussion. What would make me tilt in favour of Go instead of D, is that it has corporate backing and therefore give priority to production level stability. Even though I like the semantics of D better. Stability is important to me since I personally pay the price (literally) for technical flaws since I offer fixed priced solutions. Instead of A.A. and W.B. going defensive (and yes it is painful to see your child get a needle in the foot at the doctor to get that vaccine that will keep the child healthy in the long term) they should try to get someone into the team of leads that has an interest in software development process and software process improvement. Or at the very least, one person with real time focus. (Please note that I found it quite amusing that you claimed that I was ignorant of long running games, since I studied Anarchy Online from inception to end in a qualitative manner while trying to figure out the design properties of the design domain, from a system development perspective. You don't have to convince me, I do understand where you are coming from and enjoy reading about your perspective. ;^)
 Video games is a bigger industry than the movie industry. 
 Casual/phones
 have captured a large slice in recent years, but the rest of 
 the pie is
 almost entirely games consoles, which I don't think is a 
 diminishing
 industry so much as the casual/phone space is rather growing 
 the pie in
 overall volume. The industry is expanding as a whole.

Yes, unfortunately the revenue in the mobile app space is very low for the majority of developers which requires tools that make them very productive at the cost of technical quality. So lots of stuff is being done with cheap (and not really performant) tech to cut down on dev time. A more performant and productive language could certainly make a difference, but to get there you need to focus on that niche, otherwise it will take too many years to catch up with the alternatives (with their eco system). And the landscape keeps changing very quickly. Companies that offer 3rd party solutions fold all the time. So mobile devs are "jaded".
 I don't think anyone in the D community really has that power. 
 If Walter
 were to dictate direction that was unpopular enough, the 
 developer base
 would promptly dissolve.

Yes, some would leave, but others would join. Those who today look at D and say: - "This is kind of cool, but not quite there yet" - "when can I expect to see it land in the area where it makes me productive" - "is this cart worth pushing, can we actually make a significant improvement here or do I have to push this cart all by myself" I would imagine that there are more people sitting on the fence than not. What made Linux work out was that they were aiming for a well defined vision, Unix. Progress was easy to measure. What made Linux fail on the desktop that they did not have a well defined vision, so the community spread out on N alternatives and progress was hard to measure. This is a bit simplistic, but Open Source projects that does not have a strongly projected vision tends to wither and dissolve over time.
 the goal. Contributing to D is, in some way, a form of 
 recreation for contributors.

But you still need a clear vision and well defined goals, because for every "fun" bit there is 2 "unfun" bits. For every "excellent feature", you have to axe "2 nice to haves". (kind of)
 Are you saying I don't complain enough? :) (at least, last year 
 before I left)
 I would never want to assert authority on the language 
 direction on behalf
 of a single company, like you say, it's a niche target, 
 although a very big
 niche which I think will really benefit from D.

Actually, I think you have the passion to put forth a vision that could bring D to real time and thus make it a project that is making "fun" possible. With no "real time" person on the team I probably will take the "hobby focus" and enjoy discussing technological possibilites (such as the discussion we had about ref counting recently). If that makes A.A. upset. Great. He should be. I am implying that D needs leadership. He should take leadership. If he does not want to listen. Well, in that case I am not forcing him to read what I write. But pointing to github is pointing in the wrong direction. Github tracks missing bolts and nuts, not a skewed skeleton.
 I just make sure that people never forget that the niche 
 exists, what the
 requirements are, and that tends to result in those targets 
 being factored
 into conversations and designs.

I am perfectly cool with that. If AAA games is the vision. Good. My prime gripe is the lack of a clearly stated vision. I could go with any "system level" vision that is not covered by C++/C#.
 That's a shame, I see that as one of it's greatest (yet 
 unrealised) potentials. What are some other reasons anyone 
 would reach for a native language these days?

Scalable, low resource, servers. Servers that boot up real fast and handle many connections. I am currently musing at OSv. It is a kernel written in C++ that can run on top of KVM. Having something like Go or D on that platform could be interesting. Backing caches/databases/web services for low revenue mobile apps.
 If it's not an operating system, or some enterprising web 
 service... what
 else commands native hardware access and performance than 
 embedded
 development in a *highly* aggressive and competitive industry?

Again, I don't disagree. *smooch* ;)
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 10:04:53 UTC, Francesco Cattoglio 
wrote:
 I'm sorry, but I think you are misinterpreting how the 
 community works. Don't get me wrong, I've only be here for 2 
 years and I'm not a guru of social interactions, but to me it's 
 clear that the D "project" is a bit different from your usual 
 "leaders decide, everyone else follows".

I am not going into this, because then I would have to go down to a theoretical dicussion on group dynamics, formal/informal hiearchies and lots of different schools of thought and models. That would be a long road to walk. ;-) All groups need to define their boundaries. If they don't do it clearly, each person will define his own boundaries and you will enter a process of negotiation. Then you will have a clash of those boundaries causing various dysfunctional dynamics. Not defining your boundaries is not an option. It is going to happen. The question is: how much time do you want to spend on that process before entering a productive state?
Feb 10 2014
prev sibling next sibling parent Manu <turkeyman gmail.com> writes:
--047d7b5d90b5d2916f04f20c898f
Content-Type: text/plain; charset=UTF-8

On 10 February 2014 22:00, Szymon Gatner <noemail gmail.com> wrote:

 On Monday, 10 February 2014 at 10:53:05 UTC, Manu wrote:


 I left Remedy a year back, so I don't speak on their behalf anymore. Is
 that what you mean?


I'm very happily unemployed! :) ... but that'll probably have to change some time soon. Might have to move country again, not much happening in Australia anymore after the near total obliteration of the industry here that devastated most of my friends and colleagues. --047d7b5d90b5d2916f04f20c898f Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div class=3D"gmail_extra"><div class=3D"gmail_quote">On 1= 0 February 2014 22:00, Szymon Gatner <span dir=3D"ltr">&lt;<a href=3D"mailt= o:noemail gmail.com" target=3D"_blank">noemail gmail.com</a>&gt;</span> wro= te:<br><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;= border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:= solid;padding-left:1ex"> <div class=3D"">On Monday, 10 February 2014 at 10:53:05 UTC, Manu wrote:<br=

</div><div class=3D""><blockquote class=3D"gmail_quote" style=3D"margin:0px= 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);bor= der-left-style:solid;padding-left:1ex"> <br> I left Remedy a year back, so I don&#39;t speak on their behalf anymore. Is= <br> that what you mean?<br> <br> </blockquote> <br></div> Sorry for being OT, but where do you work now?<br> </blockquote></div><br></div><div class=3D"gmail_extra">I&#39;m very happil= y unemployed! :) ... but that&#39;ll probably have to change some time soon= .</div><div class=3D"gmail_extra">Might have to move country again, not muc= h happening in Australia anymore after the near total obliteration of the i= ndustry here that devastated most of my friends and colleagues.</div> </div> --047d7b5d90b5d2916f04f20c898f--
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 10:04:53 UTC, Francesco Cattoglio 
wrote:
 On Monday, 10 February 2014 at 08:59:28 UTC, Ola Fosheim 
 Grøstad wrote:

 The leads believe in meritocracy, that means the project will 
 flail around in any direction that is fun. That means there 
 are no rails. There is no reason to pull or push a train that 
 is not on rails. To get D to be a true better C++ you need a 
 concerted effort.

The one backing the project will shape it the most. True democracy is pure utopia. People have different interests and different ideas, which are often conflicting. In the end it's all about resources.

I forgot to comment on this. No, I don't think it is only a matter of resources. For instance, if I had the time I would most certainly consider writing a pack-rat parser for a modified subset a D that builds an AST for clang. That's actually doable for 1-3 people.
Feb 10 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 10 February 2014 at 14:24:06 UTC, Ola Fosheim Grøstad 
wrote:
 No, I don't think it is only a matter of resources. For 
 instance, if I had the time I would most certainly consider 
 writing a pack-rat parser for a modified subset a D that builds 
 an AST for clang.

"if I had the time". This exactly is the difference and reason why resources matter a lot.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 14:47:15 UTC, Dicebot wrote:
 "if I had the time". This exactly is the difference and reason 
 why resources matter a lot.

Yes, but there are enough people in these forums claiming that they desire a real time, production quality, better-than-c++ compiler to pull it off. But not for me alone. Lack of clear planning, communication of visions and establishing short term and long term measurable goals is not really a resource issue. It is a matter of taking those issue seriously. Basically a management issue. To me it would be reasonable to have: 1. short term goal: production level stability for what D is being used for today 2. long term goal: low latency, real time features/runtime Then work on 1 while planning milestones for point 2. I guess D1 was supposed to address 1, but nobody would start a project from scratch using D1 today.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 21:17:22 UTC, Xavier Bigand wrote:
 Maybe if performances are not really critical, developers can 
 already use Java or C# with a GC?

Or javascript actually, if you use the animation capabilities of the browser engine…
 IMO D have to target all applications have to be written in 
 C/C++, where performances or portability, scalability,... are 
 critical.
 Java and C# developers already have a lot of good tools that 
 C/C++ developers doesn't have, we need love too :-)
 D claims to be a system language so it's normal to expect to be 
 able to use it on critical ways easily, and maybe lesser for 
 simple applications can already done with actual proven 
 technologies.

I share your views actually… so I trying to take the view that D is not a C++ replacement, but a compiled C#. That makes it much easier to accept the current state. :-) I am no longer sure if I am able to view D as a system language. Too many features that are not really important on the low level. Too big runtime. And no strategy that points towards whole program optimization. I'd personally much prefer less features, more performance control and whole program optimization. I distinctly remember doing profiling based whole program optimization of c-programs on unix machines in the 1990s. Seriously, that's 18+ years ago.
Feb 10 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 10 February 2014 at 14:24:06 UTC, Ola Fosheim Grøstad 
wrote:
 No, I don't think it is only a matter of resources. For 
 instance, if I had the time I would most certainly consider 
 writing a pack-rat parser for a modified subset a D that builds 
 an AST for clang.

You do that :D I'll be waiting. Some people just need to run into the roadblock to notice it exists.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:06:06 UTC, deadalnix wrote:
 You do that :D

 I'll be waiting. Some people just need to run into the 
 roadblock to notice it exists.

Which roadblock?
Feb 10 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Monday, 10 February 2014 at 23:07:06 UTC, Ola Fosheim Grøstad 
wrote:
 On Monday, 10 February 2014 at 23:06:06 UTC, deadalnix wrote:
 You do that :D

 I'll be waiting. Some people just need to run into the 
 roadblock to notice it exists.

Which roadblock?

deadalanix has been working on SDC for quite a while - alternative implementation of D frontend using LLVM for code gen.
Feb 10 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 10 February 2014 at 23:07:06 UTC, Ola Fosheim Grøstad
wrote:
 On Monday, 10 February 2014 at 23:06:06 UTC, deadalnix wrote:
 You do that :D

 I'll be waiting. Some people just need to run into the 
 roadblock to notice it exists.

Which roadblock?

You seems to knows everything in and out. You tell me. If you aren't sure, please start parsing D to clang AST. After all, that sound like a great idea. I'm not sure I want to spend any time trying to convince you considering: 1 - people have been trying and it looks like a time consuming energy hungry task. 2 - You don't seems interested to actually contribute anything. So start you clang idea, if it sounds great to you. And come back enlightened.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:12:08 UTC, Dicebot wrote:
 deadalanix has been working on SDC for quite a while - 
 alternative implementation of D frontend using LLVM for code 
 gen.

Ah ok, I didn't suggest implementing D, but a subset that maps directly to C++. Then you can map directly to the AST.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:13:56 UTC, deadalnix wrote:
 You seems to knows everything in and out. You tell me. If you
 aren't sure, please start parsing D to clang AST. After all, 
 that sound like a great idea.

No, I don't know everything, but I said "modified subset of D", that would be a different language. Suitable for my needs (I don't need RTTI or exceptions etc). I know I can do it, because source-2-source compilation is not that difficult and could be the first step. And (slow) parsers are not so difficult to write with the tools we have today. After looking at the SVN repositories for clang I am becoming a bit more familiar with the source code which looked chaotic the first time I looked at it. The AST looks ok from skimming through the repository, I assume I can get it through a visualization tool and generate visual graphs of the hierarchies.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Monday, 10 February 2014 at 23:20:50 UTC, Andrei Alexandrescu 
wrote:
 Terrific. The challenge here is to adapt project management 
 theory to the realities of a volunteer project.

Yes, I understand that. I guess it often means that key people have to do the "unfun" stuff and let the more intermittent volunteers do the "fun" stuff… But I believe that "democratic design" requires very strong communication of vision if you want to create something new (otherwise you tend to end up with bastardized copies of what exist, since people will navigate towards the safe common ground).
 that. I'm all ears on advice on how to do better. I just have 
 difficulty seeing how more management would help.

Just a suggestion of some possibilities: Externally (front of web page): - More clear communication of the boundaries of the project. - Stating clearly what ground you do not cover. - Defining short term/long term goals on your front page. - Make it visible when they are met or changed. Internally (you probably do this already in some form): - Map out a list of dependencies between long term goals and use it for planning so that people who are more inclined to do proof-of-concept stuff can help out. - Some sort of social reward mechanism (borrow from MMOs?). I like the bounty stuff, but social rewards are much stronger. - Make commitments in key areas. Like stating how the project will change if you cannot meet deadlines. Like a priori stating what you will do with the GC if benchmark X is not met by date N. That could make those wanting GC push harder and those not wanting GC more complacent. Ok, maybe the wrong thing to do for the GC, but you get the idea, get the direction set. (Is it reasonable to limit features/production-quality to the shared common denominator of 3 backends? Can you somehow mitigate that?)
Feb 10 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Monday, 10 February 2014 at 23:29:25 UTC, Ola Fosheim Grøstad 
wrote:
 On Monday, 10 February 2014 at 23:13:56 UTC, deadalnix wrote:
 You seems to knows everything in and out. You tell me. If you
 aren't sure, please start parsing D to clang AST. After all, 
 that sound like a great idea.

No, I don't know everything, but I said "modified subset of D", that would be a different language. Suitable for my needs (I don't need RTTI or exceptions etc). I know I can do it, because source-2-source compilation is not that difficult and could be the first step. And (slow) parsers are not so difficult to write with the tools we have today. After looking at the SVN repositories for clang I am becoming a bit more familiar with the source code which looked chaotic the first time I looked at it. The AST looks ok from skimming through the repository, I assume I can get it through a visualization tool and generate visual graphs of the hierarchies.

Long story short, if you want to map thing on top of clang AST, you'll have to reduce significantly the subset of D you want to use, or build a full fledged front end, at which point you'd better use LLVM directly or GCC, or whatever backend suit your desires. For instance, static ifs, template constraints and CTFE require full understanding of language semantic by the frontend. Runtime reflection, is also out. That mean UDA could work but become useless. Type inference also require almost complete understanding of the language by the frontend. And even with these drastic restrictions, you'll still need a significant amount of semantic analysis in the frontend before feeding clang. You'll have to instantiate templates yourself (or bye bye alias parameters, string parameters and so on). Track variables used in closures (well, on that one you can arguably allocate every stack frame on heap and pas the whole stack frame to C++ lambda, that should work, but that will excruciatingly slow and memory consuming) to allocate them on heap. With CTFE, compile time reflection and static ifs/mixins, D wraps around itself in a way that makes it all or nothing in implementation.
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 11 February 2014 at 00:21:22 UTC, deadalnix wrote:
 With CTFE, compile time reflection and static ifs/mixins, D 
 wraps around itself in a way that makes it all or nothing in 
 implementation.

Ok, but I think I didn't communicate clearly what I am looking for. I am just looking for a "nicer" C++. I don't really need the full D semantics. I am content with the C++ subset that I can desugar into. To me D is just that, a "nicer" C++. I don't need the high level stuff. Those are "nice to have", but not "needed".
Feb 10 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 11 February 2014 at 02:15:37 UTC, Andrei Alexandrescu 
wrote:
 On 2/10/14, 6:24 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 No, I don't think it is only a matter of resources. For 
 instance, if I
 had the time

Oh, the unbelievable irony.

Not really. If you have too many outstanding issues it means you have added to many features. It means you failed to do feature freeze at an earlier stage. It could also mean that you don't give priority to mentoring. Sometimes it is better to let your best people do mentoring and help bringing "master level students" up to speed. People are not loyal to a project. People are loyal to other people. If a mentor invests time in you, you will feel a social debt. This is the principle of gifting. You can create a strategy for mentoring. One obvious one is to focus on making the code base suitable for academia. Then you can offer supervision of master students. Academics love to have good external supervisors taking some load off their backs. That means lowering the requirements for compilation speed in order to get in some high level optimization and other features that you cannot otherwise have. You can give priority to getting in support for more social bonding between developers, like give priority to an IDE that supports CSCW style collaboration (seeing the code view of others). With Skype that could make pair programming (from XP) possible. There are many options.
Feb 11 2014
prev sibling next sibling parent "Abdulhaq" <alynch4047 gmail.com> writes:
On Tuesday, 11 February 2014 at 09:42:44 UTC, Ola Fosheim Grøstad 
wrote:
 On Tuesday, 11 February 2014 at 02:15:37 UTC, Andrei

Hi Ola, as a clever guy who says he has studied project dynamics you must understand that for everyone else here it feels like you dropped into the forums from outer space, confidently asserted your strong (usually well informed) opinion about this and that all over the place and now it seems you expect the project leaders to jump. Well, you should realise that until you actually start contributing code rather just talking, it feels very pushy. There are very clever people here who have spent years contributing spare time to the code and so their opinion will always carry more weight. Andrei, who displays remarkable tolerance on these boards, momentarily lost his rag a bit with you and now you're threating to bring out your sarcasm super-powers - a strange way to win friends and influence people. You've obviously got a lot to offer but picking fights with Andrei is highly conuter productive.
Feb 11 2014
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 11 February 2014 at 12:09:23 UTC, Abdulhaq wrote:
 just talking, it feels very pushy. There are very clever people 
 here who have spent years contributing spare time to the code 
 and so their opinion will always carry more weight.

I don't mind if it feels pushy. If they are pushy, it means there is something to it that you don't want to see. I am not forcing anyone to follow opinions or read them. I want to know where this project is heading. Is it heading in a real-time direction, or not?
 You've obviously got a lot to offer but picking fights with 
 Andrei is highly conuter productive.

I did not pick a fight with him. He picked a fight with me. I am sorry, but I don't accept ad hominem. If people do that, I stand up to it. Whether is on my own behalf or on the behalf of others. If people keep doing it after being warned it is at sometimes better to drive the point home with whatever means the medium offers. It is not counter productive, I got a lot of information out of this: 1. There is no plan. 2. The main reason of slow moving is on the management side. (Bounties is not a solution, in fact they can be demotivational).
Feb 11 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 11 February 2014 at 17:12:37 UTC, Andrei Alexandrescu 
wrote:
 Also I don't think volunteer effort can't be organized. It is 
 a matter
 of people identifying themselves as part of well-defined 
 organization as
 opposed to independent crowd of collaborators.

I think we could organize ourselves better, too. It's all a matter of finding the right angle. But I think management is not what we need. We need better leadership. Big difference.

Probably. But one can't simply create leadership, this is something that comes naturally. Management is easier.
 It is a common wisdom
 that if no one feels directly responsible for an issue, no one 
 will ever
 pay attention to it.

Yah, on the trite side even I'd opine :o). I said this several times now, and I'll say it again: I have asked SEVERAL TIMES people INDIVIDUALLY to do things that are HIGH IMPACT for the D language instead of something else that was more important to them, to no avail.

Of course, because you can't ask people to fulfill duties they have not volunteered to fulfill. This is exactly what I am speaking about - amount of people actually working on the language is actually very small despite high amount of contribution. Most people just do stuff they need or are interested in and can't be obliged to do anything else. Volunteering to do stuff you don't really want is completely different thing :) And it needs to be encouraged by something more precious than tiny bounties. For example, being able to influence language-changing decisions is much more seductive reward.
 Of course, it would be learned helplessness to draw sweeping 
 conclusions from the experience so far.

 Do you need any specific proposals?

Suggestions for doing things better are gladly considered.

As I have mentioned on some occasions, I was very impressed by both simplicity and efficiency of Arch Linux "Trusted User" organization after studying it "from inside". It is group of people that are not directly affiliated with Arch developers but have volunteered to take care/responsibility about parts of ecosystem. They also have power to make decisions regarding that ecosystem by formal voting procedure (with strict quorum and success % defined). Addition of new trusted users requires sponsorship from one of existing TU's and is approved by the very same voting procedure. Usually new TU's state clearly what parts of the ecosystem they want to be responsible for during initial application and this is often taken into consideration by voters. I think D community can take some inspiration from such approach. It will allow to make decisions on more controversial topics and speed up process in general by removing bottleneck of your + Walter decision (assuming you still have veto votes in case stuff goes really bad). Also it gives clear overview of who is supposed to be responsible for what and feeling of making the difference for those who take part in it.
Feb 11 2014
prev sibling next sibling parent "Andrej Mitrovic" <andrej.mitrovich gmail.com> writes:
On Tuesday, 11 February 2014 at 17:12:37 UTC, Andrei Alexandrescu 
wrote:
 I said this several times now, and I'll say it again: I have 
 asked SEVERAL TIMES people INDIVIDUALLY to do things that are 
 HIGH IMPACT for the D language instead of something else that 
 was more important to them, to no avail.

Which things exactly?
Feb 11 2014
prev sibling next sibling parent "Dicebot" <public dicebot.lv> writes:
On Tuesday, 11 February 2014 at 17:50:54 UTC, Dicebot wrote:
 Suggestions for doing things better are gladly considered.

As I have mentioned on some occasions, I was very impressed by both simplicity and efficiency of Arch Linux "Trusted User" organization after studying it "from inside". It is group of people that are not directly affiliated with Arch developers but have volunteered to take care/responsibility about parts of ecosystem. They also have power to make decisions regarding that ecosystem by formal voting procedure (with strict quorum and success % defined). Addition of new trusted users requires sponsorship from one of existing TU's and is approved by the very same voting procedure. Usually new TU's state clearly what parts of the ecosystem they want to be responsible for during initial application and this is often taken into consideration by voters. I think D community can take some inspiration from such approach. It will allow to make decisions on more controversial topics and speed up process in general by removing bottleneck of your + Walter decision (assuming you still have veto votes in case stuff goes really bad). Also it gives clear overview of who is supposed to be responsible for what and feeling of making the difference for those who take part in it.

To give some specifics, one D example where we do have something resembling an organized process is Phobos review queue. Once I have noticed that there are lot of proposal rotting there (and that I don't like it) it was trivial to read the reiew process description, step up and proceed with all that stuff. Expectations were clear, process was (mostly) clear, same for responsibilities. And main reward for that is that I can choose what gets reviewed next and poke people about their work :)
Feb 11 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 12 February 2014 at 01:54:15 UTC, Andrei
Alexandrescu wrote:
 Yes. I think that's a disaster. We need to figure out the right 
 approach to solving that.

For a while, we were doing the review sunday. That was fun, everybody was talking about the pull in IRC and we were going from one to the other. Maybe it is time to start this tradition again ?
Feb 11 2014
prev sibling next sibling parent "Adam Wilson" <flyboynw gmail.com> writes:
On Tue, 11 Feb 2014 18:36:10 -0800, Andrei Alexandrescu  
<SeeWebsiteForEmail erdani.org> wrote:

 On 2/11/14, 6:21 PM, deadalnix wrote:
 On Wednesday, 12 February 2014 at 01:54:15 UTC, Andrei
 Alexandrescu wrote:
 Yes. I think that's a disaster. We need to figure out the right
 approach to solving that.

For a while, we were doing the review sunday. That was fun, everybody was talking about the pull in IRC and we were going from one to the other. Maybe it is time to start this tradition again ?

Are you guys up for it? I would. Andrei

I loved it. I was fun to watch the autotester go nuts and get in there and review the changes prior to committing. -- Adam Wilson GitHub/IRC: LightBender Aurora Project Coordinator
Feb 11 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 12 February 2014 at 02:21:38 UTC, deadalnix wrote:
 On Wednesday, 12 February 2014 at 01:54:15 UTC, Andrei
 Alexandrescu wrote:
 Yes. I think that's a disaster. We need to figure out the 
 right approach to solving that.

For a while, we were doing the review sunday. That was fun, everybody was talking about the pull in IRC and we were going from one to the other. Maybe it is time to start this tradition again ?

I definitively would !
Feb 11 2014
prev sibling parent "francesco cattoglio" <francesco.cattoglio gmail.com> writes:
On Wednesday, 12 February 2014 at 02:21:38 UTC, deadalnix wrote:
 On Wednesday, 12 February 2014 at 01:54:15 UTC, Andrei
 Alexandrescu wrote:
 Yes. I think that's a disaster. We need to figure out the 
 right approach to solving that.

For a while, we were doing the review sunday. That was fun, everybody was talking about the pull in IRC and we were going from one to the other. Maybe it is time to start this tradition again ?

Sounds like a really nice idea! I would come in and "listen silently" for sure! At least untill I'm knowledgeable enough :P
Feb 12 2014