www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - RFC: reference counted Throwable

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
As discussed, having exception objects being GC-allocated is clearly a 
large liability that we need to address. They prevent otherwise careful 
functions from being  nogc so they affect even apps that otherwise would 
be okay with a little litter here and there.

The Throwable hierarchy is somewhat separate from everything else, which 
makes it a great starting point for investigating an automated reference 
count approach. Here's what I'm thinking.

First, there must be some compiler flag -nogc or something, which 
triggers the RC exceptions. All modules of an application must be 
compiled with this flag if it is to work (such that one module can throw 
an exception caught by the other). Of course a lot of refinement needs 
to be added here (what happens if one tries to link modules built with 
and without -nogc, allowing people to detect the flag programmatically 
by using version(nogc) etc).

If -nogc is passed, the compiler severs the inheritance relationship 
between Throwable and Object, making it impossible to convert a 
Throwable to an Object. From then henceforth, Throwable and Object form 
a two-rooted forest. (In all likelihood we'll later add an RCObject root 
that Throwable inherits.)

Whenever a reference to a Throwable is copied about, passed to 
functions, the compiler inserts appropriately calls to e.g. incRef and 
decRef. (Compiler may assume they cancel each other for optimization 
purposes.) Implementation of these is up to the runtime library. Null 
checking may be left to either the compiler or the library (in the 
latter case, the functions must be nonmember). However it seems the 
compiler may have an advantage because it can elide some of these checks.

Once we get this going, we should accumulate good experience that we can 
later apply to generalizing this approach to more objects. Also, if 
things go well we may as well define _always_ (whether GC or not) 
Throwable to be reference counted; seems like a good fit for all programs.

Please chime in with thoughts.


Andrei
Sep 19 2014
next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.
Are you suggesting we implement ARC? -- /Jacob Carlborg
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 12:42 PM, Jacob Carlborg wrote:
 On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.
Are you suggesting we implement ARC?
Yes. -- Andrei
Sep 19 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 19 September 2014 at 22:14:08 UTC, Andrei Alexandrescu 
wrote:
 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:
 On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and
 decRef. (Compiler may assume they cancel each other for 
 optimization
 purposes.) Implementation of these is up to the runtime 
 library.
Are you suggesting we implement ARC?
Yes. -- Andrei
I don't think ARC is needed. library RC + borrowing + uniqueness/moving = WIN
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz wrote:
 On Friday, 19 September 2014 at 22:14:08 UTC, Andrei 
 Alexandrescu wrote:
 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:
 On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed 
 to
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and
 decRef. (Compiler may assume they cancel each other for 
 optimization
 purposes.) Implementation of these is up to the runtime 
 library.
Are you suggesting we implement ARC?
Yes. -- Andrei
I don't think ARC is needed. library RC + borrowing + uniqueness/moving = WIN
You can't do polymorphic entity RC (like exceptions) without at least some help from compiler because language currently does not provide tools for lifetime control of classes. At least _some_ form of ARC is necessary.
Sep 20 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:
 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
You can't do polymorphic entity RC (like exceptions) without at least some help from compiler because language currently does not provide tools for lifetime control of classes. At least _some_ form of ARC is necessary.
I think we can, using templated alias this. We're not there yet, but it's probably feasable, seeing that Igor Stepanov already implemented multiple alias this [1]. With that and maybe a little opDispatch magic, it should be possible to making wrappers that are implicitly convertible to wrappers of parents of their inner types. Granted, for exceptions there's more needed: There needs to be support for throwing and catching these wrappers, and for catching wrappers of derived types, too. But note that throw/catch itself doesn't involve copying, it's a moving operation, which might make it easier. [1] https://github.com/D-Programming-Language/dmd/pull/3998
Sep 20 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz wrote:
 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:
 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
You can't do polymorphic entity RC (like exceptions) without at least some help from compiler because language currently does not provide tools for lifetime control of classes. At least _some_ form of ARC is necessary.
I think we can, using templated alias this. We're not there yet, but it's probably feasable, seeing that Igor Stepanov already implemented multiple alias this [1]. With that and maybe a little opDispatch magic, it should be possible to making wrappers that are implicitly convertible to wrappers of parents of their inner types. Granted, for exceptions there's more needed: There needs to be support for throwing and catching these wrappers, and for catching wrappers of derived types, too. But note that throw/catch itself doesn't involve copying, it's a moving operation, which might make it easier. [1] https://github.com/D-Programming-Language/dmd/pull/3998
Thinking further about it, we don't need templated `alias this`, because we can just have one `alias this` that returns `scope!this(T)` (this needs to be there anyway to be safe). It would automatically behave like a normal class with regard to polymorphy. What's still needed then is a little magic to actually be able to throw and catch the wrapper.
Sep 20 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz wrote:
 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:
 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
You can't do polymorphic entity RC (like exceptions) without at least some help from compiler because language currently does not provide tools for lifetime control of classes. At least _some_ form of ARC is necessary.
I think we can, using templated alias this. We're not there yet, but it's probably feasable, seeing that Igor Stepanov already implemented multiple alias this [1]. With that and maybe a little opDispatch magic, it should be possible to making wrappers that are implicitly convertible to wrappers of parents of their inner types. Granted, for exceptions there's more needed: There needs to be support for throwing and catching these wrappers, and for catching wrappers of derived types, too. But note that throw/catch itself doesn't involve copying, it's a moving operation, which might make it easier. [1] https://github.com/D-Programming-Language/dmd/pull/3998
Yeah but implicitly convertible to what? If you convert it to `Throwable` your reference counting facilities are circumvented resulting in dangling exception reference at point where `Throwable` gets caught. Special casing catching such wrappers to still preserve original ref-counted type while pretending to be Throwable at call site sounds like a terrible hack, much worse than any sort of ARC complexity.
Sep 20 2014
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:19:21 UTC, Dicebot wrote:
 On Saturday, 20 September 2014 at 09:05:24 UTC, Marc Schütz 
 wrote:
 On Saturday, 20 September 2014 at 08:32:55 UTC, Dicebot wrote:
 On Saturday, 20 September 2014 at 08:20:47 UTC, Marc Schütz 
 wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
You can't do polymorphic entity RC (like exceptions) without at least some help from compiler because language currently does not provide tools for lifetime control of classes. At least _some_ form of ARC is necessary.
I think we can, using templated alias this. We're not there yet, but it's probably feasable, seeing that Igor Stepanov already implemented multiple alias this [1]. With that and maybe a little opDispatch magic, it should be possible to making wrappers that are implicitly convertible to wrappers of parents of their inner types. Granted, for exceptions there's more needed: There needs to be support for throwing and catching these wrappers, and for catching wrappers of derived types, too. But note that throw/catch itself doesn't involve copying, it's a moving operation, which might make it easier. [1] https://github.com/D-Programming-Language/dmd/pull/3998
Yeah but implicitly convertible to what? If you convert it to `Throwable` your reference counting facilities are circumvented resulting in dangling exception reference at point where `Throwable` gets caught.
As I said, throw/catch is at its core a moving operation. For classes this isn't important (they are references), but for an RC wrapper it would be, so we could specify that. Move-constructing from an lvalue in the context of D can be seen as a three-step process: 1) create a temporary, initialize with T.init 2) bit-swap the variable with the temporary 3) destroy the variable It can be seen that the value that is to be moved (the RC wrapper) must be non-const (only at the head, it may still be tail-const). Now, any generic RC type that wants to be implicitly convertible to its payload type must do this via borrowing in order to be safe (see my proposal at [1]). Using const-borrowing, we can guarantee that the wrapper will not be thrown (or moved, in general) as long as borrowed references to its payload exist: struct RC(T) { // ... T _payload; scope!(const this) borrow() { return _payload; } alias borrow this; } This already solves avoiding dangling references to the exception: No references can be left behind when the exception is thrown, and the wrapper will not be destroyed, but moved, thus not releasing the exception's memory. The second part is really "just" some way to transport the wrapper via the exception mechanism, including support for catching wrappers of derived exception types.
 Special casing catching such wrappers to still preserve 
 original ref-counted type while pretending to be Throwable at 
 call site sounds like a terrible hack, much worse than any sort 
 of ARC complexity.
IMO it would only a hack _if_ they were indeed special cased. I'm sure we can find a more generic mechanism that would allow this to be implemented cleanly. Note that the other requirements I described above (borrowing, move semantics) are also things that just happen to be usable here: they are desirable in general, exceptions are just one possible application. [1] http://wiki.dlang.org/User:Schuetzm/scope
Sep 20 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 11:50:28 UTC, Marc Schütz wrote:
 On Saturday, 20 September 2014 at 09:19:21 UTC, Dicebot wrote:
 Yeah but implicitly convertible to what? If you convert it to 
 `Throwable` your reference counting facilities are 
 circumvented resulting in dangling exception reference at 
 point where `Throwable` gets caught.
As I said, throw/catch is at its core a moving operation. For classes this isn't important (they are references), but for an RC wrapper it would be, so we could specify that. Move-constructing from an lvalue in the context of D can be seen as a three-step process: 1) create a temporary, initialize with T.init 2) bit-swap the variable with the temporary 3) destroy the variable It can be seen that the value that is to be moved (the RC wrapper) must be non-const (only at the head, it may still be tail-const). Now, any generic RC type that wants to be implicitly convertible to its payload type must do this via borrowing in order to be safe (see my proposal at [1]). Using const-borrowing, we can guarantee that the wrapper will not be thrown (or moved, in general) as long as borrowed references to its payload exist: struct RC(T) { // ... T _payload; scope!(const this) borrow() { return _payload; } alias borrow this; } This already solves avoiding dangling references to the exception: No references can be left behind when the exception is thrown, and the wrapper will not be destroyed, but moved, thus not releasing the exception's memory. The second part is really "just" some way to transport the wrapper via the exception mechanism, including support for catching wrappers of derived exception types.
 Special casing catching such wrappers to still preserve 
 original ref-counted type while pretending to be Throwable at 
 call site sounds like a terrible hack, much worse than any 
 sort of ARC complexity.
IMO it would only a hack _if_ they were indeed special cased. I'm sure we can find a more generic mechanism that would allow this to be implemented cleanly. Note that the other requirements I described above (borrowing, move semantics) are also things that just happen to be usable here: they are desirable in general, exceptions are just one possible application. [1] http://wiki.dlang.org/User:Schuetzm/scope
The mechanism might simply be to allow anything to be thrown that a) supports moving, e.g. provides a method `release()` that returns a unique expression (this requires moving and uniqueness to be specified first, which is a good idea anyway), and b) is implicitly convertible to Throwable.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 1:20 AM, "Marc Schütz" <schuetzm gmx.net>" wrote:
 On Friday, 19 September 2014 at 22:14:08 UTC, Andrei Alexandrescu wrote:
 On 9/19/14, 12:42 PM, Jacob Carlborg wrote:
 On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library.
Are you suggesting we implement ARC?
Yes. -- Andrei
I don't think ARC is needed. library RC + borrowing + uniqueness/moving = WIN
s/WIN/Rust/
Sep 20 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei 
Alexandrescu wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
s/WIN/Rust/
Well Rust does some things well. We are going in the same direction anyway (type qualifier, uniqueness) expect we do it in an ad hoc manner that is guaranteed to yield a C++ish result.
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 5:30 PM, deadalnix wrote:
 On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei Alexandrescu wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
s/WIN/Rust/
Well Rust does some things well. We are going in the same direction anyway (type qualifier, uniqueness) expect we do it in an ad hoc manner that is guaranteed to yield a C++ish result.
Rust looked a lot more exciting when I didn't know much about it. -- Andrei
Sep 20 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Rust looked a lot more exciting when I didn't know much about 
 it.
I didn't remember ever seeing you excited about Rust :-) In past you (rightfully) didn't comment much about Rust. But do you have more defined ideas about it now? Do you still think D has a chance against Rust? Bye, bearophile
Sep 20 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 02:10:13 UTC, bearophile wrote:
 Andrei Alexandrescu:

 Rust looked a lot more exciting when I didn't know much about 
 it.
I didn't remember ever seeing you excited about Rust :-) In past you (rightfully) didn't comment much about Rust. But do you have more defined ideas about it now? Do you still think D has a chance against Rust? Bye, bearophile
It is clear that rust has some serious limitations. - It interact badly with the existing world (other languages). - It is slow to compile. - Constraints too much the dev in some paradigms, which obviously won't fit all area of programming. - It bets on many unproven mechanism (ie see error handling). Some of them may be real hit, but I doubt ALL of them will be, and these which won't be will have to be worked around, adding complexity. - The macro system is plain horrible, and the only way to do code generation. Rust has many cool goodies, but it is generally too radical.
Sep 20 2014
parent reply Nick Treleaven <ntrel-public yahoo.co.uk> writes:
On 21/09/2014 03:35, deadalnix wrote:
   - It is slow to compile.
Surely that's not an inherent property of Rust?
   - Constraints too much the dev in some paradigms, which obviously
 won't fit all area of programming.
Absolutely. The unique mutable borrow rule too often prevents even reading a variable, this is a big pain IMO. I wonder if this is partly because Rust doesn't have const, only immutable.
   - The macro system is plain horrible,
The syntax is not great (I heard it may change), but I think the capabilities are actually pretty good.
 and the only way to do code generation.
I think you're wrong, you can add syntax extensions such as the compile-time regex support: http://blog.burntsushi.net/rust-regex-syntax-extensions "Luckily, the second way to write a macro is via a syntax extension (also known as a “procedural macro”). This is done by a compile time hook that lets you execute arbitrary Rust code, rewrite the abstract syntax to whatever you want and opens up access to the Rust compiler’s parser. In short, you get a lot of power. And it’s enough to implement native regexes. The obvious drawback is that they are more difficult to implement."
Sep 22 2014
parent "bearophile" <bearophileHUGS lycos.com> writes:
Nick Treleaven:

  - It is slow to compile.
Surely that's not an inherent property of Rust?
We don't know yet. Perhaps Rust type inference needs lot of computations. Bye, bearophile
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:10 PM, bearophile wrote:
 Andrei Alexandrescu:

 Rust looked a lot more exciting when I didn't know much about it.
I didn't remember ever seeing you excited about Rust :-) In past you (rightfully) didn't comment much about Rust. But do you have more defined ideas about it now? Do you still think D has a chance against Rust?
I don't think Rust has a chance against D. -- Andrei
Sep 20 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 04:50, schrieb Andrei Alexandrescu:
 On 9/20/14, 7:10 PM, bearophile wrote:
 Andrei Alexandrescu:

 Rust looked a lot more exciting when I didn't know much about it.
I didn't remember ever seeing you excited about Rust :-) In past you (rightfully) didn't comment much about Rust. But do you have more defined ideas about it now? Do you still think D has a chance against Rust?
I don't think Rust has a chance against D. -- Andrei
The real question is which language, from all that want to replace C++, will eventually get a place at an OS vendors SDK. So far, the winning ones seem to be Swift on Apple side, and .NET Native-C++/CLX on Microsoft side (who knows what are they doing with M#). Maybe someone in the commercial UNIX (FOSS is too bound with C), real time or embedded OS space? -- Paulo
Sep 20 2014
parent reply "Cliff" <cliff.s.hudson gmail.com> writes:
On Sunday, 21 September 2014 at 04:59:12 UTC, Paulo Pinto wrote:
 Am 21.09.2014 04:50, schrieb Andrei Alexandrescu:
 On 9/20/14, 7:10 PM, bearophile wrote:
 Andrei Alexandrescu:

 Rust looked a lot more exciting when I didn't know much 
 about it.
I didn't remember ever seeing you excited about Rust :-) In past you (rightfully) didn't comment much about Rust. But do you have more defined ideas about it now? Do you still think D has a chance against Rust?
I don't think Rust has a chance against D. -- Andrei
The real question is which language, from all that want to replace C++, will eventually get a place at an OS vendors SDK. So far, the winning ones seem to be Swift on Apple side, and .NET Native-C++/CLX on Microsoft side (who knows what are they doing with M#). Maybe someone in the commercial UNIX (FOSS is too bound with C), real time or embedded OS space? -- Paulo
Interop, interop, interop. Walter and Andrei are right when they talk about the importance of C++ interop - not only do you get to leverage those libraries, but it reduces the barrier to entry for D in more environments. Swift will never be more important than Objective C was - which is to say it'll be the main development language on Apple products and probably nothing else. That has real value, but the limits on it are pretty hard and fast (which says more about Apple than the language itself.) .NET suffers a similar problem in spite of the community's best efforts with Mono - it'll always be a distant 2nd (or 5th or 20th) on other platforms. And on Windows, C++ won't get supplanted by .NET absent a sea-change in the mindset of the Windows OS group - which is notoriously resistant to change (and they have a colossal existing code base which isn't likely to benefit from the kind of inflection point Apple had moving to a BSD and porting/rewriting scads of code.) So C/C++ is it for universal languages, really (outside of the web server space, where you have a large Java deployment.) I don't think D needs to be the next .NET (of any flavor) or the next Swift, and I don't see as it is being positioned that way either - the target to me is clearly C/C++. It doesn't need to compete with languages that have lesser universality, though it should (and does) borrow the good ideas from those languages. I don't think D needs to look at *replacing* C++ in the near or mid term either - it still needs to convince people it deserves a place at the table. And the easiest way to do that is to get this C++ interop story really nailed down, and make sure D's warts are smaller than C++'s. And, of course, the GC strawman that native programmers always claim is more important than it really is. I like the threads going on currently about ARC and related technologies - there's a real chance to innovate here.
Sep 20 2014
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 05:55:20 UTC, Cliff wrote:
 .NET suffers a similar problem in spite of the community's best 
 efforts with Mono - it'll always be a distant 2nd (or 5th or 
 20th) on other platforms.  And on Windows, C++ won't get 
 supplanted by .NET absent a sea-change in the mindset of the 
 Windows OS group - which is notoriously resistant to change 
 (and they have a colossal existing code base which isn't likely 
 to benefit from the kind of inflection point Apple had moving 
 to a BSD and porting/rewriting scads of code.)
A bit OT: but Mono seems to be a popular platform for desktop application on linux. I have no idea why.
Sep 20 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 08:05, schrieb deadalnix:
 On Sunday, 21 September 2014 at 05:55:20 UTC, Cliff wrote:
 .NET suffers a similar problem in spite of the community's best
 efforts with Mono - it'll always be a distant 2nd (or 5th or 20th) on
 other platforms.  And on Windows, C++ won't get supplanted by .NET
 absent a sea-change in the mindset of the Windows OS group - which is
 notoriously resistant to change (and they have a colossal existing
 code base which isn't likely to benefit from the kind of inflection
 point Apple had moving to a BSD and porting/rewriting scads of code.)
A bit OT: but Mono seems to be a popular platform for desktop application on linux. I have no idea why.
I think it may come from C++ hate that GNU/Linux enjoys, or at least it used to enjoy. Back in the early days suggesting using C++ was an heresy, although corporate developers were slowly migrating to it in commercial UNIX settings. So you got to do CORBA in C, OO in C (Gtk) and so on. When I used to take part in gtkmm discussions (early 2000), the usual C vs C++ discussions on GNOME mailing lists were quite common. Qt and KDE were a no go for many, due to licesing issues. So C++ usage was limited to those of us already into C++ that weren't that keen in using plain C. A minority, Meanwhile, Microsoft released .NET and Miguel and others, which already had ported Microsoft technologies (Bonobo(COM), Evolution(Outlook)...) started Mono. Given that they were also GNOME maintainers, there was this big discussion going on about GNOME becoming tainted with Mono. To the point GNOME created Vala, which is basically C# that compiles to C and uses gobject as runtime system. I used to be in the "we don't want .NET on GNU/Linux" side. Nowadays I can only congratulate Miguel and the others that persisted against naysayers like myself. Specially given what happened with Oracle vs Google. Which Xamarin and Microsoft took advantage of with the "Community license for open source". Slowly, as the amount of users started to increase, I guess the GNU/Linux opinion about Mono, started to change. So if you want to use a modern programming language, with bindings to most common UI frameworks and ability to generate fast code, there aren't many other options. Here I would say D could also fit the bill, but lacks visibility by the average GNU/Linux coder on the playground discussions. This is just an opinion, maybe I am too far of the real reasons. -- Paulo
Sep 20 2014
prev sibling next sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-21 07:55, Cliff wrote:

 Swift will never be more important than Objective C was - which is to
 say it'll be the main development language on Apple products and
 probably nothing else.  That has real value, but the limits on it are
 pretty hard and fast (which says more about Apple than the language
 itself.)

 .NET suffers a similar problem in spite of the community's best efforts
 with Mono - it'll always be a distant 2nd (or 5th or 20th) on other
 platforms.  And on Windows, C++ won't get supplanted by .NET absent a
 sea-change in the mindset of the Windows OS group - which is notoriously
 resistant to change (and they have a colossal existing code base which
 isn't likely to benefit from the kind of inflection point Apple had
 moving to a BSD and porting/rewriting scads of code.)
Unfortunately the user base is so large on these platforms that it's enough for these languages to only work on one particular platform and the developers will invest a huge amount work on it anyway. The languages will have a large user base even though they only work on one platform. -- /Jacob Carlborg
Sep 21 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-21 07:55, Cliff wrote:

 .NET suffers a similar problem in spite of the community's best efforts
 with Mono - it'll always be a distant 2nd (or 5th or 20th) on other
 platforms.  And on Windows, C++ won't get supplanted by .NET absent a
 sea-change in the mindset of the Windows OS group - which is notoriously
 resistant to change (and they have a colossal existing code base which
 isn't likely to benefit from the kind of inflection point Apple had
 moving to a BSD and porting/rewriting scads of code.)
Actually, Microsoft, with the help of Xamarin seems to aim for cross-paltform with Mono, at least for mobile platforms. -- /Jacob Carlborg
Sep 21 2014
prev sibling next sibling parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 01:46:13 UTC, Andrei Alexandrescu 
wrote:
 Rust looked a lot more exciting when I didn't know much about 
 it. -- Andrei
I think it is interesting that Rust is kinda converging on only having the scope semantics in the language and the rest in the library. (of course i'm biased to notice that cuz that's what I want here too!)
Sep 20 2014
prev sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 01:46:13 UTC, Andrei Alexandrescu 
wrote:
 On 9/20/14, 5:30 PM, deadalnix wrote:
 On Saturday, 20 September 2014 at 16:27:33 UTC, Andrei 
 Alexandrescu wrote:
 I don't think ARC is needed.

 library RC + borrowing + uniqueness/moving = WIN
s/WIN/Rust/
Well Rust does some things well. We are going in the same direction anyway (type qualifier, uniqueness) expect we do it in an ad hoc manner that is guaranteed to yield a C++ish result.
Rust looked a lot more exciting when I didn't know much about it. -- Andrei
Ho come on, there is a reason why I'm here and not on rust's mailing list. I'm aware of various shortcomings of rust. Reading my message as a promotion of rust is a gross strawman. And here is the thing. You and Walter closed one door to this kind of feature when discussed with Bartoz a while ago and won't admit you are wrong. Then whole thing degenerate in some ridiculous church war where all rhetorical trick and logical fallacy is good to use. That is pretty much how it degenerated with the concept vs static if discussion. The whole debate become completely ridiculous (yes, your dismissal of concept is almost as ridiculous as Bjarne's rebuttal of static if). Because at this point, this is not about what is best, but about who is right. Back to the subject, that door was closed few years ago. Now we are discussing pile of hacks again and again and again. The final complexity is WAY higher than the initial proposal. Be it Bartoz's, rust's, isolated or whatever variation of the concept, this is pretty much a given by now that it is needed. As far as I'm concerned, isolated/owned (not even talking about rust's concepts like burrowing) on the GC should probably be valid in nogc, and only the promotion to TL, shared or immutable heap should be invalid in nogc. You combine that with require that you throw isolated/owned/whatever, and the whole discussion we are having here becomes moot (as well as various it can reduce friction between the GC world and the non GC world greatly).
Sep 20 2014
prev sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Andrei Alexandrescu:

 Are you suggesting we implement ARC?
Yes. -- Andrei
I think it's better first to design & implement a first version of memory ownership management, and then later add a reference counting scheme. Bye, bearophile
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 5:17 AM, bearophile wrote:
 Andrei Alexandrescu:

 Are you suggesting we implement ARC?
Yes. -- Andrei
I think it's better first to design & implement a first version of memory ownership management, and then later add a reference counting scheme.
That's not quite informative. -- Andrei
Sep 20 2014
prev sibling next sibling parent "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:
 First, there must be some compiler flag -nogc or something, 
 which triggers the RC exceptions. All modules of an application 
 must be compiled with this flag if it is to work (such that one 
 module can throw an exception caught by the other). Of course a 
 lot of refinement needs to be added here (what happens if one 
 tries to link modules built with and without -nogc, allowing 
 people to detect the flag programmatically by using 
 version(nogc) etc).
Won't using -nogc preclude using this in Phobos/Druntime? How about instead of a new switch, change the name mangling of anything that inherits from RCObject? E.g. 'C' is used for classes, 'R' could be used for reference-counting classes. Then an ordinary version block can be used to switch between the two.
Sep 19 2014
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 8:32 AM, Andrei Alexandrescu wrote:
 As discussed, having exception objects being GC-allocated is clearly a large
 liability that we need to address. They prevent otherwise careful functions
from
 being  nogc so they affect even apps that otherwise would be okay with a little
 litter here and there.

 The Throwable hierarchy is somewhat separate from everything else, which makes
 it a great starting point for investigating an automated reference count
 approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, which triggers the
 RC exceptions. All modules of an application must be compiled with this flag if
 it is to work (such that one module can throw an exception caught by the
other).
 Of course a lot of refinement needs to be added here (what happens if one tries
 to link modules built with and without -nogc, allowing people to detect the
flag
 programmatically by using version(nogc) etc).
Having a compiler switch to change the behavior of every module in incompatible ways would be a disastrous balkanization. It has to be done in such a way that the ARC and the existing exceptions can coexist.
 If -nogc is passed, the compiler severs the inheritance relationship between
 Throwable and Object, making it impossible to convert a Throwable to an Object.
  From then henceforth, Throwable and Object form a two-rooted forest. (In all
 likelihood we'll later add an RCObject root that Throwable inherits.)

 Whenever a reference to a Throwable is copied about, passed to functions, the
 compiler inserts appropriately calls to e.g. incRef and decRef. (Compiler may
 assume they cancel each other for optimization purposes.) Implementation of
 these is up to the runtime library. Null checking may be left to either the
 compiler or the library (in the latter case, the functions must be nonmember).
 However it seems the compiler may have an advantage because it can elide some
of
 these checks.

 Once we get this going, we should accumulate good experience that we can later
 apply to generalizing this approach to more objects. Also, if things go well we
 may as well define _always_ (whether GC or not) Throwable to be reference
 counted; seems like a good fit for all programs.

 Please chime in with thoughts.


 Andrei
Sep 19 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 6:18 PM, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.
Could you please elaborate why the disaster? -- Andrei
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:
 On 9/19/14, 6:18 PM, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.
Could you please elaborate why the disaster? -- Andrei
1. Every library for D will have to come in two versions. One which works, and the other is likely untested. This will divide the D community in half. 2. RC and GC engender different styles of programming. The idea that one can successfully switch between them with merely a compiler switch is a fantasy. You cannot write non-trivial programs that function well and pay no attention to how memory management is done.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 7:20 PM, Walter Bright wrote:
 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:
 On 9/19/14, 6:18 PM, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be done
 in such a way that the ARC and the existing exceptions can coexist.
Could you please elaborate why the disaster? -- Andrei
1. Every library for D will have to come in two versions. One which works, and the other is likely untested. This will divide the D community in half.
I understand where you're coming from. And you're right that we're looking at an important change - as big as -m64 vs. -m32 or porting to a new platform. It's big. But it's needed, soon. And it would be, it seems to me, a mistake to approach this big change as a small change that we can sneak in. This judgment - that RC vs. GC would balkanize the community - has become a prejudice that is worth revisiting. Just trotting it out again won't do.
 2. RC and GC engender different styles of programming. The idea that one
 can successfully switch between them with merely a compiler switch is a
 fantasy. You cannot write non-trivial programs that function well and
 pay no attention to how memory management is done.
I think we're at a point where we owe it to ourselves to consider realizing the fantasy. Andrei
Sep 19 2014
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 06:43, schrieb Andrei Alexandrescu:
 On 9/19/14, 7:20 PM, Walter Bright wrote:
 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:
 On 9/19/14, 6:18 PM, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be
 done
 in such a way that the ARC and the existing exceptions can coexist.
Could you please elaborate why the disaster? -- Andrei
1. Every library for D will have to come in two versions. One which works, and the other is likely untested. This will divide the D community in half.
I understand where you're coming from. And you're right that we're looking at an important change - as big as -m64 vs. -m32 or porting to a new platform. It's big. But it's needed, soon. And it would be, it seems to me, a mistake to approach this big change as a small change that we can sneak in. This judgment - that RC vs. GC would balkanize the community - has become a prejudice that is worth revisiting. Just trotting it out again won't do.
 2. RC and GC engender different styles of programming. The idea that one
 can successfully switch between them with merely a compiler switch is a
 fantasy. You cannot write non-trivial programs that function well and
 pay no attention to how memory management is done.
I think we're at a point where we owe it to ourselves to consider realizing the fantasy. Andrei
This is one of the reasons why the Objective-C GC failed. Mixing Frameworks compiled with both modes never worked properly. -- Paulo
Sep 19 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 06:28:11 UTC, Paulo Pinto wrote:
 This is one of the reasons why the Objective-C GC failed.

 Mixing Frameworks compiled with both modes never worked 
 properly.
Yeah, mixing is bad unless you template eveything. But in pure nogc it would work. Except, I think most pure nogc projects want to eliminate rtti/exceptions too...
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 07:35:47 UTC, Ola Fosheim 
Grostad wrote:
 Except, I think most pure nogc projects want to eliminate 
 rtti/exceptions too...
It should be possible to do on top of plain nogc Phobos by simply hooking into druntime _d_throw function and providing some own domain-specific handler instead. Of course it doesn't help if exceptions are used to indicate casual data processing signals but it is something else to be fixed in Phobos :)
Sep 20 2014
prev sibling parent reply "Olivier Pisano" <olivier.pisano laposte.net> writes:
On Saturday, 20 September 2014 at 06:28:11 UTC, Paulo Pinto wrote:
 This is one of the reasons why the Objective-C GC failed.

 Mixing Frameworks compiled with both modes never worked 
 properly.

 --
 Paulo
Yes this was a huge failure to take into account. Linking code where Throwable inherits from Object with code where Throwable inherits from RCObject... :( If making the GC completely optional is a must, then error handling shouldn't rely on it at all, no? What about completely switching exception handling to RC ? Would it have an impact on memory safety since exeption handling mecanism is somehow "magical code generated by the compiler" ?
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:46 AM, Olivier Pisano wrote:
 If making the GC completely optional is a must, then error
 handling shouldn't rely on it at all, no? What about completely
 switching exception handling to RC ? Would it have an impact on
 memory safety since exeption handling mecanism is somehow
 "magical code generated by the compiler" ?
The more I think of it the more sensible this is. Exceptions are unlikely to create cycles, not copied extensively, and are generally short lived. So an RC scheme backed by malloc/free seems to be most appropriate. There would be breakage, though: Throwable would not be convertible to Object. I wonder what the impact in the real world that would cause. Andrei
Sep 20 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 20 Sep 2014 09:07:20 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 9/20/14, 12:46 AM, Olivier Pisano wrote:
 If making the GC completely optional is a must, then error
 handling shouldn't rely on it at all, no? What about completely
 switching exception handling to RC ? Would it have an impact on
 memory safety since exeption handling mecanism is somehow
 "magical code generated by the compiler" ?
The more I think of it the more sensible this is. Exceptions are unlikely to create cycles, not copied extensively, and are generally short lived. So an RC scheme backed by malloc/free seems to be most appropriate. There would be breakage, though: Throwable would not be convertible to Object. I wonder what the impact in the real world that would cause. Andrei
+1, replace it completely with malloc/free. However, for backwards compatibility malloced exceptions probably still have to be added as roots to the GC, at least if they refer GC allocated data. This should be somehow optional however.
Sep 21 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 09:01:45 UTC, Johannes Pfau wrote:
 +1, replace it completely with malloc/free.

 However, for backwards compatibility malloced exceptions 
 probably still
 have to be added as roots to the GC, at least if they refer GC
 allocated data. This should be somehow optional however.
Yeah, so the GC do not scan in it, and the error message disappear randomly under your feet and you segfault when trying to read. Looks like a great idea.
Sep 21 2014
parent Johannes Pfau <nospam example.com> writes:
Am Sun, 21 Sep 2014 09:35:40 +0000
schrieb "deadalnix" <deadalnix gmail.com>:

 On Sunday, 21 September 2014 at 09:01:45 UTC, Johannes Pfau wrote:
 +1, replace it completely with malloc/free.

 However, for backwards compatibility malloced exceptions 
 probably still
 have to be added as roots to the GC, at least if they refer GC
 allocated data. This should be somehow optional however.
Yeah, so the GC do not scan in it, and the error message disappear randomly under your feet and you segfault when trying to read. Looks like a great idea.
If you always need to add exceptions to GC roots the whole proposal of reference counting exceptions is useless as you depend on the GC exactly in the same way as if you allocate from the GC. Andrei initially proposed pure reference-counted exceptions in -nogc which would suffer from the same issue. And storing GC-allocated messages in exceptions is in most cases bad design. If the message is a string literal there's no GC involved and if you add 'information' to the message, making it dynamic, then: * In order to process that information you need to extract it from the message string * You create a dynamic string which might never be used (exception can be caught and ignored, etc.) The correct design is to add the information to a variable in a subclassed exception and produce the message in toString, which can work entirely without a GC. Example: BAD: throw new Exception("Timout occured, timeout = " ~ to!string(timeout)) GOOD: throw new TimeoutException("Timeout occured", timout)
Sep 21 2014
prev sibling parent reply Mike Parker <aldacron gmail.com> writes:
On 9/20/2014 1:43 PM, Andrei Alexandrescu wrote:
 On 9/19/14, 7:20 PM, Walter Bright wrote:
 On 9/19/2014 6:48 PM, Andrei Alexandrescu wrote:
 On 9/19/14, 6:18 PM, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization. It has to be
 done
 in such a way that the ARC and the existing exceptions can coexist.
Could you please elaborate why the disaster? -- Andrei
1. Every library for D will have to come in two versions. One which works, and the other is likely untested. This will divide the D community in half.
I understand where you're coming from. And you're right that we're looking at an important change - as big as -m64 vs. -m32 or porting to a new platform. It's big. But it's needed, soon. And it would be, it seems to me, a mistake to approach this big change as a small change that we can sneak in. This judgment - that RC vs. GC would balkanize the community - has become a prejudice that is worth revisiting. Just trotting it out again won't do.
Unless I'm missing something, Walter is referring specifically to the -nogc switch (or any such switch) and not to the concept of RC vs GC in general. And on that I am in strong agreement, FWIW. Recall back during the D1/D2, Phobos/Tango period, library maintainers had to either choose which D version and library to support (D1/Phobos, D1/Tango, D2/Phobos) or come up with a means of supporting all of them. Wrapper modules and mixins were a workable (though terribly annoying) solution for some cases, but not all. It was a nasty mess. I don't know yet what the ultimate impact of a -nogc switch would be on library maintenance, but I have a strong suspicion that simple wrapper modules and mixins wouldn't be enough to maintain compatibility. If there is even the slightest possibility that library maintainers will find themselves in that same situation of having to choose a or b because maintaining both is too troublesome to bother, then all the implications need to be hashed out beforehand. --- This email is free from viruses and malware because avast! Antivirus protection is active. http://www.avast.com
Sep 20 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:48 AM, Mike Parker wrote:
 I don't know yet what the ultimate impact of a -nogc switch would be on
 library maintenance, but I have a strong suspicion that simple wrapper
 modules and mixins wouldn't be enough to maintain compatibility. If
 there is even the slightest possibility that library maintainers will
 find themselves in that same situation of having to choose a or b
 because maintaining both is too troublesome to bother, then all the
 implications need to be hashed out beforehand.
We need to explore that. A possibility is to support coexistence and then have the option to use a tool statically pinpoint the uses of GC. -- Andrei
Sep 20 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei 
Alexandrescu wrote:
 We need to explore that. A possibility is to support 
 coexistence and then have the option to use a tool statically 
 pinpoint the uses of GC. -- Andrei
What, *exactly*, does "uses of GC" mean? In other words, what specifically makes GC.malloc evil that must be avoided at any cost while C malloc (+ GC.addRange most likely) is an acceptable replacement? A few things that come to mind are: 1) Obviously, GC.malloc can trigger a collection. But this can be easily disabled. 2) The GC lock? I don't know how malloc handles this though. 3) Is GC.free substantially different than C's free? 4) Programmers don't typically explicitly free GC memory... but we could. 5) Bookkeeping overhead? I know malloc has some too though, is this really a dealbreaker? 6) Public relations. ...that's all I can think of. What am I missing? Which one of these is actually causing the problem that we're supposed to be fixing here?
Sep 20 2014
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 9:31 AM, Adam D. Ruppe wrote:
 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu wrote:
 We need to explore that. A possibility is to support coexistence and
 then have the option to use a tool statically pinpoint the uses of GC.
 -- Andrei
What, *exactly*, does "uses of GC" mean? In other words, what specifically makes GC.malloc evil that must be avoided at any cost while C malloc (+ GC.addRange most likely) is an acceptable replacement? A few things that come to mind are: 1) Obviously, GC.malloc can trigger a collection. But this can be easily disabled. 2) The GC lock? I don't know how malloc handles this though. 3) Is GC.free substantially different than C's free? 4) Programmers don't typically explicitly free GC memory... but we could. 5) Bookkeeping overhead? I know malloc has some too though, is this really a dealbreaker? 6) Public relations. ....that's all I can think of. What am I missing? Which one of these is actually causing the problem that we're supposed to be fixing here?
The only problem is that GC.malloc doesn't need to be paired by a call to GC.free, whereas malloc must. Andrei
Sep 20 2014
parent reply Johannes Pfau <nospam example.com> writes:
Am Sat, 20 Sep 2014 10:17:06 -0700
schrieb Andrei Alexandrescu <SeeWebsiteForEmail erdani.org>:

 On 9/20/14, 9:31 AM, Adam D. Ruppe wrote:
 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu
 wrote:
 We need to explore that. A possibility is to support coexistence
 and then have the option to use a tool statically pinpoint the
 uses of GC. -- Andrei
What, *exactly*, does "uses of GC" mean? In other words, what specifically makes GC.malloc evil that must be avoided at any cost while C malloc (+ GC.addRange most likely) is an acceptable replacement? A few things that come to mind are: 1) Obviously, GC.malloc can trigger a collection. But this can be easily disabled. 2) The GC lock? I don't know how malloc handles this though. 3) Is GC.free substantially different than C's free? 4) Programmers don't typically explicitly free GC memory... but we could. 5) Bookkeeping overhead? I know malloc has some too though, is this really a dealbreaker? 6) Public relations. ....that's all I can think of. What am I missing? Which one of these is actually causing the problem that we're supposed to be fixing here?
The only problem is that GC.malloc doesn't need to be paired by a call to GC.free, whereas malloc must. Andrei
You're both missing another obvious point: 7) The GC needs to be implemented in D, in the druntime. Malloc is available everywhere. The GC keeps a separate heap IIRC which wastes memory if you mix D code and C code which uses malloc. Both are important points for embedded systems. A druntime without GC is much easier to port and produces smaller binaries. of course in that case, GC.addRange is not an acceptable replacement. Here 'use of GC' literally means use of GC ;-)
Sep 21 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 09:07:58 UTC, Johannes Pfau wrote:
 7) The GC needs to be implemented in D, in the druntime.
Eh, it is easy to write a stub function that just calls malloc. The main problem with that is what Andrei said: most code won't call free since it doesn't have to, so it would leak in those situations. But if you're writing code for this situation, it is easy enough to call free; you don't *have* to, but you *can* (unless it is safe code, then you can't).
Sep 21 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:31, Adam D. Ruppe wrote:
 On Saturday, 20 September 2014 at 16:15:45 UTC, Andrei Alexandrescu wrote:
 We need to explore that. A possibility is to support coexistence and
 then have the option to use a tool statically pinpoint the uses of GC.
 -- Andrei
What, *exactly*, does "uses of GC" mean? In other words, what specifically makes GC.malloc evil that must be avoided at any cost while C malloc (+ GC.addRange most likely) is an acceptable replacement? A few things that come to mind are: 1) Obviously, GC.malloc can trigger a collection. But this can be easily disabled. 2) The GC lock? I don't know how malloc handles this though. 3) Is GC.free substantially different than C's free? 4) Programmers don't typically explicitly free GC memory... but we could. 5) Bookkeeping overhead? I know malloc has some too though, is this really a dealbreaker? 6) Public relations. ...that's all I can think of. What am I missing? Which one of these is actually causing the problem that we're supposed to be fixing here?
I think it's mostly 1) and then on second place 2). -- /Jacob Carlborg
Sep 21 2014
prev sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 12:48 AM, Mike Parker wrote:
 I don't know yet what the ultimate impact of a -nogc switch would be on
 library maintenance, but I have a strong suspicion that simple wrapper
 modules and mixins wouldn't be enough to maintain compatibility. If
 there is even the slightest possibility that library maintainers will
 find themselves in that same situation of having to choose a or b
 because maintaining both is too troublesome to bother, then all the
 implications need to be hashed out beforehand.
Agreed. -- Andrei
Sep 20 2014
prev sibling parent reply "Vladimir Panteleev" <vladimir thecybershadow.net> writes:
On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright 
wrote:
 Having a compiler switch to change the behavior of every module 
 in incompatible ways would be a disastrous balkanization.
Don't -version, -debug etc. have this problem anyway? Anyway, if you change the name mangling, then you'll get link errors instead of mysterious crashes.
 It has to be done in such a way that the ARC and the existing 
 exceptions can coexist.
How?
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:
 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization.
Don't -version, -debug etc. have this problem anyway?
I submit that it is a poor practice to write code that way.
 Anyway, if you change the name mangling, then you'll get link errors instead of
 mysterious crashes.

 It has to be done in such a way that the ARC and the existing exceptions can
 coexist.
How?
Good question. It's a challenge. But it has to be done, or D will divide in half and both halves will fail. We've had this discussion numerous times before - "throw the magic compiler switch" and D becomes an ARC system and everything is wonderful. It cannot work. ARC and GC are not equivalent.
Sep 19 2014
next sibling parent reply "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright 
wrote:
 Good question. It's a challenge. But it has to be done, or D 
 will divide in half and both halves will fail.


 We've had this discussion numerous times before - "throw the 
 magic compiler switch" and D becomes an ARC system and 
 everything is wonderful. It cannot work. ARC and GC are not 
 equivalent.
I basically agree with Walter on this one, no switch please, it's maintenance nightmare for library devs. My proposal would be to permanently use ARC for Throwable, no flags. What does the GC bring to exceptions that makes it sufficiently invaluable to warrant two parallel implementations? It can't be about performance, since _thrown_ exceptions are already in the slow path... Backwards compatibility?
Sep 19 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 8:13 PM, Daniel N wrote:
 What does the GC bring to exceptions that makes it sufficiently invaluable to
 warrant two parallel implementations? It can't be about performance, since
 _thrown_ exceptions are already in the slow path... Backwards compatibility?
Backwards compatibility.
Sep 19 2014
parent reply "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 03:25:39 UTC, Walter Bright 
wrote:
 On 9/19/2014 8:13 PM, Daniel N wrote:
 What does the GC bring to exceptions that makes it 
 sufficiently invaluable to
 warrant two parallel implementations? It can't be about 
 performance, since
 _thrown_ exceptions are already in the slow path... Backwards 
 compatibility?
Backwards compatibility.
Thanks! As much as I applaud our efforts for Backwards compatibility, it only affects code which already has been written, whereas a switch would affect "all" code that will be written, imho the burden both on library and app devs would be less if we simply redefined Throwable.
Sep 19 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:45 PM, Daniel N wrote:
 As much as I applaud our efforts for Backwards compatibility, it only
 affects code which already has been written, whereas a switch would
 affect "all" code that will be written, imho the burden both on library
 and app devs would be less if we simply redefined Throwable.
We need -nogc in the same way we need -m32 and -m64. -- Andrei
Sep 19 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:13 PM, Daniel N wrote:
 On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright wrote:
 Good question. It's a challenge. But it has to be done, or D will
 divide in half and both halves will fail.


 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.
I basically agree with Walter on this one, no switch please, it's maintenance nightmare for library devs.
Why is it a maintenance nightmare?
 My proposal would be to permanently use ARC for Throwable, no flags.
How about other objects? Throwable is but the first step, and a good one to inform larger designs. We can't force RC on all objects on all applications.
 What does the GC bring to exceptions that makes it sufficiently
 invaluable to warrant two parallel implementations? It can't be about
 performance, since _thrown_ exceptions are already in the slow path...
 Backwards compatibility?
Conversion from Throwable to Object. Andrei
Sep 19 2014
parent "Daniel N" <ufo orbiting.us> writes:
On Saturday, 20 September 2014 at 04:48:33 UTC, Andrei
Alexandrescu wrote:
 I basically agree with Walter on this one, no switch please, 
 it's
 maintenance nightmare for library devs.
Why is it a maintenance nightmare?
Sorry, let's defer this, I might have overstated the negative side-effects before carefully considering methods to mitigate the impact.
 My proposal would be to permanently use ARC for Throwable, no 
 flags.
How about other objects? Throwable is but the first step, and a good one to inform larger designs. We can't force RC on all objects on all applications.
I was viewing everything from a totally different perspective: Normally what prevents an API to be nogc is exceptions, even _emplace_ falls victim of this, in my line of reasoning I was counting on the ripple effect to get us quite far, the remaining issues could be fixed by refining the API to make it possible to avoid allocations if desired. However, since you had a different goal in mind, your solution starts making more sense to me, it's also very easy to flip a switch and benchmark individual applications, maybe people from both camps will be surprised at the results.
 What does the GC bring to exceptions that makes it sufficiently
 invaluable to warrant two parallel implementations? It can't 
 be about
 performance, since _thrown_ exceptions are already in the slow 
 path...
 Backwards compatibility?
Conversion from Throwable to Object. Andrei
The conversion could allocate on the GC, which would be functionally backwards compatible and if the performance degradation is a concern the code could be updated to avoid the superfluous allocation by staying in the RCObject hierarchy. Daniel
Sep 20 2014
prev sibling next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 7:26 PM, Walter Bright wrote:
 On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:
 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright wrote:
 Having a compiler switch to change the behavior of every module in
 incompatible ways would be a disastrous balkanization.
Don't -version, -debug etc. have this problem anyway?
I submit that it is a poor practice to write code that way.
 Anyway, if you change the name mangling, then you'll get link errors
 instead of
 mysterious crashes.

 It has to be done in such a way that the ARC and the existing
 exceptions can
 coexist.
How?
Good question. It's a challenge. But it has to be done, or D will divide in half and both halves will fail.
I don't think there will be division, and I don't think the parts will fail. I see it as enrichment. We've said for years that D is usable without a GC. The time has come to make that into reality. To the extent it's automatic, we'll make it automatic. To the extent we'll require adjustment in use, we'll ask users to adjust use. The worst thing about that is to enter the design with a prejudice tied to our ankles.
 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.
I think past discussions have been inconclusive at best and must be reopened. Andrei
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 9:47 PM, Andrei Alexandrescu wrote:
 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.
I think past discussions have been inconclusive at best and must be reopened.
For starters, 1. ARC is slow and generates bloated code. 2. To deal with (1), existing ARC systems allow escapes from it. This has severe implications for memory safety. Rust's entire type system appears to be designed around trying to deal with this, and AFAIK they're the only ones who have tried. C++ shared_ptr and ObjectiveC's ARC are not memory safe. They are not. Not not not, and they don't even try. :-) 3. ARC objects require an embedded count. This means they cannot be mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC behaves, and we cannot pretend or wish this away or add a compiler switch to make it go away. None of these points were addressed in previous discussions beyond the supposition of some nonexistent compiler technology that would make them go away. D can have ref counted objects, but it will not work with a compiler switch to switch them back and forth. They'll have to coexist peacefully with GC objects.
Sep 19 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 05:24:11 UTC, Walter Bright 
wrote:
 D can have ref counted objects, but it will not work with a 
 compiler switch to switch them back and forth. They'll have to 
 coexist peacefully with GC objects.
If you don't expose the refcount, write libraries as if you have only RC, and don't expose destructors then you probably can. Just turn inc/dec into dummies.
Sep 19 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 10:32 PM, Ola Fosheim Grostad wrote:
 On Saturday, 20 September 2014 at 05:24:11 UTC, Walter Bright wrote:
 D can have ref counted objects, but it will not work with a compiler switch to
 switch them back and forth. They'll have to coexist peacefully with GC objects.
If you don't expose the refcount, write libraries as if you have only RC, and don't expose destructors then you probably can. Just turn inc/dec into dummies.
This doesn't address any of the 3 concerns.
Sep 19 2014
parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 06:01:19 UTC, Walter Bright 
wrote:
 This doesn't address any of the 3 concerns.
1. RC efficiency is architecture dependent. E.g. TSX is coming even if there is a bug in lower end CPUs. Suggest making performance oriented prototypes on different architectures before concluding. 2. As long as you have a RC pointer to an obj on the stack you can switch to regular references. No magic involved for the easy case, just semantic analysis. 3. True, but you can keep the refcount at a negative offset for new-based allocations. Besides it only affects those who do nogc and they should know what they are doing.
Sep 20 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 12:17 AM, Ola Fosheim Grostad wrote:
 On Saturday, 20 September 2014 at 06:01:19 UTC, Walter Bright wrote:
 This doesn't address any of the 3 concerns.
1. RC efficiency is architecture dependent.
Please show me the efficient assembler for the x86.
 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented prototypes on
 different architectures before concluding.
I'd love to see it.
 2. As long as you have a RC pointer to an obj on the stack you can switch to
 regular references. No magic involved for the easy case, just semantic
analysis.
As soon as you pass a reference to a function, that all goes out the window. There's a reason why Rust has invested so much effort in the notion of a "borrowed" pointer.
 3. True, but you can keep the refcount at a negative offset for new-based
 allocations. Besides it only affects those who do  nogc and they should know
 what they are doing.
If this is so simple, why doesn't everyone do it?
Sep 20 2014
parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 08:39:41 UTC, Walter Bright 
wrote:
 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented 
 prototypes on
 different architectures before concluding.
I'd love to see it.
A 128 bit CAS instruction is at about 19-25 cycles, but a transaction on the other hand can by using xbegin/xend cover both refcounting, locking and rollback of multiple objects so you need cooperation from code gen. Basically all changes between xbegin/xend are kept in cache and written to memory upon success. On failure you have a slower fallback. I don't know what that leads to in amortized cost reduction, but 30-70% might be possible if it is done right.
 As soon as you pass a reference to a function, that all goes 
 out the window. There's a reason why Rust has invested so much 
 effort in the notion of a "borrowed" pointer.
The pure nogc crowd care less about safety, but you should be able to track this using dataflow?
 3. True, but you can keep the refcount at a negative offset 
 for new-based
 allocations. Besides it only affects those who do  nogc and 
 they should know
 what they are doing.
If this is so simple, why doesn't everyone do it?
Never said performance and thread safe RC was easy. It is probably difficult to get below 10 cycles for inc/dec pairs even with excellent code gen...? And probably closer to 40 cycles for regular code gen. Just guessing.
Sep 20 2014
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 3:00 AM, Ola Fosheim Grostad wrote:
 On Saturday, 20 September 2014 at 08:39:41 UTC, Walter Bright wrote:
 E.g. TSX is coming even if there is
 a bug in lower end CPUs. Suggest making performance oriented prototypes on
 different architectures before concluding.
I'd love to see it.
A 128 bit CAS instruction is at about 19-25 cycles, but a transaction on the other hand can by using xbegin/xend cover both refcounting, locking and rollback of multiple objects so you need cooperation from code gen. Basically all changes between xbegin/xend are kept in cache and written to memory upon success. On failure you have a slower fallback. I don't know what that leads to in amortized cost reduction, but 30-70% might be possible if it is done right.
Please show me the inc/dec optimized x86 code.
 As soon as you pass a reference to a function, that all goes out the window.
 There's a reason why Rust has invested so much effort in the notion of a
 "borrowed" pointer.
The pure nogc crowd care less about safety, but you should be able to track this using dataflow?
Suppose I pass a pointer to: void foo(T* p); How do I do dataflow?
 3. True, but you can keep the refcount at a negative offset for new-based
 allocations. Besides it only affects those who do  nogc and they should know
 what they are doing.
If this is so simple, why doesn't everyone do it?
Never said performance and thread safe RC was easy. It is probably difficult to get below 10 cycles for inc/dec pairs even with excellent code gen...? And probably closer to 40 cycles for regular code gen. Just guessing.
The question was about mixing RC'd and non-RC'd objects.
Sep 20 2014
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
wrote:
 Please show me the inc/dec optimized x86 code.
There's no optimization of the inc/dec, you use regular inc/dec within a larger xbegin/xend block. You amortize the cost over all the lock-prefixed instructions you would otherwise have executed in that block. There is no locking, there is a transaction instead. Only if the transaction fails do you execute the locking fallback code.
 Suppose I pass a pointer to:

    void foo(T* p);

 How do I do dataflow?
You mean a separate compilation unit that the compiler don't have access to? You don't, obviously.
 The question was about mixing RC'd and non-RC'd objects.
Well, if you are talking about negative offset for ref counts, then people do it. E.g. there are examples of systems that provide C-compatible strings with length at offset -4 and ref count at -8. If you need to distinguish you can either use address-space info or keep a negative int in the non-RC'd object.
Sep 20 2014
next sibling parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 18:26:56 UTC, Ola Fosheim 
Grøstad wrote:
 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
 wrote:
 Please show me the inc/dec optimized x86 code.
There's no optimization of the inc/dec, you use regular inc/dec within a larger xbegin/xend block.
Actually you dont need to inc/dec if you do both in the transaction. You only need to read the refcount. If somebody else writes to the ref count the transaction fails. So, you cannot succeed a transaction while somebody else decrease the ref count. Sounds right?
Sep 20 2014
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 11:26 AM, "Ola Fosheim Grøstad" 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright wrote:
 Please show me the inc/dec optimized x86 code.
There's no optimization of the inc/dec, you use regular inc/dec within a larger xbegin/xend block. You amortize the cost over all the lock-prefixed instructions you would otherwise have executed in that block. There is no locking, there is a transaction instead. Only if the transaction fails do you execute the locking fallback code.
I strongly suggest taking a look at C++ shared_ptr<T>, compile a simple example, and examine the generated code. It is NOT AT ALL as simple as emitting an inc/dec pair. I.e. what happens if an exception is thrown between the inc and the dec? If RC was as costless and simple as you are suggesting, everyone would be doing it and would have for decades.
 Suppose I pass a pointer to:

    void foo(T* p);

 How do I do dataflow?
You mean a separate compilation unit that the compiler don't have access to? You don't, obviously.
Right. "dataflow" is not the answer.
 The question was about mixing RC'd and non-RC'd objects.
Well, if you are talking about negative offset for ref counts, then people do it. E.g. there are examples of systems that provide C-compatible strings with length at offset -4 and ref count at -8.
Now embed that object as a field in some other object.
 If you need to distinguish you can either use address-space info or keep a
 negative int in the non-RC'd object.
Now add that runtime distinguishing logic to the inc/dec code.
Sep 20 2014
parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 22:07:45 UTC, Walter Bright 
wrote:
 On 9/20/2014 11:26 AM, "Ola Fosheim Grøstad" 
 <ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Saturday, 20 September 2014 at 18:18:05 UTC, Walter Bright 
 wrote:
 Please show me the inc/dec optimized x86 code.
There's no optimization of the inc/dec, you use regular inc/dec within a larger xbegin/xend block. You amortize the cost over all the lock-prefixed instructions you would otherwise have executed in that block. There is no locking, there is a transaction instead. Only if the transaction fails do you execute the locking fallback code.
I strongly suggest taking a look at C++ shared_ptr<T>, compile a simple example, and examine the generated code. It is NOT AT ALL as simple as emitting an inc/dec pair. I.e. what happens if an exception is thrown between the inc and the dec?
C++ provides weak references. If you throw issue xabort. Nobody have said it is free, and TSX is new tech. So what you can do, who knows?
 If RC was as costless and simple as you are suggesting, 
 everyone would be doing it and would have for decades.
Transactions are not free.
 Now embed that object as a field in some other object.
Uhm, I referred to new-allocated objects.
 Now add that runtime distinguishing logic to the inc/dec code.
Test N flag?
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 10:24 PM, Walter Bright wrote:
 On 9/19/2014 9:47 PM, Andrei Alexandrescu wrote:
 We've had this discussion numerous times before - "throw the magic
 compiler switch" and D becomes an ARC system and everything is
 wonderful. It cannot work. ARC and GC are not equivalent.
I think past discussions have been inconclusive at best and must be reopened.
For starters, 1. ARC is slow and generates bloated code.
Agreed. It has the cost distributed a different way.
 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)
That's fine. You can't have everything.
 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.
In nogc mode, everything is ARC so I'm not getting this.
 None of these points were addressed in previous discussions beyond the
 supposition of some nonexistent compiler technology that would make them
 go away.
That was indeed a crappy argument, and not advanced by me. I'm not advocating for that.
 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.
We need to figure out a design. All I'm saying is we must not bring prejudice to the table. And a very basic point is: there will be a way to COMPLETELY disable the GC. That is a must. Andrei
Sep 19 2014
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 9/19/2014 10:46 PM, Andrei Alexandrescu wrote:
 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)
That's fine. You can't have everything.
You're willing to dispense with memory safety?
 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.
In nogc mode, everything is ARC so I'm not getting this.
Not even ARC languages try to make everything ARC - because it is BLOATED and SLOW. Not C++, not ObjectiveC, not Rust.
 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.
We need to figure out a design. All I'm saying is we must not bring prejudice to the table. And a very basic point is: there will be a way to COMPLETELY disable the GC. That is a must.
It doesn't have to be disabled. Just don't call it. The GC is not a bridge troll that is going to pop up and extract a toll even if you aren't allocating with it.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 11:15 PM, Walter Bright wrote:
 On 9/19/2014 10:46 PM, Andrei Alexandrescu wrote:
 2. To deal with (1), existing ARC systems allow escapes from it. This
 has severe implications for memory safety. Rust's entire type system
 appears to be designed around trying to deal with this, and AFAIK
 they're the only ones who have tried. C++ shared_ptr and ObjectiveC's
 ARC are not memory safe. They are not. Not not not, and they don't even
 try. :-)
That's fine. You can't have everything.
You're willing to dispense with memory safety?
No.
 3. ARC objects require an embedded count. This means they cannot be
 mixed with non-ARC objects. This is fundamentally DIFFERENT from how GC
 behaves, and we cannot pretend or wish this away or add a compiler
 switch to make it go away.
In nogc mode, everything is ARC so I'm not getting this.
Not even ARC languages try to make everything ARC - because it is BLOATED and SLOW. Not C++, not ObjectiveC, not Rust.
Agreed.
 D can have ref counted objects, but it will not work with a compiler
 switch to switch them back and forth. They'll have to coexist peacefully
 with GC objects.
We need to figure out a design. All I'm saying is we must not bring prejudice to the table. And a very basic point is: there will be a way to COMPLETELY disable the GC. That is a must.
It doesn't have to be disabled. Just don't call it. The GC is not a bridge troll that is going to pop up and extract a toll even if you aren't allocating with it.
Well obviously there's a hierarchy of desirability: 1. Using the GC inadvertently causes a compile-time error. 2. Using the GC inadvertently causes a link-time error. 3. Using the GC inadvertently causes a run-time error. 4. Using the GC inadvertently goes undetected. I disagree that (4) is good for everyone. Andrei
Sep 20 2014
parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 8:59 AM, Andrei Alexandrescu wrote:
 4. Using the GC inadvertently goes undetected.

 I disagree that (4) is good for everyone.
We've already implemented nogc.
Sep 20 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 05:46:44 UTC, Andrei 
Alexandrescu wrote:
 D can have ref counted objects, but it will not work with a 
 compiler
 switch to switch them back and forth. They'll have to coexist 
 peacefully
 with GC objects.
We need to figure out a design. All I'm saying is we must not bring prejudice to the table. And a very basic point is: there will be a way to COMPLETELY disable the GC. That is a must.
The ref counted objects still need to be considered as roots for the GC. This limits how much "completely" can be achieved.
Sep 20 2014
prev sibling parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 02:26:49 UTC, Walter Bright 
wrote:
 On 9/19/2014 7:04 PM, Vladimir Panteleev wrote:
 On Saturday, 20 September 2014 at 01:19:05 UTC, Walter Bright 
 wrote:
 Having a compiler switch to change the behavior of every 
 module in
 incompatible ways would be a disastrous balkanization.
Don't -version, -debug etc. have this problem anyway?
I submit that it is a poor practice to write code that way.
 Anyway, if you change the name mangling, then you'll get link 
 errors instead of
 mysterious crashes.

 It has to be done in such a way that the ARC and the existing 
 exceptions can
 coexist.
How?
Good question. It's a challenge. But it has to be done, or D will divide in half and both halves will fail. We've had this discussion numerous times before - "throw the magic compiler switch" and D becomes an ARC system and everything is wonderful. It cannot work. ARC and GC are not equivalent.
I am fully with Walter here. Only way to do it in acceptable way is to make both reference counted exceptions and current ones co-exist. Anything else means that we will need to distribute 4 builds of Phobos - debug-nogc, debug-gc, release-nogc, release-gc. Guess you can smell exponential explosion here as well as I do.
Sep 20 2014
prev sibling next sibling parent reply ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 19 Sep 2014 08:32:38 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 First, there must be some compiler flag -nogc or something, which=20
please, no compiler switches! it's partly ok for "-release", it's not very good for "-version" and it's awful to add another one which completely alters the way something works. as much as i want to make throwables nogc, it's better to change nothing than to add such switch.
Sep 19 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/19/14, 8:30 PM, ketmar via Digitalmars-d wrote:
 On Fri, 19 Sep 2014 08:32:38 -0700
 Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
 wrote:

 First, there must be some compiler flag -nogc or something, which
please, no compiler switches! it's partly ok for "-release", it's not very good for "-version" and it's awful to add another one which completely alters the way something works. as much as i want to make throwables nogc, it's better to change nothing than to add such switch.
Changing nothing is not an option. Ideas? -- Andrei
Sep 19 2014
next sibling parent ketmar via Digitalmars-d <digitalmars-d puremagic.com> writes:
On Fri, 19 Sep 2014 21:53:32 -0700
Andrei Alexandrescu via Digitalmars-d <digitalmars-d puremagic.com>
wrote:

 Changing nothing is not an option. Ideas? -- Andrei
let compiler do it's ARC magic on Throwable and allow programmer to plug in his own allocator. and drop out GC mechanics for Throwables entirely (except scanning Throwable instance for GC-managed pointers).
Sep 19 2014
prev sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 04:53:32 UTC, Andrei 
Alexandrescu wrote:
 On 9/19/14, 8:30 PM, ketmar via Digitalmars-d wrote:
 On Fri, 19 Sep 2014 08:32:38 -0700
 Andrei Alexandrescu via Digitalmars-d 
 <digitalmars-d puremagic.com>
 wrote:

 First, there must be some compiler flag -nogc or something, 
 which
please, no compiler switches! it's partly ok for "-release", it's not very good for "-version" and it's awful to add another one which completely alters the way something works. as much as i want to make throwables nogc, it's better to change nothing than to add such switch.
Changing nothing is not an option. Ideas? -- Andrei
I have been thinking about something like `ARC` interface and `ARCException : Throwable, ARC` with all the automatic ref inc/dec calls you have mentioned added for any class that implements `ARC` interface.
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 07:29:52 UTC, Dicebot wrote:
 I have been thinking about something like `ARC` interface and 
 `ARCException : Throwable, ARC` with all the automatic ref 
 inc/dec calls you have mentioned added for any class that 
 implements `ARC` interface.
..and yes, I'd expect it to be the default one used by Phobos (possibly with some backing pool allocator). It isn't critical for GC-enabled applications to use GC-based exceptions but contrary is true.
Sep 20 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Consider:

https://github.com/D-Programming-Language/phobos/blob/master/std/c/windows/com.d#L218

This is D's support for COM objects. They have AddRef() and Release(), i.e.
they 
are reference counted. They do not inherit from Object. The refcounting is 
currently manual.

D could allow an additional root (not a replacement) for thrown Exceptions.
This 
root would be based on ComObject.

As a bonus, COM objects were designed for C++, and C++ implements them well, 
i.e. they would be a compatible object structure.

catch statements could be changed to catch ComThrowable, which would not catch 
Throwables, and catches for Throwables would not catch ComThrowables.
Sep 19 2014
prev sibling next sibling parent Dmitry Olshansky <dmitry.olsh gmail.com> writes:
19-Sep-2014 19:32, Andrei Alexandrescu пишет:
 As discussed, having exception objects being GC-allocated is clearly a
 large liability that we need to address. They prevent otherwise careful
 functions from being  nogc so they affect even apps that otherwise would
 be okay with a little litter here and there.

 The Throwable hierarchy is somewhat separate from everything else, which
 makes it a great starting point for investigating an automated reference
 count approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, which
 triggers the RC exceptions. All modules of an application must be
 compiled with this flag if it is to work (such that one module can throw
 an exception caught by the other). Of course a lot of refinement needs
 to be added here (what happens if one tries to link modules built with
 and without -nogc, allowing people to detect the flag programmatically
 by using version(nogc) etc).

 If -nogc is passed, the compiler severs the inheritance relationship
 between Throwable and Object, making it impossible to convert a
 Throwable to an Object. From then henceforth, Throwable and Object form
 a two-rooted forest. (In all likelihood we'll later add an RCObject root
 that Throwable inherits.)

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.) Implementation of these is up to the runtime library. Null
 checking may be left to either the compiler or the library (in the
 latter case, the functions must be nonmember). However it seems the
 compiler may have an advantage because it can elide some of these checks.

 Once we get this going, we should accumulate good experience that we can
 later apply to generalizing this approach to more objects. Also, if
 things go well we may as well define _always_ (whether GC or not)
 Throwable to be reference counted; seems like a good fit for all programs.
All good, it wasn't a good idea to have GC-ed exceptions in the first place (It's a truly rare thing to have cycles of exceptions). It doesn' t matter GC or nogc - this must go. Speaking of RC-ed Throwable - why adding a switch? Just make a transition in 2 steps - all exceptions inherit from RCObject (but stay essentially GC-ed), then hack compiler to perform ARC on RCObject descendants (maybe even with GC.malloc/GC.free for starters, i.e. the same heap). Then we can mess with custom allocation of these. I can understand a transitory switch, say -exception-ongc for the minority of folks that indeed DO rely on exceptions being GC and not counted. -- Dmitry Olshansky
Sep 20 2014
prev sibling next sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
I'm quite a noobie in memory models but from position of user of 
D language I have some ideas about syntax of switching between 
memory models. I think that having everywhere declarations using 
wrapper structs (like RefCounted) is not very user-friendly. But 
still we need some way to say to compiler how we want to do 
memory management. I have read tutorial about Rust language. It 
looks not very clear for me, but I like the idea that type of 
memory management is included in variable declaration.

Intead of wrapper struct like scoped!A or RefCounted!A we could 
declare variables with annotations (attributes in D) to say what 
memory model we want to use. Compiler should understand this 
annotation and create some code for memory management. I thing 
it's possible to declare some interface (maybe struct with 
compile-time duck typing) to support some user-defined memory 
models or modifications of basic (implemented in a language) 
memory models.

Why I am saying about annotations? Because we can annotate some 
function, class or just block at the module scope that should use 
some sort of memory management and compiler will create 
corresponding code. In that case we don't need to put wrapper 
struct around all variables that use ref counting. We just 
annotate some function or class declaration as ref-counted and 
that's it!

It's just a concept of what I like to see in the language design 
of D in future)) Of course there are a lot of problems in a 
practice.

Also I think that we shouldn't impose some way of memory model to 
programmer and let him choose his approach but syntax should be 
simple and clear as much as it could be. Also some defaults 
should be for most common cases and for *novice* users of D.

Waiting for critic or thoughts!!!))
Sep 20 2014
next sibling parent reply "Uranuz" <neuranuz gmail.com> writes:
Also it's interesting waht is the correspondence between memory 
management, memory model and allocators? I know that 
std.allocator is in development. I guess that it should be 
considered in complex.
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 1:46 AM, Uranuz wrote:
 Also it's interesting waht is the correspondence between memory
 management, memory model and allocators? I know that std.allocator is in
 development. I guess that it should be considered in complex.
1. Memory management: when are calls to allocation and deallocation primitives made? 2. Allocators: what are the allocation and deallocation primitives, and how are they implemented? 3. Memory model: how is memory effected by programs? (of interest for compiler implementers and thread safety) There is occasional confusion among the three. I myself was in the woods for a while thinking that allocators should do memory management. Andrei
Sep 20 2014
prev sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 08:42:51 UTC, Uranuz wrote:
 I'm quite a noobie in memory models but from position of user 
 of D language I have some ideas about syntax of switching 
 between memory models. I think that having everywhere 
 declarations using wrapper structs (like RefCounted) is not 
 very user-friendly. But still we need some way to say to 
 compiler how we want to do memory management. I have read 
 tutorial about Rust language. It looks not very clear for me, 
 but I like the idea that type of memory management is included 
 in variable declaration.

 Intead of wrapper struct like scoped!A or RefCounted!A we could 
 declare variables with annotations (attributes in D) to say 
 what memory model we want to use. Compiler should understand 
 this annotation and create some code for memory management. I 
 thing it's possible to declare some interface (maybe struct 
 with compile-time duck typing) to support some user-defined 
 memory models or modifications of basic (implemented in a 
 language) memory models.

 Why I am saying about annotations? Because we can annotate some 
 function, class or just block at the module scope that should 
 use some sort of memory management and compiler will create 
 corresponding code. In that case we don't need to put wrapper 
 struct around all variables that use ref counting. We just 
 annotate some function or class declaration as ref-counted and 
 that's it!

 It's just a concept of what I like to see in the language 
 design of D in future)) Of course there are a lot of problems 
 in a practice.

 Also I think that we shouldn't impose some way of memory model 
 to programmer and let him choose his approach but syntax should 
 be simple and clear as much as it could be. Also some defaults 
 should be for most common cases and for *novice* users of D.

 Waiting for critic or thoughts!!!))
You are right that there is an intricate relationship between the various types of memory (better: ownership) management, allocators, ref-counting, uniqueness, moving etc, as you write in your other post. They interact in very specific ways. For this reason I don't think it is feasible or desirable to set the type of memory management "from the outside", the types in question need to know how their innards work. In the same vein, if your code uses a specific memory management strategy, it has to be written in a way conforming to it. Walter has already stated that: you cannot simply slap an attribute onto your code saying "please do reference counting", and expect it to work, especially not efficiently. I think we can get a lot further if we work out said relationships, and create good general purpose wrapper types that implement the different strategies accordingly. I believe, borrowing is crucial to this, which is why I made a proposal about it [1], which also deals with the other related topics. You do have a point about a simple, accessible syntax being important. Still, I don't see a big difference between annotations and RC!T wrappers in this regard (except that the former could apply to entire sections of code, which is not a good idea IMO). And I think with auto/scope/const type deduction, code is quite pleasant to read. [1] http://wiki.dlang.org/User:Schuetzm/scope
Sep 20 2014
prev sibling next sibling parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
Sep 20 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
But how would you chain them? And they have to implement Throwable's interface.
Sep 20 2014
parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Saturday, 20 September 2014 at 09:09:38 UTC, Marc Schütz wrote:
 On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
But how would you chain them?
Owning pointer to next exception.
 And they have to implement Throwable's interface.
One of their member would. Speculative design: struct ValueException { Unique!ValueException next; Unique!Throwable content; } Language would change to throw structs and catch them (without matching). At the end of a catch bloc, its destructor is called. This breaks polymorphic catch so it need sugar unfortunately.
Sep 20 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 20 September 2014 at 15:28:10 UTC, ponce wrote:
 On Saturday, 20 September 2014 at 09:09:38 UTC, Marc Schütz 
 wrote:
 On Saturday, 20 September 2014 at 09:05:15 UTC, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
But how would you chain them?
Owning pointer to next exception.
 And they have to implement Throwable's interface.
One of their member would. Speculative design: struct ValueException { Unique!ValueException next; Unique!Throwable content; } Language would change to throw structs and catch them (without matching). At the end of a catch bloc, its destructor is called. This breaks polymorphic catch so it need sugar unfortunately.
Ah, now I see, you just want a wrapper that contains a Throwable. I thought you wanted to throw _any_ value types, like in C++.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 2:05 AM, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
My knee-jerk reaction is we shouldn't. If considered, that would need to be well argued. We've derived good benefits from using polymorphic exceptions. -- Andrei
Sep 20 2014
parent reply "ponce" <contact gam3sfrommars.fr> writes:
On Saturday, 20 September 2014 at 16:41:20 UTC, Andrei 
Alexandrescu wrote:
 On 9/20/14, 2:05 AM, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
My knee-jerk reaction is we shouldn't. If considered, that would need to be well argued. We've derived good benefits from using polymorphic exceptions. -- Andrei
What do you think of the solution briefly outlined here: http://forum.dlang.org/post/iqenumncmczqvprwutmg forum.dlang.org
Sep 20 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 11:17 AM, ponce wrote:
 On Saturday, 20 September 2014 at 16:41:20 UTC, Andrei Alexandrescu wrote:
 On 9/20/14, 2:05 AM, ponce wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Coudln't we throw value types instead? (like in C++)
My knee-jerk reaction is we shouldn't. If considered, that would need to be well argued. We've derived good benefits from using polymorphic exceptions. -- Andrei
What do you think of the solution briefly outlined here: http://forum.dlang.org/post/iqenumncmczqvprwutmg forum.dlang.org
The short answer is I wouldn't advise pursuing it. -- Andrei
Sep 20 2014
prev sibling next sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.
Assuming this would eventually be implemented for regular classes, it would be nice if it could be made compatible with Objective-C ARC [1]. [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html -- /Jacob Carlborg
Sep 20 2014
next sibling parent reply "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg 
wrote:
 Assuming this would eventually be implemented for regular 
 classes, it would be nice if it could be made compatible with 
 Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html
Isn't Objective-C ARC known to be terribly efficient?
Sep 20 2014
parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 15:40, schrieb Ola Fosheim Grostad:
 On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg wrote:
 Assuming this would eventually be implemented for regular classes, it
 would be nice if it could be made compatible with Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html
Isn't Objective-C ARC known to be terribly efficient?
It requires compiler support, though. - Place retain/release at each place expected by Cocoa programming conventions, as if manually written - Do a second pass with data-flow analysis to remove retain/release pairs that don't escape current scope. If you have an Apple account search for: "Transitioning to ARC Release Notes", ARC "Memory Management Programming Guide for Core Foundation" - Cocoa conventions and allocators "WWDC 2012: Adopting Automatic Reference Counting" - Overral description "WWDC 2012: What's New in LLVM" - Some slides about ARC Optimizer -- Paulo
Sep 20 2014
next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Saturday, 20 September 2014 at 14:33:40 UTC, Paulo Pinto wrote:
 Am 20.09.2014 15:40, schrieb Ola Fosheim Grostad:
 On Saturday, 20 September 2014 at 12:27:23 UTC, Jacob Carlborg 
 wrote:
 Assuming this would eventually be implemented for regular 
 classes, it
 would be nice if it could be made compatible with Objective-C 
 ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html
Isn't Objective-C ARC known to be terribly efficient?
I'm really sorry, but that was a very confusing typo on my part. I meant "INefficient"! :-)
 It requires compiler support, though.
:-) Yes, but I was more talking about the representation in memory based on reading the source code that can be found in the repository. I read it the last time ARC was discussed… But, I am not 100% sure what code Apple actually ships.
Sep 20 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 16:33, Paulo Pinto wrote:

 It requires compiler support, though.
The first thing I asked in this thread was "Are you suggesting we implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei already wants to implement ARC. My definition of ARC is that the compiler inserts the calls to retain/release (or whatever you call them). [1] http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com -- /Jacob Carlborg
Sep 20 2014
next sibling parent reply Paulo Pinto <pjmlp progtools.org> writes:
Am 20.09.2014 17:08, schrieb Jacob Carlborg:
 On 2014-09-20 16:33, Paulo Pinto wrote:

 It requires compiler support, though.
The first thing I asked in this thread was "Are you suggesting we implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei already wants to implement ARC. My definition of ARC is that the compiler inserts the calls to retain/release (or whatever you call them). [1] http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com
I would say ARC == RC. I never saw a distinction in literature between both, before Apple used the term. With all these discussions I have been digging into old papers, and due to that I am starting to change my opinion. As it seems, RC with GC for cycle collection was more common than I thought of. In the end, what matters is having automatic memory management, be it via GC, RC or compiler dataflow analysis. From an outsider point of view, I just think that whatever the final outcome, it should be in a way that avoids the runtime episode from repeating itself. -- Paulo
Sep 20 2014
next sibling parent "Ola Fosheim Grostad" <ola.fosheim.grostad+dlang gmail.com> writes:
On Saturday, 20 September 2014 at 16:52:59 UTC, Paulo Pinto wrote:
 As it seems, RC with GC for cycle collection was more common 
 than I thought of.
I once read a paper that suggested optimizing GC by having RC on all objects and stop inc/dec ref counting when it reached 5 or something. I guess the idea was to get rid of short lived objects to make the GC run less often while cutting RC overhead.
Sep 20 2014
prev sibling parent Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:53, Paulo Pinto wrote:

 I would say ARC == RC. I never saw a distinction in literature between
 both, before Apple used the term.
I never saw the term ARC before Apple used it. I would say, ARC is a form of RC but RC doesn't not need imply ARC. BTW "Automatic Reference Counting" on Wikipedia [1] basically only talks about Apple's ARC. [1] http://en.wikipedia.org/wiki/Automatic_Reference_Counting -- /Jacob Carlborg
Sep 21 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:08 AM, Jacob Carlborg wrote:
 On 2014-09-20 16:33, Paulo Pinto wrote:

 It requires compiler support, though.
The first thing I asked in this thread was "Are you suggesting we implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei already wants to implement ARC. My definition of ARC is that the compiler inserts the calls to retain/release (or whatever you call them). [1] http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com
Please don't take me in a court of law. But yes, I am talking about the compiler inserting calls to increment and decrement reference counts. -- Andrei
Sep 20 2014
next sibling parent reply "Mike" <none none.com> writes:
On Saturday, 20 September 2014 at 16:56:32 UTC, Andrei 
Alexandrescu wrote:
 On 9/20/14, 8:08 AM, Jacob Carlborg wrote:
 On 2014-09-20 16:33, Paulo Pinto wrote:

 It requires compiler support, though.
The first thing I asked in this thread was "Are you suggesting we implement ARC?" and the answer was "Yes" [1]. So it looks like Andrei already wants to implement ARC. My definition of ARC is that the compiler inserts the calls to retain/release (or whatever you call them). [1] http://forum.dlang.org/thread/lvhiam$1pno$1 digitalmars.com#post-lvi0ve:2429il:241:40digitalmars.com
Please don't take me in a court of law. But yes, I am talking about the compiler inserting calls to increment and decrement reference counts. -- Andrei
I suggest the compiler insert calls to whatever "lifetime events" may cause an increment/decrement. Then, the druntime can provide the implementation. I believe with these runtime hooks in place platform-specific optimizations and even alternative memory managers can be experimented with. I refer you to an enlightening quote I once found on a raytracing site that really captures the beauty of software: "You know you've been raytracing too long when you've been asked how you did that thing you did by the author of the raytracer you used to do it." Just build the capability, and let the community surprise you with their ingenuity. Mike
Sep 20 2014
parent "Daniel N" <ufo orbiting.us> writes:
On Sunday, 21 September 2014 at 03:28:03 UTC, Mike wrote:
 I suggest the compiler insert calls to whatever "lifetime 
 events" may cause an increment/decrement.  Then, the druntime 
 can provide the implementation.

 I believe with these runtime hooks in place platform-specific 
 optimizations and even alternative memory managers can be 
 experimented with.
Your post gave me an idea already, so far we've only been focusing on Throwable, however... the solution for Error:s and Exception:s is actually asymmetric! For the vast majority of applications the best possible implementation for Error:s is to simply leak the memory, this is definitely not true for Exceptions(but it could be true for a small subset of applications, short-lived tools). Daniel
Sep 20 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei
We do need to know what you're proposal is for. How else can we comment on it? Paulo Pinto's comment "It requires compiler support, though" suggests, at least to me, he didn't understand you suggested ARC, i.e. RC with compiler support. -- /Jacob Carlborg
Sep 21 2014
next sibling parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 10:51, schrieb Jacob Carlborg:
 On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei
We do need to know what you're proposal is for. How else can we comment on it? Paulo Pinto's comment "It requires compiler support, though" suggests, at least to me, he didn't understand you suggested ARC, i.e. RC with compiler support.
Sorry but I guess I did understand it. There are only four ways of doing RC, regardless how you name them. - programmer writes manually calls to increment/decrement counters, like old time Cocoa and COM - library types which take advantage of operator overloading for increment/decrement operations, like *_ptr<>(), ComPtr<>() and RefCounted - automatic increment/decrement operations via special type or code pattern recognition => compiler support like Swift, Objective-C, C++/CLI, C++/CLX, Swift, Cedar, Modula-2+ - automatic increment/decrement operations via special type or code pattern recognition, followed by code removal of needless call pairs => compiler support like Objective-C and Swift So given Andrei's comment, compiler support is required. The only thing that distinguishes Apple solution from other ones, is that they made a marketing name for a common RC optimization. -- Paulo
Sep 21 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 1:51 AM, Jacob Carlborg wrote:
 On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei
We do need to know what you're proposal is for.
The title is an incomplete summary. Exceptions would be transparently reference counted. I.e. no changes in user code. A little more detail on a design Walter has put together: define a new root class RCObject. Neither Object nor RCObject inherit each other. All classes that inherit RCObject are detected by the compiler as reference counted. (For practical reasons, RCObject will implement IUnknown. There are a few random consequences of that (meaningless method QueryInterface, no class info, no monitor, COM-compatible layout) but IUnknown already has AddRef and Release and the compiler already recognizes IUnknown as a special case. Hooking RCObject there will leave us with two roots instead of adding a third one.) Throwable will inherit RCObject. The compiler inserts AddRef and Release whenever copying RCObject (and derivatives) references around, of course following a null check. The caveats discussed previously in http://goo.gl/Xl5U3z do apply. Andrei
Sep 21 2014
next sibling parent "Dicebot" <public dicebot.lv> writes:
On Sunday, 21 September 2014 at 15:03:09 UTC, Andrei Alexandrescu 
wrote:
 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:
 On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking 
 about the
 compiler inserting calls to increment and decrement reference 
 counts. --
 Andrei
We do need to know what you're proposal is for.
The title is an incomplete summary. Exceptions would be transparently reference counted. I.e. no changes in user code. A little more detail on a design Walter has put together: define a new root class RCObject. Neither Object nor RCObject inherit each other. All classes that inherit RCObject are detected by the compiler as reference counted. (For practical reasons, RCObject will implement IUnknown. There are a few random consequences of that (meaningless method QueryInterface, no class info, no monitor, COM-compatible layout) but IUnknown already has AddRef and Release and the compiler already recognizes IUnknown as a special case. Hooking RCObject there will leave us with two roots instead of adding a third one.) Throwable will inherit RCObject. The compiler inserts AddRef and Release whenever copying RCObject (and derivatives) references around, of course following a null check. The caveats discussed previously in http://goo.gl/Xl5U3z do apply. Andrei
This makes much more sense to me than initial proposal. I may even be brave enough to say that I like it :)
Sep 21 2014
prev sibling next sibling parent reply Andrej Mitrovic via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/21/14, Andrei Alexandrescu via Digitalmars-d
<digitalmars-d puremagic.com> wrote:
 (For practical reasons, RCObject will implement IUnknown.
IUnknown has this issue btw: https://issues.dlang.org/show_bug.cgi?id=12607
Sep 21 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 9:42 AM, Andrej Mitrovic via Digitalmars-d wrote:
 On 9/21/14, Andrei Alexandrescu via Digitalmars-d
 <digitalmars-d puremagic.com> wrote:
 (For practical reasons, RCObject will implement IUnknown.
IUnknown has this issue btw: https://issues.dlang.org/show_bug.cgi?id=12607
Thanks, good to know. IMHO having RCObject inherit IUnknown is more of a distraction than a benefit, but I'll let Walter be the judge of that. -- Andrei
Sep 21 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Sunday, 21 September 2014 at 18:10:53 UTC, Andrei Alexandrescu 
wrote:
 Thanks, good to know. IMHO having RCObject inherit IUnknown is 
 more of a distraction than a benefit, but I'll let Walter be 
 the judge of that. -- Andrei
Actually, I think it is a good idea to do the refcounting thing on COM objects anyway. I've used wrapper structs for that in the past, but if you don't do that right you can escape a reference which is freed, so putting it right on the object is kinda useful. Of course, IUnknown AND RCObject could be refcounted roots.
Sep 21 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
BTW hmmm what about this:

interface Foo { }

class Bar : Foo, RCObject {}

class Pwned : Foo {}

void main() {
    Foo bar = new Bar();
    /* where is bar.Release() called? */
    Foo pwned = new Pwned();
    /* better hope pwned.Release() isn't called cuz that's 
impossible */
}


I *believe* IUnknown interfaces are special and a separate 
category from regular D interfaces, so this would be statically 
disallowed.

But with the RCObject, we need to be careful about implicit 
casting to interfaces transparently killing memory safety.

of course then we might be weakening their usefulness. gah we 
just need scope references, then we can pass the interface with 
confidence that it won't be escaped. yeah yeah i know i sound 
like a broken record.
Sep 21 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 11:20 AM, Adam D. Ruppe wrote:
 BTW hmmm what about this:

 interface Foo { }

 class Bar : Foo, RCObject {}

 class Pwned : Foo {}

 void main() {
     Foo bar = new Bar();
     /* where is bar.Release() called? */
     Foo pwned = new Pwned();
     /* better hope pwned.Release() isn't called cuz that's impossible */
 }


 I *believe* IUnknown interfaces are special and a separate category from
 regular D interfaces, so this would be statically disallowed.

 But with the RCObject, we need to be careful about implicit casting to
 interfaces transparently killing memory safety.
Good point.
 of course then we might be weakening their usefulness. gah we just need
 scope references, then we can pass the interface with confidence that it
 won't be escaped. yeah yeah i know i sound like a broken record.
I don't think scope references are workable. I'll explain why in a latter post. Andrei
Sep 21 2014
prev sibling parent reply Dmitry Olshansky <dmitry.olsh gmail.com> writes:
21-Sep-2014 19:03, Andrei Alexandrescu пишет:
 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:
 On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference counts. --
 Andrei
We do need to know what you're proposal is for.
[snip]
 (For practical reasons, RCObject will implement IUnknown. There are a
 few random consequences of that (meaningless method QueryInterface, no
 class info, no monitor, COM-compatible layout) but IUnknown already has
 AddRef and Release and the compiler already recognizes IUnknown as a
 special case. Hooking RCObject there will leave us with two roots
 instead of adding a third one.)
This is actually a nice choice since COM would then (eventually) benefit from whatever automatic ref-counting we embed into the compiler. This is exactly what Visual C++ does now for new style COM object (inherited from IInspectable). -- Dmitry Olshansky
Sep 21 2014
parent Paulo Pinto <pjmlp progtools.org> writes:
Am 21.09.2014 20:17, schrieb Dmitry Olshansky:
 21-Sep-2014 19:03, Andrei Alexandrescu пишет:
 On 9/21/14, 1:51 AM, Jacob Carlborg wrote:
 On 2014-09-20 18:56, Andrei Alexandrescu wrote:

 Please don't take me in a court of law. But yes, I am talking about the
 compiler inserting calls to increment and decrement reference
 counts. --
 Andrei
We do need to know what you're proposal is for.
[snip]
 (For practical reasons, RCObject will implement IUnknown. There are a
 few random consequences of that (meaningless method QueryInterface, no
 class info, no monitor, COM-compatible layout) but IUnknown already has
 AddRef and Release and the compiler already recognizes IUnknown as a
 special case. Hooking RCObject there will leave us with two roots
 instead of adding a third one.)
This is actually a nice choice since COM would then (eventually) benefit from whatever automatic ref-counting we embed into the compiler. This is exactly what Visual C++ does now for new style COM object (inherited from IInspectable).
Specially important since all new Windows API since XP are mostly COM based. -- Paulo
Sep 21 2014
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 5:27 AM, Jacob Carlborg wrote:
 Assuming this would eventually be implemented for regular classes, it would be
 nice if it could be made compatible with Objective-C ARC [1].

 [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html
There was a long thread about that a year back or so. It wasn't looking good.
Sep 20 2014
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 9/20/2014 5:27 AM, Jacob Carlborg wrote:
 On 2014-09-19 17:32, Andrei Alexandrescu wrote:

 Whenever a reference to a Throwable is copied about, passed to
 functions, the compiler inserts appropriately calls to e.g. incRef and
 decRef. (Compiler may assume they cancel each other for optimization
 purposes.
Assuming this would eventually be implemented for regular classes, it would be nice if it could be made compatible with Objective-C ARC [1]. [1] http://clang.llvm.org/docs/AutomaticReferenceCounting.html
http://www.digitalmars.com/d/archives/digitalmars/D/draft_proposal_for_ref_counting_in_D_211885.html
Sep 20 2014
prev sibling next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
How often do you store an exception reference anyway that escapes 
a catch block? I think all this talk is overkill to solve a 
non-problem in 99% of practice.


Correct me if I'm wrong, but aren't *all* exceptions in a 
particular thread generally unreferenced at the end of a catch() 
block, unless the programmer explicitly escaped that reference?

If so, we don't need refcounting! Here's my solution:

1) Throwables are allocated in a separate thread-local memory 
pool than most other objects. The  nogc version just aborts the 
program if this pool ever runs out of memory. The gc version can 
do a collection cycle and grow the size if necessary. (Though if 
you need a lot of Throwable objects alive at once, I question wtf 
is up with your code...)

2) class Throwable adds a method, gcClone, which copies it to the 
regular GC heap for those cases when you do want to store it or 
pass it between threads or whatever.

3) A function,  system void clearExceptions() is added. You may 
manually call this when you are done handling exceptions in this 
thread.

4) Might also have  safe void prepareExceptions() which 
pre-allocates the pool. This could also be done automatically or 
on demand or whatever.



Advantages:

* Throwables are still GC managed as far as most code is 
concerned. The only time you need caution is if you call the new 
 system function to free the pool... and even then, you just make 
sure you don't escape those references before you do.

* If you want to avoid the GC, just manually clear your 
exceptions from time to time.

* Allocating/deallocating from this specialized memory pool is 
prolly faster than any other scheme anyway.

* No language changes required, this is all library stuff.

Disadvantages:

* There's a few manual steps to get all the benefit. (We could 
insert a call to clearExceptions at the end of catch blocks 
automatically, but without a static check to ensure the objects 
actually haven't escaped, this would break memory safety. But a 
manual call isn't that big of a deal)

* If you do it wrong, you'll be annoyed.

* ???



I really think this is a win. I've done a proof of concept before 
by hacking _d_newclass to use the pool based on the TypeInfo 
passed in, gives a speedup in all the simple cases I tried. But 
since that hack still uses new, it wouldn't pass the  nogc test. 
However, a library function like emplace could be  nogc and do 
the same thing. Since  nogc is an addition to the function, 
you'll be modifying it anyway, so changing "throw new" to 
whatever the new lib function is called can be done at the same 
time.

It is the catch code that have to worry about freeing the 
pool.... and even then, only if you want to avoid the GC. Normal 
code can just let the pool be collected when it is collected.

I think I'll write a little module we can play with.
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
wrote:
 How often do you store an exception reference anyway that 
 escapes a catch block? I think all this talk is overkill to 
 solve a non-problem in 99% of practice.
Pretty much any time you do fibers + async I/O : to emulate blocking API one needs to catch and store exceptions from I/O routines so that later those can be re-thrown from resumed fiber context.
Sep 20 2014
next sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 14:33:21 UTC, Dicebot wrote:
 Pretty much any time you do fibers + async I/O : to emulate 
 blocking API one needs to catch and store exceptions from I/O 
 routines so that later those can be re-thrown from resumed 
 fiber context.
Ah, indeed, and that could have a great many of them alive at once. Blargh, it'd need a real allocator to handle freeing them out of order. Nevertheless though, I still think the lifetime management there is simple enough for the user-programmer that freeing it manually isn't a big hassle and not worth making major changes to the language over.
Sep 20 2014
next sibling parent reply "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 15:05:38 UTC, Adam D. Ruppe 
wrote:
 On Saturday, 20 September 2014 at 14:33:21 UTC, Dicebot wrote:
 Pretty much any time you do fibers + async I/O : to emulate 
 blocking API one needs to catch and store exceptions from I/O 
 routines so that later those can be re-thrown from resumed 
 fiber context.
Ah, indeed, and that could have a great many of them alive at once. Blargh, it'd need a real allocator to handle freeing them out of order. Nevertheless though, I still think the lifetime management there is simple enough for the user-programmer that freeing it manually isn't a big hassle and not worth making major changes to the language over.
I don't follow. Lifetime is totally unknown there and completely up to actual application logic - from the point of view of the caller of something like `readFile()` it just a normal allocated exception that doesn't need any explicit management. Requiring any kind of manual handling would be a huge breaking change.
Sep 20 2014
parent "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 15:12:45 UTC, Dicebot wrote:
 Requiring any kind of manual handling would be a huge breaking 
 change.
I'm not talking about *requiring* it; I want it to be a garbage collected object that 1) allocating it never triggers a collection cycle (so it is a nogc allocation) and 2) you're allowed to explicitly free it if you want to, but if you don't, the gc will get around to it eventually. We have #2 already: delete (though I'd like it to not be something which is allegedly deprecated). #1 is fairly easy too: just offer a new nogc allocation function which creates garbage but does not garbage collect. This could be a new function or the new keyword could be modified to suppress collection when allocating Throwable or whatever. Maybe I'm missing something though.
Sep 20 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 8:05 AM, Adam D. Ruppe wrote:
 Nevertheless though, I still think the lifetime management there is
 simple enough for the user-programmer that freeing it manually isn't a
 big hassle and not worth making major changes to the language over.
The language will change in major ways, and this is the warmup before that. -- Andrei
Sep 20 2014
parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Saturday, 20 September 2014 at 16:55:08 UTC, Andrei 
Alexandrescu wrote:
 The language will change in major ways, and this is the warmup 
 before that. -- Andrei
Ugh. I'm very concerned about the direction this is going, replacing a system that works with something that offers little benefit and a lot of risk.
Sep 20 2014
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 10:50 AM, Adam D. Ruppe wrote:
 On Saturday, 20 September 2014 at 16:55:08 UTC, Andrei Alexandrescu wrote:
 The language will change in major ways, and this is the warmup before
 that. -- Andrei
Ugh. I'm very concerned about the direction this is going, replacing a system that works with something that offers little benefit and a lot of risk.
Fasten your seatbelt, it's gonna be a bumpy ride! :o) Andrei
Sep 20 2014
parent reply "ixid" <nuaccount gmail.com> writes:
 Fasten your seatbelt, it's gonna be a bumpy ride! :o)

 Andrei
The fundamentalness of the changes seem to be sufficient that one could argue it's D3. If you're going to make major changes wouldn't it be worth a fuller break to address some of the other unresolved and seemingly pretty major issues such as const/immutable and ref?
Sep 22 2014
next sibling parent reply "bearophile" <bearophileHUGS lycos.com> writes:
ixid:

 The fundamentalness of the changes seem to be sufficient that 
 one could argue it's D3.
This seems a good idea.
 If you're going to make major changes wouldn't it be worth a 
 fuller break to address some of the other unresolved and 
 seemingly pretty major issues such as const/immutable and ref?
This seems a bad idea, because GC and scope tracking is a sufficiently large design & implementation space. You can't face all the large problems at the same time. Bye, bearophile
Sep 22 2014
parent reply "Piotrek" <p nonexistent.pl> writes:
On Monday, 22 September 2014 at 09:13:49 UTC, bearophile wrote:
 ixid:

 The fundamentalness of the changes seem to be sufficient that 
 one could argue it's D3.
This seems a good idea.
No, it's not a good idea. Tweaking memory management shouldn't require the language branching. IMHO, this would be a suicide. Piotrek
Sep 22 2014
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Piotrek:

 No, it's not a good idea. Tweaking memory management shouldn't 
 require the language branching. IMHO, this would be a suicide.
I didn't meant the advancement as a language branching, but as a successive version that is (mostly) backwards compatible. Likewise C#6.0 is not a branching of C#5. Bye, bearophile
Sep 22 2014
parent "Piotrek" <p nonexistent.pl> writes:
On Monday, 22 September 2014 at 15:18:00 UTC, bearophile wrote:
 Piotrek:

 No, it's not a good idea. Tweaking memory management shouldn't 
 require the language branching. IMHO, this would be a suicide.
I didn't meant the advancement as a language branching, but as a successive version that is (mostly) backwards compatible. Likewise C#6.0 is not a branching of C#5. Bye, bearophile
I'm not sure how you define a branch, but I look at it from SCM pov. E.g. assuming C#6 is master/trunk/development branch then C#5 is a maintenance branch of it (same for other C# versions). That means, MS has to keep fixing all versions in parallel. Piotrek
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/22/14, 1:56 AM, ixid wrote:
 Fasten your seatbelt, it's gonna be a bumpy ride! :o)

 Andrei
The fundamentalness of the changes seem to be sufficient that one could argue it's D3.
Let's aim for not.
 If you're going to make major changes wouldn't it be
 worth a fuller break to address some of the other unresolved and
 seemingly pretty major issues such as const/immutable and ref?
What are the major issues with const/immutable and ref? Andrei
Sep 22 2014
next sibling parent "ixid" <nuaccount gmail.com> writes:
 What are the major issues with const/immutable and ref?
Const and immutable seem to be difficult to work with, Maxime wrote a piece about how difficult to use they were in practice. Ref is difficult to combine properly with generic templates, Manu covers that in another thread here. Maxime's blog: http://pointersgonewild.wordpress.com/2014/07/11/the-constness-problem/ Manu's post about the combinatorial complexity of ref for templates (which you've responded to earlier in the chain): http://forum.dlang.org/post/mailman.1381.1411381413.5783.digitalmars-d puremagic.com
Sep 22 2014
prev sibling parent Manu via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 23 September 2014 00:50, Andrei Alexandrescu via Digitalmars-d <
digitalmars-d puremagic.com> wrote:

 On 9/22/14, 1:56 AM, ixid wrote:

 Fasten your seatbelt, it's gonna be a bumpy ride! :o)
 Andrei
The fundamentalness of the changes seem to be sufficient that one could argue it's D3.
Let's aim for not. If you're going to make major changes wouldn't it be
 worth a fuller break to address some of the other unresolved and
 seemingly pretty major issues such as const/immutable and ref?
What are the major issues with const/immutable and ref?
This is precisely why I've mostly given up on this NG...
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:33 AM, Dicebot wrote:
 On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe wrote:
 How often do you store an exception reference anyway that escapes a
 catch block? I think all this talk is overkill to solve a non-problem
 in 99% of practice.
Pretty much any time you do fibers + async I/O : to emulate blocking API one needs to catch and store exceptions from I/O routines so that later those can be re-thrown from resumed fiber context.
Interesting. Are those stored as Object or Throwable/Exception? -- Andrei
Sep 20 2014
parent "Dicebot" <public dicebot.lv> writes:
On Saturday, 20 September 2014 at 16:50:08 UTC, Andrei 
Alexandrescu wrote:
 On 9/20/14, 7:33 AM, Dicebot wrote:
 On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
 wrote:
 How often do you store an exception reference anyway that 
 escapes a
 catch block? I think all this talk is overkill to solve a 
 non-problem
 in 99% of practice.
Pretty much any time you do fibers + async I/O : to emulate blocking API one needs to catch and store exceptions from I/O routines so that later those can be re-thrown from resumed fiber context.
Interesting. Are those stored as Object or Throwable/Exception? -- Andrei
I'd expect Throwable. Erasing it to Object smells like a trouble for no good. Don't know how Sonke has implemented in vibe.d though.
Sep 21 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/20/14, 7:31 AM, Adam D. Ruppe wrote:
 How often do you store an exception reference anyway that escapes a
 catch block? I think all this talk is overkill to solve a non-problem in
 99% of practice.


 Correct me if I'm wrong, but aren't *all* exceptions in a particular
 thread generally unreferenced at the end of a catch() block, unless the
 programmer explicitly escaped that reference?

 If so, we don't need refcounting! Here's my solution:
[snip] Thanks. I personally favor RC based on malloc to this. It'll be safer and simpler to use. -- Andrei
Sep 20 2014
prev sibling parent "Jakob Ovrum" <jakobovrum gmail.com> writes:
On Saturday, 20 September 2014 at 14:31:36 UTC, Adam D. Ruppe 
wrote:
 How often do you store an exception reference anyway that 
 escapes a catch block? I think all this talk is overkill to 
 solve a non-problem in 99% of practice.


 Correct me if I'm wrong, but aren't *all* exceptions in a 
 particular thread generally unreferenced at the end of a 
 catch() block, unless the programmer explicitly escaped that 
 reference?

 If so, we don't need refcounting! Here's my solution:
I proposed something very similar in this thread[1], except I proposed a solution that is fully backwards-compatible except the rare case when exceptions are escaped, in which case it causes a compile-time error. Might be worth a look. I don't think it's acceptable that code using Phobos is suddenly leaking until they add the new manual cleanup call. [1] http://forum.dlang.org/post/stlslhjndgugecvmbowd forum.dlang.org
Sep 20 2014
prev sibling next sibling parent reply "Oren Tirosh" <orent hishome.net> writes:
Hi everyone. Unlurking to make my first comment here.

Here is an idea for making RC and GC coexist peacefully. I think 
this technique may be used to make the Throwable transition to RC 
while keeping full backward compatibility.

A Throwable object would have a reference counter to track the 
number of RC references to it. It would also have a flag 
indicating whether there are any outstanding GC references to the 
object as well. This flag may be implemented by setting the MSB 
of the reference counter.

RC references would be a different type from GC references. An RC 
reference may be converted to a GC reference by setting the MSB, 
indicating that there may be one or more uncounted references to 
the object and that it should not be destroyed until successfully 
reclaimed by the GC. This RC to GC conversion may happen when a 
reference to a Throwable leaves  nogc code and enters GC land. 
When a GC operation can find no more references to the Throwable 
it would clear the counter MSB. If the number of RC references is 
zero at this point the object would be immediately destroyed.

A crucial part of making this work is to ensure that RC 
references are *not* seen by GC scanning. This may be done by 
mangling the pointer value in some way. Alternatively, if 
mark-and-sweep is replaced with count-and-sweep the number of 
counted RC references may be deducted.

One way to mangle the pointer is to decrement the value to a 
lower memory address. Incrementing the pointer won't work because 
the D garbage collector must handle inner references for slices 
etc.  Any dereferencing of the RC reference would compensate for 
this offset by incrementing the field offset. An obvious piece of 
information to put in the location reserved by decrementing the 
pointer would the the reference counter itself. Conversion of RC 
reference to GC would require incrementing the pointer and 
setting the counter MSB. Converting GC reference to RC is done by 
just decrementing the pointer.

I'm sure there are many things I'm missing here, but could 
something along these line be made to work?
Sep 20 2014
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 20 September 2014 at 19:48:14 UTC, Oren Tirosh wrote:
 Hi everyone. Unlurking to make my first comment here.

 Here is an idea for making RC and GC coexist peacefully. I 
 think this technique may be used to make the Throwable 
 transition to RC while keeping full backward compatibility.

 A Throwable object would have a reference counter to track the 
 number of RC references to it. It would also have a flag 
 indicating whether there are any outstanding GC references to 
 the object as well. This flag may be implemented by setting the 
 MSB of the reference counter.
I was thinking about this recently (and also, Andrei's talk at cppcon provided some interesting data). Most reference count are small. I think it make sense for us to propose a ref count system, that allocate on the GC heap, and that do not free when the counter saturate. It would help to: - get intrusive refcount without putting too much crap into objects (it is all about cache line). - get a way to escape RC object to the GC heap (by saturating the refcount). The GC is still there, but do kick in randomly, it simply act as a safety net. Also given the type qualifier in D, the whole synchronization thing is embeded in the type, so we can be safe and fast on that one. All of this can be done as library.
Sep 20 2014
parent reply "Oren Tirosh" <orent hishome.net> writes:
On Sunday, 21 September 2014 at 00:45:49 UTC, deadalnix wrote:
 On Saturday, 20 September 2014 at 19:48:14 UTC, Oren Tirosh 
 wrote:
 Hi everyone. Unlurking to make my first comment here.

 Here is an idea for making RC and GC coexist peacefully. I 
 think this technique may be used to make the Throwable 
 transition to RC while keeping full backward compatibility.

 A Throwable object would have a reference counter to track the 
 number of RC references to it. It would also have a flag 
 indicating whether there are any outstanding GC references to 
 the object as well. This flag may be implemented by setting 
 the MSB of the reference counter.
I was thinking about this recently (and also, Andrei's talk at cppcon provided some interesting data). Most reference count are small. I think it make sense for us to propose a ref count system, that allocate on the GC heap, and that do not free when the counter saturate.
This is not what I was proposing. What you are describing (small, saturating RC) is one of the schemes mentioned in this article: Shahriyar, Rifat, Stephen M. Blackburn, and Daniel Frampton. "Down for the count? Getting reference counting back in the ring." ACM SIGPLAN Notices. Vol. 47. No. 11. ACM, 2012. http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf Reading this article is recommended for everyone here. It demonstrates that advanced implementations of reference counting systems are viable competition for GC and even outperform them under certain workloads. The authors have implemented several reference counting and hybrid schemes for the Java runtime and benchmarked them. One of the schemes uses a small, saturating reference counter in the object header similar to what you have described.
 It would help to:
  - get intrusive refcount without putting too much crap into 
 objects (it is all about cache line).
  - get a way to escape RC object to the GC heap (by saturating 
 the refcount).
My idea is, indeed, to enable RC objects to be participate in classic GC, but not by saturation. The MSB is, by definition, big enough so the real reference count will not reach it. I suggest using it as a flag to indicate whether there are any GC references to the object. If clear, the object may be destroyed when the counter reaches 0. If set, it means that a GC cycle is still required to find if there are any more outstanding references to it. * No modification to the binary structure of Throwable. It remains fully link-compatible with object files generated by older compiler versions. * "new Throwable" is modified to allocate extra space for a counter just before the instance and returns a pointer just past it. The counter is initialized to 0 with MSB set. * The deallocator is modified to clear the counter MSB and actually deallocate the object only if the counter is zero. That's it for regular (" gc") code. In nogc code, an alternative "new" is used that sets the counter to 1 (MSB clear) and returns a pointer to the counter rather than the instance that follows it. Whenver this reference is copied, overwritten or goes out of scope the reference counter must be updated and deallocated when it reaches 0. Dereferencing the Throwable via this type of reference-counter pointer uses different offsets to compensate for the presence of the counter. This reference-counted reference may be automatically converted to a regular pointer that is safely used in code not aware of reference counting by setting the MSB of the reference counter and incrementing the pointer to point to the instance. Taking the address of a field inside an object referenced through a reference-counter pointer is either disallowed or required to set the counter MSB because reference counted references do not support internal pointers and This would provide full backward compatibility even with .o code already compiled before the feature was introduced.
 The GC is still there, but do kick in randomly, it simply act 
 as a safety net.

 Also given the type qualifier in D, the whole synchronization 
 thing is embeded in the type, so we can be safe and fast on 
 that one. All of this can be done as library.
The point of my scheme is to ensure that a thread that contains no GC references may be safely excluded from "stop the world" when GC kicks in. This is currently the biggest "lie" in the D story about GC being optional. While nogc code will not trigger collection, it is NOT excluded from random pauses caused by GC in other threads.
Sep 21 2014
parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 12:18 PM, Oren Tirosh wrote:
 Shahriyar, Rifat, Stephen M. Blackburn, and Daniel Frampton.
 "Down for the count? Getting reference counting back in the
 ring." ACM SIGPLAN Notices. Vol. 47. No. 11. ACM, 2012.
 http://users.cecs.anu.edu.au/~steveb/downloads/pdf/rc-ismm-2012.pdf

 Reading this article is recommended for everyone here.
Yah, and also all other related publications by Blackburn et comp. Great work we need to learn from. -- Andrei
Sep 21 2014
prev sibling next sibling parent "deadalnix" <deadalnix gmail.com> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu
wrote:
 As discussed, having exception objects being GC-allocated is 
 clearly a large liability that we need to address. They prevent 
 otherwise careful functions from being  nogc so they affect 
 even apps that otherwise would be okay with a little litter 
 here and there.

 The Throwable hierarchy is somewhat separate from everything 
 else, which makes it a great starting point for investigating 
 an automated reference count approach. Here's what I'm thinking.

 First, there must be some compiler flag -nogc or something, 
 which triggers the RC exceptions. All modules of an application 
 must be compiled with this flag if it is to work (such that one 
 module can throw an exception caught by the other). Of course a 
 lot of refinement needs to be added here (what happens if one 
 tries to link modules built with and without -nogc, allowing 
 people to detect the flag programmatically by using 
 version(nogc) etc).

 If -nogc is passed, the compiler severs the inheritance 
 relationship between Throwable and Object, making it impossible 
 to convert a Throwable to an Object. From then henceforth, 
 Throwable and Object form a two-rooted forest. (In all 
 likelihood we'll later add an RCObject root that Throwable 
 inherits.)

 Whenever a reference to a Throwable is copied about, passed to 
 functions, the compiler inserts appropriately calls to e.g. 
 incRef and decRef. (Compiler may assume they cancel each other 
 for optimization purposes.) Implementation of these is up to 
 the runtime library. Null checking may be left to either the 
 compiler or the library (in the latter case, the functions must 
 be nonmember). However it seems the compiler may have an 
 advantage because it can elide some of these checks.

 Once we get this going, we should accumulate good experience 
 that we can later apply to generalizing this approach to more 
 objects. Also, if things go well we may as well define _always_ 
 (whether GC or not) Throwable to be reference counted; seems 
 like a good fit for all programs.

 Please chime in with thoughts.


 Andrei
I guess it is time to put isolated back on the table. You can throw only isolated. When isolated goes out of scope without being consumed, the island is blasted out of existence.
Sep 20 2014
prev sibling next sibling parent "Chris Cain" <zshazz gmail.com> writes:
I'm just throwing a new ThisOutThereException:

What if we modified the exception hierarchy to have it so 
"Throwable"s are GC allocated but there's an alternate set of 
classes down another line that either must be manually managed or 
are RC. Thus, there's no breakage.

Something like:

class Throwable : __UnifiedThrowable ...

class NoGCThrowable : __UnifiedThrowable ...


That way any code that's catching/handling Throwables doesn't 
need to change but we start making it so that code that needs to 
be NoGC handles the NoGCThrowable types. This would mean code 
could be migrated gradually to the new method (and, if truly 
desireable, the regular throwables could be deprecated 
eventually).

Just an idea.
Sep 20 2014
prev sibling parent reply =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu 
wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things? Potentially with the help of escape analysis, which could be made part of this effort, automatically avoiding the GC seems like the wisest choice in the long run. AFAIK there has been quite a lot of efforts done here already by various persons.
Sep 21 2014
next sibling parent reply "Peter Alexander" <peter.alexander.au gmail.com> writes:
On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei 
 Alexandrescu wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things?
GC improvements are critical, but... "As discussed, having exception objects being GC-allocated is clearly a large liability that we need to address. They prevent otherwise careful functions from being nogc so they affect even apps that otherwise would be okay with a little litter here and there." No improvements to the GC can fix this. nogc needs to be usable, whether you are a GC fan or not.
Sep 21 2014
next sibling parent =?UTF-8?B?Ik5vcmRsw7Z3Ig==?= <per.nordlow gmail.com> writes:
On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
wrote:
 GC improvements are critical, but...

 "As discussed, having exception objects being GC-allocated is 
 clearly a
 large liability that we need to address. They prevent otherwise 
 careful
 functions from being  nogc so they affect even apps that 
 otherwise would
 be okay with a little litter here and there."
Ok, got it. Thanks.
Sep 21 2014
prev sibling next sibling parent Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 1:57 PM, Peter Alexander wrote:
 On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things?
GC improvements are critical, but... "As discussed, having exception objects being GC-allocated is clearly a large liability that we need to address. They prevent otherwise careful functions from being nogc so they affect even apps that otherwise would be okay with a little litter here and there." No improvements to the GC can fix this. nogc needs to be usable, whether you are a GC fan or not.
Exactly and awesomely right. -- Andrei
Sep 21 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
wrote:
 No improvements to the GC can fix this.  nogc needs to be 
 usable, whether you are a GC fan or not.
True. To fix this, we need to add a pile of hack that take care of this specific use case. The end goal is to have a pile of hack for every single use case, as this is what C++ has and C++ is successful and we want to be successful. Introducing a construct to manage ownership would obviously avoid the whole pile of hack, but hey, this would introduce complexity to the language, when the pile of hack do not. AMIRITE ?
Sep 21 2014
parent "Cliff" <cliff.s.hudson gmail.com> writes:
On Sunday, 21 September 2014 at 23:32:29 UTC, deadalnix wrote:
 On Sunday, 21 September 2014 at 20:57:24 UTC, Peter Alexander 
 wrote:
 No improvements to the GC can fix this.  nogc needs to be 
 usable, whether you are a GC fan or not.
True. To fix this, we need to add a pile of hack that take care of this specific use case. The end goal is to have a pile of hack for every single use case, as this is what C++ has and C++ is successful and we want to be successful. Introducing a construct to manage ownership would obviously avoid the whole pile of hack, but hey, this would introduce complexity to the language, when the pile of hack do not. AMIRITE ?
The devolution of conversation in this thread to snide remarks, extreme sarcasm and thinly (if at all) veiled name calling is counterproductive to solving the problem. Reasonable people can disagree without resorting to passive-aggression.
Sep 21 2014
prev sibling parent luka8088 <luka8088 owave.net> writes:
On 21.9.2014. 22:57, Peter Alexander wrote:
 On Sunday, 21 September 2014 at 19:36:01 UTC, Nordlöw wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things?
GC improvements are critical, but... "As discussed, having exception objects being GC-allocated is clearly a large liability that we need to address. They prevent otherwise careful functions from being nogc so they affect even apps that otherwise would be okay with a little litter here and there." No improvements to the GC can fix this. nogc needs to be usable, whether you are a GC fan or not.
I think that what is being suggested is that upgrading GC would widespread the point of view on what can and should be done. For example, now that ranges and mixins exist great ideas comes to mind, and without them we can only guess. I think that GC is in the same position.
Sep 22 2014
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 9/21/14, 12:35 PM, "Nordlöw" wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things?
Not sure I understand, but: an important niche is apps that COMPLETELY disallow garbage collection -- Andrei
Sep 21 2014
parent Brad Roberts via Digitalmars-d <digitalmars-d puremagic.com> writes:
On 9/21/2014 3:12 PM, Andrei Alexandrescu via Digitalmars-d wrote:
 On 9/21/14, 12:35 PM, "Nordlöw" wrote:
 On Friday, 19 September 2014 at 15:32:38 UTC, Andrei Alexandrescu wrote:
 Please chime in with thoughts.
Why don't we all focus our efforts on upgrading the current GC to a state-of-the GC making use of D's strongly typed memory model before discussing these things?
Not sure I understand, but: an important niche is apps that COMPLETELY disallow garbage collection -- Andrei
It's also far from being a zero sum game. Work on one is not necessarily taking away from the other. There needs to be more 'doing' that follows the discussions than there typically is. But progress can and needs to be made on both (and many many other) fronts.
Sep 22 2014