www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - An exegesis of Walter's reference counted slice

reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
I modified Walter's sample code to this: 
http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array 
and the reference count, and also uses  trusted minimally. I inserted 
assert()s here and there to clarify the workings. Nothing big except for 
the careful use of  trusted.

I'll use this as a basis of some exegesis.

1. The code is a bit more complicated than it should. Overall this is 
not a biggie; regular D users are not supposed to write reference 
counted slices casually. But I was bummed that e.g. I found no way to 
call emplace()  safe-ly.

2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221) 
reveals the largest issue with RC/GC integration. We need to find a fix 
for it if we want to have the GC lift cycles.

3. opIndex ("here's the magic") is the most interesting part of the 
proposal. It disallows unsafe use such as:

 safe ref int fun()
{
    auto a = RCArray!int([1, 2, 3]);
    return a[1];
}

Nice. I'd go as far as saying that barring implementation bugs, with 
DIP25 in tow, and after we fix 14221, it's impossible to get an invalid 
memory access with RCArray in  safe code. This (we offer a way to design 
 safe arrays and more generally structs that are  safe) is interesting 
and important.

That said there is a rub. The part that deallocates memory in the 
destructor must be  trusted. That is fine, but the trustworthiness of 
that code depends on the "return" attribute in opIndex. Furthermore, if 
the author of RCSlice forgets to put "return" there, the compiler won't 
help - it just allows wrong code like fun() above to compile and run 
(causing dangling pointer use).

So: does DIP25 allow safe slices? Looks that way, though a proof would 
be nice. Does it allow other safe interesting structs that own data? 
Very likely. Does it allow really sophisticated ownership schemes? We 
need to explore that. Does it protect against bugs in implementations of 
safe ownership schemes that explicitly release memory? Not too well. I 
think the prevalent idiom will be to accompany such artifacts with 
unittests that make sure unsafe uses (such as fun() above) do not compile.


Andrei
Feb 24 2015
next sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 2:40 PM, Andrei Alexandrescu wrote:
 I modified Walter's sample code to this:
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array
 and the reference count, and also uses  trusted minimally. I inserted
 assert()s here and there to clarify the workings. Nothing big except for
 the careful use of  trusted.

 I'll use this as a basis of some exegesis.

 1. The code is a bit more complicated than it should. Overall this is
 not a biggie; regular D users are not supposed to write reference
 counted slices casually. But I was bummed that e.g. I found no way to
 call emplace()  safe-ly.
I have no problem with underlying complexity of primitive types. Using Objective-C objects is just fine without understanding the implementation. Note, you need to GC.addRange all the elements if the type has references, or else you cannot have GC pointers in the array. For example, an array of class references could potentially result in those references being collected before the array is gone. Ironically, if those elements are references, but are reference counted references, then you wouldn't need to addRange. An interesting problem...
 2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221)
 reveals the largest issue with RC/GC integration. We need to find a fix
 for it if we want to have the GC lift cycles.
I think a system of making sure a piece of data is always destroyed in the same thread it was created (unless of course, the thread is gone, in which case it doesn't matter), should be fine. -Steve
Feb 24 2015
next sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...
Could you please add a detailed issue about this? Thanks. -- Andrei
Feb 24 2015
next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 12:03:36 -0800, Andrei Alexandrescu wrote:

 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...
=20 Could you please add a detailed issue about this? Thanks. -- Andrei
here's the fixed code: http://dpaste.dzfl.pl/5f7dfe237d1e btw, you have a bug in your implementation: (cast(E*) malloc(a.length * uint.sizeof))[0 .. a.length])(); should be: (cast(E*) malloc(a.length * E.sizeof))[0 .. a.length])(); sorry, i'm bad in explanations, let the code speak for itself.=
Feb 24 2015
prev sibling parent reply Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:
 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...
Could you please add a detailed issue about this? Thanks. -- Andrei
I'm not sure what you mean by 'this'. As far as I can tell, it isn't a bug in GC but a bug in your dpaste code. -Steve
Feb 26 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/26/15 8:44 AM, Steven Schveighoffer wrote:
 On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:
 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...
Could you please add a detailed issue about this? Thanks. -- Andrei
I'm not sure what you mean by 'this'. As far as I can tell, it isn't a bug in GC but a bug in your dpaste code. -Steve
I'm referring to the "interesting problem" of adding addRange for some but not all types. -- Andrei
Feb 26 2015
parent Steven Schveighoffer <schveiguy yahoo.com> writes:
On 2/26/15 11:56 AM, Andrei Alexandrescu wrote:
 On 2/26/15 8:44 AM, Steven Schveighoffer wrote:
 On 2/24/15 3:03 PM, Andrei Alexandrescu wrote:
 On 2/24/15 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.

 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...
Could you please add a detailed issue about this? Thanks. -- Andrei
I'm not sure what you mean by 'this'. As far as I can tell, it isn't a bug in GC but a bug in your dpaste code.
I'm referring to the "interesting problem" of adding addRange for some but not all types. -- Andrei
OK, but I don't know where to add that :( RC isn't supported by D today. You mean add it to a DIP? Add it to the docs? I don't think it's mechanically checkable, BTW. If you have an array of pointers, each pointer may point at a GC-allocated block, or a C malloc-allocated block. I suppose at the time of setting an element, you can addRange if necessary. But the cost for that I think far outweighs the benefit of using RC. -Steve
Feb 26 2015
prev sibling next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 14:55:14 -0500, Steven Schveighoffer wrote:

 Note, you need to GC.addRange all the elements if the type has
 references, or else you cannot have GC pointers in the array. For
 example, an array of class references could potentially result in those
 references being collected before the array is gone.
heh. yes, i hit this in my `iv.steam.PartialStream` implementation. those=20 occasional crashes was nasty.
 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting
 problem...
actually, it's safe to add GC roots and regions that contains no GC=20 references. it's a little slowdown, as GC must scan that area, but=20 nothing bad happens. yet you are right, there should be some method to distinguish "gc- agnostic" types. and compiler should be able to infer that...=
Feb 24 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 11:55 AM, Steven Schveighoffer wrote:
 Note, you need to GC.addRange all the elements if the type has references, or
 else you cannot have GC pointers in the array.
Yes. I deliberately left that out in order to keep attention focused on the use of 'return ref'.
 Ironically, if those elements are references, but are reference counted
 references, then you wouldn't need to addRange. An interesting problem...
Correct, but (aside from the performance hit) it wouldn't be a bug to scan them unnecessarily in the GC. One way is to have a scheme whereby, with introspection, a type can be determined to be ref counted. Then, using static if, doing the addRange would only be necessary if: 1. the element type contained indirections 2. those indirections were not ref counted
 I think a system of making sure a piece of data is always destroyed in the same
 thread it was created (unless of course, the thread is gone, in which case it
 doesn't matter), should be fine.
Yup, and I believe this is an orthogonal issue.
Feb 24 2015
prev sibling next sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 11:40:34 -0800, Andrei Alexandrescu wrote:

 3. opIndex ("here's the magic") is the most interesting part of the
 proposal. It disallows unsafe use such as:
=20
  safe ref int fun()
 {
     auto a =3D RCArray!int([1, 2, 3]);
     return a[1];
 }
this is the important part for those who are too lazy to read DIP25 (like=20 me). thank you.=
Feb 24 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
 But I was bummed that e.g. I found no way to call emplace() 
  safe-ly.
I assumed emplace would infer safe?
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 12:03 PM, weaselcat wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
We're going to lambdas. Ugliness is a feature.
 But I was bummed that e.g. I found no way to call emplace()  safe-ly.
I assumed emplace would infer safe?
It currently doesn't properly, probably because it uses T* instead of ref T. Andrei
Feb 24 2015
parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu 
wrote:
 On 2/24/15 12:03 PM, weaselcat wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
We're going to lambdas. Ugliness is a feature.
Stroustrup would be proud ;)
Feb 24 2015
parent reply ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:

 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu wrote:
 On 2/24/15 12:03 PM, weaselcat wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu
 wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
We're going to lambdas. Ugliness is a feature.
Stroustrup would be proud ;)
yet he made the whole language ugly, instead of making only ugly things=20 ugly. ;-)=
Feb 24 2015
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Feb 24, 2015 at 08:21:53PM +0000, ketmar via Digitalmars-d wrote:
 On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:
 
 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei Alexandrescu wrote:
 On 2/24/15 12:03 PM, weaselcat wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu
 wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
We're going to lambdas. Ugliness is a feature.
Stroustrup would be proud ;)
yet he made the whole language ugly, instead of making only ugly things ugly. ;-)
Actually, it's worse than that. He made things that shouldn't be ugly, ugly, and things that *don't* look ugly are outright wrong. For example, naked pointers have built-in, convenient syntax. They are also extremely dangerous, and almost all C++ code containing raw pointers have bugs one way or another. For-loops have very nice syntax... except that unless you're writing the most inane, trivial loops, your code probably has bugs -- off-by-1 errors, wrong loop conditions, etc.. Calling malloc/free is also easy... and your program probably has memory leak bugs too. Not to mention type-safety bugs if you're not careful about what you cast that void* into. But hey, writing: // Woohoo, bare pointer! Let's dance! void *p = malloc(1234); sure looks prettier than: // Repeat after me: Don't repeat yourself... don't repeat // yourself... don't repeat yourself... MyType *p = (MyType *)malloc(sizeof(MyType)); Well, new/delete is prettier (as far as C++ goes anyway)... But wait, you still have memory leaks. Augh... And what about cleaning up resources after you're done with them? void func() { Resource *res = acquireResource(); doStuff(res); freeResource(res); } Easy, right? Yeah, except that doStuff throws exceptions, so you have resource leakage. Solution? Write a long convoluted wrapper class with a dtor that cleans up. And a copy ctor that makes sure initializing one resource from another works correctly. And operator=() to make sure assignments work correctly. Oh but wait, your ctor is not exception-safe, so better rewrite *that* to use RAII too. Which means more copy ctors, more operator=()... oh wait, your copy ctor is not const-correct so it doesn't get called when you have a const object. Yeah, and your operator=() too. And ... After it's all said and done, what used to be a 3-line function has exploded into a monstrosity who knows how many lines long, with arcane incantations of auto_ptr<>, shared_ptr<>, and all that fancy stuff that no newbie ever has the hope of parsing, let alone understanding. http://bartoszmilewski.com/2013/09/19/edward-chands/ Favorite quote: C++ has become an extremely complex language. There are countless ways of doing the same thing — almost all of them either plain wrong, dangerous, unmaintainable, or all of the above. The problem is that most code compiles and even runs. The mistakes and shortcomings are discovered much later, often after the product has been released. Yep, that's exactly why D has ruined my life, I just can't go back to that C++ garbage anymore. T -- Don't throw out the baby with the bathwater. Use your hands...
Feb 24 2015
next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:
 	http://bartoszmilewski.com/2013/09/19/edward-chands/
I'm not generally one to read comments on blogs, but the replies to that blog are jawdropping. I have to assume most of these people are either new to C++ or are being forced to write these replies at gunpoint.
Yep, that's exactly why D has ruined my life, I just can't go 
back to
that C++ garbage anymore. Working with C++ after using D does leave one feeling... disillusioned.
Feb 24 2015
next sibling parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Feb 24, 2015 at 09:13:42PM +0000, weaselcat via Digitalmars-d wrote:
 On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:
[...]
Yep, that's exactly why D has ruined my life, I just can't go back to
that C++ garbage anymore. Working with C++ after using D does leave one feeling... disillusioned.
That's putting it rather mildly. :-P Some time ago I revisited one of my old C++ programs, hoping to optimize performance a bit by using a hash table to cache the results of certain expensive computations. I was somewhat elated that *finally* after all these years hash tables finally made it into the C++ standard... For all their warts, D's AA's have left me confident that this would be a small change... Boy was I wrong. Everything from the horrible syntax to built-in structs *not* being supported as hash keys by default, to needing to write my own hash function, *and* having to pass it around all over the place, etc.., ... after several hours of fighting with the language for something that would be just a couple dozen lines of code in D at the most, I threw up my hands and ditched the whole idea. C++11 feels a lot like "too little, too late" to me. I'd rather rewrite the whole thing in D and spare myself the pain. For all of its current flaws, D is still way ahead of C++ in terms of usability. I've stopped caring about C++ and have not bothered to find out what C++14 (or is it C++15 now?) has in store -- if past experience is anything to go by, it will just be another "too little, too late". T -- "How are you doing?" "Doing what?"
Feb 24 2015
prev sibling parent ketmar <ketmar ketmar.no-ip.org> writes:
On Tue, 24 Feb 2015 21:13:42 +0000, weaselcat wrote:

 Working with C++ after using D does leave one feeling... disillusioned.
that's why i'm not even considering c++ for writing new code anymore.=20 either D, or good old C, if D is not appropriate. actually, i haven't written a line of c++ for several years now. and i=20 wrote almost no new C code for years too -- thanks to seampless interop=20 between D and C.=
Feb 24 2015
prev sibling parent "Paulo Pinto" <pjmlp progtools.org> writes:
On Tuesday, 24 February 2015 at 20:46:28 UTC, H. S. Teoh wrote:
 On Tue, Feb 24, 2015 at 08:21:53PM +0000, ketmar via 
 Digitalmars-d wrote:
 On Tue, 24 Feb 2015 20:11:17 +0000, weaselcat wrote:
 
 On Tuesday, 24 February 2015 at 20:04:27 UTC, Andrei 
 Alexandrescu wrote:
 On 2/24/15 12:03 PM, weaselcat wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu
 wrote:
 ...
Off-topic, sorry Are we still going to get a Trusted block, or just going to use trusted lambdas?(They're kind of ugly TBH)
We're going to lambdas. Ugliness is a feature.
Stroustrup would be proud ;)
yet he made the whole language ugly, instead of making only ugly things ugly. ;-)
Actually, it's worse than that. He made things that shouldn't be ugly, ugly, and things that *don't* look ugly are outright wrong. For example, naked pointers have built-in, convenient syntax. They are also extremely dangerous, and almost all C++ code containing raw pointers have bugs one way or another. For-loops have very nice syntax... except that unless you're writing the most inane, trivial loops, your code probably has bugs -- off-by-1 errors, wrong loop conditions, etc.. Calling malloc/free is also easy... and your program probably has memory leak bugs too. Not to mention type-safety bugs if you're not careful about what you cast that void* into. But hey, writing: // Woohoo, bare pointer! Let's dance! void *p = malloc(1234); sure looks prettier than: // Repeat after me: Don't repeat yourself... don't repeat // yourself... don't repeat yourself... MyType *p = (MyType *)malloc(sizeof(MyType)); Well, new/delete is prettier (as far as C++ goes anyway)... But wait, you still have memory leaks. Augh... And what about cleaning up resources after you're done with them? void func() { Resource *res = acquireResource(); doStuff(res); freeResource(res); } Easy, right? Yeah, except that doStuff throws exceptions, so you have resource leakage. Solution? Write a long convoluted wrapper class with a dtor that cleans up. And a copy ctor that makes sure initializing one resource from another works correctly. And operator=() to make sure assignments work correctly. Oh but wait, your ctor is not exception-safe, so better rewrite *that* to use RAII too. Which means more copy ctors, more operator=()... oh wait, your copy ctor is not const-correct so it doesn't get called when you have a const object. Yeah, and your operator=() too. And ... After it's all said and done, what used to be a 3-line function has exploded into a monstrosity who knows how many lines long, with arcane incantations of auto_ptr<>, shared_ptr<>, and all that fancy stuff that no newbie ever has the hope of parsing, let alone understanding. http://bartoszmilewski.com/2013/09/19/edward-chands/ Favorite quote: C++ has become an extremely complex language. There are countless ways of doing the same thing — almost all of them either plain wrong, dangerous, unmaintainable, or all of the above. The problem is that most code compiles and even runs. The mistakes and shortcomings are discovered much later, often after the product has been released. Yep, that's exactly why D has ruined my life, I just can't go back to that C++ garbage anymore. T
Almost all those warts are caused by the C compatibility and affect also D to certain extent. Any language that tries to achieve copy-paste compatibility with C will suffer from it. -- Paulo
Feb 24 2015
prev sibling next sibling parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:
 I modified Walter's sample code to this: 
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the 
 array and the reference count, and also uses  trusted 
 minimally. I inserted assert()s here and there to clarify the 
 workings. Nothing big except for the careful use of  trusted.
Doesn't look like it follows the "philosophy behind" trusted. Conceptually, you should be able to insert exceptions everywhere outside the called trusted function and retain memory safety. The way the code is written you have to move outside the trusted function to prove memory safety.
Feb 24 2015
prev sibling next sibling parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
Andrei Alexandrescu wrote:

 I modified Walter's sample code to this:
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the array
 and the reference count, and also uses  trusted minimally. I inserted
 assert()s here and there to clarify the workings. Nothing big except for
 the careful use of  trusted.
 
 I'll use this as a basis of some exegesis.
 
 1. The code is a bit more complicated than it should. Overall this is
 not a biggie; regular D users are not supposed to write reference
 counted slices casually. But I was bummed that e.g. I found no way to
 call emplace()  safe-ly.
 
 2. Michel's point (https://issues.dlang.org/show_bug.cgi?id=14221)
 reveals the largest issue with RC/GC integration. We need to find a fix
 for it if we want to have the GC lift cycles.
 
 3. opIndex ("here's the magic") is the most interesting part of the
 proposal. It disallows unsafe use such as:
 
  safe ref int fun()
 {
     auto a = RCArray!int([1, 2, 3]);
     return a[1];
 }
 
 Nice. I'd go as far as saying that barring implementation bugs, with
 DIP25 in tow, and after we fix 14221, it's impossible to get an invalid
 memory access with RCArray in  safe code. This (we offer a way to design
  safe arrays and more generally structs that are  safe) is interesting
 and important.
 
 That said there is a rub. The part that deallocates memory in the
 destructor must be  trusted. That is fine, but the trustworthiness of
 that code depends on the "return" attribute in opIndex. Furthermore, if
 the author of RCSlice forgets to put "return" there, the compiler won't
 help - it just allows wrong code like fun() above to compile and run
 (causing dangling pointer use).
 
 So: does DIP25 allow safe slices? Looks that way, though a proof would
 be nice. Does it allow other safe interesting structs that own data?
 Very likely. Does it allow really sophisticated ownership schemes? We
 need to explore that. Does it protect against bugs in implementations of
 safe ownership schemes that explicitly release memory? Not too well. I
 think the prevalent idiom will be to accompany such artifacts with
 unittests that make sure unsafe uses (such as fun() above) do not compile.
 
 
 Andrei
Is there any plan to allow safe conversions to T[] (in restricted circumstances, of course)?
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 1:33 PM, Ivan Timokhin wrote:
 Is there any plan to allow safe conversions to T[] (in restricted
 circumstances, of course)?
We'd want to avoid it because that would necessitate the whole "scope" paraphernalia - e.g you can convert a reference counted slice to something like scope(T[]). A possible option would be to allow the user to go "I want to use GC with this slice from now on" which gives back the T[] and lets the GC take care of it. But generally I think we should let reference counted slices and built-in slices coexist. Generic range-based code is easy to write so factoring the nature of the slice away may be an option. Andrei
Feb 24 2015
parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
Andrei Alexandrescu wrote:

 On 2/24/15 1:33 PM, Ivan Timokhin wrote:
 Is there any plan to allow safe conversions to T[] (in restricted
 circumstances, of course)?
We'd want to avoid it because that would necessitate the whole "scope" paraphernalia - e.g you can convert a reference counted slice to something like scope(T[]). A possible option would be to allow the user to go "I want to use GC with this slice from now on" which gives back the T[] and lets the GC take care of it. But generally I think we should let reference counted slices and built-in slices coexist. Generic range-based code is easy to write so factoring the nature of the slice away may be an option. Andrei
Oh. So, whenever you pass a reference-counted slice around, you need to do it with the full inc/dec protocol, which, as Walter has mentioned several times already, leads to code bloat and performance hits. So... no to efficient reference counting? Also, no slicing of static arrays in safe code?
Feb 25 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/25/15 1:58 AM, Ivan Timokhin wrote:
 Oh. So, whenever you pass a reference-counted slice around, you need to do
 it with the full inc/dec protocol, which, as Walter has mentioned several
 times already, leads to code bloat and performance hits. So... no to
 efficient reference counting? Also, no slicing of static arrays in  safe
 code?
Correct. There's an issue of perception that I just figured, which explains a lot of the drama and trash talk about DIP25. DIP25 is not a borrowing mechanism. Its charter is to make reference counted structs (and a variety of other structs) usable in safe code. As RCSlice shows, with DIP25 a reference counted slice switches from unusable to usable in safe code by adding one token to an otherwise unchanged implementation. That is Remarkable, and is Good Programming Language Design(tm). Borrowing data in a scoped manner is the charter of DIP69. Given the excellent quality of DIP25, it's likely it is Here To Stay and DIP69, or any other proposal for borrowing, will work with it (and probably leverage it). Andrei
Feb 25 2015
parent reply Ivan Timokhin <timokhin.iv gmail.com> writes:
On Wed, Feb 25, 2015 at 08:06:48AM -0800, Andrei Alexandrescu wrote:
 On 2/25/15 1:58 AM, Ivan Timokhin wrote:
 Oh. So, whenever you pass a reference-counted slice around, you need to do
 it with the full inc/dec protocol, which, as Walter has mentioned several
 times already, leads to code bloat and performance hits. So... no to
 efficient reference counting? Also, no slicing of static arrays in  safe
 code?
Correct. There's an issue of perception that I just figured, which explains a lot of the drama and trash talk about DIP25. DIP25 is not a borrowing mechanism. Its charter is to make reference counted structs (and a variety of other structs) usable in safe code. As RCSlice shows, with DIP25 a reference counted slice switches from unusable to usable in safe code by adding one token to an otherwise unchanged implementation. That is Remarkable, and is Good Programming Language Design(tm). Borrowing data in a scoped manner is the charter of DIP69. Given the excellent quality of DIP25, it's likely it is Here To Stay and DIP69, or any other proposal for borrowing, will work with it (and probably leverage it). Andrei
OK, thank you. I wasn't asking about DIP25 in particular (I think I understand what it's about), but about your (and Walter's) plans on the topic in general. I apologise if it sounded offensive in any way.
Feb 25 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/25/15 11:04 AM, Ivan Timokhin wrote:
 OK, thank you. I wasn't asking about DIP25 in particular (I think I
 understand what it's about), but about your (and Walter's) plans on
 the topic in general.
The medium-term plan is two-pronged: (a) immediately apply DIP25 to http://wiki.dlang.org/?title=DIP74, which will propose safe reference counted class objects; (b) examine scoped borrowing (DIP69 and friends) opportunities post DIP25. Walter and I are working on DIP74 right now. Andrei
Feb 25 2015
parent "weaselcat" <weaselcat gmail.com> writes:
  Wednesday, 25 February 2015 at 20:01:51 UTC, Andrei Alexandrescu 
wrote:
 On 2/25/15 11:04 AM, Ivan Timokhin wrote:
 OK, thank you. I wasn't asking about DIP25 in particular (I 
 think I
 understand what it's about), but about your (and Walter's) 
 plans on
 the topic in general.
The medium-term plan is two-pronged: (a) immediately apply DIP25 to http://wiki.dlang.org/?title=DIP74, which will propose safe reference counted class objects; (b) examine scoped borrowing (DIP69 and friends) opportunities post DIP25. Walter and I are working on DIP74 right now. Andrei
As someone who directly complained about this, this is awesome. Can't wait to read DIP74.
Feb 25 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:
 So: does DIP25 allow safe slices? Looks that way
I believe it does, but at the cost of forced reference counting. As I pointed out, the `ref` solution is not applicable to slices, therefore it needs to return an RCArray. This in turn forces an inc/dec whenever it is sliced or copied, and - maybe worse - it requires all consumers to support RCArray (or whatever other idiosyncratic RC implementation users come up with).
Feb 24 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 2/24/15 1:55 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" 
wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu wrote:
 So: does DIP25 allow safe slices? Looks that way
I believe it does, but at the cost of forced reference counting. As I pointed out, the `ref` solution
What is the ref solution?
 is not applicable to slices, therefore
 it needs to return an RCArray. This in turn forces an inc/dec whenever
 it is sliced or copied, and - maybe worse - it requires all consumers to
 support RCArray (or whatever other idiosyncratic RC implementation users
 come up with).
It seems to me it's time to have a UniqueArray proof of concept. Andrei
Feb 24 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Tuesday, 24 February 2015 at 22:11:36 UTC, Andrei Alexandrescu 
wrote:
 On 2/24/15 1:55 PM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei 
 Alexandrescu wrote:
 So: does DIP25 allow safe slices? Looks that way
I believe it does, but at the cost of forced reference counting. As I pointed out, the `ref` solution
What is the ref solution?
DIP25, in that it applies the semantics of `scope` to `ref`. The safety gained is thereby restricted to `ref`, but does not apply to slices, pointers and class references. That's the reason - as you already replied to Ivan - that RCArray cannot return a real slice.
 is not applicable to slices, therefore
 it needs to return an RCArray. This in turn forces an inc/dec 
 whenever
 it is sliced or copied, and - maybe worse - it requires all 
 consumers to
 support RCArray (or whatever other idiosyncratic RC 
 implementation users
 come up with).
It seems to me it's time to have a UniqueArray proof of concept.
How is that related to what I wrote above?
Feb 25 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Tuesday, 24 February 2015 at 19:40:35 UTC, Andrei Alexandrescu 
wrote:
 I modified Walter's sample code to this: 
 http://dpaste.dzfl.pl/f3d854feede9. It uses malloc for both the 
 array and the reference count, and also uses  trusted 
 minimally. I inserted assert()s here and there to clarify the 
 workings. Nothing big except for the careful use of  trusted.
Using malloc make the GC blind. It means you can't stored anything GCed in this. Which means you either need to enforce this at the interface level (I have no idea how) or remove all these trusted in the code. Generally using malloc is not the right way forward. I do think that, even for RCed resource, we want them backed by the GC. It will allow for cycle collection. The main problem with the GC is not allocation, it is collection, and collection won't fire if one do not leak.
 2. Michel's point 
 (https://issues.dlang.org/show_bug.cgi?id=14221) reveals the 
 largest issue with RC/GC integration. We need to find a fix for 
 it if we want to have the GC lift cycles.
There is a set of feature that create implicit sharing. GC is only one of them. They either need to be fixed (and here, you'll get my point about simple fix aggregating to something more complex) or make this thread safe.
 So: does DIP25 allow safe slices? Looks that way, though a 
 proof would be nice. Does it allow other safe interesting 
 structs that own data? Very likely.
As long as you don't plan to own an arbitrary sub graph.
 Does it allow really sophisticated ownership schemes? We need 
 to explore that.
Limited, unless you wrap all indirection in some sort of struct down the road.
 Does it protect against bugs in implementations of safe 
 ownership schemes that explicitly release memory? Not too well. 
 I think the prevalent idiom will be to accompany such artifacts 
 with unittests that make sure unsafe uses (such as fun() above) 
 do not compile.
I though we wanted to do better than C++...
Feb 24 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 2:45 PM, deadalnix wrote:
 Using malloc make the GC blind. It means you can't stored anything GCed in
this.
GC.addRange() solves this problem.
 Generally using malloc is not the right way forward. I do think that, even for
 RCed resource, we want them backed by the GC. It will allow for cycle
 collection.
If cycles are not possible, malloc/free should be used. Cycles are not possible in objects without indirections, and we can assert they are not possible it the type can be introspected at compile time to see if cycles are not possible.
 So: does DIP25 allow safe slices? Looks that way, though a proof would be
 nice. Does it allow other safe interesting structs that own data? Very likely.
As long as you don't plan to own an arbitrary sub graph.
To have an arbitrary sub graph be memory safe with return ref, the interface to it will have to be constructed to only allow access by values or return refs.
 Does it protect against bugs in implementations of safe ownership schemes that
 explicitly release memory? Not too well. I think the prevalent idiom will be
 to accompany such artifacts with unittests that make sure unsafe uses (such as
 fun() above) do not compile.
I though we wanted to do better than C++...
No language offers a way to check system code for safety. Not Java, not Rust, not C#, not nobody, not no how. What is offered is safety for the client's usage of it. C++ doesn't have that.
Feb 24 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 25 February 2015 at 00:11:28 UTC, Walter Bright 
wrote:
 Generally using malloc is not the right way forward. I do 
 think that, even for
 RCed resource, we want them backed by the GC. It will allow 
 for cycle
 collection.
If cycles are not possible, malloc/free should be used. Cycles are not possible in objects without indirections, and we can assert they are not possible it the type can be introspected at compile time to see if cycles are not possible.
Why ? There is very little chance that malloc + GC.addRange becomes any faster that GC.malloc in the first place.
 So: does DIP25 allow safe slices? Looks that way, though a 
 proof would be
 nice. Does it allow other safe interesting structs that own 
 data? Very likely.
As long as you don't plan to own an arbitrary sub graph.
To have an arbitrary sub graph be memory safe with return ref, the interface to it will have to be constructed to only allow access by values or return refs.
That means all library code must be duplicated. I could explain you why this is bad, but someone already made a very good point about it there: http://www.drdobbs.com/cpp/type-qualifiers-and-wild-cards/231902461
 Does it protect against bugs in implementations of safe 
 ownership schemes that
 explicitly release memory? Not too well. I think the 
 prevalent idiom will be
 to accompany such artifacts with unittests that make sure 
 unsafe uses (such as
 fun() above) do not compile.
I though we wanted to do better than C++...
No language offers a way to check system code for safety. Not Java, not Rust, not C#, not nobody, not no how. What is offered is safety for the client's usage of it. C++ doesn't have that.
A language is an API to the system you are running on. A very expressive and complex one, but an API nevertheless. A good API will make correct and safe use easy and incorrect and unsafe use convoluted, so they are done on purpose and not by mistake. That is not about proving system code correct (impossible) that is about making unprovable code harder to spit out than provable one (or getting an error).
Feb 24 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 2/24/2015 4:41 PM, deadalnix wrote:
 Why ? There is very little chance that malloc + GC.addRange becomes any faster
 that GC.malloc in the first place.
1. GC is designed to do GC, which is more than malloc/free is designed to do. Less work usually is faster. 2. There are many malloc/free implementations available, and people can and do swap them out to find one that works best for their application. Also, compiler vendors often expend a great deal of effort making malloc/free work well. Why not take advantage of that? It's like it's pretty hard to compete with the system memcpy() - people expend enormous effort on that.
 To have an arbitrary sub graph be memory safe with return ref, the interface
 to it will have to be constructed to only allow access by values or return
refs.
That means all library code must be duplicated.
Why?
 but someone already made a very good point about it there:
 http://www.drdobbs.com/cpp/type-qualifiers-and-wild-cards/231902461
Don't see how that applies.
Feb 24 2015