www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP77 - Fix unsafe RC pass by 'ref'

reply Walter Bright <newshound2 digitalmars.com> writes:
http://wiki.dlang.org/DIP77
Apr 08 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
Deceptively simple solution for a hard problem.
Apr 08 2015
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
That is pretty much the old Rust solution called boxing. This sound like the right way forward to me.
Apr 08 2015
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 5:30 PM, deadalnix wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
That is pretty much the old Rust solution called boxing. This sound like the right way forward to me.
Yes, it's equivalent to it. I had started with doing an INC/DEC pair, but that implied adding more logic to detect how to do an INC/DEC for a particular type. The logic for doing a copy is much more straightforward. The RC object should also be designed in such a way that the copy is not costly. Function purity is again showing what an advantage it is. Andrei's idea was to not do the copy for system opAssign's, thus providing C++ equivalence for those folks that need it and don't care about guaranteed memory safety.
Apr 08 2015
prev sibling parent reply "Brad Anderson" <eco gnuk.net> writes:
On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
That is pretty much the old Rust solution called boxing. This sound like the right way forward to me.
Any idea why they abandoned it?
Apr 08 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 6:19 PM, Brad Anderson wrote:
 On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
That is pretty much the old Rust solution called boxing. This sound like the right way forward to me.
Any idea why they abandoned it?
No, but I can guess. It's less efficient.
Apr 08 2015
parent "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 01:35:14 UTC, Walter Bright wrote:
 On 4/8/2015 6:19 PM, Brad Anderson wrote:
 On Thursday, 9 April 2015 at 00:30:46 UTC, deadalnix wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:
 http://wiki.dlang.org/DIP77
That is pretty much the old Rust solution called boxing. This sound like the right way forward to me.
Any idea why they abandoned it?
No, but I can guess. It's less efficient.
This and the fact that they want to prevent multiple writable burrow for concurrency reasons, so that seemed like the correct way forward for them.
Apr 08 2015
prev sibling next sibling parent reply "weaselcat" <weaselcat gmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
Can this be clarified a little?
The lifetime of tmp will be the same as that of a temporary with 
a destructor.
Does that mean the lifetime of tmp is tied to the scope of the function being called, or the current scope calling the function?
Apr 08 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/8/2015 5:49 PM, weaselcat wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
Can this be clarified a little?
 The lifetime of tmp will be the same as that of a temporary with a destructor.
Does that mean the lifetime of tmp is tied to the scope of the function being called, or the current scope calling the function?
The same as that of a tmp being returned from a function - to the end of the expression.
Apr 08 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 01:35:54 UTC, Walter Bright wrote:
 The same as that of a tmp being returned from a function - to 
 the end of the expression.
But how will this work with anything that makes RCArrays reachable through indirections...? Is the compiler going to do a recursive scan and create temporaries of all RC-pointers that are reachable? (e.g. RCArrays of RCArrays)
Apr 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Thursday, 9 April 2015 at 01:35:54 UTC, Walter Bright wrote:
 The same as that of a tmp being returned from a function - to the end of the
 expression.
But how will this work with anything that makes RCArrays reachable through indirections...? Is the compiler going to do a recursive scan and create temporaries of all RC-pointers that are reachable? (e.g. RCArrays of RCArrays)
It examines the types of all mutable values available to the function. Note the discussion of the effect of purity in the DIP.
Apr 09 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 21:47:47 UTC, Walter Bright wrote:
 On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
 indirections...? Is the compiler going to do a recursive scan 
 and create
 temporaries of all RC-pointers that are reachable?

 (e.g. RCArrays of RCArrays)
It examines the types of all mutable values available to the function.
So if I pass in a graph, linked list or a tree, then it will scan the entire structure at runtime and build a large set of temporaries before calling the function?
Apr 09 2015
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 05:56:40 UTC, Ola Fosheim Grøstad 
wrote:
 On Thursday, 9 April 2015 at 21:47:47 UTC, Walter Bright wrote:
 On 4/9/2015 12:58 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
 indirections...? Is the compiler going to do a recursive scan 
 and create
 temporaries of all RC-pointers that are reachable?

 (e.g. RCArrays of RCArrays)
It examines the types of all mutable values available to the function.
So if I pass in a graph, linked list or a tree, then it will scan the entire structure at runtime and build a large set of temporaries before calling the function?
No, but indeed, passing down indirection require boxing too. The DIP is not clear about it.
Apr 09 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 06:02:35 UTC, deadalnix wrote:
 No, but indeed, passing down indirection require boxing too. 
 The DIP is not clear about it.
Not sure what you mean by boxing. Usually it means that you wrap a value type in an object to turn it into a reference type. In Rust I think it means the same as unique_ptr ( http://rustbyexample.com/box.html ). How will DIP77 work out if you represent a directed acyclic graph with ref counted edges? (e.g. one RCArray per node) Basically the problem that needs to be solved is very close to determining if there is no aliasing between parameters. The compile time solution could be either whole program pointer analysis or heavy type constraints (linear typing). A general runtime solution would be rather inefficient IMO. I think memory safety in D probably could be better solved, and with less effort, by a separate tool if DMD could be made to generate high level IR (a D VM that preserves high level information better than say LLVM).
Apr 10 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 10:56 PM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 So if I pass in a graph, linked list or a tree, then it will scan the entire
 structure at runtime and build a large set of temporaries before calling the
 function?
Nope. It'll scan at compile time the types reachable through the parameters. If any of those types are RCO's that have a type that matches a ref parameter, the ref parameter's RCO will get copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 07:01:47 UTC, Walter Bright wrote:
 Nope. It'll scan at compile time the types reachable through 
 the parameters. If any of those types are RCO's that have a 
 type that matches a ref parameter, the ref parameter's RCO will 
 get copied.
But how can this work? If I provide 2 ref parameters to different Node instances (not a RCO) in the same graph. How can I then be sure that it will not disappear? E.g. ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X" ref2 -> Node"X"
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 10 April 2015 at 07:01:47 UTC, Walter Bright wrote:
 Nope. It'll scan at compile time the types reachable through the parameters.
 If any of those types are RCO's that have a type that matches a ref parameter,
 the ref parameter's RCO will get copied.
But how can this work? If I provide 2 ref parameters to different Node instances (not a RCO) in the same graph. How can I then be sure that it will not disappear? E.g. ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X" ref2 -> Node"X"
Because with two parameters, if one is a ref coming from an RCO rc1, and the other is a ref to a type that is the root of a graph that contains an RCO type, then the rc1 is copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 08:47:09 UTC, Walter Bright wrote:
 On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
 ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
 ref2 -> Node"X"
Because with two parameters, if one is a ref coming from an RCO rc1, and the other is a ref to a type that is the root of a graph that contains an RCO type, then the rc1 is copied.
ref1 -> RCO1 -> N1 -> RCRef1 -> RCO2 -> N2 -> RCRef3 -> RCO3 -> N3 ref2 -> N3 N1..N3 represent resources (like a file). process(ref1,ref2){ ref1.stuff.stuff.stuff.release(); // Node2 and Node3 is released. ref2.read(); // fails/garbage }
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:14 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 10 April 2015 at 08:47:09 UTC, Walter Bright wrote:
 On 4/10/2015 12:10 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?=
 ref1 -> Node->RCRef->Edge(RCO)->Node->RCRef->Edge(RCO)->Node"X"
 ref2 -> Node"X"
Because with two parameters, if one is a ref coming from an RCO rc1, and the other is a ref to a type that is the root of a graph that contains an RCO type, then the rc1 is copied.
ref1 -> RCO1 -> N1 -> RCRef1 -> RCO2 -> N2 -> RCRef3 -> RCO3 -> N3 ref2 -> N3 N1..N3 represent resources (like a file). process(ref1,ref2){ ref1.stuff.stuff.stuff.release(); // Node2 and Node3 is released. ref2.read(); // fails/garbage }
ref2 is copied.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 09:15:11 UTC, Walter Bright wrote:
 ref2 is copied.
You cannot copy a resource like a memfile. It is already closed and the memory is gone.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:27 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 10 April 2015 at 09:15:11 UTC, Walter Bright wrote:
 ref2 is copied.
You cannot copy a resource like a memfile. It is already closed and the memory is gone.
Please read the DIP again. The copy happens earlier.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 09:28:57 UTC, Walter Bright wrote:
 Please read the DIP again. The copy happens earlier.
But one cannot keep a stateful resource alive by copying memory. One would have to take ownership to keep it alive. In order to do that you need a model for ownership.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:54 AM, "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= 
<ola.fosheim.grostad+dlang gmail.com>" wrote:
 On Friday, 10 April 2015 at 09:28:57 UTC, Walter Bright wrote:
 Please read the DIP again. The copy happens earlier.
But one cannot keep a stateful resource alive by copying memory. One would have to take ownership to keep it alive. In order to do that you need a model for ownership.
That's what ref counting is about.
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:
 That's what ref counting is about.
That would require pervasive ref-counting.
Apr 10 2015
parent reply Marco Leise <Marco.Leise gmx.de> writes:
Am Fri, 10 Apr 2015 14:04:39 +0000
schrieb "Ola Fosheim Gr=C3=B8stad"
<ola.fosheim.grostad+dlang gmail.com>:

 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:
 That's what ref counting is about.
=20 That would require pervasive ref-counting.
You make is sound like there was an alternative to ref counting for resources. Is there? --=20 Marco
Apr 11 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 11 April 2015 at 18:20:51 UTC, Marco Leise wrote:
 Am Fri, 10 Apr 2015 14:04:39 +0000
 schrieb "Ola Fosheim Grøstad"
 <ola.fosheim.grostad+dlang gmail.com>:

 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:
 That's what ref counting is about.
That would require pervasive ref-counting.
You make is sound like there was an alternative to ref counting for resources. Is there?
Ownership, GC.
Apr 11 2015
parent Marco Leise <Marco.Leise gmx.de> writes:
Am Sat, 11 Apr 2015 20:42:52 +0000
schrieb "deadalnix" <deadalnix gmail.com>:

 On Saturday, 11 April 2015 at 18:20:51 UTC, Marco Leise wrote:
 Am Fri, 10 Apr 2015 14:04:39 +0000
 schrieb "Ola Fosheim Gr=C3=B8stad"
 <ola.fosheim.grostad+dlang gmail.com>:

 On Friday, 10 April 2015 at 10:28:03 UTC, Walter Bright wrote:
 That's what ref counting is about.
=20 That would require pervasive ref-counting.
You make is sound like there was an alternative to ref counting for resources. Is there?
=20 Ownership, GC.
That doesn't convince me yet. GC is just incorrect to use here. D's GC like any other that is non-deterministic will hold on to many external resources for longer than is feasible. (Unique) ownership on the other hand isn't really what we are looking at here, right? I mean, the DIP talks about what happens when you are dealing with multiple references to the same item. That leaves us with RC, which I think is a good choice. --=20 Marco
May 01 2015
prev sibling next sibling parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
I've only skimmed this, but: 1. I don't think Objective-C uses autorelease pools for any other reason than to make manual ref counting easier. See also John McCall's reflections on ARC: http://forum.dlang.org/thread/mcqcor$aa$1 digitalmars.com?page=2#post-hgmhgirfervrsvcghchw:40forum.dlang.org 2. How will this work with "yield"? Why not just implement the more generic solution (shared pointers with move/borrow or WPO) ?
Apr 09 2015
parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 09:05:10 UTC, Ola Fosheim Grøstad 
wrote:
 2. How will this work with "yield"?
You yield both caller and callee, so you'll get caller's boxing in the yield.
 Why not just implement the more generic solution (shared 
 pointers with move/borrow or WPO) ?
I don't think this is possible.
Apr 09 2015
parent "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Thursday, 9 April 2015 at 18:31:24 UTC, deadalnix wrote:
 On Thursday, 9 April 2015 at 09:05:10 UTC, Ola Fosheim Grøstad 
 wrote:
 2. How will this work with "yield"?
You yield both caller and callee, so you'll get caller's boxing in the yield.
But the coroutine stack and everything on it will be intact when it yields, including references to array elements...?
 Why not just implement the more generic solution (shared 
 pointers with move/borrow or WPO) ?
I don't think this is possible.
It should be possible with pointer analysis, but the easier approach is just to ban non-const ref parameters (c++ style) for rc-pointer-objects, so maybe D should provide "head const" after all...
Apr 09 2015
prev sibling next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-08 23:10:37 +0000, Walter Bright <newshound2 digitalmars.com> said:

 http://wiki.dlang.org/DIP77
In the definition of a Reference Counted Object: """ An object is assumed to be reference counted if it has a postblit and a destructor, and does not have an opAssign marked system. """ Why should it not have an opAssign marked system? And what happens if the struct has a postblit but it is disabled? Will the compiler forbid you from passing it by ref in cases where it'd need to make a copy, or will it just not be a RCO? More generally, is it right to add implicit copying just because a struct has a postblit and a destructor? If someone implemented a by-value container in D (such as those found in C++), this behaviour of the compiler would trash the performance by silently doing useless unnecessary copies. You won't even get memory-safety as a benefit: if the container allocates from the GC it's safe anyway, otherwise you're referencing deallocated memory with your ref parameter (copying the struct would just make a copy elsewhere, not retain the memory of the original). I think you're assuming too much from the presence of a postblit and a destructor. This implicit copy behaviour should not be trigged by seemingly unrelated clues. Instead of doing that: auto tmp = rc; the compiler should insert this: auto tmp = rc.opPin(); RCArray can implement opPin by returning a copy of itself. A by-value container can implement opPin by returning a dummy struct that retains the container's memory until the dummy struct's destructor is called. Alternatively someone could make a dummy "void opPin() system {}" to signal it isn't safe to pass internal references around (only in system code would the implicit call to opPin compile). If you were writing a layout-compatible D version of std::vector, you'd likely have to use a system opPin because there's no way you can "pin" that memory and guaranty memory-safety when passing references around. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Apr 09 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 5:05 AM, Michel Fortin wrote:
 Why should it not have an opAssign marked  system?
"Andrei's idea was to not do the copy for system opAssign's, thus providing C++ equivalence for those folks that need it and don't care about guaranteed memory safety."
 And what happens if the struct has a postblit but it is  disabled? Will the
 compiler forbid you from passing it by ref in cases where it'd need to make a
 copy, or will it just not be a RCO?
It wouldn't be an RCO.
 More generally, is it right to add implicit copying just because a struct has a
 postblit and a destructor? If someone implemented a by-value container in D
 (such as those found in C++), this behaviour of the compiler would trash the
 performance by silently doing useless unnecessary copies. You won't even get
 memory-safety as a benefit: if the container allocates from the GC it's safe
 anyway, otherwise you're referencing deallocated memory with your ref parameter
 (copying the struct would just make a copy elsewhere, not retain the memory of
 the original).
The only real purpose to a postblit is to support ref counting. Why would a by-value container use a postblit and not ref count?
 I think you're assuming too much from the presence of a postblit and a
 destructor. This implicit copy behaviour should not be trigged by seemingly
 unrelated clues. Instead of doing that:

      auto tmp = rc;

 the compiler should insert this:

      auto tmp = rc.opPin();

 RCArray can implement opPin by returning a copy of itself. A by-value container
 can implement opPin by returning a dummy struct that retains the container's
 memory until the dummy struct's destructor is called. Alternatively someone
 could make a dummy "void opPin()  system {}" to signal it isn't safe to pass
 internal references around (only in system code would the implicit call to
opPin
 compile). If you were writing a layout-compatible D version of std::vector,
 you'd likely have to use a  system opPin because there's no way you can "pin"
 that memory and guaranty memory-safety when passing references around.
My first impression is that's too complicated for the user to get right.
Apr 09 2015
next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:
 The only real purpose to a postblit is to support ref counting. 
 Why would a by-value container use a postblit and not ref count?
A struct could have a postblit defined if you are implementing something like std::vector, where you you copy the memory when the struct is copied. I'm not sure why you would want to do such a thing in D, though. If allocating memory is your concern, you probably don't want any allocation, including malloc.
Apr 09 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 11:53 AM, w0rp wrote:
 On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:
 The only real purpose to a postblit is to support ref counting. Why would a
 by-value container use a postblit and not ref count?
A struct could have a postblit defined if you are implementing something like std::vector, where you you copy the memory when the struct is copied. I'm not sure why you would want to do such a thing in D, though.
I'm not sure why you'd do that, either. Just make the memory part ref counted, then when modifying it, make the copy then if your ref count > 1. If you want to interface with std::vector, make opAssign system, after all, you're dealing with C++ :-)
Apr 09 2015
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:
 On 4/9/2015 5:05 AM, Michel Fortin wrote:
 Why should it not have an opAssign marked  system?
"Andrei's idea was to not do the copy for system opAssign's, thus providing C++ equivalence for those folks that need it and don't care about guaranteed memory safety."
Why not bind this behavior to extern(C++) ?
 The only real purpose to a postblit is to support ref counting. 
 Why would a by-value container use a postblit and not ref count?
In C++, the only real purpose of template were generic containers, and that didn't ended up well. It is dangerous, at language level, to reason from usage instead of first principle (not that usage do not matter, usage should serve as a guideline for the principles) or thing like C++ templates happens.
 My first impression is that's too complicated for the user to 
 get right.
Yeah, I don't think opPin is a the right way to go. It is always easy and tempting to add new stuff to make X or Y work, but at the end, it only create language complexity explosion. There are some inefficiencies involved here, but I trust the compiler to be able to optimize it away for the most part, and we have a backdoor for thoses who want to bypass safety. More generally, this is why I oppose the return attribute + adding op on reference type in favor of a principle scope proposal. The first one is 2 language features for a less general end result.
Apr 09 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/9/2015 1:21 PM, deadalnix wrote:
 On Thursday, 9 April 2015 at 18:44:10 UTC, Walter Bright wrote:
 On 4/9/2015 5:05 AM, Michel Fortin wrote:
 Why should it not have an opAssign marked  system?
"Andrei's idea was to not do the copy for system opAssign's, thus providing C++ equivalence for those folks that need it and don't care about guaranteed memory safety."
Why not bind this behavior to extern(C++) ?
Because the charter of system is "the user supplies the memory safety", which fits here perfectly. extern(C++) carries a lot of other stuff with it.
 It is dangerous, at language level, to reason from usage instead of first
 principle (not that usage do not matter, usage should serve as a guideline for
 the principles) or thing like C++ templates happens.
On the other hand, throwing things in the language just because you can, with no idea what they are good for or how they will be used, doesn't end well.
 My first impression is that's too complicated for the user to get right.
Yeah, I don't think opPin is a the right way to go. It is always easy and tempting to add new stuff to make X or Y work, but at the end, it only create language complexity explosion.
Glad you agree. I'm afraid of making something that is technically correct, but unusable.
 There are some inefficiencies involved here, but I trust the compiler to be
able
 to optimize it away for the most part, and we have a backdoor for thoses who
 want to bypass safety.
Yes. Interestingly, this is a superset of what Rust does. Rust's has the idea of only one mutable reference at a time, for efficiency. As the DIP points out, if the function only has const references available other than the one mutable one, it doesn't do the copy, either. I was curious how Rust handled the global variable issue. Turns out it doesn't - mutable global variables are marked as "unsafe" and the checker gives up on it.
 More generally, this is why I oppose the return attribute + adding op on
 reference type in favor of a principle scope proposal. The first one is 2
 language features for a less general end result.
I understand where you're coming from on this. The design was picked for being as simple as possible with the least disruption.
Apr 09 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 9 April 2015 at 22:06:01 UTC, Walter Bright wrote:
 I was curious how Rust handled the global variable issue. Turns 
 out it doesn't - mutable global variables are marked as 
 "unsafe" and the checker gives up on it.
I think it's safe to say that if even Rust with its linear type system and advanced borrow checker cannot deal with it, noone can, short of whole program analysis. And it's easy to understand why once you realize that it's all about aliasing. That's why I also suggest to make borrowing from mutable globals unsafe.
Apr 10 2015
prev sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-09 18:43:25 +0000, Walter Bright <newshound2 digitalmars.com> said:

 The only real purpose to a postblit is to support ref counting. Why 
 would a by-value container use a postblit and not ref count?
Because reference counting has tradoffs, or because someone translates an exiting C++ program and want to keep the semantics the same to make the translation simpler. If you want to limit postblit to the reference counting use case I guess that's fine, although this limitation should probably be clarified somewhere because to someone with a C++ background it looks a lot like a substitute for a copy-constructor. But this goes beyond containers. Destructors and postblits are implicitly created whenever a field contained within a struct has them. So you could have this: struct Big { RCArray!T array; int[1000] ints; } and now implicit copies of Big are made because Big implicitly gained a destructor and postblit from its array field. But at least you gain memory safety. ...or maybe not. Let's add another field to Big: struct Handle { // shouldn't be able to make copies of this disable this(this) {} } struct Big { RCArray!T array; int[1000] ints; Handle handle; } Now Big's implicit postblit becomes disabled too (because Handle can't handle copying), therefore it is no longer a RCO object and the compiler will no longer create temporary copies. Now you've trashed memory-safety. To be fair, my opPin idea suffers a similar fate here: there is no opPin defined for Big, which creates a hole in memory-safety. To compensate, the compiler would have to call opPin individually for each field that defines it. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Apr 10 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 4:49 AM, Michel Fortin wrote:
 ...or maybe not. Let's add another field to Big:

      struct Handle
      {
          // shouldn't be able to make copies of this
           disable this(this) {}
      }

      struct Big
      {
          RCArray!T array;
          int[1000] ints;
          Handle handle;
      }

 Now Big's implicit postblit becomes disabled too (because Handle can't handle
 copying), therefore it is no longer a RCO object and the compiler will no
longer
 create temporary copies. Now you've trashed memory-safety.
The copy would be made of array, not Big.
Apr 10 2015
prev sibling parent "Martin Nowak" <code dawg.eu> writes:
On Thursday, 9 April 2015 at 12:05:16 UTC, Michel Fortin wrote:
 """
 An object is assumed to be reference counted if it has a 
 postblit and a destructor, and does not have an opAssign marked 
  system.
 """
It seems to only fit to RCO, I'm not positive though. We should formalize this, by documenting it and by adding isRCO traits to phobos.
Apr 10 2015
prev sibling next sibling parent "Zach the Mystic" <reachzach gggmail.com> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
From the DIP: T foo(ref Payload payload); ... RC rc; foo(rc.payload); rewrite foo(rc) as: auto tmp = rc; foo(rc.payload); The only question I had was whether there was an advantage to rewriting the statement in its own scope: { auto tmp = rc; foo(rc.payload); } You can save some stack space that way, but I don't know how hard it is to implement.
Apr 09 2015
prev sibling next sibling parent reply "ixid" <adamsibson hotmail.com> writes:
 http://wiki.dlang.org/DIP77
We seem to be incrementally edging towards Rust's memory management system or something similar and retracing their steps along the path to get there. Hypothetically how close could we get to being able to opt into a Rust world from the GC world or C worlds of memory management?
Apr 10 2015
next sibling parent "w0rp" <devw0rp gmail.com> writes:
On Friday, 10 April 2015 at 09:16:38 UTC, ixid wrote:
 http://wiki.dlang.org/DIP77
We seem to be incrementally edging towards Rust's memory management system or something similar and retracing their steps along the path to get there. Hypothetically how close could we get to being able to opt into a Rust world from the GC world or C worlds of memory management?
extern(Rust) embraceExtendExtinguish();
Apr 10 2015
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:16 AM, ixid wrote:
 http://wiki.dlang.org/DIP77
We seem to be incrementally edging towards Rust's memory management system or something similar and retracing their steps along the path to get there. Hypothetically how close could we get to being able to opt into a Rust world from the GC world or C worlds of memory management?
As I explained to deadalnix in this thread, D's semantics in this regard are a superset of Rust's.
Apr 10 2015
prev sibling parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/10/15 2:16 AM, ixid wrote:
 http://wiki.dlang.org/DIP77
We seem to be incrementally edging towards Rust's memory management system or something similar and retracing their steps along the path to get there. Hypothetically how close could we get to being able to opt into a Rust world from the GC world or C worlds of memory management?
We hope to do better than Rust. -- Andrei
Apr 10 2015
parent reply "Ola Fosheim =?UTF-8?B?R3LDuHN0YWQi?= writes:
On Friday, 10 April 2015 at 15:44:32 UTC, Andrei Alexandrescu 
wrote:
 We hope to do better than Rust. -- Andrei
Hope does not cut it. A design that breaks down even on a linked list (e.g. you have to turn to ref-counted pointers everywhere) and that makes findByRef(container,node) break should not have made it to a DIP in the first place. D needs clean semantics on what is a value. The only sensible definition for a copyable value is that you cannot take the identity in _any way_ from a value or anything that can be reached from it. D needs clean semantics on what is unique ownership. Ref counting is not it. Pervasive ref counting requires WPO to work out ok. C++14 with sanitizer tooling is a lot more attractive than this direction you are going for.
Apr 11 2015
next sibling parent "w0rp" <devw0rp gmail.com> writes:
I was thinking about this again, and how the first 'ref' would 
create a copy in order to keep the object around. At first I 
thought I didn't like that, but then I realised that it's 
actually not far from what I wanted for taking r-values by 
reference. I commonly write functions which take values either by 
reference or by value in this way.

void foo(ref T value);
void foo(T value) { foo(value); }

Which obviously results in a combinatorial explosion of 
overloads, which you can write automatically some times with 
'auto ref'.

My question is, in the greater context of the language, supposing 
we had this copying behaviour for a first ref for reference 
counted objects, will it putting something in the language for 
copying r-values into functions, or will it provide a path to 
implement such a thing?
Apr 11 2015
prev sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 11 April 2015 at 08:35:18 UTC, Ola Fosheim Grøstad 
wrote:
 On Friday, 10 April 2015 at 15:44:32 UTC, Andrei Alexandrescu 
 wrote:
 We hope to do better than Rust. -- Andrei
Hope does not cut it. A design that breaks down even on a linked list (e.g. you have to turn to ref-counted pointers everywhere) and that makes findByRef(container,node) break should not have made it to a DIP in the first place.
a bidirectional linked list cannot be implemented in safe rust afaik
Apr 11 2015
prev sibling next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
So someone passes an RCO via ref to avoid the inc/dec, and because that imposes safety issues we turn it into some sort of pass by value under the hood, defeating the purpose, and provide an opt-out via system opAssign. Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs? Then the only thing missing to make this equally powerful, would be an optimization opportunity for the compiler to elide copies of pass-by-value RCOs, e.g. it could avoid calling the postblit when the function retains the refcount.
Apr 10 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 3:02 AM, Martin Nowak wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
So someone passes an RCO via ref to avoid the inc/dec, and because that imposes safety issues we turn it into some sort of pass by value under the hood, defeating the purpose, and provide an opt-out via system opAssign.
Or you could pass it by const ref (which is what Rust essentially does).
 Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs?
That's what we have now. It's not good enough.
Apr 10 2015
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/10/2015 12:29 PM, Walter Bright wrote:
 So someone passes an RCO via ref to avoid the inc/dec, and because
 that imposes
 safety issues we turn it into some sort of pass by value under the hood,
 defeating the purpose, and provide an opt-out via  system opAssign.
Or you could pass it by const ref (which is what Rust essentially does).
Maybe I'm missing something, but this proposal seems to make `ref RCO` fairly useless, because it creates a copy anyhow.
 Wouldn't it more straightforward to make pass-by-ref unsafe ( system)
 for RCOs?
That's what we have now. It's not good enough.
Assigning a RefCounted is marked system, pass-by-ref is safe. What's missing, you want to be able to use RefCounted in safe code? Why not pass it by value then? That would pin the object and you could elide the additional inc/dec just like you propose to elide the temporary copies. It's more efficient to pass a smart pointer like RefCounted by-value anyhow.
Apr 10 2015
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:20 PM, Martin Nowak wrote:
 On 04/10/2015 12:29 PM, Walter Bright wrote:
 So someone passes an RCO via ref to avoid the inc/dec, and because
 that imposes
 safety issues we turn it into some sort of pass by value under the hood,
 defeating the purpose, and provide an opt-out via  system opAssign.
Or you could pass it by const ref (which is what Rust essentially does).
Maybe I'm missing something, but this proposal seems to make `ref RCO` fairly useless, because it creates a copy anyhow.
Not if you read the copy elision section.
 Assigning a RefCounted is marked  system,
That's kind of throwing in the towel, isn't it?
 pass-by-ref is  safe.
 What's missing, you want to be able to use RefCounted in  safe code? Why
 not pass it by value then? That would pin the object and you could elide
 the additional inc/dec just like you propose to elide the temporary copies.
 It's more efficient to pass a smart pointer like RefCounted by-value anyhow.
Apr 10 2015
prev sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Friday, 10 April 2015 at 21:20:32 UTC, Martin Nowak wrote:
 Assigning a RefCounted is marked  system, pass-by-ref is  safe.
 What's missing, you want to be able to use RefCounted in  safe 
 code? Why
 not pass it by value then?
This actually sounds quite logical to me - passing an RC object by ref doesn't sound very safe to begin with.
Apr 10 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
So someone passes an RCO via ref to avoid the inc/dec, and because that imposes safety issues we turn it into some sort of pass by value under the hood, defeating the purpose, and provide an opt-out via system opAssign. Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs? Then the only thing missing to make this equally powerful, would be an optimization opportunity for the compiler to elide copies of pass-by-value RCOs, e.g. it could avoid calling the postblit when the function retains the refcount.
Only the first pass by ref create a copy. You can then pass the ref down all you want without copy. That is an acceptable cost IMO.
Apr 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 23:12:55 UTC, deadalnix wrote:
 On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:
 http://wiki.dlang.org/DIP77
So someone passes an RCO via ref to avoid the inc/dec, and because that imposes safety issues we turn it into some sort of pass by value under the hood, defeating the purpose, and provide an opt-out via system opAssign. Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs? Then the only thing missing to make this equally powerful, would be an optimization opportunity for the compiler to elide copies of pass-by-value RCOs, e.g. it could avoid calling the postblit when the function retains the refcount.
Only the first pass by ref create a copy. You can then pass the ref down all you want without copy. That is an acceptable cost IMO.
It's not acceptable that it happens behind the user's back. Costly operations must be explicit.
Apr 11 2015
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 It's not acceptable that it happens behind the user's back. Costly operations
 must be explicit.
Don't know of a better solution.
Apr 11 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:
 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 It's not acceptable that it happens behind the user's back. 
 Costly operations
 must be explicit.
Don't know of a better solution.
How about this? http://wiki.dlang.org/User:Schuetzm/scope3#.40safe-ty_violations_with_borrowing Btw, I also made other changes: No implied scope for safe functions, no overloading on scope (instead postblit and destructor are skipped), and added a terminology section (rather important!).
Apr 11 2015
next sibling parent reply "matovitch" <camille.brugel laposte.net> writes:
On Saturday, 11 April 2015 at 11:33:51 UTC, Marc Schütz wrote:
 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:
 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 It's not acceptable that it happens behind the user's back. 
 Costly operations
 must be explicit.
Don't know of a better solution.
How about this? http://wiki.dlang.org/User:Schuetzm/scope3#.40safe-ty_violations_with_borrowing Btw, I also made other changes: No implied scope for safe functions, no overloading on scope (instead postblit and destructor are skipped), and added a terminology section (rather important!).
Just passing, a bit off topic and clearly not familiar enough with the discussed subject but in case you missed it the last rust blog post is an nice and motivated introduction to their ownership system : http://blog.rust-lang.org/2015/04/10/Fearless-Concurrency.html I need to read about Marc's scope proposal...I am not convinced by the dip77, what about raii managed classes with costly opAssign operator (like in w0rn example) ? The goal of passing an object by reference is to avoid the copy and here you guess the parameter is ref-counted and made a (pseudo-)copy ? Then any allocating opAssign should be marked system ?
Apr 11 2015
parent reply Andrei Alexandrescu <SeeWebsiteForEmail erdani.org> writes:
On 4/11/15 5:35 AM, matovitch wrote:
 I am not convinced by the dip77, what about raii managed classes with
 costly opAssign operator (like in w0rn example) ? The goal of passing an
 object by reference is to avoid the copy and here you guess the
 parameter is ref-counted and made a (pseudo-)copy ? Then any allocating
 opAssign should be marked  system ?
Expensive opAssign or expensive postblit? -- Andrei
Apr 11 2015
parent "matovitch" <camille.brugel laposte.net> writes:
On Sunday, 12 April 2015 at 04:18:54 UTC, Andrei Alexandrescu 
wrote:
 On 4/11/15 5:35 AM, matovitch wrote:
 I am not convinced by the dip77, what about raii managed 
 classes with
 costly opAssign operator (like in w0rn example) ? The goal of 
 passing an
 object by reference is to avoid the copy and here you guess the
 parameter is ref-counted and made a (pseudo-)copy ? Then any 
 allocating
 opAssign should be marked  system ?
Expensive opAssign or expensive postblit? -- Andrei
Well both in the example given by w0rp. I guess D as no "rule of five" but still a "rule of three" (which is in fact rule of four with the copy-and-swap idiom) about coding posblit, opAssign and destructor together ?
Apr 12 2015
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright wrote:
 On 4/11/2015 2:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 It's not acceptable that it happens behind the user's back. Costly operations
 must be explicit.
Don't know of a better solution.
How about this? http://wiki.dlang.org/User:Schuetzm/scope3#.40safe-ty_violations_with_borrowing Btw, I also made other changes: No implied scope for safe functions, no overloading on scope (instead postblit and destructor are skipped), and added a terminology section (rather important!).
A quick read of this suggests it is doing the Rust model of only one mutable reference at a time. Is this really viable with D?
Apr 12 2015
next sibling parent "matovitch" <camille.brugel laposte.net> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:
 A quick read of this suggests it is doing the Rust model of 
 only one mutable reference at a time. Is this really viable 
 with D?
I would love to see a storage class to add the rust ownership model to D. I surely don't know if it's feasible.
Apr 12 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:
 On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 http://wiki.dlang.org/User:Schuetzm/scope3#.40safety_violations_with_borrowing
A quick read of this suggests it is doing the Rust model of only one mutable reference at a time. Is this really viable with D?
I think it is. You have to keep in mind that this is opt-in; it only applies to `scope` variables. I would agree that as a default, it wouldn't fit D at all. I think that, for all its usefulness, it'd be a tad too limiting for my taste if I were forced to use it everywhere. In Rust, it needs to be ubiquitous, because their goal is to get by without a GC. We on the other hand just want a way to make the GC easier to avoid. We also want to be able to use these different programming styles side by side, therefore an opt-in solution is good. Besides, I too want to recommend the article Matovitch linked to, and Yehuda Katz' article mentioned therein, because they make a good point demonstrating that it gives more correctness guarantees than just memory safety: http://forum.dlang.org/post/ttjzihuffrdlgqonqiwi forum.dlang.org It's a great building block for lots of nice things. In fact, it's quite likely the only way to get these guarantees once you look beyond reference counting.
Apr 12 2015
parent reply "bearophile" <bearophileHUGS lycos.com> writes:
Marc Schütz:

 You have to keep in mind that this is opt-in; it only applies 
 to `scope` variables. I would agree that as a default, it 
 wouldn't fit D at all. I think that, for all its usefulness, 
 it'd be a tad too limiting for my taste if I were forced to use 
 it everywhere.
I think D Zen asks for safety on default and opt-out on request. See also " safe by default": https://issues.dlang.org/show_bug.cgi?id=13838 Bye, bearophile
Apr 12 2015
parent "Martin Nowak" <code dawg.eu> writes:
On Sunday, 12 April 2015 at 12:06:50 UTC, bearophile wrote:
 I think D Zen asks for safety on default and opt-out on request.
Marc talked about unique ownership vs. shared ownership, not about safe vs. unsafe.
Apr 12 2015
prev sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Sunday, 12 April 2015 at 07:17:07 UTC, Walter Bright wrote:
 On 4/11/2015 4:33 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 On Saturday, 11 April 2015 at 09:41:07 UTC, Walter Bright 
 wrote:
A quick read of this suggests it is doing the Rust model of only one mutable reference at a time. Is this really viable with D?
It's a very good ownership model IMO, because it has no runtime overhead and often sharing/escaping isn't needed. Would at least be interesting to explore this in more detail for scoped variables. As a side note, I find that unique ownership is strangely missing from our discussions, yet it's a perfect ownership model for resources such as a File. https://github.com/D-Programming-Language/phobos/pull/3171
Apr 12 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Sunday, 12 April 2015 at 13:12:05 UTC, Martin Nowak wrote:
 As a side note, I find that unique ownership is strangely 
 missing from our discussions, yet it's a perfect ownership 
 model for resources such as a File.
 https://github.com/D-Programming-Language/phobos/pull/3171
I'm aware of it, but didn't want to open another can of worms. I have a vague idea what to do about it, based on deadalnix' "isolated" idea and the already existing uniqueness concept. I need to make up my mind about moving though, whether the benefits of making the compiler know about it are worth the added complexity.
Apr 12 2015
prev sibling next sibling parent reply "w0rp" <devw0rp gmail.com> writes:
On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:
 On Friday, 10 April 2015 at 23:12:55 UTC, deadalnix wrote:
 On Friday, 10 April 2015 at 10:02:01 UTC, Martin Nowak wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:
 http://wiki.dlang.org/DIP77
So someone passes an RCO via ref to avoid the inc/dec, and because that imposes safety issues we turn it into some sort of pass by value under the hood, defeating the purpose, and provide an opt-out via system opAssign. Wouldn't it more straightforward to make pass-by-ref unsafe ( system) for RCOs? Then the only thing missing to make this equally powerful, would be an optimization opportunity for the compiler to elide copies of pass-by-value RCOs, e.g. it could avoid calling the postblit when the function retains the refcount.
Only the first pass by ref create a copy. You can then pass the ref down all you want without copy. That is an acceptable cost IMO.
It's not acceptable that it happens behind the user's back. Costly operations must be explicit.
If we somehow know for a fact that the object really is a reference counted object, then the cost should be acceptable. Running the postblit will consist of incrementing a reference count and nothing else. There's no other way I can think of that permits passing the object safely. I think the one thing in the DIP I'm not sure of is the definition of what is a reference counted object. Say you write this. struct Widget { byte* data; size_t length; trusted opAssign(ref const(Widget) other) { length = other.length; data = malloc(length); memcpy(cast(void*) data, cast(void*) other.data, length); } trusted this(this) { auto oldData = data; data = malloc(length); memcpy(cast(void*) data, cast(void*) oldData, length); } trusted ~this() { free(data); } } Why would you want to do this? Who knows. According to DIP77 the object above is defined to be a reference counted object, when it isn't. If we're rejecting the C++ answer of "don't write code that way," then the same rejection should apply here. We need to be able to say with 100% certainty that we are dealing with a reference counted object, or at least an object where we know the postblit is so trivial the cost of calling it will be small.
Apr 11 2015
parent "weaselcat" <weaselcat gmail.com> writes:
On Saturday, 11 April 2015 at 10:14:00 UTC, w0rp wrote:

 I think the one thing in the DIP I'm not sure of is the 
 definition of what is a reference counted object.
I believe it's still defined by DIP74? http://wiki.dlang.org/DIP74
Apr 11 2015
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:
 It's not acceptable that it happens behind the user's back. 
 Costly operations must be explicit.
Nothing is costly here.
Apr 11 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 20:40:20 UTC, deadalnix wrote:
 On Saturday, 11 April 2015 at 09:28:46 UTC, Marc Schütz wrote:
 It's not acceptable that it happens behind the user's back. 
 Costly operations must be explicit.
Nothing is costly here.
Of course it is: http://forum.dlang.org/post/iehijjgdlfouplcngrjy forum.dlang.org
Apr 12 2015
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In general, this is way too much focused on reference counting. The DIP does define "RCO" in more general terms, but still, it feels like the wrong approach. You also seem to look at Rust's type system from the same POV ("only one mutable reference at a time, for efficiency"), but that's not the point at all: it's about safety in general, not efficiency for RC. It encompasses so much more: memory safety, race-free sharing of data, avoiding iterator invalidation... DIP77 cannot help with general iterator invalidation at all. Another thing that just feels wrong is the automagic copying. We're passing something by `ref`, and the compiler inserts a copy behind our back?! And this isn't even visible to the user... Because the topic is correctness, safe/ system seems the correct way to approach the problem. The compiler should just treat unsafe pass-by-ref as system, as Martin suggests. Basically, instead of inserting a copy, pass-by-ref will be system, when no copy would need to be inserted, it will be safe. I also think that the unsafety detection heuristic is too simplistic. When scope/return is extended to pointers and other kinds of references, there would probably be way too many false positives. A more sophisticated analysis is necessary that can more realistically determine when mutable aliasing can occur. E.g., it can also take uniqueness into account. Besides, the address and slice operators also need to be taken into account, and these can appear outside of function calls.
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In general, this is way too much focused on reference counting. The DIP does define "RCO" in more general terms, but still, it feels like the wrong approach. You also seem to look at Rust's type system from the same POV ("only one mutable reference at a time, for efficiency"), but that's not the point at all: it's about safety in general, not efficiency for RC. It encompasses so much more: memory safety, race-free sharing of data, avoiding iterator invalidation... DIP77 cannot help with general iterator invalidation at all.
D uses ranges, not iterators.
 Another thing that just feels wrong is the automagic copying. We're passing
 something by `ref`, and the compiler inserts a copy behind our back?! And this
 isn't even visible to the user...

 Because the topic is correctness,  safe/ system seems the correct way to
 approach the problem. The compiler should just treat unsafe pass-by-ref as
  system, as Martin suggests. Basically, instead of inserting a copy,
pass-by-ref
 will be  system, when no copy would need to be inserted, it will be  safe.
A difficulty with that is the semantics will change as the compiler improves and gets better at realizing no copy would be needed. Having the safety be "implementation defined" would be unhappy.
 I also think that the unsafety detection heuristic is too simplistic. When
 scope/return is extended to pointers and other kinds of references, there would
 probably be way too many false positives. A more sophisticated analysis is
 necessary that can more realistically determine when mutable aliasing can
occur.
 E.g., it can also take uniqueness into account.

 Besides, the address and slice operators also need to be taken into account,
and
 these can appear outside of function calls.
I expect any use of RCO's necessarily means the implementation of the RCO is unsafe (after all, it calls free()), but a safe interface is provided. The safe interface to an RCO is tightly controlled with 'return ref'.
Apr 10 2015
parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 17:43:00 UTC, Walter Bright wrote:
 On 4/10/2015 7:23 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 On Wednesday, 8 April 2015 at 23:11:08 UTC, Walter Bright 
 wrote:
 http://wiki.dlang.org/DIP77
In general, this is way too much focused on reference counting. The DIP does define "RCO" in more general terms, but still, it feels like the wrong approach. You also seem to look at Rust's type system from the same POV ("only one mutable reference at a time, for efficiency"), but that's not the point at all: it's about safety in general, not efficiency for RC. It encompasses so much more: memory safety, race-free sharing of data, avoiding iterator invalidation... DIP77 cannot help with general iterator invalidation at all.
D uses ranges, not iterators.
I used "iterator invalidation" because it is an established term. You get the same problems with ranges, once the contents aren't GC managed. And even if they are, your code is merely memory safe, but not necessarily correct. stdin.byLine() without idup is a prime example.
 Another thing that just feels wrong is the automagic copying. 
 We're passing
 something by `ref`, and the compiler inserts a copy behind our 
 back?! And this
 isn't even visible to the user...

 Because the topic is correctness,  safe/ system seems the 
 correct way to
 approach the problem. The compiler should just treat unsafe 
 pass-by-ref as
  system, as Martin suggests. Basically, instead of inserting a 
 copy, pass-by-ref
 will be  system, when no copy would need to be inserted, it 
 will be  safe.
A difficulty with that is the semantics will change as the compiler improves and gets better at realizing no copy would be needed. Having the safety be "implementation defined" would be unhappy.
Agreed, it needs to be specified. I'm aware that this is difficult. Though, why can't we use the same approach we use in other cases? Be restrictive at first, later accept more, as we refine the specification... At some point we'll find the sweet spot between false negatives and complexity of specification, and can stabilize on it, so that other compilers have a non-moving target.
 I also think that the unsafety detection heuristic is too 
 simplistic. When
 scope/return is extended to pointers and other kinds of 
 references, there would
 probably be way too many false positives. A more sophisticated 
 analysis is
 necessary that can more realistically determine when mutable 
 aliasing can occur.
 E.g., it can also take uniqueness into account.

 Besides, the address and slice operators also need to be taken 
 into account, and
 these can appear outside of function calls.
I expect any use of RCO's necessarily means the implementation of the RCO is unsafe (after all, it calls free()), but a safe interface is provided. The safe interface to an RCO is tightly controlled with 'return ref'.
Uhm... how is this related to what I wrote? Example: void foo() safe { RCArray!int arr = [0,1,2]; { int* p = &arr[0]; // legal under new scope rules RCArray!int other; arr = other; writeln(*p); // OOPS } } This shows that with the extended scope rules (i.e. scope not restricted to `ref`), the borrowing can happen everywhere, not just on function calls. By your rules, every address and slice operator applied to a mutable local would need to trigger the copying, because the resulting (borrowed) reference exists in the same scope as the original variable (owner). I'd prefer instead to make writing to the owner unsafe, as long as borrowed references to it exist (this also includes potential writes, like passing a mutable ref to another function). This doesn't require copying behind the user's back, and allows the user to choose the solution best suited for the situation: manual copying, deferring the write to the owner, trusted...
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 Example:

 void foo()  safe {
      RCArray!int arr = [0,1,2];
      {
          int* p = &arr[0];  // legal under new scope rules
This would be a bad design of an RCO. RCO's must be constructed to not allow pointers to the payload other than by ref.
          RCArray!int other;
          arr = other;
          writeln(*p);       // OOPS
      }
 }
Apr 10 2015
next sibling parent reply "Martin Nowak" <code dawg.eu> writes:
On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:
 This would be a bad design of an RCO. RCO's must be constructed 
 to not allow pointers to the payload other than by ref.
And taking the address of that is already unsafe.
Apr 11 2015
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Saturday, 11 April 2015 at 09:15:19 UTC, Martin Nowak wrote:
 On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:
 This would be a bad design of an RCO. RCO's must be 
 constructed to not allow pointers to the payload other than by 
 ref.
And taking the address of that is already unsafe.
It isn't under my proposal, which Walter has already accepted the most important part of.
Apr 11 2015
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:
 On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= 
 <schuetzm gmx.net>" wrote:
 Example:

 void foo()  safe {
     RCArray!int arr = [0,1,2];
     {
         int* p = &arr[0];  // legal under new scope rules
This would be a bad design of an RCO. RCO's must be constructed to not allow pointers to the payload other than by ref.
There is no reason for this restriction. But if this is your opinion, why did you agree to "implement 'scope' and 'return' for arrays, classes, and pointers"? http://forum.dlang.org/post/mfhkbm$2vbk$1 digitalmars.com It makes no sense to implement that, but not allow it to be used safely.
Apr 11 2015
parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/11/2015 2:18 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>" wrote:
 On Friday, 10 April 2015 at 21:26:14 UTC, Walter Bright wrote:
 On 4/10/2015 11:28 AM, "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net>"
wrote:
 Example:

 void foo()  safe {
     RCArray!int arr = [0,1,2];
     {
         int* p = &arr[0];  // legal under new scope rules
This would be a bad design of an RCO. RCO's must be constructed to not allow pointers to the payload other than by ref.
There is no reason for this restriction.
The reason is to prevent unsafe access of the payload.
 But if this is your opinion, why did
 you agree to "implement 'scope' and 'return' for arrays, classes, and
pointers"?

 http://forum.dlang.org/post/mfhkbm$2vbk$1 digitalmars.com

 It makes no sense to implement that, but not allow it to be used safely.
In your example posted upthread, it wasn't safe. Perhaps scoped pointers can't be made safe for RCOs, or perhaps the proposal needs more thinking. I don't know at the moment.
Apr 11 2015
prev sibling next sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 01:10 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In the first problem example: struct S { RCArray!T array; } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s.array[0]); // pass by ref } void foo(ref S s, ref T t) { s.array = RCArray!T([]); // drop the old s.array t.doSomething(); // oops, t is gone } What do you do to pin s.array? auto tmp = s; or auto tmp = s.array;
Apr 10 2015
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:11 PM, Martin Nowak wrote:
 On 04/09/2015 01:10 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In the first problem example: struct S { RCArray!T array; } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s.array[0]); // pass by ref } void foo(ref S s, ref T t) { s.array = RCArray!T([]); // drop the old s.array t.doSomething(); // oops, t is gone } What do you do to pin s.array? auto tmp = s; or auto tmp = s.array;
The latter.
Apr 10 2015
next sibling parent reply Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-10 21:29:19 +0000, Walter Bright <newshound2 digitalmars.com> said:

 On 4/10/2015 2:11 PM, Martin Nowak wrote:
 On 04/09/2015 01:10 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In the first problem example: struct S { RCArray!T array; } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s.array[0]); // pass by ref } void foo(ref S s, ref T t) { s.array = RCArray!T([]); // drop the old s.array t.doSomething(); // oops, t is gone } What do you do to pin s.array? auto tmp = s; or auto tmp = s.array;
The latter.
And how is it pinned in this case? struct S { private RCArray!T array; ref T opIndex(int index) return { return array[index]; } void clear() { s.array = RCArray!T([]); } } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s[0]); // pass by ref } void foo(ref S s, ref T t) { s.clear(); // drop the old s.array t.doSomething(); // oops, t is gone } -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Apr 10 2015
next sibling parent "weaselcat" <weaselcat gmail.com> writes:
On Friday, 10 April 2015 at 21:50:13 UTC, Michel Fortin wrote:
 On 2015-04-10 21:29:19 +0000, Walter Bright 
 <newshound2 digitalmars.com> said:

 On 4/10/2015 2:11 PM, Martin Nowak wrote:
 On 04/09/2015 01:10 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
In the first problem example: struct S { RCArray!T array; } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s.array[0]); // pass by ref } void foo(ref S s, ref T t) { s.array = RCArray!T([]); // drop the old s.array t.doSomething(); // oops, t is gone } What do you do to pin s.array? auto tmp = s; or auto tmp = s.array;
The latter.
And how is it pinned in this case? struct S { private RCArray!T array; ref T opIndex(int index) return { return array[index]; } void clear() { s.array = RCArray!T([]); } } void main() { auto s = S(RCArray!T([T()])); // s.array's refcount is now 1 foo(s, s[0]); // pass by ref } void foo(ref S s, ref T t) { s.clear(); // drop the old s.array t.doSomething(); // oops, t is gone }
AFAICT it would create a temporary copy of s.array, which would keep t valid.
 Nope. It'll scan at compile time the types reachable through 
 the parameters. If any of those types are RCO's that have a 
 type that matches a ref parameter, the
ref parameter's RCO will get copied.
Apr 10 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 2:50 PM, Michel Fortin wrote:
 And how is it pinned in this case?

   struct S {
       private RCArray!T array;
       ref T opIndex(int index) return { return array[index]; }
       void clear() { s.array = RCArray!T([]); }
   }
   void main() {
       auto s = S(RCArray!T([T()])); // s.array's refcount is now 1
       foo(s, s[0]);           // pass by ref
The s[0] is preceded by tmp=s; Because T is reachable from typeof(first argument s)
   }
   void foo(ref S s, ref T t) {
       s.clear();            // drop the old s.array
       t.doSomething();              // oops, t is gone
   }
Apr 10 2015
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/10/2015 11:29 PM, Walter Bright wrote:
 
 The latter.
Can you update that part in the DIP, it wasn't clear that the temporary selectively pins RCO fields of a normal struct passed by ref.
Apr 10 2015
next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Friday, 10 April 2015 at 23:18:59 UTC, Martin Nowak wrote:
 On 04/10/2015 11:29 PM, Walter Bright wrote:
 
 The latter.
Can you update that part in the DIP, it wasn't clear that the temporary selectively pins RCO fields of a normal struct passed by ref.
If a struct has RCO fields, shouldn't it be an RCO itself, and as such be pinned ? It sounds like this is implied in the DIP.
Apr 10 2015
parent Michel Fortin <michel.fortin michelf.ca> writes:
On 2015-04-10 23:22:17 +0000, "deadalnix" <deadalnix gmail.com> said:

 On Friday, 10 April 2015 at 23:18:59 UTC, Martin Nowak wrote:
 On 04/10/2015 11:29 PM, Walter Bright wrote:
 
 The latter.
Can you update that part in the DIP, it wasn't clear that the temporary selectively pins RCO fields of a normal struct passed by ref.
If a struct has RCO fields, shouldn't it be an RCO itself, and as such be pinned ?
Not necessarily. A disabled postblit could make it no longer RCO (including a disabled postblit in one of the fields).
 It sounds like this is implied in the DIP.
That's what I thought too. But when confronted to a case where that wouldn't work Walter said in this thread that the compiler would make a temporary of the fields. So I'm not too sure what to think anymore. The DIP should clarify what happens with disabled postblit and RCO fields inside non-RCO structs. -- Michel Fortin michel.fortin michelf.ca http://michelf.ca
Apr 10 2015
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 4/10/2015 4:18 PM, Martin Nowak wrote:
 On 04/10/2015 11:29 PM, Walter Bright wrote:
 The latter.
Can you update that part in the DIP, it wasn't clear that the temporary selectively pins RCO fields of a normal struct passed by ref.
It pins RCO objects, however they are derived.
Apr 10 2015
prev sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 04/09/2015 01:10 AM, Walter Bright wrote:
 http://wiki.dlang.org/DIP77
It's a very interesting proposal to tackle this specific problem. As with all the scope/lifetime related stuff, I find it extremely difficult to anticipate all the needs and foresee how this will integrate with the rest of the language, or even the other scope related DIPs. At this point it might be better to make a dmd branch and throw up a prototype for all the scope/ref-counted work. The litmus test for a successful design would be safe, efficient, nogc containers.
Apr 11 2015