www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.announce - Ownership and Borrowing in D

reply Mike Parker <aldacron gmail.com> writes:
In the Draft Review for Walter's DIP, Argument Ownership and 
Function Calls (which in the next half hour or so will be 
starting a community review as DIP 1021), some folks asked for 
some information about the bigger picture. In response, Walter 
has put together a post outlining his current thinking on how to 
add Ownership and Borrowing to D.

This is just an overview and he's currently working on a more 
comprehensive proposal, but it should provide some context for 
those who want it when reviewing DIP 1021.

The blog:
https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/

Reddit:
https://www.reddit.com/r/programming/comments/cdifbu/ownership_and_borrowing_in_d/
Jul 15
next sibling parent reply Mike Franklin <slavo5150 yahoo.com> writes:
On Monday, 15 July 2019 at 14:58:55 UTC, Mike Parker wrote:
 In the Draft Review for Walter's DIP, Argument Ownership and 
 Function Calls (which in the next half hour or so will be 
 starting a community review as DIP 1021), some folks asked for 
 some information about the bigger picture. In response, Walter 
 has put together a post outlining his current thinking on how 
 to add Ownership and Borrowing to D.
Thank you, Walter. This is very exciting. I've said that if I could have my ideal language, it would be Dust, and now I just might get it :) Mike
Jul 15
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 7:07 PM, Mike Franklin wrote:
 Thank you, Walter.  This is very exciting.  I've said that if I could have
my 
 ideal language, it would be Dust, and now I just might get it :)
Now I just have to deliver the goods!
Jul 15
next sibling parent Newbie2019 <newbie2019 gmail.com> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:
 Now I just have to deliver the goods!
Very exciting about d have the live and borrow owner feature. How do you plan to handle move ? dose it depends on DIP1014 ?
Jul 16
prev sibling next sibling parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:
 Now I just have to deliver the goods!
Lately, I've been thinking about the possibility of an alternative ownership system for D, one that would be radically different from what you're considering, but still aim for the same features (memory safety, compile-time checking, zero-cost abstraction), based on a `unique` qualifier. If I were to write a formal proposal for it, how interested would you be in comparing the two schemes (DIP 1021 and eventually Rust "one mutable ref" rule, vs unique qualifier)? Like, I want to make my pitch, but I don't want to spend huge amount of effort on it if you're just going to go with DIP 1021 anyway.
Jul 17
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/17/2019 12:11 PM, Olivier FAURE wrote:
 On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:
 Now I just have to deliver the goods!
Lately, I've been thinking about the possibility of an alternative ownership system for D, one that would be radically different from what you're considering, but still aim for the same features (memory safety, compile-time checking, zero-cost abstraction), based on a `unique` qualifier. If I were to write a formal proposal for it, how interested would you be in comparing the two schemes (DIP 1021 and eventually Rust "one mutable ref" rule, vs unique qualifier)? Like, I want to make my pitch, but I don't want to spend huge amount of effort on it if you're just going to go with DIP 1021 anyway.
Any competing system would need to not be 'opt-in' on a type by type basis. I.e. the central feature of an live function is the user will not be able to write memory unsafe code within that function. I'm interested to see your design. But I suggest you move quickly, as I've suggested people who talked about this propose a design for the last 10 years, nothing has happened, and we can't wait any longer.
Jul 17
next sibling parent Timon Gehr <timon.gehr gmx.ch> writes:
On 17.07.19 22:59, Walter Bright wrote:
 
 Any competing system would need to not be 'opt-in' on a type by type 
 basis. I.e. the central feature of an  live function is the user will 
 not be able to write memory unsafe code within that function.
Those two things are not the least bit at odds with each other. You only need to do the additional checks for types that actually need it. (And there you need to _always_ do them in safe code, not just in specially-annotated " live" functions.) A dynamic array of integers that is owned by the garbage collector doesn't need any O/B semantics. A malloc-backed array that wants to borrow out its contents needs to be able to restrict safe access patterns to ensure that the memory stays alive for the duration of the borrow. Furthermore, making built-in types change meaning based on function attributes is just not a good idea, because you will get needless friction at the interface between functions with the attribute and functions without. Anyway, it makes no sense to have a variable of type `int*` that owns the `int` it points to (especially in safe code), because that type doesn't track how to deallocate the memory. Ownership and borrowing should be supported for user-defined types, not raw pointers. Rust doesn't track ownership for built-in pointers. In Rust, raw pointer access is unsafe.
Jul 19
prev sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Wednesday, 17 July 2019 at 20:59:27 UTC, Walter Bright wrote:
 I'm interested to see your design. But I suggest you move 
 quickly, as I've suggested people who talked about this propose 
 a design for the last 10 years, nothing has happened, and we 
 can't wait any longer.
For the record, I've been busy with WebAssembly proposals and haven't had the time to write a proper DIP this week. That said, here's a **very, very rough** draft that conveys what the proposed design is about: https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088 I'll try to make a PR before next week-end.
Jul 21
prev sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 16 July 2019 at 06:12:42 UTC, Walter Bright wrote:
 Now I just have to deliver the goods!
I am very excited about this, together with unique these are some very desired semantics. I have a practical question. In spasm I have to solve the lifetime of javascript objects that are referenced from D. The way it works is that objects in Javascript are put in a big array and the index - the handle if you will - is passed to D. When D code wants to do anything with the js object, it passes the handle to a js glue function which looks up the object in the table and does what it needs to. The trick is finding an efficient mechanism to free the js objects in the js array. At first I tried reference counting - which worked, but introduced a lot of code bloat. Now I have settled on having the wrapper struct (JsHandle) of the handle be non-copyable, and pass it around by either moving or using `ref scope` with dip1000 and safe. Then if you need to have the handle around for longer you can always wrap it in a RC wrapper. This works remarkably well. It gives cheap, efficient and safe code and still allows multiple long-lived references with RC if that is what is wanted. There are 2 issues I currently have. First some code: --- alias Handle = uint; struct JsHandle { nothrow: package Handle handle; alias handle this; ~this() { if (handle != invalid) spasm_removeObject(handle); } disable this(this); } struct HTMLElement { spasm.bindings.dom.Element _parent; alias _parent this; this(JsHandle h) { _parent = .Element(h); } // ... functions } struct HTMLCanvasElement { spasm.bindings.html.HTMLElement _parent; alias _parent this; this(JsHandle h) { _parent = .HTMLElement(h); } // ... functions } --- 1) Sometimes the web api's accept an Nullable/Optional, which means I have to wrap, for instance, the HTMLElement into an Optional type and pass that. Of course, I want to maintain the unique reference constraint and enforce that the Optional won't outlive the original HTMLElement. Which I can't, since they are all non-pointers and I can't store a scope ref inside a Optional. The solution I have now is to create a Nullable!(T*). This does solve the unique semantics, at the expense of having an api that accepts both Nullable!(T) and Nullable!(T*). Pretty ugly. 2) the other issue is that I sometimes want to "upcast" a HTMLElement to a HTMLCanvasElement. I can simply do this by moving the handle around, but I want the HTMLCanvasElement not to outlive the HTMLElement. Again, the solution is to involve pointers. Since all wrapper structs have the same memory layout (only a uint), I used this hack: --- template as(Target) { auto as(Source)(scope return ref Source s) { return cast(Target*)&s; } } --- Anyway my question is whether the ownership and borrowing semantics ever gets applied to non-pointers?
Jul 20
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:
 Anyway my question is whether the ownership and borrowing semantics ever gets 
 applied to non-pointers?
Currently, no. The programmer would be expected to include the correct copyctor/opAssign/destructors which would take care of it. We're likely going to add a move constructor, too.
Jul 20
next sibling parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:
 On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:
 Anyway my question is whether the ownership and borrowing 
 semantics ever gets applied to non-pointers?
Currently, no. The programmer would be expected to include the correct copyctor/opAssign/destructors which would take care of it.
Do you mean to keep track of ownership/borrowedness manually? Sure, that will work. But how would I then get the compiler to complain when a borrowed one outlives the owned one?
Jul 20
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:
 Do you mean to keep track of ownership/borrowedness manually?
No, that's what the copyctor/opAssign/dtor semantics so.
Jul 20
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 21.07.19 02:17, Walter Bright wrote:
 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:
 Do you mean to keep track of ownership/borrowedness manually?
No, that's what the copyctor/opAssign/dtor semantics so.
This is not true.
Jul 22
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 23 July 2019 at 04:02:50 UTC, Timon Gehr wrote:
 On 21.07.19 02:17, Walter Bright wrote:
 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:
 Do you mean to keep track of ownership/borrowedness manually?
No, that's what the copyctor/opAssign/dtor semantics so.
This is not true.
I thought as much. Thanks for the confirmation. I am considering moving to pointers to benefit from the future semantics. It's just that I don't like pointers that much...
Jul 23
parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 23.07.19 10:20, Sebastiaan Koppe wrote:
 On Tuesday, 23 July 2019 at 04:02:50 UTC, Timon Gehr wrote:
 On 21.07.19 02:17, Walter Bright wrote:
 On 7/20/2019 3:39 PM, Sebastiaan Koppe wrote:
 Do you mean to keep track of ownership/borrowedness manually?
No, that's what the copyctor/opAssign/dtor semantics so.
This is not true.
I thought as much. Thanks for the confirmation. I am considering moving to pointers to benefit from the future semantics. It's just that I don't like pointers that much...
I think tying ownership/borrowing semantics to pointers instead of structs makes no sense; it's not necessary and it's not sufficient. Your use case illustrates why it is not sufficient.
Jul 23
parent reply Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 23 July 2019 at 14:18:27 UTC, Timon Gehr wrote:
 I think tying ownership/borrowing semantics to pointers instead 
 of structs makes no sense; it's not necessary and it's not 
 sufficient. Your use case illustrates why it is not sufficient.
That's as good a time as any to plug my own proposal draft: https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088 Any thoughts?
Jul 23
parent reply Sebastiaan Koppe <mail skoppe.eu> writes:
On Tuesday, 23 July 2019 at 19:15:44 UTC, Olivier FAURE wrote:
 That's as good a time as any to plug my own proposal draft:

 https://gist.github.com/PoignardAzur/9896ddb17b9f6d6f3d0fa5e6fe1a7088

 Any thoughts?
So in a nutshell, a variable with the unique qualifier ensures that there are no other references to that data during the lifetime of said variable? In other words, you can only take 1 addressOf/ref at a time? Does it prevent more that just use-after-free? Random idea: what about an owned destructor? One that only runs on the original object, not on any of its copies. Which implies scope of course, so that nothing of the copies or its internals are allowed to escape.
Jul 23
parent Olivier FAURE <couteaubleu gmail.com> writes:
On Tuesday, 23 July 2019 at 22:01:56 UTC, Sebastiaan Koppe wrote:
 So in a nutshell, a variable with the unique qualifier ensures 
 that there are no other references to that data during the 
 lifetime of said variable? In other words, you can only take 1 
 addressOf/ref at a time?

 Does it prevent more that just use-after-free?
Can you ask again on the Github thread? I don't want to hijack this thread for 12 pages. (also, can you expand your question? I don't get what you're asking)
Jul 23
prev sibling next sibling parent reply Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:

 We're likely going to add a move constructor, too.
And deprecate opPostMove?
Jul 21
parent Max Haughton <maxhaton gmail.com> writes:
On Sunday, 21 July 2019 at 20:20:10 UTC, Per Nordlöw wrote:
 On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:

 We're likely going to add a move constructor, too.
And deprecate opPostMove?
Seems likely, especially given that DIP1014 hasn't been implemented in the language or made it into the spec yet.
Jul 21
prev sibling parent 12345swordy <alexanderheistermann gmail.com> writes:
On Saturday, 20 July 2019 at 21:56:50 UTC, Walter Bright wrote:
 On 7/20/2019 5:42 AM, Sebastiaan Koppe wrote:
 Anyway my question is whether the ownership and borrowing 
 semantics ever gets applied to non-pointers?
Currently, no. The programmer would be expected to include the correct copyctor/opAssign/destructors which would take care of it. We're likely going to add a move constructor, too.
Make sure that it is a separate pull request from the dip, as c++ std::string binding is currently block by the lack of this.
Jul 21
prev sibling next sibling parent reply rikki cattermole <rikki cattermole.co.nz> writes:
Wonderful quote from you Walter!

 The careful reader will notice something peculiar in what I wrote: “as
if”. What do I mean by that weasel wording? Is there some skullduggery going
on? Why yes, there is. Computer languages are full of “as if” dirty deeds
under the hood, like the money you deposit in your bank account isn’t
actually there (I apologize if this is a rude shock to anyone), and this
isn’t any different. Read on!
I copied that over to Oceania's Discord server to talk about (very well received).
Jul 15
parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 8:52 PM, rikki cattermole wrote:
 Wonderful quote from you Walter!
 
 The careful reader will notice something peculiar in what I wrote: “as
if”. 
 What do I mean by that weasel wording? Is there some skullduggery going on? 
 Why yes, there is. Computer languages are full of “as if” dirty deeds
under 
 the hood, like the money you deposit in your bank account isn’t actually
there 
 (I apologize if this is a rude shock to anyone), and this isn’t any
different. 
 Read on!
I copied that over to Oceania's Discord server to talk about (very well received).
Thanks for letting me know. I enjoyed writing it.
Jul 15
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Now on the front page of Hacker News at number 9!

https://news.ycombinator.com/news
Jul 15
prev sibling next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 7/15/2019 7:58 AM, Mike Parker wrote:
 The blog:
 https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/
 
 Reddit:
 https://www.reddit.com/r/programming/comments/cdifbu/ownership_and_borrowing_in_d/
Hacker News: https://news.ycombinator.com/item?id=20441519
Jul 16
prev sibling next sibling parent Olivier FAURE <couteaubleu gmail.com> writes:
On Monday, 15 July 2019 at 14:58:55 UTC, Mike Parker wrote:
 some folks asked for some information about the bigger picture.
As one of the folks that asked for that information, thank you Walter for posting this :) The system described is a little rough around the edges, but it's really helpful to inform the discussion around DIP 1021!
Jul 17
prev sibling parent "Nick Sabalausky (Abscissa)" <SeeWebsiteToContactMe semitwist.com> writes:
On 7/15/19 10:58 AM, Mike Parker wrote:
 
 The blog:
 https://dlang.org/blog/2019/07/15/ownership-and-borrowing-in-d/
 
Very interesting! One formatting issue though: "D has a history of using function attributes to alter the semantics of a function&mdash;for example" Note the "&mdash;".
Jul 18