www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP1000: 'return scope' ambiguity and why you can't make opIndex work

reply Dennis <dkorpel gmail.com> writes:
You may have seen my previous dip1000 posts:
- [dip1000 + pure is a DEADLY 
COMBO](https://forum.dlang.org/thread/jnkdcngzytgtobihzggj forum.dlang.org)
- [DIP1000: The return of 'Extend Return Scope 
Semantics'](https://forum.dlang.org/thread/zzovywgswjmwneqwbdnm forum.dlang.org)

Consider this part 3 in the "fixing dip1000 series", but it's 
about a different bug.



dip25 and dip1000 are supposed to provide *simple* lifetime 
tracking that's still good enough to be useful. In the previous 
thread [Atila Neves 
mentioned](https://forum.dlang.org/post/azdzxsxyuipovrlmbhbb forum.dlang.org)
that [Lifetime Annotations like in
Rust](https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-
unction-signatures) are to be avoided. Is it simple though?

[On Wednesday, 26 May 2021 at 15:29:32 UTC, Paul Backus 
wrote:](https://forum.dlang.org/post/tsjffxwlwitaawiniztv forum.dlang.org)
 Of course, D's vision here is severely hampered in practice by
 the poor quality of its documentation (raise your hand if you
 can explain what ["return ref parameter semantics with
 additional scope parameter semantics"][1] actually means). But
 that's the idea.

 [1]:
 https://dlang.org/spec/function.html#ref-return-scope-parameters
Working on dip1000 made me finally able to "raise my hand", so here's how it works: Function parameters of a type with pointers have three possible lifetimes: infinite, scope, or return scope. You might have heard that `scope` is "not transitive" and think that there's only one layer to it. However, the key insight is that there's actually *two layers* when `ref` comes into play: then the parameter's *address* itself also has a lifetime in addition to the *value*. It can be demonstrated with a linked list: ```D safe: struct Node { int x; Node* next; } // First layer: returning the address of the node int* get0(return ref Node node) { return &node.x; } // Second layer: returning a value of the node int* get1(ref return scope Node node) { return &node.next.x; } // Third layer and beyond: this is where scope checking ends int* get2(ref scope Node node) { return &node.next.next.x; } ``` The lifetimes are determined as follows: | Lifetime | `ref` address | value of pointer type | |---------------|-----------------------|-----------------------| | infinite | never | default | | current scope | default | with `scope` keyword | | return scope | with `return` keyword | with `return scope` | A few code examples: ```D safe: int* v0( int* x) {return x;} // allowed, no lifetime restrictions int* v1(return int* x) {return x;} // allowed, returned value is `scope` int* v2( scope int* x) {return x;} // not allowed, x is `scope` int* v3(return scope int* x) {return x;} // allowed, equivalent to v1 int* r0( ref int x) {return &x;} // not allowed, `ref` is always scope int* r1(scope ref int x) {return &x;} // not allowed, `scope` does nothing here int* r2(return ref int x) {return &x;} // allowed, return applies to `ref` ``` As you can see, `scope` always applies to the pointer value and not to the `ref`, since `ref` is inherently `scope`. No ambiguity there. But what if we have a `ref int*`: does `return` apply to the address of the `ref` or the `int*` value? That's where those confusing lines from the specification come in, which distinguishes "return ref semantics" and "return scope semantics". It turns out there are three important factors: whether the function's return type is `ref`, whether the parameter is `ref`, and whether the parameter is annotated `scope`. Here's a table: **Does the `return` attribute apply to the parameter's `ref` or the pointer value?** | | `scope` | no `scope` | |----------------------------------|-----------|------------| | `ref` return type / `ref` param | **`ref`** | **`ref`** | | value return type / `ref` param | **value** | **`ref`** | | `ref` return type / value param | **value** | **value** | | value return type / value param | **value** | **value** | If you're still confused, I don't blame you: I'm still confusing myself regularly when reading signatures with `return` and `ref`. Anyway, is this difficulty problematic? [On Wednesday, 15 May 2019 at 08:32:09 UTC, Walter Bright wrote:](https://forum.dlang.org/post/qbgf95$2071$1 digitalmars.com)
 On 5/15/2019 12:21 AM, Dukc wrote:
 Could be worth a try even without docs, but in the long run we
 definitely need some explaining.
True, but I've tried fairly hard with the error messages. Please post your experiences with them. Also, there shouldn't be any caveats with using it. If it passes the compiler, it should be good to go. (Much like const and pure.)
All you need to do is see if the compiler complains, try adding `return` and/or `scope`, and see if the errors go away. Well... ```D safe: struct S { int x; } int* f(ref return scope S s) { return &s.x; // Error: returning `&s.x` escapes a reference to parameter `s` // perhaps annotate the parameter with `return` } ``` That's a confusing supplemental error, the parameter *is* annotated `return`. The actual problem is that `return` applies to the value, not the `ref` parameter, since there is no `ref` return. ```D struct T { int x; int* y; // <- pointer member added } int* g(ref return scope T t) { return &t.x; // No error } ``` And now the compiler accepts invalid code. Indeed, even the compiler doesn't always know what the `return` storage class actually applies to. See [bugzilla issue 21868](https://issues.dlang.org/show_bug.cgi?id=21868). While fixing [issue 21868](https://issues.dlang.org/show_bug.cgi?id=21868), the CI uncovered that [dub package 'automem' relies on the current accepts-invalid behavior](https://github.com/dlang/dmd/pull/12665#issuecomment-858836483). Here's the reduced code: ```D struct Vector { float[] _elements; ref float opIndex(size_t i) scope return { return this._elements[i]; } } ``` With the patch I made, the error becomes: ``` source/automem/vector.d(212,25): Error: scope parameter `this` may not be returned source/automem/vector.d(212,25): note that `return` applies to `ref`, not the value ``` My new supplemental error message is working, yay! But how to fix it? One way is to pass the `Vector` by value instead of by reference, but `opIndex` must be a member function to work as an operator overload and member functions pass `this` by reference. Another way is to return by value instead of by reference, but that means accessing array elements introduces a copy, and `&vector[0]` won't work anymore. dip1000 simply can't express a 'return scope' `opIndex` returning by `ref`. So it turns out the double duty of the `return` storage class is neither simple, nor expressive enough. Do you have any ideas how to move forward, and express the `Vector.opIndex` method without making the attribute soup worse? Keep in mind that dip25 (with `return ref`) is already in the language, but dip1000 (with `return scope`) is still behind a preview switch.
Jun 18
next sibling parent reply Steven Schveighoffer <schveiguy gmail.com> writes:
On 6/18/21 11:44 AM, Dennis wrote:
 If you're still confused, I don't blame you: I'm still confusing myself 
 regularly when reading signatures with `return` and `ref`.
I have a headache reading this post, and it makes me want to never use DIP1000. We are creeping towards having as much confusion and pain as Rust, without the benefit. I strongly believe we should implement DIP1000 in an expressive manner, instead of relying on confusing conventions -- just make a type constructor to signify lifetime management and be done. -Steve
Jun 18
parent Per =?UTF-8?B?Tm9yZGzDtnc=?= <per.nordlow gmail.com> writes:
On Friday, 18 June 2021 at 17:00:14 UTC, Steven Schveighoffer 
wrote:
 I strongly believe we should implement DIP1000 in an expressive 
 manner, instead of relying on confusing conventions -- just 
 make a type constructor to signify lifetime management and be 
 done.
Yes, I'm thinking about that too. I wonder if there's a performance penalty for such an expressive management we are currently not paying for.
Jul 06
prev sibling next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 18 June 2021 at 15:44:02 UTC, Dennis wrote:
 If you're still confused, I don't blame you: I'm still 
 confusing myself regularly when reading signatures with 
 `return` and `ref`. Anyway, is this difficulty problematic?
I am getting the same feeling from this as I am getting from certain aspects in C++ (e.g. intricate details of constructors). Thank you for explaining it, but I also think I will not remember it. I think stuff like this is what programmers will throw into a bucket labeled "I will figure this out later" and just apply keywords until it compiles... I've suggested that one might want to make the function signatures more readable and keep "auxiliary stuff" on a separate line: https://forum.dlang.org/thread/nzwobsazsawxvxbxhoue forum.dlang.org I personally think explicit lifetimes are easier to read, because I don't actually have to remember what keywords signify. It also makes it possible to expand the capabilities of the compiler over time.
Jun 18
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Friday, 18 June 2021 at 17:02:41 UTC, Ola Fosheim Grøstad 
wrote:
 [snip]

 I've suggested that one might want to make the function 
 signatures more readable and keep "auxiliary stuff" on a 
 separate line:

 https://forum.dlang.org/thread/nzwobsazsawxvxbxhoue forum.dlang.org

 I personally think explicit lifetimes are easier to read, 
 because I don't actually have to remember what keywords signify.

 It also makes it possible to expand the capabilities of the 
 compiler over time.
I am sympathetic to this. scope is relatively simple, but once you start getting into more combinations it requires a bit of mental energy.
Jun 18
prev sibling parent reply Bradley Chatha <sealabjaster gmail.com> writes:
On Friday, 18 June 2021 at 17:02:41 UTC, Ola Fosheim Grøstad 
wrote:
 I've suggested that one might want to make the function 
 signatures more readable and keep "auxiliary stuff" on a 
 separate line:

 https://forum.dlang.org/thread/nzwobsazsawxvxbxhoue forum.dlang.org

 I personally think explicit lifetimes are easier to read, 
 because I don't actually have to remember what keywords signify.

 It also makes it possible to expand the capabilities of the 
 compiler over time.
Being able to perform explicit, sort of 'algebra-esque' expressions of lifetime seems like a much more reasonable idea than the current magical keyword combinations. What are the chances though that the path/syntax can be changed at this point though, mostly in regards to convincing people? Not just for this suggestion, but any suggestion/criticism towards DIP 1000 in general? My main worry is that we'll end up with an inflexible, hard to understand system that doesn't even do the job right. Yet another tacked on feature for the language, etc. I've not been terribly optimistic for a quite a while now about the general direction things like this end up going, so I'm not getting my hopes up in anyway.
Jun 19
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 19 June 2021 at 10:40:45 UTC, Bradley Chatha wrote:
 What are the chances though that the path/syntax can be changed 
 at this point though, mostly in regards to convincing people? 
 Not just for this suggestion, but any suggestion/criticism 
 towards DIP 1000 in general?
I think it is mostly up to the other compiler devs to convince Walter? I don't think there is anything I can do anyway.
 My main worry is that we'll end up with an inflexible, hard to 
 understand system that doesn't even do the job right. Yet 
 another tacked on feature for the language, etc.
This is a good reason to have an experimental branch and let new features sit there until people get enough experience with them.
Jun 19
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 19 June 2021 at 13:47:06 UTC, Ola Fosheim Grøstad 
wrote:
 This is a good reason to have an experimental branch and let 
 new features sit there until people get enough experience with 
 them.
DIP1000 is still behind a preview switch, which you could consider an experimental branch, and it has been for five years now. I don't think incubation time is the problem here, but rather the lack of documentation and thorough testing. Now we have some code bases (most notably Phobos) that compile with -dip1000 but only because of hacks and bugs such as [inout implies return](https://issues.dlang.org/show_bug.cgi?id=22027), [pure implies scope](https://issues.dlang.org/show_bug.cgi?id=20150), [address of ref isn't scope](https://issues.dlang.org/show_bug.cgi?id=20245), [conflation of return-ref and return-scope](https://issues.dlang.org/show_bug.cgi?id=21868), [getting a slice to a scope static array is allowed](https://issues.dlang.org/show_bug.cgi?id=20505), [separate compilation isn't checked](https://issues.dlang.org/show_bug.cgi?id=20023), ... Those were great in the short term because they minimize the amount of breaking changes when turning on -dip1000. In the long term, all this code relying on accepts-invalid bugs is annoying to fix though.
Jun 19
next sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 19 June 2021 at 14:37:24 UTC, Dennis wrote:
 Those were great in the short term because they minimize the 
 amount of breaking changes when turning on -dip1000. In the 
 long term, all this code relying on accepts-invalid bugs is 
 annoying to fix though.
And the worst of it, the bugs cannot be easily fixed because they would break Phobos, as you outlined in the first post of your series. I think it was a mistake to declare Phobos `-dip1000` compilant with all those issues still around. I'd much rather have a non-Phobos `-dip1000` that behaves as it's supposed to.
Jun 19
prev sibling parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 19 June 2021 at 14:37:24 UTC, Dennis wrote:
 DIP1000 is still behind a preview switch, which you could 
 consider an experimental branch, and it has been for five years 
 now. I don't think incubation time is the problem here, but 
 rather the lack of documentation and thorough testing.
I don't know if people use switches all that much? With an experimental branch most hobbyists would use it to get the latest features, so I think it would lead to more use of upcoming features... but I don't know for sure.
Jun 19
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On Friday, 18 June 2021 at 15:44:02 UTC, Dennis wrote:
 **Does the `return` attribute apply to the parameter's `ref` or 
 the pointer value?**
 |                                  | `scope`   | no `scope` |
 |----------------------------------|-----------|------------|
 | `ref` return type / `ref` param  | **`ref`** | **`ref`**  |
 | value return type / `ref` param  | **value** | **`ref`**  |
 | `ref` return type / value param  | **value** | **value**  |
 | value return type / value param  | **value** | **value**  |
[...]
 Here's the reduced code:
 ```D
 struct Vector {
     float[] _elements;
     ref float opIndex(size_t i) scope return {
         return this._elements[i];
     }
 }
 ```

 With the patch I made, the error becomes:
 ```
 source/automem/vector.d(212,25): Error: scope parameter `this` 
 may not be returned
 source/automem/vector.d(212,25):        note that `return` 
 applies to `ref`, not the value
 ```
Geez, this isn't easy. I had to go step by step to make sense of that error, so maybe this can help others understand: `opIndex` has a half-hidden parameter: `return ref scope this`. Depending on the `opIndex`'s return type, the `return` part of the `this` parameter can either bind to its `ref` part or to its `scope` part. In pseudo code, it can be either `(return ref) (not-return scope) this` or `(not-return ref) (return scope) this`. `opIndex` has a `ref` return type. According to the table above, that means `return` binds to the `ref` part of `ref scope this`. I.e., it's `(return ref) (not-return scope) this`. `(return ref) this` means `opIndex` may return a `ref` to `this` or `this._elements` (same address). `(not-return scope) this` means it cannot return a `ref` to the elements of `this._elements`, because that would be returning a `scope` pointer which hasn't been annotated with `return`. As far as I understand, `opIndex` could return `&this._elements[i]` by value. Then the `return` would bind to the `scope` part of `ref scope this`, making `&this._elements[i]` a `return scope` pointer. But `float*` would be an awkward return type for `opIndex`. Geez, this isn't easy.
Jun 18
parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 18 June 2021 at 17:04:02 UTC, ag0aep6g wrote:
 Geez, this isn't easy.
I know right? When I started to get the hang of it I was like "I should write a tutorial about this" followed closely by "how am I going to explain this in one go to someone who hasn't spelunked dmd/escape.d and looked at the relevant spec a dozen times?" For this post I hoped to get across the idea that dmd has concepts of 'escaping by reference' for `ref int` and 'escaping by value' for `int*`, and that it currently sometimes goes wrong when you mix them. But there is so much more to cover: - constructors act like they return `this` by `ref`, but still have `return scope` semantics - `out` acts like `ref` - `in` acts like... I don't know. With -preview=in it's implementation defined whether it's `ref scope` or just `scope`, so is it also implementation defined what `return` applies to then? - `auto ref`... Don't know how that works internally. - `ref` in foreach is actually *not* inerhently scope like in parameters, and [it has its own hole](https://issues.dlang.org/show_bug.cgi?id=22040). - when `scope` is inferred, could it change the meaning of `return` to apply to the value instead of the `ref`? - ... who knows what I missed Learning a complex system could be rewarding if afterwards you can write expressive code with lifetime tracking, but in the case of dip1000, after all your learning efforts you still can't write a routine that splits a `scope string` into a `scope(string)[]` because dip1000 simply can't express that.
Jun 18
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Friday, 18 June 2021 at 18:31:40 UTC, Dennis wrote:
 Learning a complex system could be rewarding if afterwards you 
 can write expressive code with lifetime tracking, but in the 
 case of dip1000, after all your learning efforts you still 
 can't write a routine that splits a `scope string` into a 
 `scope(string)[]` because dip1000 simply can't express that.
I think this is the most significant issue. There is now way to extend it later without making signatures even more complicated.
Jun 19
prev sibling next sibling parent Dukc <ajieskola gmail.com> writes:
On Friday, 18 June 2021 at 15:44:02 UTC, Dennis wrote:
 [snip]
Wow, if nothing else you're doing a great job documenting DIP1000 with your posts. Thanks! With regular pointers and `ref` parameters, I think we should change the semantics of `scope ref` to be simply same as `ref`, i.e. no binding `scope` to the underlying pointer. Other than that, the semantics you explained are understandable IMO. I'd prefer to call the `return scope` storage class just a `return` storage class. Your post shows they are the same except for the corner cases with `ref scope` I just recommended ditching. Do you agree? Of course, we also need to be able to annotate the `this` pointer as return. Simplest answer IMO: have `return` storage class for a function declaration to always bind to the `this` argument, compiler error if there is none. `return` storage class for the returned value makes no sense anyway.
Jun 18
prev sibling next sibling parent reply ag0aep6g <anonymous example.com> writes:
On 18.06.21 17:44, Dennis wrote:
 So it turns out the double duty of the `return` storage class is neither 
 simple, nor expressive enough. Do you have any ideas how to move 
 forward, and express the `Vector.opIndex` method without making the 
 attribute soup worse? Keep in mind that dip25 (with `return ref`) is 
 already in the language, but dip1000 (with `return scope`) is still 
 behind a preview switch.
A quick and easy fix could be introducing `return(ref)` and `return(scope)`, allowing the programmer to pick what `return` binds to. Then `opIndex` can be written this way: ---- ref float opIndex(size_t i) return(scope) { return this._elements[i]; } ---- But: * That's still hard to figure out, especially with methods because `ref this` is invisible. * It doesn't address the underlying issues: one level of `scope` is not enough, and treating `ref` different from other indirections is confusing. I'm afraid DIPs 25 and 1000 are falling short.
Jun 19
next sibling parent reply Dukc <ajieskola gmail.com> writes:
On Saturday, 19 June 2021 at 09:43:18 UTC, ag0aep6g wrote:
 A quick and easy fix could be introducing `return(ref)` and 
 `return(scope)`, allowing the programmer to pick what `return` 
 binds to. Then `opIndex` can be written this way:

 ----
 ref float opIndex(size_t i) return(scope) {
     return this._elements[i];
 }
 ----
No thanks. The compiler still has ambiguity between ```d ref float opIndex(return(scope) ref __this, size_t i); return(scope) ref float opIndex(ref __this, size_t i); ``` It's simpler and better if `return` attribute outside the parameter list always binds to `this`. Unless there is some reason you might want to annotate the return type as `return`? I can't think of any.
 I'm afraid DIPs 25 and 1000 are falling short.
In the sense that you can't have deep `scope`, true. But I think DIP1000 was deliberately designed to not address that, for simplicity. I suggest that we leave fixing that for a potential future DIP. Just by fixing that `opIndex` example we still have the ability to define custom types that can be used for deep lifetime checking.
Jun 19
next sibling parent reply ag0aep6g <anonymous example.com> writes:
On Saturday, 19 June 2021 at 14:39:01 UTC, Dukc wrote:
 It's simpler and better if `return` attribute outside the 
 parameter list always binds to `this`. Unless there is some 
 reason you might want to annotate the return type as `return`? 
 I can't think of any.
I think you misunderstand. `return` does bind to `this`. But it can bind to two different aspects of `this`: (1) `ref` or (2) `scope`. Currently, you cannot freely choose which aspect of `this` you want to be `return`. The `opIndex` example fails because `return` is applied to the wrong aspect, and the programmer can't override it.
Jun 19
parent reply Dukc <ajieskola gmail.com> writes:
On Saturday, 19 June 2021 at 14:48:45 UTC, ag0aep6g wrote:
 I think you misunderstand. `return` does bind to `this`. But it 
 can bind to two different aspects of `this`: (1) `ref` or (2) 
 `scope`.

 Currently, you cannot freely choose which aspect of `this` you 
 want to be `return`. The `opIndex` example fails because 
 `return` is applied to the wrong aspect, and the programmer 
 can't override it.
Ah thanks, let's try again. So the member function rewritten would look like ```d ref float opIndex(return ref float[] vector, size_t i){ return vector[i]; } ``` So the problem is that `return` allows returning the vector array itself by reference, but not any of it's elements. I think this function should compile, `return` or no. DIP1000 is not supposed to be transitive, so it should only check returning `vector` itself by reference, but not anything that it refers to. However, the user still may want to annotate the `vector` parameter as `return`, because that lets the client code to ensure that the reference to `vector[i]` won't live longer than `vector`.
Jun 19
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 19 June 2021 at 15:18:20 UTC, Dukc wrote:
 ```d
 ref float opIndex(return ref float[] vector, size_t i){
    return vector[i];
 }
 ```

 So the problem is that `return` allows returning the vector 
 array itself by reference, but not any of it's elements.

 I think this function should compile, `return` or no. DIP1000 
 is not supposed to be transitive, so it should only check 
 returning `vector` itself by reference, but not anything that 
 it refers to.
It currently does compile. `float[] vector` is assumed to have infinite lifetime here, so you can return its elements by ref just fine. You cannot call this `opIndex` on a slice of stack memory, then your signature has to add the `scope` keyword: ```D ref float opIndex(return ref scope float[] vector, size_t i){ return vector[i]; } ``` And then the compiler needs to know that the returned float's lifetime is bound to the stack memory you put in, so then you want `return scope` semantics (which you can't get here because there's a `ref` return and a `ref` param.
Jun 19
parent Dukc <ajieskola gmail.com> writes:
On Saturday, 19 June 2021 at 15:27:24 UTC, Dennis wrote:
 It currently does compile. `float[] vector` is assumed to have 
 infinite lifetime here, so you can return its elements by ref 
 just fine. You cannot call this `opIndex` on a slice of stack 
 memory, then your signature has to add the `scope` keyword:


 ```D
 ref float opIndex(return ref scope float[] vector, size_t i){
    return vector[i];
 }
 ```
 And then the compiler needs to know that the returned float's 
 lifetime is bound to the stack memory you put in, so then you 
 want `return scope` semantics (which you can't get here because 
 there's a `ref` return and a `ref` param.
I'm not sure that `scope ref` should have any different semantics from `ref` anyway. If `vector` were passed by value it'd work with stack memory. But for that to work with operator overloading in same time, we'd need a non-`ref` `this` argument, or overloading with `static` or UFCS functions. Okay, now I'm not sure what would be the best course of action.
Jun 19
prev sibling parent Dennis <dkorpel gmail.com> writes:
On Saturday, 19 June 2021 at 14:39:01 UTC, Dukc wrote:
 No thanks. The compiler still has ambiguity between

 ```d
 ref float opIndex(return(scope) ref __this, size_t i);
 return(scope) ref float opIndex(ref __this, size_t i);
 ```
 It's simpler and better if `return` attribute outside the 
 parameter list always binds to `this`.
There is no ambiguity, `return` and `scope` outside the parameter list *always* bind to `this`, whether you put them on the left or right. ```D scope int* f(); // Error: function `onlineapp.f` functions cannot be `scope` int* g() scope; // Error: function `onlineapp.g` functions cannot be `scope` ``` (Nice error message btw)
Jun 19
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 19 June 2021 at 09:43:18 UTC, ag0aep6g wrote:
 But:

 * That's still hard to figure out, especially with methods 
 because `ref this` is invisible.
On top of that, `scope` could be invisible behind `in`. Here's an idea: could `return` apply to both the `ref` and value if there is `return scope`? | | `scope` | no `scope` | |---------------------------------|-----------|------------| | `ref` return type / `ref` param | **both** | **`ref`** | | value return type / `ref` param | **both** | **`ref`** | | `ref` return type / value param | **value** | **value** | | value return type / value param | **value** | **value** | The table would simply become: | | `scope` | no `scope` | |-------------|-----------|------------| | `ref` param | **both** | **`ref`** | | value param | **value** | **value** | You could even make this work then: ```D struct Vector { float[4] small; // return ref float[] large; // return scope bool isSmall; ref float opIndex(size_t i) return scope { return isSmall ? small[i] : large[i]; // dynamically choose } } ``` I *think* this is sound because there is no way a `ref` outlives its `scope` members, so the lifetime of the returned value is simply the one of the `ref` (which is the smaller one). However, I feel like there's a caveat I overlooked. Can the solution really be this simple?
Jun 19
parent reply Dukc <ajieskola gmail.com> writes:
On Saturday, 19 June 2021 at 19:40:36 UTC, Dennis wrote:
 I *think* this is sound because there is no way a `ref` 
 outlives its `scope` members, so the lifetime of the returned 
 value is simply the one of the `ref` (which is the smaller 
 one). However, I feel like there's a caveat I overlooked. Can 
 the solution really be this simple?
Well, I had to think this one really hard. Your solution has one disadvantage: ```d struct CustomPtr { private ubyte* ptr; //works, but suboptimal ubyte* opUnary(string op)() if(op=="*") return scope {return ptr;} } ``` With your solution, DIP1000 will prevent the return value of `*customPtr` outliving `customPtr`. We would ideally want only to prevent `*customPtr` outliving whatever `customPtr.ptr` points to, which is what happens now if I understood the table right. But I can see no flaw in changing the field "`ref` arg / `ref` return / `scope`" to "both". Whether the complexity of the current semantics are worth it because of the issue I mentioned, I'm not sure at all. It's highly likely there are more corner cases lurking about. I have a feeling that we should go with your proposal, or continue brainstorming. Deriving the `scope` semantics from return value makes too much assumptions about the intention.
Jun 21
parent reply Dennis <dkorpel gmail.com> writes:
On Monday, 21 June 2021 at 16:10:53 UTC, Dukc wrote:
 With your solution, DIP1000 will prevent the return value of 
 `*customPtr` outliving `customPtr`. We would ideally want only 
 to prevent `*customPtr` outliving whatever `customPtr.ptr` 
 points to, which is what happens now if I understood the table 
 right.
That was my first reservation when considering it, but so far I failed to find a scenario where the `scope` outlives its `ref`. Take for example this: ```D safe: struct CustomPtr { private ubyte* ptr; ubyte* get() return scope {return ptr;} } void f(scope ubyte* ptr0) { scope ubyte* ptr1; ptr1 = ptr0; // fine { CustomPtr c; c.ptr = ptr0; ptr1 = c.get(); // scope variable `c` assigned to `ptr1` with longer lifetime } } ``` While ptr0 can be assigned to ptr1, when going through a `CustomPtr c`, dmd still sees it as assigning "scope variable `c`".
Jun 21
parent Dukc <ajieskola gmail.com> writes:
On Monday, 21 June 2021 at 16:56:30 UTC, Dennis wrote:
 While ptr0 can be assigned to ptr1, when going through a 
 `CustomPtr c`, dmd still sees it as assigning "scope variable 
 `c`".
I was about to say it might be that you need to write ```d void f(scope ubyte* ptr0) { scope ubyte* ptr1; ptr1 = ptr0; // fine { auto c = CustomPtr(ptr0); ptr1 = c.get();with longer lifetime } } ``` ...instead, but that fails too, at least on 2.096 release canditate. As does this one: ```d ubyte* get(ref return scope ubyte* ptr){return ptr;} void f(scope ubyte* ptr0) { scope ubyte* ptr1; ptr1 = ptr0; // fine { auto c = ptr0; ptr1 = c.get(); } } ``` I believe we can conclude that the field "`ref` argument / value return / `scope`" already reads either "`ref`" or "both", at least when you're returning a pointer. That being the case, your solution makes perfect sense.
Jun 21
prev sibling next sibling parent nkm1 <t4nk074 openmailbox.org> writes:
On Friday, 18 June 2021 at 15:44:02 UTC, Dennis wrote:

 If you're still confused, I don't blame you
I didn't understand anything. This is so bad.
Jun 21
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/18/2021 8:44 AM, Dennis wrote:
 Here's the reduced code:
 ```D
 struct Vector {
      float[] _elements;
      ref float opIndex(size_t i) scope return {
          return this._elements[i];
      }
 }
 ```
It *always* helps to reduce these examples to the bare minimum, i.e. strip all the higher level constructs out. The compiler absolutely must be consistent in this. The above reduces to: ref int test(ref scope return int* p) { return *p; } Take a moment to satisfy yourself that it is logically the same. [] was replaced by *, float by int, i no longer needed, don't confuse things with operator overloading, replace `this` with a ref parameter making it a non-member function, and then strip away the struct wrapper. This currently compiles with master and -dip1000 without error. Examining Dennis' table, it has a [ref return type] and a [scope value], and so the `return` applies to the pointer type. And so it is working as intended and is not a bug. I totally understand that this is headache-inducing. The aspirin is realizing that 100% of these examples can be rewritten using *only* the following terms: int i; int* p; return i; return *p; return &i; return &p; Write the example using these, sprinkling in ref, return, and scope. If anything else appears, like class, struct, this, [], delegate, etc., more work is needed to reduce it. Hope this helps!
Jul 05
parent reply ag0aep6g <anonymous example.com> writes:
On Monday, 5 July 2021 at 09:28:47 UTC, Walter Bright wrote:
   ref int test(ref scope return int* p)
   {
      return *p;
   }
[...]
 Examining Dennis' table, it has a [ref return type] and a 
 [scope value], and so the `return` applies to the pointer type.
I think you're misreading something there. For reference, the table:
 **Does the `return` attribute apply to the parameter's `ref` or 
 the pointer value?**
 |                                  | `scope`   | no `scope` |
 |----------------------------------|-----------|------------|
 | `ref` return type / `ref` param  | **`ref`** | **`ref`**  |
 | value return type / `ref` param  | **value** | **`ref`**  |
 | `ref` return type / value param  | **value** | **value**  |
 | value return type / value param  | **value** | **value**  |
Your test function has a `ref` return type and a `ref` parameter, so we're looking at the first row. That row says `return` applies to the `ref` part of the parameter (with and without `scope`).
Jul 05
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/5/2021 3:01 AM, ag0aep6g wrote:
 On Monday, 5 July 2021 at 09:28:47 UTC, Walter Bright wrote:
   ref int test(ref scope return int* p)
   {
      return *p;
   }
[...]
 Examining Dennis' table, it has a [ref return type] and a [scope value], and 
 so the `return` applies to the pointer type.
I think you're misreading something there. For reference, the table:
 **Does the `return` attribute apply to the parameter's `ref` or the pointer 
 value?**
 |                                  |
`scope`   | no `scope` |
 |----------------------------------|-----------|------------|
 | `ref` return type / `ref` param  | **`ref`** | **`ref`**  |
 | value return type / `ref` param  | **value** | **`ref`**  |
 | `ref` return type / value param  | **value** | **value**  |
 | value return type / value param  | **value** | **value**  |
Your test function has a `ref` return type and a `ref` parameter, so we're looking at the first row. That row says `return` applies to the `ref` part of the parameter (with and without `scope`).
I think you might be right.
Jul 05
next sibling parent reply claptrap <clap trap.com> writes:
On Monday, 5 July 2021 at 10:44:54 UTC, Walter Bright wrote:
 On 7/5/2021 3:01 AM, ag0aep6g wrote:
 On Monday, 5 July 2021 at 09:28:47 UTC, Walter Bright wrote:
 Your test function has a `ref` return type and a `ref` 
 parameter, so we're looking at the first row. That row says 
 `return` applies to the `ref` part of the parameter (with and 
 without `scope`).
I think you might be right.
How the heck are us mere mortals supposed to use this stuff if you struggle to understand what it means?
Jul 05
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/5/2021 4:36 AM, claptrap wrote:
 How the heck are us mere mortals supposed to use this stuff if you struggle to 
 understand what it means?
I make mistakes all the time. That's why D has unit tests built in, and an acceptance test suite. One reason I like writing software is once I get it right, it stays right every time. Actually, I find Dennis' table a tad confusing. An alternate formulation is: --------- Consider the ambiguity of a `return ref scope` parameter. Which is it: 1. `return ref` and `scope` 2. `ref` and `return scope` ? The ambiguity is resolved by looking at the function return. If it returns by `ref`, then it's (1), otherwise (2). This is evident in the table above. This rule is arbitrary, precluding things like creating a `return ref` and `return scope` for the same parameter. ----------
Jul 05
next sibling parent reply claptrap <clap trap.com> writes:
On Tuesday, 6 July 2021 at 05:15:24 UTC, Walter Bright wrote:
 On 7/5/2021 4:36 AM, claptrap wrote:
 How the heck are us mere mortals supposed to use this stuff if 
 you struggle to understand what it means?
I make mistakes all the time. That's why D has unit tests built in, and an acceptance test suite.
It wasnt a criticism of you, it was like if the most experienced and talented developers are struggling to make sense of it how can the average programmer (me) ever hope to use it correctly. It just seems that whenever a DIP1000/safe thread comes up nobody really understands it. Like it's the exact antithesis of "easy to use correctly"
Jul 06
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Tuesday, 6 July 2021 at 09:06:47 UTC, claptrap wrote:
 It wasnt a criticism of you, it was like if the most 
 experienced and talented developers are struggling to make 
 sense of it how can the average programmer (me) ever hope to 
 use it correctly.
It is also a question of whether one wants to support people who use the language occasionally or only want to support full time users. Languages like Ada and C++ is is the full time camp, but that only works if you have high commercial adoption, so not a good strategy in general.
Jul 06
prev sibling parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 6 July 2021 at 05:15:24 UTC, Walter Bright wrote:
 This rule is arbitrary, precluding things like creating a 
 `return ref` and
 `return scope` for the same parameter.
My question is: why? Why can't we allow a ref parameter to return either its address or its value? Given that lifetime(address) <= lifetime(value) and the returned value has lifetime(address), I'm struggling to see where it goes wrong.
Jul 06
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 7/6/2021 2:23 AM, Dennis wrote:
 On Tuesday, 6 July 2021 at 05:15:24 UTC, Walter Bright wrote:
 This rule is arbitrary, precluding things like creating a `return ref` and
 `return scope` for the same parameter.
My question is: why? Why can't we allow a ref parameter to return either its address or its value? Given that lifetime(address) <= lifetime(value) and the returned value has lifetime(address), I'm struggling to see where it goes wrong.
It doesn't make sense to return either p or the address of p with the same function. What kind of a function does that? We could trick the code into doing that, but that would introduce all kinds of bugs into the lifetime analysis code. The real problem is not being able to return the value of p as a ref, if p is also passed by ref. I'm going to look into addressing that.
Jul 06
parent Dennis <dkorpel gmail.com> writes:
On Tuesday, 6 July 2021 at 09:37:30 UTC, Walter Bright wrote:
 It doesn't make sense to return either p or the address of p 
 with the same function. What kind of a function does that?
Here's an example: https://forum.dlang.org/post/uxtqfrbomcsfzzbefkyw forum.dlang.org But such a function should not be the norm. The point is that users can simply add `return` and think "now I'm allowed to return `this` or `&this.x` or `this.arr[0]` or whatever" without delving into the whole "am I returning an address or value and am I allowed to" conundrum.
 We could trick the code into doing that, but that would 
 introduce all kinds of bugs into the lifetime analysis code.
Do you have an example?
 The real problem is not being able to return the value of p as 
 a ref, if p is also passed by ref.

 I'm going to look into addressing that.
Awesome!
Jul 06
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Added PRs:

https://github.com/dlang/dmd/pull/12817
https://github.com/dlang/dmd/pull/12819

This, of course, breaks:

  struct Vector {
     float[] _elements;
     ref float opIndex(size_t i) scope return {
         return this._elements[i];
     }
  }

as Dennis points out. It still is the correct fix, however, we just have to
find 
a way to get opIndex to work.
Jul 06