www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP1000: The return of 'Extend Return Scope Semantics'

reply Dennis <dkorpel gmail.com> writes:


The `return` attribute allows you to return a scope variable.
The compiler knows that the returned value has the same lifetime 
as the argument passed to the function.
```D
// ┌───────────<──────────┐
int* identity(return int* x)  safe {
     return x;
}

void main()  safe {
     int x;
     //   ┌──────<──────┐
     int* y = identity(&x);
     static int* z; // global variable
     z = y;         // error! scope variable `y` assigned to 
non-scope `z`
}
```

What if you are not using a return value, but setting it to 
another parameter, e.g. using `ref` or `out`?

```D
//                          
┌─────────<─────────┐
void assign(ref scope int* target, return int* source)  safe {
     target = source;
}
```
The compiler knows that `source` cannot be returned since it's a 
`void` function, so it makes the first parameter, _but only the 
first parameter_, the destination of the `return` parameter 
`source`.
```D
void main()  safe {
     int x;
     int* y;
     //     ┌─<─┐
     assign(y, &x); // allowed
}
```

This feature has been proposed and implemented by Walter Bright: 
[Issue 19097](https://issues.dlang.org/show_bug.cgi?id=19097), 
[fix Issue 19097 - Extend Return Scope 
Semantics](https://github.com/dlang/dmd/pull/8504).
It was merged on January 2019, but in August 2018 there was a 
discussion about it:
[Re: Is  safe still a 
work-in-progress?](https://forum.dlang.org/post/plja2k$28r0$1 digitalmars.com)

Mike Franklin:
 Why not the first `ref` parameter regardless of whether it's 
 the absolute first in the list.  Why not the last `ref` 
 parameter?  Why not all `ref` parameters?
Walter Bright:
 Good question. If this fairly restricted form solves the 
 problems, then there is no need for the more flexible form. 
 Things can always be made more flexible in the future, but 
 tightening things can be pretty disruptive. Hence, unless there 
 is an obvious and fairly strong case case for the flexibility, 
 then it should be avoided for now.
So did the restricted form turn out to be good enough? Well, on March 23 2019, [Build entire Phobos with -preview=dip1000 https://github.com/dlang/phobos/pull/6931) was merged, so it seems to be. Hurray! Except... It turns out we're not quite done yet, because Druntime and Phobos still rely on a pretty major accepts-invalid bug to compile with -dip1000, see: [dip1000 + pure is a DEADLY COMBO](https://forum.dlang.org/thread/jnkdcngzytgtobihzggj forum.dlang.org) The good news is, Per Nordlöw and I have been working on [fixing Phobos](https://github.com/dlang/phobos/pull/8076), and Druntime [[1]](https://github.com/dlang/druntime/pull/3471), [[2]](https://github.com/dlang/druntime/pull/3472), [[3]](https://github.com/dlang/druntime/pull/3474), [[4]](https://github.com/dlang/druntime/pull/3480) (also thanks to [aG0aep6G](https://github.com/dlang/phobos/pull/8109), [Walter Bright](https://github.com/dlang/phobos/pull/8104), [Iain Buclaw](https://github.com/dlang/phobos/pull/8103), and reviewers). During this, we did stumble on cases where the restriction came up, such as [BigUint.divMod](https://github.com/dlang/phobos/pull/8081#iss ecomment-842976462) having `out` parameters after the input parameters. More importantly: [It turns out](https://github.com/dlang/druntime/pull/3480#issuecomment-845480887) that `core.lifetime: move` (which is [also in Phobos](https://github.com/dlang/phobos/blob/460ed9c6198b1c84d54ef98b4d0240bc672b89b5/std/algorithm mutation.d#L1075)), has a signature that's incompatible with -dip1000 in its current form: ```D void move(T)(ref T source, ref T target) ``` We want to express: ```D // ┌─────────>─────────┐ void move(T)(return T source, ref scope T target) ``` But there's currently now way of making `target` the return scope destination when it's not the first parameter. So unless we want to leave `move` ` system` or create a `move2` function with parameters reversed, we're back at the drawing board for "Extend Return Scope Parameters". Two earlier proposals were: ```D // Mike Franklin's proposal: void move(T)(return(target) T source, ref scope T target) // Steven Schveighoffer's proposal: void move(T)(return T source, __sink ref scope T target) ``` What do you think?
May 25
next sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So unless 
 we want to leave `move` ` system` or create a `move2` function 
 with parameters reversed, we're back at the drawing board for 
 "Extend Return Scope Parameters". Two earlier proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?
Good overview, thanks for the post. If we don't really need a generic solution for that, i.e., can expect new code to use the first param and `move` to be the only problematic existing code, a possibility would be to make `move` and `forward` compiler intrinsics. The compiler can handle those much more efficiently than a non-trivial library solution with according template bloat.
May 25
next sibling parent Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:
 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use the 
 first param and `move` to be the only problematic existing 
 code, a possibility would be to make `move` and `forward` 
 compiler intrinsics. The compiler can handle those much more 
 efficiently than a non-trivial library solution with according 
 template bloat.
Even without moving the implementation inside the compiler, an easy fix would be to make the druntime function a special case in dmd. I recently found out Walter [did something similar](https://github.com/dlang/dmd//commit/ab5a18635adaa3e2271e0f4b569500a89ac74235#diff-88cc711d774668de0af3461742be476454e8374545722dfd7 457a3264aecf32R804) to make `char[].dup` implicitly convert to `immutable(char)[]`. It's ugly, but it does move `-dip1000` forward.
May 25
prev sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:
 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So unless 
 we want to leave `move` ` system` or create a `move2` function 
 with parameters reversed, we're back at the drawing board for 
 "Extend Return Scope Parameters". Two earlier proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?
Good overview, thanks for the post. If we don't really need a generic solution for that, i.e., can expect new code to use the first param and `move` to be the only problematic existing code,
This sounds like a really annoying and arbitrary limitation. Among other things, it interferes with the API designer's ability to freely choose which parameter they want to serve as the pseudo-`this` for UFCS. Being able to make that choice is important both for aesthetic reasons, but more importantly to enable "extension methods" that allow writing generic code that accepts both custom types, and built-in types which are not otherwise extensible. Also, what about C interop? APIs written in C often contain functions which otherwise obey the semantics of `scope` and `return`, but will not respect arbitrary limitations on the order of parameters.
May 25
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:
 On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:
 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So 
 unless we want to leave `move` ` system` or create a `move2` 
 function with parameters reversed, we're back at the drawing 
 board for "Extend Return Scope Parameters". Two earlier 
 proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?
Good overview, thanks for the post. If we don't really need a generic solution for that, i.e., can expect new code to use the first param and `move` to be the only problematic existing code,
This sounds like a really annoying and arbitrary limitation.
It's not arbritrary at all - the purpose is to avoid this: https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures Reasonable people of course can disagree on whether or not avoiding that is a good idea.
May 26
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 [snip]

 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.
I also couldn't help but think of Rust's lifetime annotations when reading the OP's post. Rust's is more expressive though. If you try to limit it, then there obviously will be cases that you can't cover everything without making it more complicated. I can't help but think that return should be something like lifetime(return) instead. Then you can have lifetime(variableA), lifetime(variableB). It's a bit more confusing to have return(variableA)/return(variableB).
May 26
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 26 May 2021 at 13:01:24 UTC, jmh530 wrote:
 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 [snip]

 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.
I also couldn't help but think of Rust's lifetime annotations when reading the OP's post. Rust's is more expressive though. If you try to limit it, then there obviously will be cases that you can't cover everything without making it more complicated. I can't help but think that return should be something like lifetime(return) instead. Then you can have lifetime(variableA), lifetime(variableB). It's a bit more confusing to have return(variableA)/return(variableB).
D's general philosophy w.r.t. memory safety so far has been to try and balance ease of use and comprehension with expressiveness. D's `scope` and `return` are less expressive than Rust's lifetimes, but that expressiveness may be worth giving up if it's easier to write correct ` safe` and DIP1000-compliant code than it is to write correct Rust code. Of course, D's vision here is severely hampered in practice by the poor quality of its documentation (raise your hand if you can explain what ["return ref parameter semantics with additional scope parameter semantics"][1] actually means). But that's the idea. [1]: https://dlang.org/spec/function.html#ref-return-scope-parameters
May 26
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 26 May 2021 at 15:29:32 UTC, Paul Backus wrote:
 [snip]

 D's general philosophy w.r.t. memory safety so far has been to 
 try and balance ease of use and comprehension with 
 expressiveness. D's `scope` and `return` are less expressive 
 than Rust's lifetimes, but that expressiveness may be worth 
 giving up if it's easier to write correct ` safe` and 
 DIP1000-compliant code than it is to write correct Rust code.
Correct me if I'm wrong, but Rust bakes in D's `scope` to everything unless you override it with a lifetime annotation. So I don't think the issue is `scope` per se. `return` is what is used to tie the lifetime of a parameter to the lifetime of the output. I agree that in the simple example it is easier to write correct code.
 Of course, D's vision here is severely hampered in practice by 
 the poor quality of its documentation (raise your hand if you 
 can explain what ["return ref parameter semantics with 
 additional scope parameter semantics"][1] actually means). But 
 that's the idea.

 [1]: 
 https://dlang.org/spec/function.html#ref-return-scope-parameters
+1
May 26
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2021 8:29 AM, Paul Backus wrote:
 Of course, D's vision here is severely hampered in practice by the poor
quality 
 of its documentation (raise your hand if you can explain what ["return ref 
 parameter semantics with additional scope parameter semantics"][1] actually 
 means). But that's the idea.
 
 [1]: https://dlang.org/spec/function.html#ref-return-scope-parameters
The beauty of open source is anyone can submit PRs to improve it!
Jun 10
prev sibling next sibling parent Zach Attack! <reachzach gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.
Proposal: Solve the problem incrementally. Start with a single pre-named routing channel (i.e. lifetime) supported directly by the compiler. Call it ` __in1` and ` __out1` (these are parameter attributes). The compiler will track this lifetime in addition to the `scope` and `return` lifetimes. It’s possible that this one additional channel will solve all use cases. If later on, someone provides an undeniably compelling use case for additional lifetime channels, simply add ` __in2` and ` __out2`, ` __in3` and ` __out3`, etc. as the case requires. The assumption upon which this solution is based is that Rust’s user-named "dictionary of lifetimes" is **complete and total OVERKILL**, that the actual number of lifetimes a function ever really must track is so small that the tracking mechanism can be incrementally baked into the compiler itself. (If I’m not mistaken, a baked-in lifetime requires only two bits of compiler memory per parameter to track, and only one bit for other variables.) For the most part, users would hardly ever see ` __in1` and ` __out1` since they would be inferred by templates, and rarely used anyway. But for cases like phobos’ `move`, the facility would be available. Hope you like my suggestion. Zach
May 26
prev sibling next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:
 On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:
 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use 
 the first param and `move` to be the only problematic 
 existing code,
This sounds like a really annoying and arbitrary limitation.
It's not arbritrary at all - the purpose is to avoid this: https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures
Why must it always be the first parameter, instead of always being the last parameter, etc.? The choice of **which** parameter gets special treatment is arbitrary, and is the part I was complaining about because of how it interacts with UFCS.
May 26
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2021 5:44 PM, tsbockman wrote:
 Why must it always be the first parameter, instead of always being the last 
 parameter, etc.? The choice of **which** parameter gets special treatment is 
 arbitrary, and is the part I was complaining about because of how it interacts 
 with UFCS.
It is not arbitrary. It actually fits with common Phobos usage. It is especially true if one writes member functions for a container, doing things like: stack.push(value); where value is a pointer.
Jun 10
parent reply ag0aep6g <anonymous example.com> writes:
On 11.06.21 04:58, Walter Bright wrote:
 It is not arbitrary. It actually fits with common Phobos usage. It is 
 especially true if one writes member functions for a container, doing 
 things like:
 
     stack.push(value);
 
 where value is a pointer.
You can't defend the "first parameter" rule with member functions. If you want to support returning through `this`, you can easily define that without bringing the order of the parameters into it. The actual point of the "first parameter" rule is obviously UFCS. You want support a free function `push` that is called via UFCS. `push` is not a member function then.
Jun 10
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2021 10:53 PM, ag0aep6g wrote:
 You can't defend the "first parameter" rule with member functions. If you want 
 to support returning through `this`, you can easily define that without
bringing 
 the order of the parameters into it.
 The actual point of the "first parameter" rule is obviously UFCS. You want 
 support a free function `push` that is called via UFCS. `push` is not a member 
 function then.
The whole idea of UFCS is member functions and regular functions are interchangeable, hence `this` is the first parameter.
Jun 11
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2021 10:53 PM, ag0aep6g wrote:
 [...]
BTW, another "first parameter" rule is assignment operators: a += b; a ~= b; a = b; The right side (second parameter) is assigned to the left side (first parameter).
Jun 11
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:
 This sounds like a really annoying and arbitrary limitation.
It's not arbritrary at all - the purpose is to avoid this: https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures Reasonable people of course can disagree on whether or not avoiding that is a good idea.
I understand that motivation, but at the same time, it's not necessarily good to reduce flexibility for the programmer. I think I would rather have to clearly annotate my target ref/out parameters every time, than rely on a "magic" rule like "the first parameter is the targeted one". Of course, that doesn't mean that I want to have Rust-style lifetime annotation if we can have something simpler and easier! But it would certainly be very nice to have the full scope of possibilities that Rust allows. It strikes me that what D should aim for is notation that is clear and unambiguous in intent, but which scales nicely with the complexity of what the programmer wants to achieve (so e.g. if you want to ensure a struct instance outlives a pointer/reference that it wraps, you might have to do more annotation, but if you want to make sure that in input parameter outlives a returned parameter, that's simple and easy).
May 27
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 27 May 2021 at 19:03:53 UTC, Joseph Rushton Wakeling 
wrote:
 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 [...]
I understand that motivation, but at the same time, it's not necessarily good to reduce flexibility for the programmer. I think I would rather have to clearly annotate my target ref/out parameters every time, than rely on a "magic" rule like "the first parameter is the targeted one". [...]
I agree with all of this. The problem is of course that pesky idea that will neatly tie everything up.
Jun 01
prev sibling parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.
I find Rust's approach more intuitive, even though I am not a Rust programmer... In general, there is way too many words to remember the meaning of on D function signatures. But I guess system is still available...
May 27
parent reply vitoroak <carvalhogvm gmail.com> writes:
On Thursday, 27 May 2021 at 19:57:49 UTC, Ola Fosheim Grostad 
wrote:
 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:
 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.
I find Rust's approach more intuitive, even though I am not a Rust programmer... In general, there is way too many words to remember the meaning of on D function signatures. But I guess system is still available...
I do find it more intuitive in Rust too, and lifetime parameters in Rust also apply to structs so you can have a struct that has the same lifetime as the reference it receives, so you can have a Mutex for example return a LockGuard struct that will not outlive the Mutex.
May 27
parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Thursday, 27 May 2021 at 20:11:37 UTC, vitoroak wrote:
 I do find it more intuitive in Rust too, and lifetime 
 parameters in Rust also apply to structs so you can have a 
 struct that has the same lifetime as the reference it receives, 
 so you can have a Mutex for example return a LockGuard struct 
 that will not outlive the Mutex.
Sounds useful. I have never used lifetimes myself, only read about it (including some theory), but I know that lifetime annotations predates Rust, I think it was used in a ML dialect. So it isn't unique to Rust and I assume universities will teach it... So, it will most likely become standard? Dunno.
May 27
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 27 May 2021 at 20:21:58 UTC, Ola Fosheim Grostad 
wrote:
 On Thursday, 27 May 2021 at 20:11:37 UTC, vitoroak wrote:
 I do find it more intuitive in Rust too, and lifetime 
 parameters in Rust also apply to structs so you can have a 
 struct that has the same lifetime as the reference it 
 receives, so you can have a Mutex for example return a 
 LockGuard struct that will not outlive the Mutex.
Sounds useful. I have never used lifetimes myself, only read about it (including some theory), but I know that lifetime annotations predates Rust, I think it was used in a ML dialect. So it isn't unique to Rust and I assume universities will teach it... So, it will most likely become standard? Dunno.
It goes back to Cyclone, https://cyclone.thelanguage.org/
May 29
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:
 It goes back to Cyclone, https://cyclone.thelanguage.org/
There you go, with original papers and all. I still think local GC + ARC for shared is a more convenient path for safe users, but who am I to judge if people want affine/linear types.
May 29
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Saturday, 29 May 2021 at 12:39:48 UTC, Ola Fosheim Grøstad 
wrote:
 On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:
 It goes back to Cyclone, https://cyclone.thelanguage.org/
There you go, with original papers and all. I still think local GC + ARC for shared is a more convenient path for safe users, but who am I to judge if people want affine/linear types.
I would rather bet on GC + affine/linear types like Haskell is pursuing. In any case, there is even research trying to make use of C2X annotations to retrofit it into C, https://plv.mpi-sws.org/refinedc/
May 29
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 29 May 2021 at 18:21:15 UTC, Paulo Pinto wrote:
 I would rather bet on GC + affine/linear types like Haskell is 
 pursuing.
Haskell is a functional language, so they are in a much better position to separating the heap and doing shape analysis.
 In any case, there is even research trying to make use of C2X 
 annotations to retrofit it into C,

 https://plv.mpi-sws.org/refinedc/
Looks more or less like other languages that focus on full formal verification using Hoare logic.
May 30
prev sibling parent Elronnd <elronnd elronnd.net> writes:
On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:
 On Thursday, 27 May 2021 at 20:21:58 UTC, Ola Fosheim Grostad 
 wrote:
 I know that lifetime annotations predates Rust, I think it was 
 used in a ML dialect.
It goes back to Cyclone, https://cyclone.thelanguage.org/
Also clean (https://clean.cs.ru.nl/Clean). Don't know which came first, but clean is certainly more of an ml (cyclone is almost backwards-compatible with c).
Jun 11
prev sibling next sibling parent reply Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:


 The `return` attribute allows you to return a scope variable.
 The compiler knows that the returned value has the same 
 lifetime as the argument passed to the function.
 ```D
 // ┌───────────<──────────┐
 int* identity(return int* x)  safe {
     return x;
 }

 [...]
I think the Franklin solution is the best one presented here. Similarly with inout, any keywords in this context (i.e. non-trivial and non-local effect on the code) should aim for some syntax like this, even if the existing rules technically mean you could go without, flexibility and redundancy are a good thing (especially for people new to the language).
May 25
parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:50:29 UTC, Max Haughton wrote:
 I think the Franklin solution is the best one presented here.
It looks nice, but it's also the one with the most implementation work. It requires a change to the parser (+libdparse), new mangling rules, and if we want to support it: it opens the door for multiple return parameters with different destinations. Looking at the implementation [dmd.escape(603)](https://github.com/dlang/dmd/blob/a7cf02c8bb124f5d5a05b9d238f71d2c0773dc51/src dmd/escape.d#L603), it's not going to be me who will be able to implement that any time soon.
May 25
parent Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 25 May 2021 at 12:48:55 UTC, Dennis wrote:
 On Tuesday, 25 May 2021 at 11:50:29 UTC, Max Haughton wrote:
 I think the Franklin solution is the best one presented here.
It looks nice, but it's also the one with the most implementation work. It requires a change to the parser (+libdparse), new mangling rules, and if we want to support it: it opens the door for multiple return parameters with different destinations. Looking at the implementation [dmd.escape(603)](https://github.com/dlang/dmd/blob/a7cf02c8bb124f5d5a05b9d238f71d2c0773dc51/src dmd/escape.d#L603), it's not going to be me who will be able to implement that any time soon.
This is unfortunately true. That being said I think "too hard to implement" is being said a lot these days, and it has nothing to do with documentation (in light of the other thread).
May 25
prev sibling next sibling parent zjh <fqbqrr 163.com> writes:
The meaning of function has limited the position of `return`,
So`void move(T)(return(target) T source, ref scope T target)`is 
enough.
`return(target)`can be added to any parameter.
May 25
prev sibling next sibling parent reply zjh <fqbqrr 163.com> writes:
The meaning of function has limited the position of `return`,
So`void move(T)(return(target) T source, ref scope T target)`is 
enough.
`return(target)`can be added to any parameter.
May 25
parent reply zjh <fqbqrr 163.com> writes:
We can limit each parameter to `one return` at most.
It's not a bad thing to have multiple `return areas`. My `C++` 
function `f(A,B,C,D,...)`,every even parameter` can be used as 
the return value.
May 25
parent zjh <fqbqrr 163.com> writes:
On Tuesday, 25 May 2021 at 13:18:16 UTC, zjh wrote:

Maybe we can simplify it with a symbol ` (a)/ (b)`, then ` a/ b`, 
and so on.Indicating the direction of the pointer to `track the 
direction` of the pointer(`A-->B`).
May 26
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/25/21 7:24 AM, Dennis wrote:
 But there's currently now way of making `target` the return scope 
 destination when it's not the first parameter. So unless we want to 
 leave `move` ` system` or create a `move2` function with parameters 
 reversed, we're back at the drawing board for "Extend Return Scope 
 Parameters". Two earlier proposals were:
 
 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)
 
 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```
 
 What do you think?
Re-reading that thread, there is more to the proposal than just the attribute -- the fact that the __sink attribute would be inferred on templates. So no attribute would be necessary on `move`, even if the attribute is added for non-templates/non-auto functions (ditto for Mike's proposal). That being said, Mike's proposal is obviously more flexible, as it allows multiple return parameters that link to multiple input parameters. But the cases in which those are present are super-rare. I also don't like having a keyword (return) match with an attribute ( __something), as they seem unrelated, and the only reason an attribute would be to avoid adding a keyword. But that's a bit of bikeshedding (we could actually add a __keyword anyway since double-underscores are reserved). -Steve
May 25
prev sibling next sibling parent sighoya <sighoya gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?
I prefer Mike's proposal, too. It is more readable to some who isn't that involved in the topic, like me :) for instance. As a compromise, what about this?: ```D void move(T)( __source return T source, __sink ref scope T target) ``` ```D void move(T)( __source("sourceA") return T sourceA, __sink("targetA") ref scope T targetA, __source("sourceB") return T sourceB, __sink("targetB") ref scope T targetB) ```
May 25
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:


 The `return` attribute allows you to return a scope variable.
 The compiler knows that the returned value has the same 
 lifetime as the argument passed to the function.
 ```D
 // ┌───────────<──────────┐
 int* identity(return int* x)  safe {
     return x;
 }

 [...]
These are *very* good questions.
May 26
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```
Here's a simple extension idea: https://github.com/dlang/dmd/pull/12601 Simply skip over `return` parameters when looking for the 'first' parameter, since making a `return` parameter its own destination has no use. It's still constraining the parameter order, but may be good enough to move forward with `move`.
May 29
next sibling parent ZachAttack! <reachzach gmail.com> writes:
On Saturday, 29 May 2021 at 11:32:50 UTC, Dennis wrote:
 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```
Here's a simple extension idea: https://github.com/dlang/dmd/pull/12601 Simply skip over `return` parameters when looking for the 'first' parameter, since making a `return` parameter its own destination has no use. It's still constraining the parameter order, but may be good enough to move forward with `move`.
It might be more accurate to say, "A simple extension idea, built upon several other 'simple extension ideas,' such that the keyword which activates the feature is now completely counterintuitive!" Apart from the counterintuitive meaning, the other downside, as I see it, would be if the `return` keyword were stretched to the point of exhaustion, and yet still proved insufficient. At that point, a more robust solution would have to be added, and the new solution would have to exist side-by-side with existing overextended uses of `return`. Perhaps in the end, `return` alone, totally out of its element, will nonetheless cover all necessary use cases. But if not, for example, the system I proposed my prior post (https://forum.dlang.org/post/cdurilferuuvwgciqdtb forum.dlang.org) could replace the above extension of `return` in a far more intuitive way. The system uses designated attributes to refer to built-in lifetime routing channels. If we use, for example, ` in1` and ` out1` for channel 1, the `move` function would look like this: ```D // ┌─────────────>───────────┐ void move(T)( in1 T source, out1 ref scope T target) ``` Under this system, the `return` attribute can still be defined—it is a routing channel which automatically attaches to the first viable target, starting with the return reference and then through... whatever the sequence is.... The main point is that if `return` is stretched as far as possible, the day may come when it will be forced to live side-by-side with a replacement which covers much of the same territory. It's likely a manageable problem, a small stain on future incarnations of the language. But worth mentioning.
May 30
prev sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 29 May 2021 at 11:32:50 UTC, Dennis wrote:
 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:
 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```
Here's a simple extension idea: https://github.com/dlang/dmd/pull/12601 Simply skip over `return` parameters when looking for the 'first' parameter, since making a `return` parameter its own destination has no use. It's still constraining the parameter order, but may be good enough to move forward with `move`.
This is the way to go IMO. The potential problem is that if we later on decide we need a more complex solution, it's going to be more complicated than if we do it now. So we kind of have to choose right away unless we want `move2`. Still, ignoring return ref when choosing the output argument sounds the best compromise for me.
Jun 11
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/25/2021 4:24 AM, Dennis wrote:
 [...]
Thanks for a good explanation of the issue. Well done! In practice from what I see, the normal use is: stack.push(value) container.add(element) array.append(element) buffer.write(value) and rather overwhelmingly so, although I did not collect statistics. (The `this` reference is considered the first parameter.) The move example is one counterexample. There's tremendous value in doing the common case in a simple manner, rather than trying to accommodate every case by adding a great deal of complexity to the language and compiler. There's value not just in simplicity, but a follow on effect in having diverse code organized in predictable ways. Hence a simple solution: Make move() trusted. Write an safe alternative to move() with the parameters swapped.
Jun 10
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:
 Hence a simple solution:

 Make move()  trusted.
I don't think that makes sense. `move` is ` safe` anyway for non-scope inputs, and for scope inputs you wouldn't want it to lie about its interface (pretending it's `scope` when it isn't). Also you don't want it to call a ` system` move constructor in ` safe` code.
 Write an  safe alternative to move() with the parameters 
 swapped.
What would that alternative be called?
Jun 11
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2021 3:27 AM, Dennis wrote:
 On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:
 Hence a simple solution:

 Make move()  trusted.
I don't think that makes sense. `move` is ` safe` anyway for non-scope inputs, and for scope inputs you wouldn't want it to lie about its interface (pretending it's `scope` when it isn't). Also you don't want it to call a ` system` move constructor in ` safe` code.
 Write an  safe alternative to move() with the parameters swapped.
What would that alternative be called?
Consider: a = b; Naturally, it moves right to left. We already have a word for that in the library, "emplace".
Jun 12
parent Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 23:21:02 UTC, Walter Bright wrote:
 Consider:

    a = b;

 Naturally, it moves right to left. We already have a word for 
 that in the library, "emplace".
`emplace` is currently used to construct an object on top of uninitialized memory. I don't think adding an overload to `emplace` that acts as move is going to work; even if it doesn't give overload conflicts, it will become difficult to determine whether `emplace(x, y)` is a move that destructs x or emplacement that initializes x with y. Also not that currently `moveEmplace` exists, and it has the same (source, target) parameter order as move. How about defining `moveTo(source, target)` and `moveFrom(target, source)` with an `alias move = moveTo` for backwards compatibility?
Jun 15
prev sibling parent reply ZachAttack! <reachzach gmail.com> writes:
On Friday, 11 June 2021 at 10:27:19 UTC, Dennis wrote:
 On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:
 Hence a simple solution:

 Make move()  trusted.
I don't think that makes sense. `move` is ` safe` anyway for non-scope inputs, and for scope inputs you wouldn't want it to lie about its interface (pretending it's `scope` when it isn't). Also you don't want it to call a ` system` move constructor in ` safe` code.
 Write an  safe alternative to move() with the parameters 
 swapped.
What would that alternative be called?
Challenge: Provide a code example illustrating how the existing variant of `move` compiles when it shouldn't—or errors when it shouldn't—while the alternative form works correctly. I actually couldn't do it... it may be harder than we suspect.
Jun 13
parent ZachAttack! <reachzach gmail.com> writes:
On Sunday, 13 June 2021 at 11:59:49 UTC, ZachAttack! wrote:
 Challenge: Provide a code example illustrating how the existing 
 variant of `move` compiles when it shouldn't—or errors when it 
 shouldn't—while the alternative form works correctly.

 I actually couldn't do it... it may be harder than we suspect.
Ok I found out how to do it. It requires the arguments being passed to `move` to be qualified with `scope`: ```d void move1(ref int* source, scope ref int* target) safe { target = source; } void move2(scope ref int* target, return ref int* source) safe { target = source; } void main() safe { scope int* src; scope int* tgt; move2(tgt, src); // pass move1(src, tgt); // Error: scope variable `src` assigned to non-scope parameter () trusted{ move1(src, tgt); }(); // passes correctly } ``` The `source` parameter in `move1` above cannot be marked `scope` without an error. Thus, the error when `move1` is called is unavoidable. Apparently, only those `move`s involving `scope` variables—a very small percentage—will be *falsely* flagged by the compiler as unsafe. If this is the only price that is being paid for the language limitation, it would seem impossible to justify a language change. Therefore, I suspect Walter's "simple" solution, involving no language change, is the best one. However, marking `move` (or functions like it) as ` system` is overkill. Since only a very small percentage of calls to `move` and similar functions will be flagged wrongly, I believe the right solution is to *document* functions of this form, saying that otherwise ` safe` calls to them may occasionally need to be wrapped in ` trusted` blocks, to avoid the compiler falsely identifying them as having escaping pointers.
Jun 13
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 11.06.21 05:12, Walter Bright wrote:
 Hence a simple solution:
 
 Make move()  trusted.
You can't make `move` trusted. Consider a simplified `move`: ```d void move(ref return scope int* source, ref scope int* target) safe { target = source; /* error */ } ``` An trusted function must still obey the spec. The spec says that the value of `source` cannot be assigned to `target`. The compiler assumes that that holds. If you abuse trusted to break that assumption, undefined behavior follows. For example, you would allow the following: ```d int* target; void f() safe { int local; int* source = &local; move(source, target); /* uh-oh */ } ```
 Write an  safe alternative to move() with the parameters swapped.
If you could make `move` trusted, there would be no need for an safe alternative. safe and trusted are the same from the perspective of the caller (unless you apply trusted incorrectly).
Jun 12
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2021 12:33 AM, ag0aep6g wrote:
 On 11.06.21 05:12, Walter Bright wrote:
 Hence a simple solution:

 Make move()  trusted.
You can't make `move` trusted.
You're right. system then.
 Write an  safe alternative to move() with the parameters swapped.
If you could make `move` trusted, there would be no need for an safe alternative.
Right. But a compiler feature to support one function seems excessive.
Jun 12
parent ag0aep6g <anonymous example.com> writes:
On 13.06.21 04:58, Walter Bright wrote:
 You're right.  system then.
Even an system function must obey the spec (as long as it's called correctly). An system `move` still wouldn't be allowed to do `target = source;` when `source` is `scope`. A way forward could be leaving `scope` off: ---- void move(ref int* source, ref int* target) safe { target = source; /* ok now */ } ---- Then one has to use trusted when calling `move` one `scope` things: ---- void main() safe { int* source1; int* target1; move(source1, target1); /* non-scope things just work */ int local; int* source2 = &local; () trusted { /* scope things need trusted */ scope int* target2; move(source2, target2); } (); } ---- As always, applying trusted correctly is hard and prone to errors. If you don't want to change the language, it seems to me that replacing `move` with a function that has the parameters swapped is the way to go.
Jun 13
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
Just a question: How will DIP1000 work with custom pointer types?

As in, you don't want the lifetime of the pointer-object, but the 
pointed-to-object.
Jun 12
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 08:23:10 UTC, Ola Fosheim Grøstad 
wrote:
 Just a question: How will DIP1000 work with custom pointer 
 types?

 As in, you don't want the lifetime of the pointer-object, but 
 the pointed-to-object.
`scope` currently applies to the pointed-to-object. For structs, it is applied to each member that is a type with pointers. For `struct Ptr {int* ptr;}`, a `scope Ptr` is treated the same as a `scope int*`. For `struct Handle { int handle; }`, a `scope Handle` is treated like `scope int` (which gets its `scope` stripped away because it has no pointers).
Jun 12
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 18:12:34 UTC, Dennis wrote:
 `scope` currently applies to the pointed-to-object. For 
 structs, it is applied to each member that is a type with 
 pointers. For `struct Ptr {int* ptr;}`, a `scope Ptr` is 
 treated the same as a `scope int*`. For `struct Handle { int 
 handle; }`, a `scope Handle` is treated like `scope int` (which 
 gets its `scope` stripped away because it has no pointers).
Ok, I get it (?) So basically it works if one always transfer smart pointers by value. Although, something like a linked list would not work as it isn't transitive and there is no way to make it transitive? Or?
Jun 12
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 18:18:18 UTC, Ola Fosheim Grøstad 
wrote:
 So basically it works if one always transfer smart pointers by 
 value.
You can pass them by `ref`, which is like a non-null pointer with built-in `scope`. So if you have a `ref scope int*`, you actually have two layers of `scope`: both the pointer variable and the pointed-to-object have a restricted lifetime. You can't take the address of a `ref scope int*` though, since that would result in a double-scope `int**` which dip1000 can't express.
 Although, something like a linked list would not work as it 
 isn't transitive and there is no way to make it transitive? Or?
If you give public access to the `Node* next` member, then you can freely escape `node.next.next` indeed. If you encapsulate it and only allow accessing nodes through `return scope` member functions, then you can make a fully `scope` linked list with e.g. all the nodes allocated on the stack.
Jun 12
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 18:35:02 UTC, Dennis wrote:
 If you give public access to the `Node* next` member, then you 
 can freely escape `node.next.next` indeed. If you encapsulate 
 it and only allow accessing nodes through `return scope` member 
 functions, then you can make a fully `scope` linked list with 
 e.g. all the nodes allocated on the stack.
There should be some way for an ADT to state that objects they produce have the same lifetime as themselves. So if I had a linked list wrapper. And you call ```wrapper.find_node(…)``` then the wrapper object should be able to signal that the returned node has the same lifetime.
Jun 12
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 19:12:27 UTC, Ola Fosheim Grøstad 
wrote:
 There should be some way for an ADT to state that objects they 
 produce have the same lifetime as themselves.

 So if I had a linked list wrapper. And you call 
 ```wrapper.find_node(…)``` then the wrapper object should be 
 able to signal that the returned node has the same lifetime.
That sounds like `return scope`, as long as you wrap the linked list by value and not by pointer.
Jun 12
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 22:23:45 UTC, Dennis wrote:
 That sounds like `return scope`, as long as you wrap the linked 
 list by value and not by pointer.
Ok, thanks! I guess I will have to implement a test case and see how it works out.
Jun 12