www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - DIP1000: The return of 'Extend Return Scope Semantics'

reply Dennis <dkorpel gmail.com> writes:


The `return` attribute allows you to return a scope variable.
The compiler knows that the returned value has the same lifetime 
as the argument passed to the function.
```D
// ┌───────────<──────────┐
int* identity(return int* x)  safe {
     return x;
}

void main()  safe {
     int x;
     //   ┌──────<──────┐
     int* y = identity(&x);
     static int* z; // global variable
     z = y;         // error! scope variable `y` assigned to 
non-scope `z`
}
```

What if you are not using a return value, but setting it to 
another parameter, e.g. using `ref` or `out`?

```D
//                          
┌─────────<─────────┐
void assign(ref scope int* target, return int* source)  safe {
     target = source;
}
```
The compiler knows that `source` cannot be returned since it's a 
`void` function, so it makes the first parameter, _but only the 
first parameter_, the destination of the `return` parameter 
`source`.
```D
void main()  safe {
     int x;
     int* y;
     //     ┌─<─┐
     assign(y, &x); // allowed
}
```

This feature has been proposed and implemented by Walter Bright: 
[Issue 19097](https://issues.dlang.org/show_bug.cgi?id=19097), 
[fix Issue 19097 - Extend Return Scope 
Semantics](https://github.com/dlang/dmd/pull/8504).
It was merged on January 2019, but in August 2018 there was a 
discussion about it:
[Re: Is  safe still a 
work-in-progress?](https://forum.dlang.org/post/plja2k$28r0$1 digitalmars.com)

Mike Franklin:

 Why not the first `ref` parameter regardless of whether it's 
 the absolute first in the list.  Why not the last `ref` 
 parameter?  Why not all `ref` parameters?


Walter Bright:

 Good question. If this fairly restricted form solves the 
 problems, then there is no need for the more flexible form. 
 Things can always be made more flexible in the future, but 
 tightening things can be pretty disruptive. Hence, unless there 
 is an obvious and fairly strong case case for the flexibility, 
 then it should be avoided for now.


So did the restricted form turn out to be good enough? Well, on 
March 23 2019, [Build entire Phobos with -preview=dip1000 
https://github.com/dlang/phobos/pull/6931) was merged, so 
it seems to be. Hurray!

Except... It turns out we're not quite done yet, because Druntime 
and Phobos still rely on a pretty major accepts-invalid bug to 
compile with -dip1000, see:
[dip1000 + pure is a DEADLY 
COMBO](https://forum.dlang.org/thread/jnkdcngzytgtobihzggj forum.dlang.org)

The good news is, Per Nordlöw and I have been working on [fixing 
Phobos](https://github.com/dlang/phobos/pull/8076), and Druntime 
[[1]](https://github.com/dlang/druntime/pull/3471), 
[[2]](https://github.com/dlang/druntime/pull/3472), 
[[3]](https://github.com/dlang/druntime/pull/3474), 
[[4]](https://github.com/dlang/druntime/pull/3480) (also thanks 
to [aG0aep6G](https://github.com/dlang/phobos/pull/8109), [Walter 
Bright](https://github.com/dlang/phobos/pull/8104), [Iain 
Buclaw](https://github.com/dlang/phobos/pull/8103), and 
reviewers). During this, we did stumble on cases where the 
restriction came up, such as 
[BigUint.divMod](https://github.com/dlang/phobos/pull/8081#iss
ecomment-842976462) having `out` parameters after the input parameters. More
importantly:



[It turns 
out](https://github.com/dlang/druntime/pull/3480#issuecomment-845480887) that
`core.lifetime: move` (which is [also in
Phobos](https://github.com/dlang/phobos/blob/460ed9c6198b1c84d54ef98b4d0240bc672b89b5/std/algorithm
mutation.d#L1075)), has a signature that's incompatible with -dip1000 in its
current form:

```D
void move(T)(ref T source, ref T target)
```

We want to express:
```D
//                      
┌─────────>─────────┐
void move(T)(return T source, ref scope T target)
```

But there's currently now way of making `target` the return scope 
destination when it's not the first parameter. So unless we want 
to leave `move` ` system` or create a `move2` function with 
parameters reversed, we're back at the drawing board for "Extend 
Return Scope Parameters". Two earlier proposals were:

```D
// Mike Franklin's proposal:
void move(T)(return(target) T source, ref scope T target)

// Steven Schveighoffer's proposal:
void move(T)(return T source,  __sink ref scope T target)
```

What do you think?
May 25 2021
next sibling parent reply kinke <noone nowhere.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So unless 
 we want to leave `move` ` system` or create a `move2` function 
 with parameters reversed, we're back at the drawing board for 
 "Extend Return Scope Parameters". Two earlier proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?


Good overview, thanks for the post. If we don't really need a 
generic solution for that, i.e., can expect new code to use the 
first param and `move` to be the only problematic existing code, 
a possibility would be to make `move` and `forward` compiler 
intrinsics. The compiler can handle those much more efficiently 
than a non-trivial library solution with according template bloat.
May 25 2021
next sibling parent Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:

 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use the 
 first param and `move` to be the only problematic existing 
 code, a possibility would be to make `move` and `forward` 
 compiler intrinsics. The compiler can handle those much more 
 efficiently than a non-trivial library solution with according 
 template bloat.


Even without moving the implementation inside the compiler, an 
easy fix would be to make the druntime function a special case in 
dmd. I recently found out Walter [did something 
similar](https://github.com/dlang/dmd//commit/ab5a18635adaa3e2271e0f4b569500a89ac74235#diff-88cc711d774668de0af3461742be476454e8374545722dfd7
457a3264aecf32R804) to make `char[].dup` implicitly convert to
`immutable(char)[]`. It's ugly, but it does move `-dip1000` forward.
May 25 2021
prev sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:

 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So unless 
 we want to leave `move` ` system` or create a `move2` function 
 with parameters reversed, we're back at the drawing board for 
 "Extend Return Scope Parameters". Two earlier proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?


 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use the 
 first param and `move` to be the only problematic existing code,


This sounds like a really annoying and arbitrary limitation. 
Among other things, it interferes with the API designer's ability 
to freely choose which parameter they want to serve as the 
pseudo-`this` for UFCS.

Being able to make that choice is important both for aesthetic 
reasons, but more importantly to enable "extension methods" that 
allow writing generic code that accepts both custom types, and 
built-in types which are not otherwise extensible.

Also, what about C interop? APIs written in C often contain 
functions which otherwise obey the semantics of `scope` and 
`return`, but will not respect arbitrary limitations on the order 
of parameters.
May 25 2021
parent reply Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:

 On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:

 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 But there's currently now way of making `target` the return 
 scope destination when it's not the first parameter. So 
 unless we want to leave `move` ` system` or create a `move2` 
 function with parameters reversed, we're back at the drawing 
 board for "Extend Return Scope Parameters". Two earlier 
 proposals were:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?


 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use 
 the first param and `move` to be the only problematic existing 
 code,


 This sounds like a really annoying and arbitrary limitation.


It's not arbritrary at all - the purpose is to avoid this:

https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

Reasonable people of course can disagree on whether or not 
avoiding that is a good idea.
May 26 2021
next sibling parent reply jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 [snip]

 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


I also couldn't help but think of Rust's lifetime annotations 
when reading the OP's post. Rust's is more expressive though. If 
you try to limit it, then there obviously will be cases that you 
can't cover everything without making it more complicated. I 
can't help but think that return should be something like 
lifetime(return) instead. Then you can have lifetime(variableA), 
lifetime(variableB). It's a bit more confusing to have 
return(variableA)/return(variableB).
May 26 2021
parent reply Paul Backus <snarwin gmail.com> writes:
On Wednesday, 26 May 2021 at 13:01:24 UTC, jmh530 wrote:

 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 [snip]

 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


 I also couldn't help but think of Rust's lifetime annotations 
 when reading the OP's post. Rust's is more expressive though. 
 If you try to limit it, then there obviously will be cases that 
 you can't cover everything without making it more complicated. 
 I can't help but think that return should be something like 
 lifetime(return) instead. Then you can have 
 lifetime(variableA), lifetime(variableB). It's a bit more 
 confusing to have return(variableA)/return(variableB).


D's general philosophy w.r.t. memory safety so far has been to 
try and balance ease of use and comprehension with 
expressiveness. D's `scope` and `return` are less expressive than 
Rust's lifetimes, but that expressiveness may be worth giving up 
if it's easier to write correct ` safe` and DIP1000-compliant 
code than it is to write correct Rust code.

Of course, D's vision here is severely hampered in practice by 
the poor quality of its documentation (raise your hand if you can 
explain what ["return ref parameter semantics with additional 
scope parameter semantics"][1] actually means). But that's the 
idea.

[1]: 
https://dlang.org/spec/function.html#ref-return-scope-parameters
May 26 2021
next sibling parent jmh530 <john.michael.hall gmail.com> writes:
On Wednesday, 26 May 2021 at 15:29:32 UTC, Paul Backus wrote:

 [snip]

 D's general philosophy w.r.t. memory safety so far has been to 
 try and balance ease of use and comprehension with 
 expressiveness. D's `scope` and `return` are less expressive 
 than Rust's lifetimes, but that expressiveness may be worth 
 giving up if it's easier to write correct ` safe` and 
 DIP1000-compliant code than it is to write correct Rust code.


Correct me if I'm wrong, but Rust bakes in D's `scope` to 
everything unless you override it with a lifetime annotation. So 
I don't think the issue is `scope` per se. `return` is what is 
used to tie the lifetime of a parameter to the lifetime of the 
output. I agree that in the simple example it is easier to write 
correct code.


 Of course, D's vision here is severely hampered in practice by 
 the poor quality of its documentation (raise your hand if you 
 can explain what ["return ref parameter semantics with 
 additional scope parameter semantics"][1] actually means). But 
 that's the idea.

 [1]: 
 https://dlang.org/spec/function.html#ref-return-scope-parameters


+1
May 26 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2021 8:29 AM, Paul Backus wrote:

 Of course, D's vision here is severely hampered in practice by the poor
quality 
 of its documentation (raise your hand if you can explain what ["return ref 
 parameter semantics with additional scope parameter semantics"][1] actually 
 means). But that's the idea.
 
 [1]: https://dlang.org/spec/function.html#ref-return-scope-parameters


The beauty of open source is anyone can submit PRs to improve it!
Jun 10 2021
prev sibling next sibling parent Zach Attack! <reachzach gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


Proposal: Solve the problem incrementally. Start with a single 
pre-named routing channel (i.e. lifetime) supported directly by 
the compiler. Call it ` __in1`  and ` __out1` (these are 
parameter attributes). The compiler will track this lifetime in 
addition to the `scope` and `return` lifetimes. It’s possible 
that this one additional channel will solve all use cases.

If later on, someone provides an undeniably compelling use case 
for additional lifetime channels, simply add ` __in2` and 
` __out2`, ` __in3` and ` __out3`, etc. as the case requires.

The assumption upon which this solution is based is that Rust’s 
user-named "dictionary of lifetimes" is **complete and total 
OVERKILL**, that the actual number of lifetimes a function ever 
really must track is so small that the tracking mechanism can be 
incrementally baked into the compiler itself. (If I’m not 
mistaken, a baked-in lifetime requires only two bits of compiler 
memory per parameter to track, and only one bit for other 
variables.)

For the most part, users would hardly ever see ` __in1` and 
` __out1` since they would be inferred by templates, and rarely 
used anyway. But for cases like phobos’ `move`, the facility 
would be available.

Hope you like my suggestion.

Zach
May 26 2021
prev sibling next sibling parent reply tsbockman <thomas.bockman gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:

 On Tuesday, 25 May 2021 at 11:46:12 UTC, kinke wrote:

 Good overview, thanks for the post. If we don't really need a 
 generic solution for that, i.e., can expect new code to use 
 the first param and `move` to be the only problematic 
 existing code,


 This sounds like a really annoying and arbitrary limitation.


 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures


Why must it always be the first parameter, instead of always 
being the last parameter, etc.? The choice of **which** parameter 
gets special treatment is arbitrary, and is the part I was 
complaining about because of how it interacts with UFCS.
May 26 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/26/2021 5:44 PM, tsbockman wrote:

 Why must it always be the first parameter, instead of always being the last 
 parameter, etc.? The choice of **which** parameter gets special treatment is 
 arbitrary, and is the part I was complaining about because of how it interacts 
 with UFCS.


It is not arbitrary. It actually fits with common Phobos usage. It is
especially 
true if one writes member functions for a container, doing things like:

    stack.push(value);

where value is a pointer.
Jun 10 2021
parent reply ag0aep6g <anonymous example.com> writes:
On 11.06.21 04:58, Walter Bright wrote:

 It is not arbitrary. It actually fits with common Phobos usage. It is 
 especially true if one writes member functions for a container, doing 
 things like:
 
     stack.push(value);
 
 where value is a pointer.


You can't defend the "first parameter" rule with member functions. If 
you want to support returning through `this`, you can easily define that 
without bringing the order of the parameters into it.

The actual point of the "first parameter" rule is obviously UFCS. You 
want support a free function `push` that is called via UFCS. `push` is 
not a member function then.
Jun 10 2021
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2021 10:53 PM, ag0aep6g wrote:

 You can't defend the "first parameter" rule with member functions. If you want 
 to support returning through `this`, you can easily define that without
bringing 
 the order of the parameters into it.
 The actual point of the "first parameter" rule is obviously UFCS. You want 
 support a free function `push` that is called via UFCS. `push` is not a member 
 function then.


The whole idea of UFCS is member functions and regular functions are 
interchangeable, hence `this` is the first parameter.
Jun 11 2021
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 6/10/2021 10:53 PM, ag0aep6g wrote:

 [...]


BTW, another "first parameter" rule is assignment operators:

    a += b;
    a ~= b;
    a = b;

The right side (second parameter) is assigned to the left side (first
parameter).
Jun 11 2021
prev sibling next sibling parent reply Joseph Rushton Wakeling <joseph.wakeling webdrake.net> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 On Tuesday, 25 May 2021 at 22:19:23 UTC, tsbockman wrote:

 This sounds like a really annoying and arbitrary limitation.


 It's not arbritrary at all - the purpose is to avoid this:

 https://carols10cents.github.io/book/ch10-03-lifetime-syntax.html#lifetime-annotations-in-function-signatures

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


I understand that motivation, but at the same time, it's not 
necessarily good to reduce flexibility for the programmer.  I 
think I would rather have to clearly annotate my target ref/out 
parameters every time, than rely on a "magic" rule like "the 
first parameter is the targeted one".

Of course, that doesn't mean that I want to have Rust-style 
lifetime annotation if we can have something simpler and easier!  
But it would certainly be very nice to have the full scope of 
possibilities that Rust allows.  It strikes me that what D should 
aim for is notation that is clear and unambiguous in intent, but 
which scales nicely with the complexity of what the programmer 
wants to achieve (so e.g. if you want to ensure a struct instance 
outlives a pointer/reference that it wraps, you might have to do 
more annotation, but if you want to make sure that in input 
parameter outlives a returned parameter, that's simple and easy).
May 27 2021
parent Atila Neves <atila.neves gmail.com> writes:
On Thursday, 27 May 2021 at 19:03:53 UTC, Joseph Rushton Wakeling 
wrote:

 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 [...]


 I understand that motivation, but at the same time, it's not 
 necessarily good to reduce flexibility for the programmer.  I 
 think I would rather have to clearly annotate my target ref/out 
 parameters every time, than rely on a "magic" rule like "the 
 first parameter is the targeted one".

 [...]


I agree with all of this. The problem is of course that pesky 
idea that will neatly tie everything up.
Jun 01 2021
prev sibling parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


I find Rust's approach more intuitive, even though I am not a 
Rust programmer...

In general, there is way too many words to remember the meaning 
of on D function signatures.

But I guess  system is still available...
May 27 2021
parent reply vitoroak <carvalhogvm gmail.com> writes:
On Thursday, 27 May 2021 at 19:57:49 UTC, Ola Fosheim Grostad 
wrote:

 On Wednesday, 26 May 2021 at 12:36:42 UTC, Atila Neves wrote:

 Reasonable people of course can disagree on whether or not 
 avoiding that is a good idea.


 I find Rust's approach more intuitive, even though I am not a 
 Rust programmer...

 In general, there is way too many words to remember the meaning 
 of on D function signatures.

 But I guess  system is still available...


I do find it more intuitive in Rust too, and lifetime parameters 
in Rust also apply to structs so you can have a struct that has 
the same lifetime as the reference it receives, so you can have a 
Mutex for example return a LockGuard struct that will not outlive 
the Mutex.
May 27 2021
parent reply Ola Fosheim Grostad <ola.fosheim.grostad gmail.com> writes:
On Thursday, 27 May 2021 at 20:11:37 UTC, vitoroak wrote:

 I do find it more intuitive in Rust too, and lifetime 
 parameters in Rust also apply to structs so you can have a 
 struct that has the same lifetime as the reference it receives, 
 so you can have a Mutex for example return a LockGuard struct 
 that will not outlive the Mutex.


Sounds useful. I have never used lifetimes myself, only read 
about it (including some theory), but I know that lifetime 
annotations predates Rust, I think it was used in a ML dialect. 
So it isn't unique to Rust and I assume universities will teach 
it... So, it will most likely become standard? Dunno.
May 27 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Thursday, 27 May 2021 at 20:21:58 UTC, Ola Fosheim Grostad 
wrote:

 On Thursday, 27 May 2021 at 20:11:37 UTC, vitoroak wrote:

 I do find it more intuitive in Rust too, and lifetime 
 parameters in Rust also apply to structs so you can have a 
 struct that has the same lifetime as the reference it 
 receives, so you can have a Mutex for example return a 
 LockGuard struct that will not outlive the Mutex.


 Sounds useful. I have never used lifetimes myself, only read 
 about it (including some theory), but I know that lifetime 
 annotations predates Rust, I think it was used in a ML dialect. 
 So it isn't unique to Rust and I assume universities will teach 
 it... So, it will most likely become standard? Dunno.


It goes back to Cyclone, https://cyclone.thelanguage.org/
May 29 2021
next sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:

 It goes back to Cyclone, https://cyclone.thelanguage.org/


There you go, with original papers and all.

I still think local GC + ARC for shared is a more convenient path 
for  safe users, but who am I to judge if people want 
affine/linear types.
May 29 2021
parent reply Paulo Pinto <pjmlp progtools.org> writes:
On Saturday, 29 May 2021 at 12:39:48 UTC, Ola Fosheim Grøstad 
wrote:

 On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:

 It goes back to Cyclone, https://cyclone.thelanguage.org/


 There you go, with original papers and all.

 I still think local GC + ARC for shared is a more convenient 
 path for  safe users, but who am I to judge if people want 
 affine/linear types.


I would rather bet on GC + affine/linear types like Haskell is 
pursuing.

In any case, there is even research trying to make use of C2X 
annotations to retrofit it into C,

https://plv.mpi-sws.org/refinedc/
May 29 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 29 May 2021 at 18:21:15 UTC, Paulo Pinto wrote:

 I would rather bet on GC + affine/linear types like Haskell is 
 pursuing.


Haskell is a functional language, so they are in a much better 
position to separating the heap and doing shape analysis.


 In any case, there is even research trying to make use of C2X 
 annotations to retrofit it into C,

 https://plv.mpi-sws.org/refinedc/


Looks more or less like other languages that focus on full formal 
verification using Hoare logic.
May 30 2021
prev sibling parent Elronnd <elronnd elronnd.net> writes:
On Saturday, 29 May 2021 at 12:32:49 UTC, Paulo Pinto wrote:

 On Thursday, 27 May 2021 at 20:21:58 UTC, Ola Fosheim Grostad 
 wrote:

 I know that lifetime annotations predates Rust, I think it was 
 used in a ML dialect.


 It goes back to Cyclone, https://cyclone.thelanguage.org/


Also clean (https://clean.cs.ru.nl/Clean).  Don't know which came 
first, but clean is certainly more of an ml (cyclone is almost 
backwards-compatible with c).
Jun 11 2021
prev sibling next sibling parent reply Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:



 The `return` attribute allows you to return a scope variable.
 The compiler knows that the returned value has the same 
 lifetime as the argument passed to the function.
 ```D
 // ┌───────────<──────────┐
 int* identity(return int* x)  safe {
     return x;
 }

 [...]


I think the Franklin solution is the best one presented here. 
Similarly with inout, any keywords in this context (i.e. 
non-trivial and non-local effect on the code) should aim for some 
syntax like this, even if the existing rules technically mean you 
could go without, flexibility and redundancy are a good thing 
(especially for people new to the language).
May 25 2021
parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:50:29 UTC, Max Haughton wrote:

 I think the Franklin solution is the best one presented here.


It looks nice, but it's also the one with the most implementation 
work. It requires a change to the parser (+libdparse), new 
mangling rules, and if we want to support it: it opens the door 
for multiple return parameters with different destinations.

Looking at the implementation 
[dmd.escape(603)](https://github.com/dlang/dmd/blob/a7cf02c8bb124f5d5a05b9d238f71d2c0773dc51/src
dmd/escape.d#L603), it's not going to be me who will be able to implement that
any time soon.
May 25 2021
parent Max Haughton <maxhaton gmail.com> writes:
On Tuesday, 25 May 2021 at 12:48:55 UTC, Dennis wrote:

 On Tuesday, 25 May 2021 at 11:50:29 UTC, Max Haughton wrote:

 I think the Franklin solution is the best one presented here.


 It looks nice, but it's also the one with the most 
 implementation work. It requires a change to the parser 
 (+libdparse), new mangling rules, and if we want to support it: 
 it opens the door for multiple return parameters with different 
 destinations.

 Looking at the implementation 
 [dmd.escape(603)](https://github.com/dlang/dmd/blob/a7cf02c8bb124f5d5a05b9d238f71d2c0773dc51/src
dmd/escape.d#L603), it's not going to be me who will be able to implement that
any time soon.


This is unfortunately true.

That being said I think "too hard to implement" is being said a 
lot these days, and it has nothing to do with documentation (in 
light of the other thread).
May 25 2021
prev sibling next sibling parent zjh <fqbqrr 163.com> writes:
The meaning of function has limited the position of `return`,
So`void move(T)(return(target) T source, ref scope T target)`is 
enough.
`return(target)`can be added to any parameter.
May 25 2021
prev sibling next sibling parent reply zjh <fqbqrr 163.com> writes:
The meaning of function has limited the position of `return`,
So`void move(T)(return(target) T source, ref scope T target)`is 
enough.
`return(target)`can be added to any parameter.
May 25 2021
parent reply zjh <fqbqrr 163.com> writes:
We can limit each parameter to `one return` at most.
It's not a bad thing to have multiple `return areas`. My `C++` 
function `f(A,B,C,D,...)`,every even parameter` can be used as 
the return value.
May 25 2021
parent zjh <fqbqrr 163.com> writes:
On Tuesday, 25 May 2021 at 13:18:16 UTC, zjh wrote:

Maybe we can simplify it with a symbol ` (a)/ (b)`, then ` a/ b`, 
and so on.Indicating the direction of the pointer to `track the 
direction` of the pointer(`A-->B`).
May 26 2021
prev sibling next sibling parent Steven Schveighoffer <schveiguy gmail.com> writes:
On 5/25/21 7:24 AM, Dennis wrote:

 But there's currently now way of making `target` the return scope 
 destination when it's not the first parameter. So unless we want to 
 leave `move` ` system` or create a `move2` function with parameters 
 reversed, we're back at the drawing board for "Extend Return Scope 
 Parameters". Two earlier proposals were:
 
 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)
 
 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```
 
 What do you think?


Re-reading that thread, there is more to the proposal than just the 
attribute -- the fact that the  __sink attribute would be inferred on 
templates. So no attribute would be necessary on `move`, even if the 
attribute is added for non-templates/non-auto functions (ditto for 
Mike's proposal).

That being said, Mike's proposal is obviously more flexible, as it 
allows multiple return parameters that link to multiple input 
parameters. But the cases in which those are present are super-rare.

I also don't like having a keyword (return) match with an attribute 
( __something), as they seem unrelated, and the only reason an attribute 
would be to avoid adding a keyword. But that's a bit of bikeshedding (we 
could actually add a __keyword anyway since double-underscores are 
reserved).

-Steve
May 25 2021
prev sibling next sibling parent sighoya <sighoya gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 ```D
 // Mike Franklin's proposal:
 void move(T)(return(target) T source, ref scope T target)

 // Steven Schveighoffer's proposal:
 void move(T)(return T source,  __sink ref scope T target)
 ```

 What do you think?


I prefer Mike's proposal, too. It is more readable to some who 
isn't that involved in the topic, like me :) for instance.

As a compromise, what about this?:

```D
void move(T)( __source return T source,  __sink ref scope T 
target)
```

```D
void move(T)( __source("sourceA") return T sourceA, 
 __sink("targetA") ref scope T targetA,  __source("sourceB") 
return T sourceB,  __sink("targetB") ref scope T targetB)
```
May 25 2021
prev sibling next sibling parent Atila Neves <atila.neves gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:



 The `return` attribute allows you to return a scope variable.
 The compiler knows that the returned value has the same 
 lifetime as the argument passed to the function.
 ```D
 // ┌───────────<──────────┐
 int* identity(return int* x)  safe {
     return x;
 }

 [...]


These are *very* good questions.
May 26 2021
prev sibling next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```


Here's a simple extension idea: 
https://github.com/dlang/dmd/pull/12601

Simply skip over `return` parameters when looking for the 'first' 
parameter, since making a `return` parameter its own destination 
has no use. It's still constraining the parameter order, but may 
be good enough to move forward with `move`.
May 29 2021
next sibling parent ZachAttack! <reachzach gmail.com> writes:
On Saturday, 29 May 2021 at 11:32:50 UTC, Dennis wrote:

 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```


 Here's a simple extension idea: 
 https://github.com/dlang/dmd/pull/12601

 Simply skip over `return` parameters when looking for the 
 'first' parameter, since making a `return` parameter its own 
 destination has no use. It's still constraining the parameter 
 order, but may be good enough to move forward with `move`.


It might be more accurate to say, "A simple extension idea, built 
upon several other 'simple extension ideas,' such that the 
keyword which activates the feature is now completely 
counterintuitive!"

Apart from the counterintuitive meaning, the other downside, as I 
see it, would be if the `return` keyword were stretched to the 
point of exhaustion, and yet still proved insufficient. At that 
point, a more robust solution would have to be added, and the new 
solution would have to exist side-by-side with existing 
overextended uses of `return`.

Perhaps in the end, `return` alone, totally out of its element, 
will nonetheless cover all necessary use cases. But if not, for 
example, the system I proposed my prior post 
(https://forum.dlang.org/post/cdurilferuuvwgciqdtb forum.dlang.org) could
replace the above extension of `return` in a far more intuitive way. The system
uses designated attributes to refer to built-in lifetime routing channels. If
we use, for example, ` in1` and ` out1` for channel 1, the `move` function
would look like this:
```D
//                   
┌─────────────>───────────┐
void move(T)( in1 T source,  out1 ref scope T target)
```
Under this system, the `return` attribute can still be defined—it 
is a routing channel which automatically attaches to the first 
viable target, starting with the return reference and then 
through... whatever the sequence is....

The main point is that if `return` is stretched as far as 
possible, the day may come when it will be forced to live 
side-by-side with a replacement which covers much of the same 
territory.

It's likely a manageable problem, a small stain on future 
incarnations of the language. But worth mentioning.
May 30 2021
prev sibling parent Dukc <ajieskola gmail.com> writes:
On Saturday, 29 May 2021 at 11:32:50 UTC, Dennis wrote:

 On Tuesday, 25 May 2021 at 11:24:59 UTC, Dennis wrote:

 We want to express:
 ```D
 //                      
┌─────────>─────────┐
 void move(T)(return T source, ref scope T target)
 ```


 Here's a simple extension idea: 
 https://github.com/dlang/dmd/pull/12601

 Simply skip over `return` parameters when looking for the 
 'first' parameter, since making a `return` parameter its own 
 destination has no use. It's still constraining the parameter 
 order, but may be good enough to move forward with `move`.


This is the way to go IMO.

The potential problem is that if we later on decide we need a 
more complex solution, it's going to be more complicated than if 
we do it now. So we kind of have to choose right away unless we 
want `move2`.

Still, ignoring return ref when choosing the output argument 
sounds the best compromise for me.
Jun 11 2021
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 5/25/2021 4:24 AM, Dennis wrote:

 [...]


Thanks for a good explanation of the issue. Well done!

In practice from what I see, the normal use is:

    stack.push(value)
    container.add(element)
    array.append(element)
    buffer.write(value)

and rather overwhelmingly so, although I did not collect statistics. (The
`this` 
reference is considered the first parameter.)

The move example is one counterexample.

There's tremendous value in doing the common case in a simple manner, rather 
than trying to accommodate every case by adding a great deal of complexity to 
the language and compiler. There's value not just in simplicity, but a follow
on 
effect in having diverse code organized in predictable ways.

Hence a simple solution:

Make move()  trusted.

Write an  safe alternative to move() with the parameters swapped.
Jun 10 2021
next sibling parent reply Dennis <dkorpel gmail.com> writes:
On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:

 Hence a simple solution:

 Make move()  trusted.


I don't think that makes sense. `move` is ` safe` anyway for 
non-scope inputs, and for scope inputs you wouldn't want it to 
lie about its interface (pretending it's `scope` when it isn't). 
Also you don't want it to call a ` system` move constructor in 
` safe` code.


 Write an  safe alternative to move() with the parameters 
 swapped.


What would that alternative be called?
Jun 11 2021
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/11/2021 3:27 AM, Dennis wrote:

 On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:

 Hence a simple solution:

 Make move()  trusted.


 
 I don't think that makes sense. `move` is ` safe` anyway for non-scope inputs, 
 and for scope inputs you wouldn't want it to lie about its interface
(pretending 
 it's `scope` when it isn't). Also you don't want it to call a ` system` move 
 constructor in ` safe` code.
 

 Write an  safe alternative to move() with the parameters swapped.


 
 What would that alternative be called?
 


Consider:

    a = b;

Naturally, it moves right to left. We already have a word for that in the 
library, "emplace".
Jun 12 2021
parent Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 23:21:02 UTC, Walter Bright wrote:

 Consider:

    a = b;

 Naturally, it moves right to left. We already have a word for 
 that in the library, "emplace".


`emplace` is currently used to construct an object on top of 
uninitialized memory. I don't think adding an overload to 
`emplace` that acts as move is going to work; even if it doesn't 
give overload conflicts, it will become difficult to determine 
whether `emplace(x, y)` is a move that destructs x or emplacement 
that initializes x with y.

Also not that currently `moveEmplace` exists, and it has the same 
(source, target) parameter order as move.

How about defining `moveTo(source, target)` and `moveFrom(target, 
source)` with an `alias move = moveTo` for backwards 
compatibility?
Jun 15 2021
prev sibling parent reply ZachAttack! <reachzach gmail.com> writes:
On Friday, 11 June 2021 at 10:27:19 UTC, Dennis wrote:

 On Friday, 11 June 2021 at 03:12:29 UTC, Walter Bright wrote:

 Hence a simple solution:

 Make move()  trusted.


 I don't think that makes sense. `move` is ` safe` anyway for 
 non-scope inputs, and for scope inputs you wouldn't want it to 
 lie about its interface (pretending it's `scope` when it 
 isn't). Also you don't want it to call a ` system` move 
 constructor in ` safe` code.


 Write an  safe alternative to move() with the parameters 
 swapped.


 What would that alternative be called?


Challenge: Provide a code example illustrating how the existing 
variant of `move` compiles when it shouldn't—or errors when it 
shouldn't—while the alternative form works correctly.

I actually couldn't do it... it may be harder than we suspect.
Jun 13 2021
parent ZachAttack! <reachzach gmail.com> writes:
On Sunday, 13 June 2021 at 11:59:49 UTC, ZachAttack! wrote:

 Challenge: Provide a code example illustrating how the existing 
 variant of `move` compiles when it shouldn't—or errors when it 
 shouldn't—while the alternative form works correctly.

 I actually couldn't do it... it may be harder than we suspect.


Ok I found out how to do it. It requires the arguments being 
passed to `move` to be qualified with `scope`:

```d
void move1(ref int* source, scope ref int* target)  safe {
     target = source;
}

void move2(scope ref int* target, return ref int* source)  safe {
     target = source;
}

void main()  safe {
     scope int* src;
     scope int* tgt;
     move2(tgt, src); // pass
     move1(src, tgt); // Error: scope variable `src` assigned to 
non-scope parameter
     ()  trusted{ move1(src, tgt); }(); // passes correctly
}
```
The `source` parameter in `move1` above cannot be marked `scope` 
without an error. Thus, the error when `move1` is called is 
unavoidable.

Apparently, only those `move`s involving `scope` variables—a very 
small percentage—will be *falsely* flagged by the compiler as 
unsafe.

If this is the only price that is being paid for the language 
limitation, it would seem impossible to justify a language change.

Therefore, I suspect Walter's "simple" solution, involving no 
language change, is the best one.

However, marking `move` (or functions like it) as ` system` is 
overkill.

Since only a very small percentage of calls to `move` and similar 
functions will be flagged wrongly, I believe the right solution 
is to *document* functions of this form, saying that otherwise 
` safe` calls to them may occasionally need to be wrapped in 
` trusted` blocks, to avoid the compiler falsely identifying them 
as having escaping pointers.
Jun 13 2021
prev sibling parent reply ag0aep6g <anonymous example.com> writes:
On 11.06.21 05:12, Walter Bright wrote:

 Hence a simple solution:
 
 Make move()  trusted.


You can't make `move`  trusted.

Consider a simplified `move`:

```d
void move(ref return scope int* source, ref scope int* target)  safe
{
     target = source; /* error */
}
```

An  trusted function must still obey the spec. The spec says that the 
value of `source` cannot be assigned to `target`. The compiler assumes 
that that holds. If you abuse  trusted to break that assumption, 
undefined behavior follows.

For example, you would allow the following:

```d
int* target;
void f()  safe
{
     int local;
     int* source = &local;
     move(source, target); /* uh-oh */
}
```


 Write an  safe alternative to move() with the parameters swapped.


If you could make `move`  trusted, there would be no need for an  safe 
alternative.  safe and  trusted are the same from the perspective of the 
caller (unless you apply  trusted incorrectly).
Jun 12 2021
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 6/12/2021 12:33 AM, ag0aep6g wrote:

 On 11.06.21 05:12, Walter Bright wrote:

 Hence a simple solution:

 Make move()  trusted.


 
 You can't make `move`  trusted.


You're right.  system then.



 Write an  safe alternative to move() with the parameters swapped.


 If you could make `move`  trusted, there would be no need for an  safe 
 alternative.


Right. But a compiler feature to support one function seems excessive.
Jun 12 2021
parent ag0aep6g <anonymous example.com> writes:
On 13.06.21 04:58, Walter Bright wrote:

 You're right.  system then.


Even an  system function must obey the spec (as long as it's called 
correctly). An  system `move` still wouldn't be allowed to do `target = 
source;` when `source` is `scope`.

A way forward could be leaving `scope` off:

----
void move(ref int* source, ref int* target)  safe
{
     target = source; /* ok now */
}
----

Then one has to use  trusted when calling `move` one `scope` things:

----
void main()  safe
{
     int* source1;
     int* target1;
     move(source1, target1); /* non-scope things just work */

     int local;
     int* source2 = &local;
     ()  trusted { /* scope things need  trusted */
         scope int* target2;
         move(source2, target2);
     } ();
}
----

As always, applying  trusted correctly is hard and prone to errors.

If you don't want to change the language, it seems to me that replacing 
`move` with a function that has the parameters swapped is the way to go.
Jun 13 2021
prev sibling parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
Just a question: How will DIP1000 work with custom pointer types?

As in, you don't want the lifetime of the pointer-object, but the 
pointed-to-object.
Jun 12 2021
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 08:23:10 UTC, Ola Fosheim Grøstad 
wrote:

 Just a question: How will DIP1000 work with custom pointer 
 types?

 As in, you don't want the lifetime of the pointer-object, but 
 the pointed-to-object.


`scope` currently applies to the pointed-to-object. For structs, 
it is applied to each member that is a type with pointers. For 
`struct Ptr {int* ptr;}`, a `scope Ptr` is treated the same as a 
`scope int*`. For `struct Handle { int handle; }`, a `scope 
Handle` is treated like `scope int` (which gets its `scope` 
stripped away because it has no pointers).
Jun 12 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 18:12:34 UTC, Dennis wrote:

 `scope` currently applies to the pointed-to-object. For 
 structs, it is applied to each member that is a type with 
 pointers. For `struct Ptr {int* ptr;}`, a `scope Ptr` is 
 treated the same as a `scope int*`. For `struct Handle { int 
 handle; }`, a `scope Handle` is treated like `scope int` (which 
 gets its `scope` stripped away because it has no pointers).


Ok, I get it (?) So basically it works if one always transfer 
smart pointers by value. Although, something like a linked list 
would not work as it isn't transitive and there is no way to make 
it transitive? Or?
Jun 12 2021
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 18:18:18 UTC, Ola Fosheim Grøstad 
wrote:

 So basically it works if one always transfer smart pointers by 
 value.


You can pass them by `ref`, which is like a non-null pointer with 
built-in `scope`. So if you have a `ref scope int*`, you actually 
have two layers of `scope`: both the pointer variable and the 
pointed-to-object have a restricted lifetime. You can't take the 
address of a `ref scope int*` though, since that would result in 
a double-scope `int**` which dip1000 can't express.


 Although, something like a linked list would not work as it 
 isn't transitive and there is no way to make it transitive? Or?


If you give public access to the `Node* next` member, then you 
can freely escape `node.next.next` indeed. If you encapsulate it 
and only allow accessing nodes through `return scope` member 
functions, then you can make a fully `scope` linked list with 
e.g. all the nodes allocated on the stack.
Jun 12 2021
parent reply Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 18:35:02 UTC, Dennis wrote:

 If you give public access to the `Node* next` member, then you 
 can freely escape `node.next.next` indeed. If you encapsulate 
 it and only allow accessing nodes through `return scope` member 
 functions, then you can make a fully `scope` linked list with 
 e.g. all the nodes allocated on the stack.


There should be some way for an ADT to state that objects they 
produce have the same lifetime as themselves.

So if I had a linked list wrapper. And you call 
```wrapper.find_node(…)``` then the wrapper object should be able 
to signal that the returned node has the same lifetime.
Jun 12 2021
parent reply Dennis <dkorpel gmail.com> writes:
On Saturday, 12 June 2021 at 19:12:27 UTC, Ola Fosheim Grøstad 
wrote:

 There should be some way for an ADT to state that objects they 
 produce have the same lifetime as themselves.

 So if I had a linked list wrapper. And you call 
 ```wrapper.find_node(…)``` then the wrapper object should be 
 able to signal that the returned node has the same lifetime.


That sounds like `return scope`, as long as you wrap the linked 
list by value and not by pointer.
Jun 12 2021
parent Ola Fosheim =?UTF-8?B?R3LDuHN0YWQ=?= <ola.fosheim.grostad gmail.com> writes:
On Saturday, 12 June 2021 at 22:23:45 UTC, Dennis wrote:

 That sounds like `return scope`, as long as you wrap the linked 
 list by value and not by pointer.


Ok, thanks! I guess I will have to implement a test case and see 
how it works out.
Jun 12 2021