www.digitalmars.com         C & C++   DMDScript  

digitalmars.D.internals - DIP1000 discussion and testing

reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: DIP1000 discussion and testing

--wOClvSkNK3LhkNpiKGvc9cBvr8w7r5rHG
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

So, there is a pull request (https://github.com/dlang/dmd/pull/5972)
which is intended to implement DIP1000
(https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md).

Problem is that Walter wants to prioritize moving on with it (which is
reasonable) but the PR is too complicated to be reviewed by most other
developers.

Trying to solve it, I have been experimenting lately with various
acceptance tests trying to come up with useful design snippets enabled
by that PR. Rationale is that if actual semantics are clear and examples
are all validated as included test cases, review of the code itself
becomes less important.

I'd like to have more open feedback about it because in its current
shape PR5972 doesn't seem useful enough to me to be merged at all - and
much less powerful than DIP1000 seemed to promise.

I'll start separate sub-threads with more details.


--wOClvSkNK3LhkNpiKGvc9cBvr8w7r5rHG--
Oct 16 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: DIP1000 discussion and testing: RC pointer snippet
References: <nu00a6$t5i$1 digitalmars.com>
In-Reply-To: <nu00a6$t5i$1 digitalmars.com>

--vraXSChUTnhqU5bkpie8rqsDHcDoUNcxK
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 04:45 PM, Dicebot wrote:
 I'll start separate sub-threads with more details.=20
This snippet is a variation of RCSlice example from the DIP: ------------------------------------------------------------ safe unittest { safe static struct RCPointer (T) { private T* payload; private size_t* count; this (T initializer) { this.payload =3D new T; *this.payload =3D initializer; this.count =3D new size_t; *count =3D 1; } this (this) { if (this.count) ++(*this.count); } trusted ~this () { if (this.count && !--(*this.count)) { delete payload; delete count; } } // must bind lifetime of return value to lifetime of `this` T* get () return scope { return this.payload; } alias get this; } auto ptr =3D RCPointer!int(42); auto leak2 =3D ptr.get(); // must not compile unless spec says 'auto'= // can deduce scope int* leak1 =3D ptr.get(); // must not compile void foo (RCPointer!int) { assert(*(ptr.count) =3D=3D 2); } foo(ptr); void bar (ref int) { assert(*(ptr.count) =3D=3D 1); } bar(*ptr); } ------------------------------------------------------------ DIP1000 says that `return scope` applied to a method means applying it to aggregate `this`. It also says that struct is viewed as a composition of all its fields when it comes to lifetime. Natural consequence is that it should be possible to define `get` method like in this example to ensure that pointer won't outlive struct instance it came from. This snippet currently compiles, failing to reject both "leak" lines. --vraXSChUTnhqU5bkpie8rqsDHcDoUNcxK--
Oct 16 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2016 6:55 AM, Dicebot wrote:
 This snippet currently compiles, failing to reject both "leak" lines.
It compiles because 'scope' is inferred for both leak1 and leak2.
Oct 16 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: RC pointer snippet
References: <nu00a6$t5i$1 digitalmars.com> <nu00tf$u0a$1 digitalmars.com>
 <nu0lrm$1tm1$1 digitalmars.com>
In-Reply-To: <nu0lrm$1tm1$1 digitalmars.com>

--U458ROC3lnDWdCSgJxEfITovUVTM2FHdP
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 10:53 PM, Walter Bright wrote:
 On 10/16/2016 6:55 AM, Dicebot wrote:
 This snippet currently compiles, failing to reject both "leak" lines.
=20 It compiles because 'scope' is inferred for both leak1 and leak2.
Where does it come from? I don't see any mention of such deduction in https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md and it doesn't look like a good idea at all. --U458ROC3lnDWdCSgJxEfITovUVTM2FHdP--
Oct 16 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2016 1:03 PM, Dicebot wrote:
 Where does it come from?
It comes from trying to compile code with the new safety checks, and having it fail to compile and require adding annotations all over the place. The idea is to infer such annotations in the obvious places. 'return' was already successfully inferred in many places, this just extends an existing practice.
 I don't see any mention of such deduction in
 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
It's one of those things that becomes necessary once trying to implement it.
 and it doesn't look like a good idea at all.
Inferring 'scope' and 'return' as much as possible makes safe much more palatable. All I can say is try to break it!
Oct 16 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: RC pointer snippet
References: <nu00a6$t5i$1 digitalmars.com> <nu00tf$u0a$1 digitalmars.com>
 <nu0lrm$1tm1$1 digitalmars.com> <nu0me7$1umq$1 digitalmars.com>
 <nu0n44$1vmf$1 digitalmars.com>
In-Reply-To: <nu0n44$1vmf$1 digitalmars.com>

--xVnORgmRRjBGIbpf4awD8kdCOumKN6k5p
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 11:15 PM, Walter Bright wrote:
 On 10/16/2016 1:03 PM, Dicebot wrote:
 Where does it come from?
=20 It comes from trying to compile code with the new safety checks, and having it fail to compile and require adding annotations all over the place. The idea is to infer such annotations in the obvious places. 'return' was already successfully inferred in many places, this just extends an existing practice. =20 =20
 I don't see any mention of such deduction in
 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
=20 It's one of those things that becomes necessary once trying to implemen=
t
 it.
Walter, there is no way we can merge your PR if it implements things not mentioned in DIP. If you don't want to spend time adjusting it yourself, please, at least notify me about what has changed. That will save both of us a lot of time.
 and it doesn't look like a good idea at all.
=20 Inferring 'scope' and 'return' as much as possible makes safe much mor=
e
 palatable. All I can say is try to break it!
It also makes reasoning about what your code does much harder, adding unacceptable amount of magic to it. I'd much prefer to rely on `auto` and fix rest of cases myself through deprecations than go with current "helpful" behavior. Such semantics are also unprecedented in D as far as I can remember. Before storage classes would never be inferred for variables with explicit type. That is highly confusing. Right now I think this is a bad decision and should be reconsidered. Would like to now what other developers thing though. --xVnORgmRRjBGIbpf4awD8kdCOumKN6k5p--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 9:32 AM, Dicebot wrote:
 On 10/16/2016 11:15 PM, Walter Bright wrote:
 On 10/16/2016 1:03 PM, Dicebot wrote:
 Where does it come from?
It comes from trying to compile code with the new safety checks, and having it fail to compile and require adding annotations all over the place. The idea is to infer such annotations in the obvious places. 'return' was already successfully inferred in many places, this just extends an existing practice.
 I don't see any mention of such deduction in
 https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md
It's one of those things that becomes necessary once trying to implement it.
Walter, there is no way we can merge your PR if it implements things not mentioned in DIP. If you don't want to spend time adjusting it yourself, please, at least notify me about what has changed. That will save both of us a lot of time.
Ok. But it's pretty simple. A local variable that is initialized with a scoped value is tagged with scope.
 Inferring 'scope' and 'return' as much as possible makes  safe much more
 palatable. All I can say is try to break it!
It also makes reasoning about what your code does much harder, adding unacceptable amount of magic to it. I'd much prefer to rely on `auto` and fix rest of cases myself through deprecations than go with current "helpful" behavior. Such semantics are also unprecedented in D as far as I can remember. Before storage classes would never be inferred for variables with explicit type. That is highly confusing.
It's already done by DIP25, and D has been very successful in inferring safe, trusted, system, pure, nogc, and nothrow. I'd like to go even further and infer const, but that's a more complex task.
 Right now I think this is a bad decision and should be reconsidered.
 Would like to now what other developers thing though.
It's long been my belief that if lifetime annotations had to be explicit, few would be interested in using such a "bondage and discipline" language. As further evidence of that, Rust goes to great lengths to infer lifetime attributes. Annotations don't excite people.
Oct 20 2016
parent reply meppl <mephisto nordhoff-online.de> writes:
On Thursday, 20 October 2016 at 21:43:59 UTC, Walter Bright wrote:
 It's long been my belief that if lifetime annotations had to be 
 explicit, few would be interested in using such a "bondage and 
 discipline" language. As further evidence of that, Rust goes to 
 great lengths to infer lifetime attributes. Annotations don't 
 excite people.
explicit lifetime annotations... isnt it possible to upgrade this scope-annotation system afterwards? for example: hidden scope after dip1000 is implemented, a new storage_class gets introduced - lets say „jailed“. If „jailed“ appears in code, the compiler scans the AST for the route the pointer takes within one module or within one package and adds the needed hidden_scope annotations automatically. it would look like that: void func( T* t) safe; // second version of this function exists: // void func( hidden_scope T* t) safe; void bar( jailed T* t) safe { func(t); // ok, compiler added hidden_scope for void func( T* t) itself } how it looks like in DIP1000: void func(scope T* t) safe; void bar(scope T* t) safe { func(t); // ok } if an opaque function (of different module/package) appears, that opaque function must have the scope/jailed annotation again.
Oct 20 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 01:46:20 UTC, meppl wrote:
 void bar( jailed T* t)  safe {
    func(t); // ok, compiler added hidden_scope for void func( 
 T* t) itself
 }
As said in DIP1000 and implemented in the PR, the compiler infers scope for lambda and template function parameters. Not sure what you're proposing. For sure we want to extend any attribute inferrence to functions, but how to make that work with separate compilation is still undecided. Talking more about inferrence would be an OT discussion though.
Oct 21 2016
next sibling parent meppl <mephisto nordhoff-online.de> writes:
On Friday, 21 October 2016 at 12:04:33 UTC, Martin Nowak wrote:
 On Friday, 21 October 2016 at 01:46:20 UTC, meppl wrote:
 void bar( jailed T* t)  safe {
    func(t); // ok, compiler added hidden_scope for void func( 
 T* t) itself
 }
As said in DIP1000 and implemented in the PR, the compiler infers scope for lambda and template function parameters. Not sure what you're proposing. For sure we want to extend any attribute inferrence to functions, but how to make that work with separate compilation is still undecided. Talking more about inferrence would be an OT discussion though.
I just assumed that inference for DIP1000 (for non-template private functions) is _possible_ to implement afterwards with backward compatibility. I want to know, if it is possible, because DIP1000 tells people to scatter another keyword („scope“) all over the place in the code.
Oct 21 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 5:04 AM, Martin Nowak wrote:
 For sure we want to extend any attribute inferrence to functions, but how to
 make that work with separate compilation is still undecided. Talking more about
 inferrence would be an OT discussion though.
I know how to make it (inference on regular functions) work, but it is a separate topic, and would only further confuse things here.
Oct 22 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: DIP1000 discussion and testing: borrowing a range
References: <nu00a6$t5i$1 digitalmars.com>
In-Reply-To: <nu00a6$t5i$1 digitalmars.com>

--LiNlUgC1X3TLk2kb3sSe8FRpCr6wGUpqb
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 04:45 PM, Dicebot wrote:
 I'll start separate sub-threads with more details.
This is probably most ambitious snippet I have come up with so far. It seems to be possible within current DIP1000 definition but doesn't work for other reasons I wanted to discuss: ------------------------------------------------------------ safe unittest { safe static struct Owner { int data; // must bind lifetime of return pointer/ref to one of `this` // Error: function has 'return' but does not return any // indirections auto borrow () return scope { struct Range { int* data; enum empty =3D false; auto front () { return *this.data; } void popFront() { (*this.data)++; } } return Range(&this.data); // Error: cannot take address of // parameter this in safe function borrow } } Owner instance; // must work, lifetime of 'instance' exceeds one of x : scope x =3D instance.borrow(); // must not work, infinite lifetime auto y =3D instance.borrow(); } ------------------------------------------------------------ First error simply to be simply a bug, because it contradicts "struct is a composition of its fields" statement from DIP1000. Second error is an existing limitation of safe code (can't take pointer from ref) which seems extremely limiting now that tools to enforce scope lifetime come into play. --LiNlUgC1X3TLk2kb3sSe8FRpCr6wGUpqb--
Oct 16 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2016 7:02 AM, Dicebot wrote:
     Owner instance;
     // must work, lifetime of 'instance' exceeds one of x :
     scope x = instance.borrow();
     // must not work, infinite lifetime
     auto y  = instance.borrow();
 }
 Second error is an existing limitation of  safe code (can't take pointer
 from ref) which seems extremely limiting now that tools to enforce scope
 lifetime come into play.
It works because 'scope' is inferred as a storage class when an expression that needs scope is used to initialize a local. y gets a subset of the life of instance. Doing such inference makes using -transition=safe far more palatable. BTW, reducing the examples to their minimums helps a lot.
Oct 16 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: borrowing a range
References: <nu00a6$t5i$1 digitalmars.com> <nu01af$upk$1 digitalmars.com>
 <nu0khd$1rja$1 digitalmars.com>
In-Reply-To: <nu0khd$1rja$1 digitalmars.com>

--nrGwsjB1wN20noea94PIuQLJGotE1S75o
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 10:30 PM, Walter Bright wrote:
 On 10/16/2016 7:02 AM, Dicebot wrote:
     Owner instance;
     // must work, lifetime of 'instance' exceeds one of x :
     scope x =3D instance.borrow();
     // must not work, infinite lifetime
     auto y  =3D instance.borrow();
 }
 It works because 'scope' is inferred as a storage class when an
 expression that needs scope is used to initialize a local. y gets a
 subset of the life of instance. Doing such inference makes using
 -transition=3Dsafe far more palatable.
Great, thanks for explanation, this will need to be mentioned clearly with example in DIP document (I'll make a PR once other uncertainties are cleared too). However, it also keeps compiling if I change it to use explicit non-scope type: // move Range definition outside and mark `borrow` trusted Range y =3D instance.borrow(); // compiles, but must require `scope Range= y`
 BTW, reducing the examples to their minimums helps a lot.
This helps in debugging but doesn't help in defining good acceptance tests. I am looking for the latter and that means "minimal snippet that looks like useful code", not "minimal reduced test case" - and this is exactly what is needed as part of scope PR test cases to evaluate it. Now, I would be glad to provide minimal test cases for ease of debugging too, but that is not possible until I can reason which behavior is bug and which one is intentional. Getting your comments about why certain things don't work helps to understand. --nrGwsjB1wN20noea94PIuQLJGotE1S75o--
Oct 16 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2016 1:00 PM, Dicebot wrote:
 Range y = instance.borrow(); // compiles, but must require `scope Range y`
'scope' is inferred for 'y'.
 Now, I would be glad to provide minimal test cases for ease of debugging
 too, but that is not possible until I can reason which behavior is bug
 and which one is intentional. Getting your comments about why certain
 things don't work helps to understand.
When saying "this should compile" or "this should not compile" a minimized example is best when reasoning about it. Once we have understanding on what should and should not compile, a more meta example exploiting this would then be useful.
Oct 16 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: borrowing a range
References: <nu00a6$t5i$1 digitalmars.com> <nu01af$upk$1 digitalmars.com>
 <nu0khd$1rja$1 digitalmars.com>
In-Reply-To: <nu0khd$1rja$1 digitalmars.com>

--btH5bvnHD6apbRFnqrGaQlnvXnQBm6Slu
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 10:30 PM, Walter Bright wrote:
 It works because 'scope' is inferred as a storage class when an
 expression that needs scope is used to initialize a local. y gets a
 subset of the life of instance. Doing such inference makes using
 -transition=3Dsafe far more palatable.
=20
 BTW, reducing the examples to their minimums helps a lot.
This compiles too: ``` int* x; safe unittest { safe static struct S { int data; int* borrow () return scope trusted { return &this.data; } } S instance; x =3D instance.borrow(); } ``` Now, _this_ must be wrong, right? --btH5bvnHD6apbRFnqrGaQlnvXnQBm6Slu--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 9:44 AM, Dicebot wrote:
 This compiles too:

 ```
 int* x;

  safe unittest
 {
      safe static struct S
     {
         int data;

         int* borrow () return scope  trusted {
             return &this.data;
         }
     }

     S instance;
     x = instance.borrow();
 }
 ```

 Now, _this_ must be wrong, right?
It compiles because you added ' trusted' to the bad boy function, disabling the check preventing taking the address of a 'ref'. The compiler has no reason to believe that the return value of instance.borrow() has any relationship to 'instance' - it trusted the programmer to not do that! The practical result is that a container that is passed by ref can only control its uses by returning by ref, not by *.
Oct 20 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: borrowing a range
References: <nu00a6$t5i$1 digitalmars.com> <nu01af$upk$1 digitalmars.com>
 <nu0khd$1rja$1 digitalmars.com> <nuas8n$30f2$1 digitalmars.com>
 <nuc827$2dj2$1 digitalmars.com>
In-Reply-To: <nuc827$2dj2$1 digitalmars.com>

--JOpCCw1jLSMXRFu7hSrOOwshdfK4Gji8V
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/21/2016 08:11 AM, Walter Bright wrote:
 On 10/20/2016 9:44 AM, Dicebot wrote:
 This compiles too:

 ```
 int* x;

  safe unittest
 {
      safe static struct S
     {
         int data;

         int* borrow () return scope  trusted {
             return &this.data;
         }
     }

     S instance;
     x =3D instance.borrow();
 }
 ```

 Now, _this_ must be wrong, right?
=20 =20 It compiles because you added ' trusted' to the bad boy function, disabling the check preventing taking the address of a 'ref'. The compiler has no reason to believe that the return value of instance.borrow() has any relationship to 'instance' - it trusted the programmer to not do that!
Wait what? This is in direct contradiction with what DIP1000 states and what I have been trying to write example of all this time. By existing specification annotating method with `return scope` means that lifetime of return value is same as that method `this`. Are you saying that existing system forgets that `instance` is `this` the very moment `borrow` methods finishes? If yes, I am afraid it is a rather useless feature, much worse than one may think reading DIP1000.
 The practical result is that a container that is passed by ref can only=
 control its uses by returning by ref, not by *.
1) Why? You still haven't answered how scope pointer is different from ref in safe code. 2) Does that imply that lifetime algebra as described in DIP1000 only applies to function internal arguments and variables? As it is not possible to neither have `ref` variables, nor take address of `ref` return value, lifetime of return value will always be limited to single expression. --JOpCCw1jLSMXRFu7hSrOOwshdfK4Gji8V--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 10:46 PM, Dicebot wrote:
 Wait what?
I've said it multiple times. 'ref' cannot be converted to '*' and be safe. Hence the compiler disallows it. This is exactly why we need to start with trivial examples.
 This is in direct contradiction with what DIP1000 states and
 what I have been trying to write example of all this time.
I know what you've been trying to write, and each time I point out the same thing.
 By existing
 specification annotating method with `return scope` means that lifetime
 of return value is same as that method `this`.
If this is a class this then return scope is appropriate, because this is a pointer. But when this is a ref, scope then applies to the struct's fields.
 Are you saying that existing system forgets that `instance` is `this`
 the very moment `borrow` methods finishes? If yes, I am afraid it is a
 rather useless feature, much worse than one may think reading DIP1000.
By making borrow trusted in order to defeat the checking, you defeated the connection between 'instance' the argument and the return value from the function. Do not convert 'ref' to '*'. It won't work.
 The practical result is that a container that is passed by ref can only
 control its uses by returning by ref, not by *.
1) Why? You still haven't answered how scope pointer is different from ref in safe code.
I've been trying to answer. There are two things at work for variable 'v': 1. v's address 2. v's value 'ref' deals with the first, 'scope' with the second. It is very, very, very important to think about whether you are dealing with v's address or v's value, otherwise you will get hopelessly lost. Next, recall that 'this' for a struct is an address. 'this' for a class is a value.
 2) Does that imply that lifetime algebra as described in DIP1000 only
 applies to function internal arguments and variables? As it is not
 possible to neither have `ref` variables, nor take address of `ref`
 return value, lifetime of return value will always be limited to single
 expression.
That's what scope variables are for - storing addresses in. I implore you to look at the test cases with the PR. They are all trivial and easy to follow.
Oct 22 2016
parent Martin Nowak <code dawg.eu> writes:
On Saturday, 22 October 2016 at 07:38:18 UTC, Walter Bright wrote:
 If this is a class this then return scope is appropriate, 
 because this is a pointer. But when this is a ref, scope then 
 applies to the struct's fields.
Mmh, and if wanted to return a reference to the struct itself, we'd need to resort to return ref? Would it be possible to conflate return scope and return ref, at least for that use case?
 Are you saying that existing system forgets that `instance` is 
 `this`
 the very moment `borrow` methods finishes? If yes, I am afraid 
 it is a
 rather useless feature, much worse than one may think reading 
 DIP1000.
By making borrow trusted in order to defeat the checking, you defeated the connection between 'instance' the argument and the return value from the function. Do not convert 'ref' to '*'. It won't work.
As said further down in this thread, it seems that returning casted refs (or structs containing pointers) that alias internal memory, but are bound/scoped to the lifetime of the comtainer, is a major use-case. Why can't we use this explicit scope annotation on a trusted method to make this work?
 The practical result is that a container that is passed by 
 ref can only
 control its uses by returning by ref, not by *.
1) Why? You still haven't answered how scope pointer is different from ref in safe code.
I've been trying to answer. There are two things at work for variable 'v': 1. v's address 2. v's value 'ref' deals with the first, 'scope' with the second. It is very, very, very important to think about whether you are dealing with v's address or v's value, otherwise you will get hopelessly lost.
Could you elaborate a bit on that?
 Next, recall that 'this' for a struct is an address. 'this' for 
 a class is a value.
The other way round, right?
 That's what scope variables are for - storing addresses in.

 I implore you to look at the test cases with the PR. They are 
 all trivial and easy to follow.
I hope you agree that we must check the design against our actual high level goals. While the breaking down the rules is a good approach for a formal view, being able to puzzle together the pieces to useful idioms is essential.
Oct 26 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/16/2016 7:02 AM, Dicebot wrote:
 	// [1] Error: function has 'return' but does not return any
  	// indirections
         auto borrow () return scope
         {
             struct Range
             {
                 int* data;
             }

             return Range(&this.data); // [2] Error: cannot take address of
                                       // parameter this in  safe
 function borrow
         }

 First error [1] simply to be simply a bug, because it contradicts "struct is
 a composition of its fields" statement from DIP1000.
I'd ignore that error because it is a cascaded error. (It is helpful to label errors so it's easier to see which you're referring to. I added such annotations to the quoted part.) As for error [2], 'ref' is meant to be a restricted pointer. Converting a restricted pointer to a regular pointer, which is what &this does, undermines the whole point of having 'ref', and so is disallowed in safe code. Consider: safe int* foo(return ref int r) { return &r; // Error: cannot take address of parameter r in safe function foo } Making it trusted works. But even if trusted, escape checking is still done, even in system code: trusted int* foo(return ref int r) { return &r; // no error, 'cuz it's trusted } int* bar() { int r; return foo(r); // Error: escaping reference to local variable r }
Oct 16 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing: borrowing a range
References: <nu00a6$t5i$1 digitalmars.com> <nu01af$upk$1 digitalmars.com>
 <nu0lop$1thf$1 digitalmars.com>
In-Reply-To: <nu0lop$1thf$1 digitalmars.com>

--ET4EUDLrjMcIvwXHrpAKeXu9CQsi48MI2
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/16/2016 10:51 PM, Walter Bright wrote:
 As for error [2], 'ref' is meant to be a restricted pointer. Converting=
 a restricted pointer to a regular pointer, which is what &this does,
 undermines the whole point of having 'ref', and so is disallowed in
  safe code.
=20
 Consider:
=20
    safe int* foo(return ref int r) {
     return &r; // Error: cannot take address of parameter r in  safe
 function foo
   }
This must not compile indeed. But this is perfectly fine: safe scope int* foo (return scope ref int r) { return &r; // taking address of r is fine because returned pointer can't outlive the reference } As far as I can see right now, from the safety PoV scoped pointer is strictly equivalent to ref. --ET4EUDLrjMcIvwXHrpAKeXu9CQsi48MI2--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 9:37 AM, Dicebot wrote:
 But this is perfectly fine:

  safe scope int* foo (return scope ref int r) {
     return &r; // taking address of r is fine because returned pointer
 can't outlive the reference
 }
Nope: Error: function foo functions cannot be scope If foo was a member function, the 'scope' would be allowed and would apply to the 'this' reference.
Oct 20 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 06:09:36 UTC, Walter Bright wrote:
  safe scope int* foo (return scope ref int r) {
     return &r; // taking address of r is fine because returned 
 pointer
 can't outlive the reference
 }
Nope: Error: function foo functions cannot be scope
Scope was supposed to apply to the return value, comare https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#scope-function-returns. Let's maybe work with a better equivalent example. safe scope Klass get(return scope ref Unique!Klass);
Oct 21 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 10:38 AM, Martin Nowak wrote:
 Scope was supposed to apply to the return value, comare
 https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#scope-function-returns.


 Let's maybe work with a better equivalent example.

  safe scope Klass get(return scope ref Unique!Klass);
Please use simple examples. 'scope' has nothing to do with templates, and adding in templates just distracts and confuses.
Oct 22 2016
prev sibling next sibling parent Kagamin <spam here.lot> writes:
What about mandatory  system attribute? 
https://github.com/dlang/dmd/pull/5972#issuecomment-242227591
Oct 19 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 16 October 2016 at 13:45:42 UTC, Dicebot wrote:
 So, there is a pull request 
 (https://github.com/dlang/dmd/pull/5972)
 which is intended to implement DIP1000
 (https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md).
Can we first get a high level overview about existing discussions and the goals/problems of DIP1000. Not sure this forum is the right place for detailed DIP/PR reviews. The DIP is named Scoped Pointers, but seems to address any escaping references (including classes). Will it fully enable safe reference counted memory management? What pieces of the puzzle are still missing? I read that RC `opAssign` and `destroy` would remain unsafe b/c destroying the owner might leave a dangling ref parameter in `fun(ref RCS rc, ref int ri)`. It seems that unsafe assignment/destruction of RCs would be very limiting. Do we at least have an idea how to tackle this problem? In the RCClass idea an additional addRef/decRef call was proposed. Also see https://github.com/dlang/DIPs/pull/35#issuecomment-252345548.
Oct 19 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/19/2016 10:48 AM, Martin Nowak wrote:
 Can we first get a high level overview about existing discussions and the 
goals/problems of DIP1000. Simply put, it provides the means to guarantee that a pointer supplied as a function argument does not escape the scope of that function. Special consideration is given to functions that return their arguments.
 The DIP is named Scoped Pointers, but seems to address any escaping references
 (including classes).
It includes classes, delegates, and dynamic arrays.
 Will it fully enable  safe reference counted memory management?
No, because an RC design also requires the preincrement as noted below.
 What pieces of the puzzle are still missing?
The preincrement thing, but that is not DIP1000.

All.
 I read that RC `opAssign` and `destroy` would remain unsafe b/c destroying the
 owner might leave a dangling ref parameter in `fun(ref RCS rc, ref int ri)`.
 It seems that unsafe assignment/destruction of RCs would be very limiting.
 Do we at least have an idea how to tackle this problem? In the RCClass idea an
 additional addRef/decRef call was proposed.
 Also see https://github.com/dlang/DIPs/pull/35#issuecomment-252345548.
The solution to that has been proposed and forgotten a couple of times. It is to have the compiler insert code to preemptively increment the reference count, then reassigning the RC object will not invalidate references to its internals. This is beyond the scope of DIP1000, though. DIP1000 is necessary for memory safety even without reference counting.
Oct 20 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Thursday, 20 October 2016 at 09:00:02 UTC, Walter Bright wrote:
 I read that RC `opAssign` and `destroy` would remain unsafe 
 b/c destroying the
 owner might leave a dangling ref parameter in `fun(ref RCS rc, 
 ref int ri)`.
 It seems that unsafe assignment/destruction of RCs would be 
 very limiting.
 Do we at least have an idea how to tackle this problem? In the 
 RCClass idea an
 additional addRef/decRef call was proposed.
 Also see 
 https://github.com/dlang/DIPs/pull/35#issuecomment-252345548.
The solution to that has been proposed and forgotten a couple of times. It is to have the compiler insert code to preemptively increment the reference count, then reassigning the RC object will not invalidate references to its internals. This is beyond the scope of DIP1000, though. DIP1000 is necessary for memory safety even without reference counting.
Thought a bit more about this. The essence of the problem is a simple pointer aliasing problem, namely calling void dangling(ref A a, ref B b) is unsafe if A is a "container" that could own a B. This does not only apply to RC but also things like arrays with deterministic MM, unions/variants (changing type of b's pointee), or implementations of Nullable/Optional using heap allocated memory. Fortunately any pointer aliasing through untyped memory/pointers in A (ubyte[], void*) can only result in a dangling b pointer, if doing unsafe operations on a (or incorrectly marked trusted ones). For pointer aliasing through typed fields of A, the compiler can detect a possible aliasing and mark the function call as unsafe. That would allow operations on a that could free Bs to be trustable. This aliasing problem seems only slightly related to scope/escape analysis. A container using scope in a safe manner, should only escape scoped references. So the statement of the problem is actually void dangling(ref A a, scope ref B b). Copying a before the call couldn't solve all those problems, e.g. Unique isn't copyable, a union/variant could be in a reference type field of A. But it seems that we can detect all such unsafe function calls and the problem only scratches on scope b/c we want to know whether the lifetime of b might be limited attached to a. So seems indeed fair enough to leave this aside for now. What would be the plan for RC.opAssign? Making it trusted after DIP1000 and implementing the aliasing detection later, trusted opAssign but conservatively unsafe any call with multiple references (at least one of which being scoped), or system opAssign and changing it to trusted later? It seems that the aliasing detect wouldn't be too hard to implememt (even implementable as druntime template similar to std.traits.hasAliasing). Am I right that conservatively detecting the aliasing would allow a slightly limited subset of safe RC usage? If so going w/ trusted RC.opAssign and the aliasing detection seems like a good milestone for 2.073.
Oct 21 2016
next sibling parent Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 10:11:28 UTC, Martin Nowak wrote:
 It seems that the aliasing detect wouldn't be too hard to 
 implememt (even implementable as druntime template similar to 
 std.traits.hasAliasing). Am I right that conservatively 
 detecting the aliasing would allow a slightly limited subset of 
  safe RC usage?
 If so going w/  trusted RC.opAssign and the aliasing detection 
 seems like a good milestone for 2.073.
Nice side effect of http://forum.dlang.org/post/fzawzedvokbvhmdspxsh forum.dlang.org, it's now clear that a good (not too conservative) aliasing detection is also needed for the lifetime of function return values. So we can reuse that for detecting the dangling ref problem.
Oct 21 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 3:11 AM, Martin Nowak wrote:
 What would be the plan for RC.opAssign?
The plan is rather blunt. Andrei is going to design an RC object, and then I am going to figure out how to make it work (because it will need a bit of compiler help, specifically, dealing with the pre-increment case). You are correct in that RC is only tangentially related to scope.
Oct 22 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 16 October 2016 at 13:45:42 UTC, Dicebot wrote:
 Problem is that Walter wants to prioritize moving on with it 
 (which is reasonable) but the PR is too complicated to be 
 reviewed by most other developers.
Just through with my first read of DIP1000. First impressions interesting, incoherent, might or might not solve our problems. Will give it some more time in the next few days. But IMHO we should ensure that we have sound specs (backed by tests) that solve the __right__ use-cases before discussing implementations. As part of the implementation we'll have to transform this into a more easily understandable spec anyhow, so we might as well do that in the beginning.
Oct 19 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/19/2016 3:12 PM, Martin Nowak wrote:
 On Sunday, 16 October 2016 at 13:45:42 UTC, Dicebot wrote:
 Problem is that Walter wants to prioritize moving on with it (which is
 reasonable) but the PR is too complicated to be reviewed by most other
 developers.
Just through with my first read of DIP1000. First impressions interesting, incoherent, might or might not solve our problems. Will give it some more time in the next few days. But IMHO we should ensure that we have sound specs (backed by tests) that solve the __right__ use-cases before discussing implementations. As part of the implementation we'll have to transform this into a more easily understandable spec anyhow, so we might as well do that in the beginning.
I suggest looking over the test cases that come with the PR. They are straightforward. Whether they are the right use cases or not is debatable, but they each illustrate an existing safety hole in D today that the PR plugs.
Oct 20 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
In-Reply-To: <nua19j$1fvb$1 digitalmars.com>

--40X9ck5NCmx7OFRF0Wkj3q1McsEq3Ddam
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/20/2016 12:03 PM, Walter Bright wrote:
 I suggest looking over the test cases that come with the PR. They are
 straightforward. Whether they are the right use cases or not is
 debatable, but they each illustrate an existing safety hole in D today
 that the PR plugs.
If I was to judge by available test cases only, I would have decided DIP1000 is waste of time and brings nothing useful to the table. They are terrible in showing implementation goals being low level regression testing artifacts. Really, do you think anyone will ever be impressed seeing code like `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can allow much more interesting things and tests should show that too, not just synthetic artifacts. --40X9ck5NCmx7OFRF0Wkj3q1McsEq3Ddam--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 12:27 PM, Dicebot wrote:
 If I was to judge by available test cases only, I would have decided
 DIP1000 is waste of time and brings nothing useful to the table. They
 are terrible in showing implementation goals being low level regression
 testing artifacts.

 Really, do you think anyone will ever be impressed seeing code like
 `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can
 allow much more interesting things and tests should show that too, not
 just synthetic artifacts.
They are not meant to impress people, nor are they regression tests. They illustrate closing of existing safety holes. int* bar() { int p; return foo1(&p); } is a currently undetected bug. DIP1000 detects it. DIP1000 also closes safety bugs that you posted to Bugzilla (listed in the PR for it). D must have these fixes, one way or another. Yes, the test cases are very minimal. But understanding them is necessary in order to understand more complex cases - I don't think we've gotten anywhere by using complex cases to understand the mechanics of it.
Oct 20 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
In-Reply-To: <nubddb$ulr$1 digitalmars.com>

--DcGiQ6EdFx2bc2mf2kVpEjlvmI86q51u7
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/21/2016 12:36 AM, Walter Bright wrote:
 On 10/20/2016 12:27 PM, Dicebot wrote:
 If I was to judge by available test cases only, I would have decided
 DIP1000 is waste of time and brings nothing useful to the table. They
 are terrible in showing implementation goals being low level regressio=
n
 testing artifacts.

 Really, do you think anyone will ever be impressed seeing code like
 `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can=
 allow much more interesting things and tests should show that too, not=
 just synthetic artifacts.
=20 They are not meant to impress people, nor are they regression tests. They illustrate closing of existing safety holes. =20 int* bar() { int p; return foo1(&p); } =20 is a currently undetected bug. DIP1000 detects it. DIP1000 also closes safety bugs that you posted to Bugzilla (listed in the PR for it).
Yes, but that is not enough for such a major change. If we want to finally implement `scope`, it has to be _awesome_, plain good is not good enough.
 Yes, the test cases are very minimal. But understanding them is
 necessary in order to understand more complex cases - I don't think
 we've gotten anywhere by using complex cases to understand the mechanic=
s
 of it.
Can you give a clear answer regarding borrowing snippets I am posting? Like one of: - Such idioms are not supposed to work with DIP1000 at all (why?) and you only want to focus on fixing existing safe holes - It is supposed to work, but I am doing it wrong (please show how to write it in that case) - It is supposed to work, but bugs need fixing - It is supposed to work, but some parts will be implemented later (which ones?) So far you have only given answers regarding technical trivialities without taking actual example to the heart. --DcGiQ6EdFx2bc2mf2kVpEjlvmI86q51u7--
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 3:52 PM, Dicebot wrote:
 On 10/21/2016 12:36 AM, Walter Bright wrote:
 On 10/20/2016 12:27 PM, Dicebot wrote:
 If I was to judge by available test cases only, I would have decided
 DIP1000 is waste of time and brings nothing useful to the table. They
 are terrible in showing implementation goals being low level regression
 testing artifacts.

 Really, do you think anyone will ever be impressed seeing code like
 `int* foo1(return scope int* p) { return p; }`? DIP1000 as defined can
 allow much more interesting things and tests should show that too, not
 just synthetic artifacts.
They are not meant to impress people, nor are they regression tests. They illustrate closing of existing safety holes. int* bar() { int p; return foo1(&p); } is a currently undetected bug. DIP1000 detects it. DIP1000 also closes safety bugs that you posted to Bugzilla (listed in the PR for it).
Yes, but that is not enough for such a major change. If we want to finally implement `scope`, it has to be _awesome_, plain good is not good enough.
Actually, I consider it a minor change, and I *want* the fix for this bug to be as minor as possible. Furthermore, Andrei, I, Bartosz, and others have struggled with this particular problem for about 10 years. This is the best (and simplest, by far) solution we've come up with. If you can think of a better one, I'm interested.
 Yes, the test cases are very minimal. But understanding them is
 necessary in order to understand more complex cases - I don't think
 we've gotten anywhere by using complex cases to understand the mechanics
 of it.
Can you give a clear answer regarding borrowing snippets I am posting? Like one of: - Such idioms are not supposed to work with DIP1000 at all (why?) and you only want to focus on fixing existing safe holes - It is supposed to work, but I am doing it wrong (please show how to write it in that case) - It is supposed to work, but bugs need fixing - It is supposed to work, but some parts will be implemented later (which ones?) So far you have only given answers regarding technical trivialities
That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.
 without taking actual example to the heart.
1. D has no notion of "borrowing" and describing DIP1000 in those terms is going to mislead. There is no "borrow checker". 2. This design has nothing in common with Rust's design. Any notions brought from Rust will mislead. 3. The design utterly relies on a 'ref' value not being returned as a '*'. Any attempts to circumvent this will lose the checking. The rationale for this is a dramatic simplification of the design. 4. The effects of 'ref' and 'scope' are not transitive. 5. 'scope' is ignored if applied to a type with no indirections (this is necessary to support generic code). 6. What does 'return ref scope' mean? If the function returns by ref, then the 'return' applies to the 'ref'. Otherwise, it applies to the 'scope'. 7. 'scope' applies to the value of a type, 'ref' applies to its address. Your example failed to apply 'scope' to the 'int*' return, because the struct had no pointer types in it. 8. A container passed by 'ref' can return a 'ref' to the container's value or indirect contents, or by '*' to indirect contents, but it cannot return the address of the container by '*'. (This is a restatement of item 3.)
Oct 20 2016
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 06:04:41 UTC, Walter Bright wrote:
 3. The design utterly relies on a 'ref' value not being 
 returned as a '*'. Any attempts to circumvent this will lose 
 the checking. The rationale for this is a dramatic 
 simplification of the design.
DIP1000 mentions that disallowing &var in safe code is overly restrictive. This got me fairly confused, please help to clarify https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md escaping-via-return or my understanding of it.
Oct 21 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 3:21 AM, Martin Nowak wrote:
 On Friday, 21 October 2016 at 06:04:41 UTC, Walter Bright wrote:
 3. The design utterly relies on a 'ref' value not being returned as a '*'. Any
 attempts to circumvent this will lose the checking. The rationale for this is
 a dramatic simplification of the design.
DIP1000 mentions that disallowing &var in safe code is overly restrictive. This got me fairly confused, please help to clarify https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#escaping-via-return or my understanding of it.
In order to simplify the text, I used &t as an example, even though safe code doesn't allow taking the address of a local. But, this is allowed: int[10] sa; int[] da = sa[]; and is the semantic equivalent of taking the address of a local. But I didn't want to confuse things by introducing arrays, I wanted to be reductionist as much as possible. But the examples in the PR tests use the arrays.
Oct 22 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 06:04:41 UTC, Walter Bright wrote:
 So far you have only given answers regarding technical 
 trivialities
That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.
I think we all agree on which overall goal we're trying to achieve, safe deterministic memory management. So the primitives of DIP1000 need to prove that they properly cover the reference escaping part of that goal. 1. Default initializers have infinite lifetime. lifetime(e.init) = ∞ Non default initializers are treated as default init followed by an assignment. 2. The sticky lifetime axiom. Assignment `v = e` changes lifetime(v) to lifetime(e) if v is a pointer or reference. 3. The safe reference axiom. Can't assign an expression e to a variable v if that would change lifetime(v) so that lifetime(v) < reachability(v). 4. The conservative lifetime axiom. lifetime(func(args)) = min(∞, argmin_{e ∈ A}(lifetime(e))) with A = {a ∈ args | canAlias(returnType(func), typeof(a))} The lifetime of a function call expression, is the minimum of infinity and the lifetime of all arguments that could be referenced by the return type. (Shouldn't be type here b/c scope is a storage class and not part of the type) 5. The conservative escape axiom. The reachability of non-scope ref or pointer paramters is assumed to be ∞. Hence it's not allowed to assign expressions with finite lifetime to non-scope parameters, though the compiler might infer scope for parameters. If this works out we guarantee lifetime(v) ≥ reachability(v) in safe code. The rest should be deducible using the rules from https://github.com/dlang/DIPs/blob/731255ed7dc37d596fec0552013f0e12c68f7bea/DIPs/DIP1000.md#aggregates. The problem that remains could be stated as lifetime tunneling. Given `v2 = func(v1)` lifetime(v2) = lifetime(v1) according to 5. Changing lifetime(v1) by `v1 = e` can change lifetime(v2) to < reachability(v2).
Oct 21 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Friday, 21 October 2016 at 11:39:27 UTC, Martin Nowak wrote:
 The problem that remains could be stated as lifetime tunneling.

 Given `v2 = func(v1)` lifetime(v2) = lifetime(v1) according to 
 5.
 Changing lifetime(v1) by `v1 = e` can change lifetime(v2) to < 
 reachability(v2).
This doesn't manifest easily because there are no named reference variables and converting references to pointers isn't allowed in safe code. But the void dangling(ref A a, scope ref B b) is one way to run into this. Can we at least enforce that any value/reference returned by `func(v1)` cannot be bound to a v2 in the calling scope while still referencing v1?
Oct 21 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 4:54 AM, Martin Nowak wrote:
 On Friday, 21 October 2016 at 11:39:27 UTC, Martin Nowak wrote:
 The problem that remains could be stated as lifetime tunneling.

 Given `v2 = func(v1)` lifetime(v2) = lifetime(v1) according to 5.
 Changing lifetime(v1) by `v1 = e` can change lifetime(v2) to <
reachability(v2).
This doesn't manifest easily because there are no named reference variables and converting references to pointers isn't allowed in safe code. But the void dangling(ref A a, scope ref B b) is one way to run into this. Can we at least enforce that any value/reference returned by `func(v1)` cannot be bound to a v2 in the calling scope while still referencing v1?
I don't really understand the question. Can you recast it in terms of a short example?
Oct 22 2016
prev sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
In-Reply-To: <nucb5n$2hv7$1 digitalmars.com>

--Hjurrn9jjE25PFBstrpIBQMciRmKQfCOD
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/21/2016 09:04 AM, Walter Bright wrote:
 On 10/20/2016 3:52 PM, Dicebot wrote:
 So far you have only given answers regarding technical trivialities
=20 That is true, but we must reach mutual understanding on how those trivialities work before we can do more. So far, we have not.
Well, it doesn't work that way. So far all you explanations made zero sense until I have only finally understood what your intentions are thanks to next answer ..
 1. D has no notion of "borrowing" and describing DIP1000 in those terms=
 is going to mislead. There is no "borrow checker".
=20
 2. This design has nothing in common with Rust's design. Any notions
 brought from Rust will mislead.
=20
 3. The design utterly relies on a 'ref' value not being returned as a
 '*'. Any attempts to circumvent this will lose the checking. The
 rationale for this is a dramatic simplification of the design.
=20
 4. The effects of 'ref' and 'scope' are not transitive.
=20
 5. 'scope' is ignored if applied to a type with no indirections (this i=
s
 necessary to support generic code).
=20
 6. What does 'return ref scope' mean? If the function returns by ref,
 then the 'return' applies to the 'ref'. Otherwise, it applies to the
 'scope'.
=20
 7. 'scope' applies to the value of a type, 'ref' applies to its address=
=2E
 Your example failed to apply 'scope' to the 'int*' return, because the
 struct had no pointer types in it.
=20
 8. A container passed by 'ref' can return a 'ref' to the container's
 value or indirect contents, or by '*' to indirect contents, but it
 cannot return the address of the container by '*'. (This is a
 restatement of item 3.)
=2E. and means two things for me: 1) DIP1000 has to be changes a lot because it doesn't list most of those limitation, leading to natural conclusion that borrowing can be emulated as a user code concept. Most importantly, the statement "The design utterly relies on a 'ref' value not being returned as a '*'. Any attempts to circumvent this will lose the checking" is not mentioned at a= ll. 2) The whole thing is very disappointing and won't revive dead body of safe as I hoped it will. It won't allow to fix `Unique`, won't enable new RNG design Joe has been going, won't allow easy skipping of reference count inc/dec - the list goes on. Leaving aside issue 2, I'd like to give a try at updating DIP1000 this weekend to be more in match with what you have just explained. --Hjurrn9jjE25PFBstrpIBQMciRmKQfCOD--
Oct 21 2016
next sibling parent reply meppl <mephisto nordhoff-online.de> writes:
On Friday, 21 October 2016 at 15:38:39 UTC, Dicebot wrote:
 2) The whole thing is very disappointing ...
 ... won't allow easy skipping of reference count inc/dec - ...
Isn't that - in the short term - impossible anyway? If a struct leaves the scope the reference counter has to be increased. But how does compiled code of a library know, if the foreign struct is an RC-object - without doing any runtime checks? So, any function has to exist twice, one time for RC and one time for no-RC? I mean, it's probably a big topic that doesn't have to be solved in that DIP, too.
Oct 21 2016
parent Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <jmlynifzlzgzghwmtogn forum.dlang.org>
In-Reply-To: <jmlynifzlzgzghwmtogn forum.dlang.org>

--oCpUjQ2k5163goje2KJIkGxpJ1m935FID
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/21/2016 09:43 PM, meppl wrote:
 On Friday, 21 October 2016 at 15:38:39 UTC, Dicebot wrote:
 2) The whole thing is very disappointing ...
 ... won't allow easy skipping of reference count inc/dec - ...
=20 Isn't that - in the short term - impossible anyway? If a struct leaves the scope the reference counter has to be increased. But how does compiled code of a library know, if the foreign struct is an RC-object =
-
 without doing any runtime checks? So, any function has to exist twice,
 one time for RC and one time for no-RC?
 I mean, it's probably a big topic that doesn't have to be solved in tha=
t
 DIP, too.
It would be possible with even most simple borrowing semantics, see http://forum.dlang.org/post/nu00tf$u0a$1 digitalmars.com - if one can bind lifetime of a scope pointer to lifetime of stack variable, it becomes OK to implicitly convert reference counted struct to scope pointer for passing to other functions. So you only define function taking scope pointer if it doesn't store it (and it works with both RC and other data sourceS) or taking RC pointer explicitly if does indeed store it. But with the system Walter is proposing it is indeed impossible. --oCpUjQ2k5163goje2KJIkGxpJ1m935FID--
Oct 22 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 8:38 AM, Dicebot wrote:
 1) DIP1000 has to be changes a lot because it doesn't list most of those
 limitation, leading to natural conclusion that borrowing can be emulated
 as a user code concept. Most importantly, the statement "The design
 utterly relies on a 'ref' value not being returned as a '*'. Any
 attempts to circumvent this will lose the checking" is not mentioned at all.
I believe they are implicit (taking the & of a 'ref' variable has never been allowed in safe code), but they are worth reiterating. It also is apparent now that it should explicitly say it has nothing in common with Rust's scheme.
 2) The whole thing is very disappointing and won't revive dead body of
  safe as I hoped it will.
What it will do is close the numerous safe holes that have been identified.
 It won't allow to fix `Unique`, won't enable
 new RNG design Joe has been going, won't allow easy skipping of
 reference count inc/dec - the list goes on.
You're right, scope won't help at all in inc/dec skipping. But all is not lost. I have written up a short article on how the compiler can do inc/dec skipping, but it'll have to wait until we have an actual RC design. The idea is to have a memory model that always inserts inc/dec, then allow the compiler to remove them as long as it can prove it behaves "as if" the inc/dec happened.
 Leaving aside issue 2, I'd like to give a try at updating DIP1000 this
 weekend to be more in match with what you have just explained.
Thank you! Your help is invaluable.
Oct 22 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
In-Reply-To: <nuf5qv$18ca$1 digitalmars.com>

--b81vMXWWtGD5UxNjJeQtQQmPtuS1XQCDM
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/22/2016 10:52 AM, Walter Bright wrote:
 It won't allow to fix `Unique`, won't enable
 new RNG design Joe has been going, won't allow easy skipping of
 reference count inc/dec - the list goes on.
=20 You're right, scope won't help at all in inc/dec skipping. But all is not lost. I have written up a short article on how the compiler can do inc/dec skipping, but it'll have to wait until we have an actual RC des=
ign. Beauty of borrowing is that it allows to to inc/dec skipping with no special RC support from compiler and works the very same for stack allocated data. What you are going at is simply replacing GC with (compiler enhanced) RC and that is not good at all. Would you at least be willing to consider an alternative proposal before `scope` branch is merged for release? I have a feeling that with relatively few changes to DIP1000 rules we can both have the cookie and eat it (i.e. get a way to express non-transitive borrowing), more in spirit of original proposal by Marc. --b81vMXWWtGD5UxNjJeQtQQmPtuS1XQCDM--
Oct 22 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/22/2016 3:50 PM, Dicebot wrote:
 Beauty of borrowing is that it allows to to inc/dec skipping with no
 special RC support from compiler and works the very same for stack
 allocated data. What you are going at is simply replacing GC with
 (compiler enhanced) RC and that is not good at all.
I don't see why supporting something beyond RC and GC is necessary.
 Would you at least be willing to consider an alternative proposal before
 `scope` branch is merged for release? I have a feeling that with
 relatively few changes to DIP1000 rules we can both have the cookie and
 eat it (i.e. get a way to express non-transitive borrowing), more in
 spirit of original proposal by Marc.
It'd have to be a big improvement over DIP1000, without sacrificing easy use. I find Rust's system to be seriously restrictive, off-putting, and hear many reports of users fighting its borrow checker. (I have not written any Rust code myself.) At some point we need to move forward. We are already years late. Another month has slipped by. I am also not convinced that you cannot do what you want to do with DIP1000. I do not understand why it is necessary to return the address of a value passed by ref, when one can return by ref.
Oct 23 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
 <nugqg3$ld8$1 digitalmars.com> <nujrme$1pu9$1 digitalmars.com>
In-Reply-To: <nujrme$1pu9$1 digitalmars.com>

--25nCmsKSB6u2n7qhxX7mjXjbrSg0BMd6d
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/24/2016 05:29 AM, Walter Bright wrote:
 On 10/22/2016 3:50 PM, Dicebot wrote:
 Beauty of borrowing is that it allows to to inc/dec skipping with no
 special RC support from compiler and works the very same for stack
 allocated data. What you are going at is simply replacing GC with
 (compiler enhanced) RC and that is not good at all.
=20 I don't see why supporting something beyond RC and GC is necessary.
Both GC and RC are forms of automatic memory management with unavoidable associated overhead. Both have their uses but are unacceptable in most of the system code. Which means that would would either need to duplicate all Phobos/druntime functions to work with both RC and plain pointers or it will become even less suitable for low level code than it is now.
 Would you at least be willing to consider an alternative proposal befo=
re
 `scope` branch is merged for release? I have a feeling that with
 relatively few changes to DIP1000 rules we can both have the cookie an=
d
 eat it (i.e. get a way to express non-transitive borrowing), more in
 spirit of original proposal by Marc.
=20 It'd have to be a big improvement over DIP1000, without sacrificing eas=
y
 use. I find Rust's system to be seriously restrictive, off-putting, and=
 hear many reports of users fighting its borrow checker. (I have not
 written any Rust code myself.)
The problem is DIP1000 is a lie. I started to rewrite it to match your implemented semantics but that would simply result in throwing much of its content away. Some of crucial things DIP1000 promises to deliver but are not possible with design as in https://github.com/dlang/dmd/pull/5972= : - system code like std.internal.scopebuffer can be made safe - reference counting systems need not adjust the count when passing references that do not escape That means that any counter-proposal that actually allows to do those things would already be considerably better. Now, I do agree that we don't want to go with amount of complexity Rust has. Something that makes basic idioms possible at 10% of complexity price would be a much better trade-off. Thus I'd propose to keep most of the stuff in DIP1000 as it is (including non-transitivity), but enhance/tweak lifetime rules, for example: - allow taking address of a `ref` and define lifetime of such expression to be same of `ref` lifetime - `return scope` of a struct methods binds returns value lifetime to one of the struct instance itself, not composition of its fields It is of similar complexity as existing proposal but is good enough to allow design patterns I have been trying to do.
 At some point we need to move forward. We are already years late.
 Another month has slipped by.
Current implementation does not bring us closer to feature rich compiler-verified safety in any way. What is even worse, going with it will close the option of using same syntax for something better and we will be stuck with it forever. I have a feeling that you completely misunderstand why Rust system has all the hype. Would it help if I explained some of compiler-verified safety patterns there? I have put some time into studying Rust last year.=
 I am also not convinced that you cannot do what you want to do with
 DIP1000. I do not understand why it is necessary to return the address
 of a value passed by ref, when one can return by ref.
You are author of the proposal. It is your obligation to convince everyone that it is possible to do what you claim possible, not mine. If you think it can be done, show me the code. --25nCmsKSB6u2n7qhxX7mjXjbrSg0BMd6d--
Oct 23 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/23/2016 11:49 PM, Dicebot wrote:
 On 10/24/2016 05:29 AM, Walter Bright wrote:
 On 10/22/2016 3:50 PM, Dicebot wrote:
 Beauty of borrowing is that it allows to to inc/dec skipping with no
 special RC support from compiler and works the very same for stack
 allocated data. What you are going at is simply replacing GC with
 (compiler enhanced) RC and that is not good at all.
I don't see why supporting something beyond RC and GC is necessary.
Both GC and RC are forms of automatic memory management with unavoidable associated overhead. Both have their uses but are unacceptable in most of the system code. Which means that would would either need to duplicate all Phobos/druntime functions to work with both RC and plain pointers or it will become even less suitable for low level code than it is now.
I'm afraid that is a bit vague. Can you give a more concrete example?
 The problem is DIP1000 is a lie. I started to rewrite it to match your
 implemented semantics but that would simply result in throwing much of
 its content away. Some of crucial things DIP1000 promises to deliver but
 are not possible with design as in https://github.com/dlang/dmd/pull/5972 :

 -  system code like std.internal.scopebuffer can be made  safe
 - reference counting systems need not adjust the count when passing
 references that do not escape

 That means that any counter-proposal that actually allows to do those
 things would already be considerably better.

 Now, I do agree that we don't want to go with amount of complexity Rust
 has. Something that makes basic idioms possible at 10% of complexity
 price would be a much better trade-off. Thus I'd propose to keep most of
 the stuff in DIP1000 as it is (including non-transitivity), but
 enhance/tweak lifetime rules, for example:

 - allow taking address of a `ref` and define lifetime of such expression
 to be same of `ref` lifetime
Again, why? Why can't the function return by 'ref'?
 - `return scope` of a struct methods binds returns value lifetime to one
 of the struct instance itself, not composition of its fields
'ref' return already does that.
 It is of similar complexity as existing proposal but is good enough to
 allow design patterns I have been trying to do.
Your design patterns all hinge on requiring conversion of a ref to a *. Why is this necessary, as opposed to using return by 'ref' ?
 At some point we need to move forward. We are already years late.
 Another month has slipped by.
Current implementation does not bring us closer to feature rich compiler-verified safety in any way.
The numerous safety bugs it corrects, some reported by you, argues otherwise.
 What is even worse, going with it
 will close the option of using same syntax for something better and we
 will be stuck with it forever.

 I have a feeling that you completely misunderstand why Rust system has
 all the hype. Would it help if I explained some of compiler-verified
 safety patterns there? I have put some time into studying Rust last year.
Your understanding of Rust is likely better than mine. But I do know (and Rust documentation says this) that one has to rethink design patterns in order to work with the borrow checker, it cannot just be slapped onto any existing pattern, even if that pattern is correct. Trying to implement a Rust pattern in D is likely to be incorrect as well.
 I am also not convinced that you cannot do what you want to do with
 DIP1000. I do not understand why it is necessary to return the address
 of a value passed by ref, when one can return by ref.
You are author of the proposal. It is your obligation to convince everyone that it is possible to do what you claim possible, not mine. If you think it can be done, show me the code.
I don't know what you have in mind. I already showed you how it can help with unique pointers. The only concrete issue you have with it is that ref cannot be returned by *. I ask again why is this important as opposed to returning by ref.
Oct 24 2016
next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
 <nugqg3$ld8$1 digitalmars.com> <nujrme$1pu9$1 digitalmars.com>
 <nukaue$2d5l$1 digitalmars.com> <nulojm$1li3$1 digitalmars.com>
In-Reply-To: <nulojm$1li3$1 digitalmars.com>

--sbBsSsMcu5C65WDITokEqjPBmejA1VGpf
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/24/2016 09:49 PM, Walter Bright wrote:
 ...
I think it is obvious that there is fundamental misunderstanding on both sides here about goals and need of the other one, is is better to make a step back and resolve that before continuing same discussion in circles. Yesterday we had a very nice live discussion with Martin about current scope semantics and my concerns about it. He has mentioned a planned call with you later on same topic - let's wait for that and return to discussing details after if something will yet remain unclear. --sbBsSsMcu5C65WDITokEqjPBmejA1VGpf--
Oct 26 2016
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 26 October 2016 at 09:59:07 UTC, Dicebot wrote:
 Yesterday we had a very nice live discussion with Martin about 
 current scope semantics and my concerns about it. He has 
 mentioned a planned call with you later on same topic - let's 
 wait for that and return to discussing details after if 
 something will yet remain unclear.
Yes, let's do that on tuesday if that suits all of you, will send around an invite. I think we're all in the topic now and finally close to mutual understanding ;).
Oct 29 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
I think I thought of a solution. Currently, the design ignores 'scope' if it is 
applied to a variable with no indirections. But we can make it apply, in that 
the 'scope' of a function's return value is tied to the 'scope' of the
argument, 
even if that argument has no indirections.

This should make Dicebot's design work.
Oct 27 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
 <nugqg3$ld8$1 digitalmars.com> <nujrme$1pu9$1 digitalmars.com>
 <nukaue$2d5l$1 digitalmars.com> <nulojm$1li3$1 digitalmars.com>
 <nusi80$4h0$1 digitalmars.com>
In-Reply-To: <nusi80$4h0$1 digitalmars.com>

--AXhBLeigNXa5Omm19t2D6vhe9olpbrFt1
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/27/2016 12:43 PM, Walter Bright wrote:
 I think I thought of a solution. Currently, the design ignores 'scope'
 if it is applied to a variable with no indirections. But we can make it=
 apply, in that the 'scope' of a function's return value is tied to the
 'scope' of the argument, even if that argument has no indirections.
=20
 This should make Dicebot's design work.
This is the step in the right direction but doesn't fix more general issue. While RNG engine example doesn't have any indirections, that won't be true in all cases where one needs to return a scoped range. Most obvious problem is an owning container type: struct Tree (T) { private Node!T* head; ~this ( ) { delete this.head; } TreeRange range ( ); // returned range must not outlive // this struct instance } There really needs to be a way to bind returned value lifetime to `this` explicitly. No special cases, no magic, just one of straightforward use cases. --AXhBLeigNXa5Omm19t2D6vhe9olpbrFt1--
Oct 29 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/29/2016 4:30 AM, Dicebot wrote:
 Most obvious problem is an owning container type:

 struct Tree (T)
 {
     private Node!T* head;

     ~this ( ) { delete this.head; }

     TreeRange range ( ); // returned range must not outlive
                          // this struct instance
 }

 There really needs to be a way to bind returned value lifetime to `this`
 explicitly. No special cases, no magic, just one of straightforward use
 cases.
This is covered by my proposed modification: struct Tree { TreeRange range ( ) scope; }
Oct 29 2016
parent reply Dicebot <public dicebot.lv> writes:
On Sunday, 30 October 2016 at 02:17:36 UTC, Walter Bright wrote:
 This is covered by my proposed modification:

 struct Tree
 {
     TreeRange range ( ) scope;
 }
Maybe I misunderstand what you propose in that case? You have said "if it is applied to a variable with no indirections", but `struct Tree` in my example does contain indirection in form of `head` pointer (as struct is composition of fields).
Oct 29 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/29/2016 10:33 PM, Dicebot wrote:
 On Sunday, 30 October 2016 at 02:17:36 UTC, Walter Bright wrote:
 This is covered by my proposed modification:

 struct Tree
 {
     TreeRange range ( ) scope;
 }
Maybe I misunderstand what you propose in that case? You have said "if it is applied to a variable with no indirections", but `struct Tree` in my example does contain indirection in form of `head` pointer (as struct is composition of fields).
Since the caller cannot see inside the function, it must rely on the interface. If the returned pointer is or is not derived from the value of the scope parameter is immaterial, the caller will act as if it is - meaning that the scope of the returned value will be restricted to being the same scope of the argument. It's just tracking the scope of expressions: return x; return x + 3; return i ? x : y; so they work through function calls: return foo(x); return foo(x + 3); return foo(x, y); The 'return scope' and 'return ref' make that work so that the compiler only has to examine the function's interface, not its implementation.
Oct 30 2016
parent reply Dicebot <public dicebot.lv> writes:
On Monday, 31 October 2016 at 06:43:48 UTC, Walter Bright wrote:
 If the returned pointer is or is not derived from the value of 
 the  scope parameter is immaterial, the caller will act as if 
 it is - meaning that the scope of the returned value will be 
 restricted to being the same scope of the argument.
Is it how you want to change the implementation or how you intend it to work now? Because currently it acts differently and no lifetime restriction ever happens in absence of explicit `return scope` annotation: int* foo ( scope int* ); int* global; void main () { scope int* ptr; global = foo(ptr); // compiles } ----------------------------- Let's get back to the old snippet as you haven't answered my question: struct Tree { Node* head; TreeRange range ( ) scope; } Per my understanding, it is roughly equivalent to this: struct Tree { Node* head; } TreeRange range ( scope ref Tree this ); With current rules `scope` here is completely ignored because `this` is already a reference and thus is not allowed to escape. And `TreeRange` lifetime has no relation with `this` lifetime, because such relation is only defined by `return scope`/`return ref`. Now in http://forum.dlang.org/post/nusi80$4h0$1 digitalmars.com you propose to make so that such `scope` annotation does indeed transfer lifetime: ".. the design ignores 'scope' if it is applied to a variable with no indirections. But we can make it apply". But that does not seem applicable to discussed example as struct Tree does contain indirections. Another issue is that would be completely out of line from existing difference between `scope` and `return scope`.
Oct 31 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/31/2016 1:08 AM, Dicebot wrote:
 On Monday, 31 October 2016 at 06:43:48 UTC, Walter Bright wrote:
 If the returned pointer is or is not derived from the value of the  scope
 parameter is immaterial, the caller will act as if it is - meaning that the
 scope of the returned value will be restricted to being the same scope of the
 argument.
Is it how you want to change the implementation or how you intend it to work now? Because currently it acts differently and no lifetime restriction ever happens in absence of explicit `return scope` annotation: int* foo ( scope int* ); int* global; void main () { scope int* ptr; global = foo(ptr); // compiles }
Amending it as follows: safe int* foo ( return scope int* ); int* global; safe void main () { scope int* ptr; global = foo(ptr); // compiles } Produces: ..\dmd bug -transition=safe DMD v2.069 DEBUG bug.d(6): Error: scope variable ptr assigned to global with longer lifetime which works as intended and is the current behavior. 'scope' means the value does not escape the function. 'return scope' means the value does not escape the function, but may be returned. It's analogous to 'ref' and 'return ref' parameters. It works the same way, except it pertains to the value of the argument rather than its address. I'm a bit baffled why I am unable to explain this.
 -----------------------------

 Let's get back to the old snippet as you haven't answered my question:

 struct Tree
 {
     Node* head;

     TreeRange range ( ) scope;
 }

 Per my understanding, it is roughly equivalent to this:

 struct Tree { Node* head; }
 TreeRange range ( scope ref Tree this );
Yes. Note that you can ALWAYS rewrite member functions as normal functions with the 'this' parameter as a regular parameter. There is nothing magically different about them, they MUST MUST MUST behave the same way. Otherwise, nothing will make sense.
 With current rules `scope` here is completely ignored because `this` is already
 a reference and thus is not allowed to escape. And `TreeRange` lifetime has no
 relation with `this` lifetime, because such relation is only defined by `return
 scope`/`return ref`.
Again, 'ref' refers to the ADDRESS of the argument. 'scope' refers to the VALUE of the argument. 1. 'return ref' says that the function returns that parameter by 'ref' (or is treated as if it did). 2. 'return scope' says that the function returns that parameter by value (or is treated as if it did). 3. 'return ref scope' is treated as 'return ref' and 'scope' if the function returns by ref. 4. 'return ref scope' is treated as 'ref' and 'return scope' if the function does not return by ref. Therefore, TreeRange range(scope ref Tree this); is case 4, and the return value of 'range' will have the scope of the value of the argument corresponding to 'this', which corresponds to the scope of the argument.
 Now in http://forum.dlang.org/post/nusi80$4h0$1 digitalmars.com you propose to
 make so that such `scope` annotation does indeed transfer lifetime: ".. the
 design ignores 'scope' if it is applied to a variable with no indirections. But
 we can make it apply". But that does not seem applicable to discussed example
as
 struct Tree does contain indirections.
My proposed change is to make it apply even if it does not contain indirections.
 Another issue is that would be completely out of line from existing difference
 between `scope` and `return scope`.
I don't think that is at all the case.
Oct 31 2016
parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
 <nugqg3$ld8$1 digitalmars.com> <nujrme$1pu9$1 digitalmars.com>
 <nukaue$2d5l$1 digitalmars.com> <nulojm$1li3$1 digitalmars.com>
 <nusi80$4h0$1 digitalmars.com> <nv2197$2n1r$1 digitalmars.com>
 <nv3l7t$208p$1 digitalmars.com> <aihxaamczpyxgrqelxoz forum.dlang.org>
 <nv6p72$1an0$1 digitalmars.com> <ibyhjqcdhhbwzggamoqo forum.dlang.org>
 <nv74ao$1s38$1 digitalmars.com>
In-Reply-To: <nv74ao$1s38$1 digitalmars.com>

--OxX0R3PGWCHXebaSJUrkK6hfrs7abtJvJ
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/31/2016 11:53 AM, Walter Bright wrote:
 I'm a bit baffled why I am unable to explain this.
Walter it is very uncomfortable to see that are trying to explain me my own words (".. no lifetime restriction ever happens in absence of explicit `return scope` annotation.. ") as if I don't understand that. It sometimes feels like you spot some random keywords and completely ignore actual content of the question. You are unable to explain simply because you don't explain what was asked. I know the difference between `scope` and `return scope`, and I ask questions exactly because I know it and last proposal doesn't make sense with it. See below:
 4. 'return ref scope' is treated as 'ref' and 'return scope' if the
 function does not return by ref.
=20
 Therefore,
=20
     TreeRange range(scope ref Tree this);
=20
 is case 4, and the return value of 'range' will have the scope of the
 value of the argument corresponding to 'this', which corresponds to the=
 scope of the argument.
This makes no sense because declaration is not `TreeRange range(return ref scope Tree this)`, it is `TreeRange range(ref scope Tree this)` (absence of `return` is critical!) Does that mean that you have actually meant you first example to look like this: struct Tree { TreeRange range () return ref scope } ? --OxX0R3PGWCHXebaSJUrkK6hfrs7abtJvJ--
Oct 31 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/31/2016 3:13 AM, Dicebot wrote:
 4. 'return ref scope' is treated as 'ref' and 'return scope' if the
 function does not return by ref.

 Therefore,

     TreeRange range(scope ref Tree this);

 is case 4, and the return value of 'range' will have the scope of the
 value of the argument corresponding to 'this', which corresponds to the
 scope of the argument.
This makes no sense because declaration is not `TreeRange range(return ref scope Tree this)`, it is `TreeRange range(ref scope Tree this)` (absence of `return` is critical!) Does that mean that you have actually meant you first example to look like this: struct Tree { TreeRange range () return ref scope } ?
Yeah, sorry, the 'return' is necessary as you said.
Oct 31 2016
parent Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <nubhr3$15k6$1 digitalmars.com> <nucb5n$2hv7$1 digitalmars.com>
 <nudcq0$1bio$1 digitalmars.com> <nuf5qv$18ca$1 digitalmars.com>
 <nugqg3$ld8$1 digitalmars.com> <nujrme$1pu9$1 digitalmars.com>
 <nukaue$2d5l$1 digitalmars.com> <nulojm$1li3$1 digitalmars.com>
 <nusi80$4h0$1 digitalmars.com> <nv2197$2n1r$1 digitalmars.com>
 <nv3l7t$208p$1 digitalmars.com> <aihxaamczpyxgrqelxoz forum.dlang.org>
 <nv6p72$1an0$1 digitalmars.com> <ibyhjqcdhhbwzggamoqo forum.dlang.org>
 <nv74ao$1s38$1 digitalmars.com> <nv75gs$1u5u$1 digitalmars.com>
 <nv8uv0$1kft$1 digitalmars.com>
In-Reply-To: <nv8uv0$1kft$1 digitalmars.com>

--LlJcOU3XAs8gn6QL9IXqv9RH0jOWIbBl7
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 11/01/2016 04:34 AM, Walter Bright wrote:
 Yeah, sorry, the 'return' is necessary as you said.
Great, this brings my understanding back to state of harmony again :) --LlJcOU3XAs8gn6QL9IXqv9RH0jOWIbBl7--
Nov 01 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Monday, 24 October 2016 at 19:49:10 UTC, Walter Bright wrote:
 On 10/23/2016 11:49 PM, Dicebot wrote:
 On 10/24/2016 05:29 AM, Walter Bright wrote:
 On 10/22/2016 3:50 PM, Dicebot wrote:
 Beauty of borrowing is that it allows to to inc/dec skipping 
 with no
 special RC support from compiler and works the very same for 
 stack
 allocated data. What you are going at is simply replacing GC 
 with
 (compiler enhanced) RC and that is not good at all.
I don't see why supporting something beyond RC and GC is necessary.
Both GC and RC are forms of automatic memory management with unavoidable associated overhead. Both have their uses but are unacceptable in most of the system code. Which means that would would either need to duplicate all Phobos/druntime functions to work with both RC and plain pointers or it will become even less suitable for low level code than it is now.
I'm afraid that is a bit vague. Can you give a more concrete example?
Also important are Unique ownership (handing out scoped references to internal (untyped) memory) and ranges that cannot outlive their containers.
 - allow taking address of a `ref` and define lifetime of such 
 expression
 to be same of `ref` lifetime
Again, why? Why can't the function return by 'ref'?
While returning by ref is good enough to prevent escaping, and we do use it in RefCounted for structs, it doesn't allow you to bind that returned value anywhere. So you could only chain method calls s.get.call.getNested.doSome.foo.bar;. While that might be fine as a first step, it doesn't seem to need scope/return scope, and is fairly limiting in the mid-term.
 - `return scope` of a struct methods binds returns value 
 lifetime to one
 of the struct instance itself, not composition of its fields
'ref' return already does that.
Would it be possible to conflate the two different annotatons with similar but slightly different semantic? Couldn't return scope on the method mean, that a reference to the whole struct can be returned.
Oct 29 2016
prev sibling parent reply Stefam Koch <uplink.coder googlemail.com> writes:
On Monday, 24 October 2016 at 02:29:35 UTC, Walter Bright wrote:

 At some point we need to move forward. We are already years 
 late. Another month has slipped by.
That argument can go two ways. Since we are already years late another month won't matter that much. D's main advantage is hindsight and careful consideration of complexity versus power! We should stick to those values and trying to beat someone in a marketing hype.
Oct 24 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 1:13 AM, Stefam Koch wrote:
 That argument can go two ways.
 Since we are already years late another month won't matter that much.
That doesn't work as an argument. Being later makes things worse, not irrelevant. Memory safety is constantly growing in importance. It is a key advantage D can have over C/C++ - one they will have great difficulty matching.
 D's main advantage is hindsight and careful consideration of complexity versus
 power!
 We should stick to those values and trying to beat someone in a marketing hype.
It's not a question of beating. It is a question of becoming irrelevant. We lose design wins constantly because of this issue.
Oct 24 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 8:38 AM, Dicebot wrote:
 It won't allow to fix `Unique`
What it does mean is given a unique pointer u, int foo(scope Unique u); it is guaranteed that u's value does not escape from foo(). So, Unique u = malloc(); foo(u); free(u); // guaranteed to not leave a dangling pointer
Oct 22 2016
parent Martin Nowak <code dawg.eu> writes:
On Saturday, 22 October 2016 at 08:10:51 UTC, Walter Bright wrote:
 On 10/21/2016 8:38 AM, Dicebot wrote:
    int foo(scope Unique u);

 it is guaranteed that u's value does not escape from foo(). So,

    Unique u = malloc();
    foo(u);
    free(u); // guaranteed to not leave a dangling pointer
That's unsafe code calling safe code, but the more relevant case is safe code interfacing with trusted code.
Oct 29 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Thursday, 20 October 2016 at 21:36:44 UTC, Walter Bright wrote:
 On 10/20/2016 12:27 PM, Dicebot wrote:
   int* bar()
   {
     int p;
     return foo1(&p);
   }
This is a problem I had reading DIP1000. At first it decomposes language constructs into primitives then sets up rules for the primitives. Such a mathematical approach is nice, but it isn't exercised throughout the DIP. For example the most interesting function semantics just refer "See section dedicated to discussing methods.", but the Function section (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/D P1000.md#functions) doesn't contain formal lifetime rules for functions. Weirdly the Pointer section (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/ IP1000.md#pointers) seems to contain lifetime rules for function return values. At this point the DIP regresses a bit into showing examples with short notes whether they are errors or correct. It's somewhat difficult to extract a mental model from that. The DIP seems to define the lifetime of function return values as the longest lifetime that can be conservatively assumed given the function arguments. You mentioned an article and a talk, do you have any cleaner/more detailed explanation for functions Walter?
Oct 20 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/20/2016 7:57 PM, Martin Nowak wrote:
 For example the most interesting function semantics just refer "See section
 dedicated to discussing methods.", but the Function section
 (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#functions)
 doesn't contain formal lifetime rules for functions.
 Weirdly the Pointer section
 (https://github.com/dlang/DIPs/blob/6f9f50ee579bd0ccc11e5dfeef035cbbb095e0d9/DIPs/DIP1000.md#pointers)
 seems to contain lifetime rules for function return values.
 At this point the DIP regresses a bit into showing examples with short notes
 whether they are errors or correct.
 It's somewhat difficult to extract a mental model from that.
What you're looking at is I'm not very good at articulating the mental model I have for it. I don't know how to write 'formal' rules for these things.
 The DIP seems to define the lifetime of function return values as the longest
 lifetime that can be conservatively assumed given the function arguments.
That's right.
 You mentioned an article and a talk, do you have any cleaner/more detailed
 explanation for functions  Walter?
I haven't written either, yet, though I need to get busy with it Real Soon Now. It'll be informal, obviously, but I plan to use trivial examples to make it as clear as possible. I hate slides with more than about 5 lines of code in them - I lose track of the presentation when I'm trying to understand the code, and I presume my audience will, too.
Oct 20 2016
parent reply pineapple <meapineapple gmail.com> writes:
On Friday, 21 October 2016 at 05:41:31 UTC, Walter Bright wrote:
 What you're looking at is I'm not very good at articulating the 
 mental model I have for it. I don't know how to write 'formal' 
 rules for these things.
If we're going to go through the DIP process, you're going to have to try. There's this big language enhancement in the pipeline and I still have only a tenuous understanding of what it's meant to do and an even more tenuous understanding of how it's meant to be an improvement. Your mental model needs to be stated in concise, unambiguous terms before the rest of us can catch up and be able to consent to having the language we all use modified in such a way. The DIP process is annoying and obtuse as hell, but it does exist for a reason. Your proposals should be held to the same standards as the rest of us, because changes to the language affect all of us.
Oct 21 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/21/2016 1:30 AM, pineapple wrote:
 There's this big language enhancement in the pipeline
But it's not big. It's the same as 'return ref', which is already there, except it's 'return scope'. The examples on how it works all boil down to a couple lines of code.
Oct 22 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Saturday, 22 October 2016 at 07:19:11 UTC, Walter Bright wrote:
 On 10/21/2016 1:30 AM, pineapple wrote:
 There's this big language enhancement in the pipeline
But it's not big. It's the same as 'return ref', which is already there, except it's 'return scope'. The examples on how it works all boil down to a couple lines of code.
Most people seem to disagree with that opinion. I was looking at the DIP, and checked out the P.R. branch to test it, and I cannot see how it could work if `scope` is not transitive. It is just pushing the problem one level down. For example: ``` alias FunDG = void delegate () safe; struct Escaper { FunDG DG; } FunDG escapeDg1 (scope FunDG d) safe { Escaper e; e.DG = d; return e.DG; } ``` This compiles, and escapes a `scope delegate`. Used `./src/dmd -transition=safe -dip25 -run test.d` to compile, on commit: `48b7815 - (HEAD, walter/return-scope) add lifetime checks (9 days ago) <Walter Bright>`. Did I miss something ? Also, as others already mentioned, the DIP is quite confusing. I read the whole thread and AFAICS people got confused because of https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md#escaping-via-return which mentions that "taking the address of a local [...] is restrictive." and continue with "The `scope` storage class was introduced which annotates the pointer.", which doesn't quite relate to the previous sentence if scope is not meant to remove this restriction. The examples also use a mix of ` safe` and non-annotated function, which doesn't help understanding since it's explicitly stated that those checks are only done in safe code (why?). Finally, I agree with Dicebot that having scope infered when the type is specified is extremely confusing, and makes writing test cases, and code in general, much more complicated.
Oct 22 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/22/2016 8:52 AM, Mathias Lang wrote:
 I was looking at the DIP, and checked out the P.R. branch to test it, and I
 cannot see how it could work if `scope` is not transitive.
 It is just pushing the problem one level down.

 For example:

 ```
 alias FunDG = void delegate ()  safe;

 struct Escaper
 {
     FunDG DG;
 }

 FunDG escapeDg1 (scope FunDG d)  safe
 {
     Escaper e;
     e.DG = d;
     return e.DG;
 }
 ```

 This compiles, and escapes a `scope delegate`.
1. It is not an example of transitivity. e and d are at the same level, being a field does not make it a level down. 2. You're right that the above should error, I'll look into it.
 Finally, I agree with Dicebot that having scope infered when the type is
 specified is extremely confusing,
What is confusing about it? scope int* p = ...; int* q = p; // q becomes 'scope' as well The compiler already does tons of inference for types and attributes. People should be well used to it by now :-)
 and makes writing test cases, and code in
 general, much more complicated.
Actually, it makes things much simpler allowing the compiler to take care of it. The whole scope thing would be fairly unusable if one had to manually add all the annotations.
Oct 22 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/22/2016 7:34 PM, Walter Bright wrote:
 2. You're right that the above should error, I'll look into it.
Added the fix to the PR.
Oct 22 2016
parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Sunday, 23 October 2016 at 06:42:41 UTC, Walter Bright wrote:
 On 10/22/2016 7:34 PM, Walter Bright wrote:
 2. You're right that the above should error, I'll look into it.
Added the fix to the PR.
Somehow my previous (mailed yesterday afternoon) message didn't go through, so I'm going to paste it here again: Just looked at it. The diff shows: +fail_compilation/retscope.d(198): Error: scope variable e may not be returned +void* escapeDg1(scope void* d) safe +{ + Escaper e; + e.DG = d; + return e.DG; // L198 +} So it means that the storage class of 'e' changes during semantic analysis of the function. I don't think that's something we want to ever do. Crafting a test case that exploit this was trivial: void* escape3 (scope void* p) safe { Escaper e; scope dg = () { return e.e; }; e.e = p; return dg(); } Not to mention, if we ever relax the restriction on taking the address of a local variable, as the DIP seems to hint as a goal, you can just do: Escaper e; Escaper* pe = &e; And you'll have the same issue.
Oct 24 2016
next sibling parent Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org> <nua19j$1fvb$1 digitalmars.com>
 <nub5qg$hks$1 digitalmars.com> <nubddb$ulr$1 digitalmars.com>
 <rxhugjriispyocejnscj forum.dlang.org> <nuc9q8$2g0o$1 digitalmars.com>
 <wrlfmfvelrduhlqugaus forum.dlang.org> <nuf3tc$14t5$1 digitalmars.com>
 <avaeijcpibvujqdosnge forum.dlang.org> <nuh7ij$1512$1 digitalmars.com>
 <nuhm4s$1uva$1 digitalmars.com> <fngwxfoeudirjwkfuxlw forum.dlang.org>
In-Reply-To: <fngwxfoeudirjwkfuxlw forum.dlang.org>

--6Puj17VNp24qpiH5a0I0fGLqI1DWQsKFM
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/24/2016 12:13 PM, Mathias Lang wrote:
 Crafting a test case that exploit this was trivial:
 void* escape3 (scope void* p)  safe
 {
     Escaper e;
     scope dg =3D () { return e.e; };
     e.e =3D p;
     return dg();
 }
I think it is quite possible to keep non-transitive semantics to enable designing correct trusted wrappers, the problem comes from trying to magically work as valid safe code. Really, all we need is for developer to be able to say that some pointer/reference may only be used as long as host aggregate exists. With that transitivity can be emulated by removing direct access to reference-type fields and marking matching getters as scope. --6Puj17VNp24qpiH5a0I0fGLqI1DWQsKFM--
Oct 24 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 2:13 AM, Mathias Lang wrote:
 So it means that the storage class of 'e' changes during semantic analysis of
 the function. I don't think that's something we want to ever do.
Why? D already infers lots of things.
 Crafting a test case that exploit this was trivial:
 void* escape3 (scope void* p)  safe
 {
     Escaper e;
     scope dg = () { return e.e; };
     e.e = p;
     return dg();
 }
I'll investigate that one.
Oct 24 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 12:30 PM, Walter Bright wrote:
 Why? D already infers lots of things.
Like safe, trusted, system, nogc, pure, nothrow, return. Consider also that Rust does extensive lifetime attribute inference.
Oct 24 2016
parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Monday, 24 October 2016 at 20:57:08 UTC, Walter Bright wrote:
 On 10/24/2016 12:30 PM, Walter Bright wrote:
 Why? D already infers lots of things.
Like safe, trusted, system, nogc, pure, nothrow, return. Consider also that Rust does extensive lifetime attribute inference.
Don't get me wrong, I love inference. I am afraid my feedback may have mixed up two things, so I'm going to reinstate them here: 1) As highlighted in the previous post, we should not change the STC / type of a declaration after it is declared. It's a can of worms and will break in many ways. 2) I don't know a single place in the language where a single "something" (type / attributes / STC) is infered. It's either all or nothing (e.g. you cannot have a function infering safety but not infering nothrow / nogc, short of tricking the compiler by changing the function's code). Also, I don't know of a single place in the language where a type provided explicitly is infered. For example, if you do `char[] s = "Hello World";`, `const` or `immutable` is not infered, and you get an error, and that's a good thing. Changing it would make code way less readable and harder to reason about.
Oct 24 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 2:11 PM, Mathias Lang wrote:
 1) As highlighted in the previous post, we should not change the STC / type of
a
 declaration after it is declared. It's a can of worms and will break in many
ways.
The definition of a storage class is a bit fuzzy. 'scope' does not say where the variable is stored - it does not say "put it on the stack", "put it in the code segment", "put it on the heap". It's more of an attribute, just like nothrow, pure, etc.
 2) I don't know a single place in the language where a single "something" (type
 / attributes / STC) is infered. It's either all or nothing (e.g. you cannot
have
 a function infering safety but not infering nothrow /  nogc, short of tricking
 the compiler by changing the function's code). Also, I don't know of a single
 place in the language where a type provided explicitly is infered. For example,
 if you do `char[] s = "Hello World";`, `const` or `immutable` is not infered,
 and you get an error, and that's a good thing. Changing it would make code way
 less readable and harder to reason about.
'scope' is not a type, although it does affect function type signatures the same as safe, pure, nothrow and nogc do, as well as affecting covariance/contravariance in an equivalent manner.
Oct 24 2016
parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Monday, 24 October 2016 at 21:30:50 UTC, Walter Bright wrote:
 On 10/24/2016 2:11 PM, Mathias Lang wrote:
 1) As highlighted in the previous post, we should not change 
 the STC / type of a
 declaration after it is declared. It's a can of worms and will 
 break in many ways.
The definition of a storage class is a bit fuzzy. 'scope' does not say where the variable is stored - it does not say "put it on the stack", "put it in the code segment", "put it on the heap". It's more of an attribute, just like nothrow, pure, etc.
`scope o = new Object;` If you prefer to compare it to `nothrow` or `pure`, that's also fine by me. I still don't know of any place where an attribute can be added to a pre-existing variable declaration.
 'scope' is not a type, although it does affect function type 
 signatures the same as  safe, pure, nothrow and  nogc do, as 
 well as affecting covariance/contravariance in an equivalent 
 manner.
The DIP is very clear that `scope` is not a type qualifier. The parallel with `const` inference was drawn as a reference to https://forum.dlang.org/post/nubdqu$v6u$1 digitalmars.com (and your later post). I see both inference as getting in the way of code readability if not requested by the user explicitly. On Monday, 24 October 2016 at 21:40:48 UTC, Walter Bright wrote:
 
 Please wait until I investigate that one.
Fair enough, then I'll leave the point of adding attributes for later.
 On Monday, 24 October 2016 at 19:32:45 UTC, Walter Bright 
 wrote:
 On 10/24/2016 9:49 AM, Mathias Lang wrote:
Users will have to learn `scope` semantic to write ` safe` code. If one doesn't want to learn `scope` semantics, one just doesn't use it, `scope` is opt-in after all. I fail to see any benefit for someone that doesn't want to learn `scope`,
Writing system code can be done without paying attention to that stuff, and there is value in that. safe code comes into play when writing larger scale, longer lived (!) code, and code that is worked on by a team.
Could you explain the value there is in not checking `scope` in ` system` ? Also, how does that value offset the user confusion and possible bugs caused by people trying to use `scope` in ` system` code without realizing it doesn't do any check ? Quick note: I notice you said ` system`. The DIP said "only checked in safe code". Will trusted by checked ? Also, I don't quite see what safe has to do with code lifetime, or collaboration.
Oct 24 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 3:10 PM, Mathias Lang wrote:
 `scope o = new Object;`
 If you prefer to compare it to `nothrow` or `pure`, that's also fine by me.
 I still don't know of any place where an attribute can be added to a
 pre-existing variable declaration.
You're right, that is a bit of a special case, inherited from D1. But storing it on the stack is an optimization, it behaves "as if" it was allocated on the heap.
 I see both inference as getting in the way of code readability if not requested
 by the user explicitly.
At this point, we'll just have to disagree on that.
 Could you explain the value there is in not checking `scope` in ` system` ?
Quick and dirty code, for one.
 Also, how does that value offset the user confusion and possible bugs caused by
 people trying to use `scope` in ` system` code without realizing it doesn't do
 any check ?
It still checks the interface.
 Quick note: I notice you said ` system`. The DIP said "only checked in  safe
 code". Will  trusted by checked ?
No, because trusted code is system code.
 Also, I don't quite see what  safe has to do with code lifetime,
Wouldn't you want to take more care when you write code that will be used for years, rather than some throwaway piece of code?
  or collaboration.
Because it enables your collaborators to understand your code better.
Oct 24 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
Now that we have https://github.com/dlang/dmd/pull/6253 I came 
back to test it a bit.

First I have a question: What is going to happen with `scope foo 
= new Object` ?
Are we keeping it the way it currently is ? It's something we 
highly rely on at work and it looks like it could work nicely 
with the enhancements, but it wasn't mentioned anywhere.

And from a quick test (Using `-dip25 -transition=safe` and 
5928249 which is the commit of this P.R.), the following code:

```
void main ()  safe
{
     scope o = new Object();
}
```

Results in the following error:
scope2.d(3): Error: delete o is not  safe but is used in  safe 
function main

Which works with the compiler I'm currently using, 2.071.2.

By the way, I reiterate my point that tying `scope` verification 
to ` safe` function is highly misleading and not necessary. I 
originally crafted a couple of test case, and when everything 
worked, I was left puzzled for a minute before realizing I forgot 
` safe`.

Second point, on testing your P.R., I found that the following 
code compiles:

```
alias FunDG = void delegate ()  safe;

void main ()  safe
{
     int x = 42;
     scope FunDG f = () { assert(x == 42); };
     return fun(&f);
}

FunDG fun (scope FunDG* ptr)  safe
{
     return *ptr;
}
```

As well as the following:
```
alias FunDG = void* delegate ()  safe;

void main ()  safe
{
     void* x = fwd();
}

void* fwd ()  safe
{
     int x = 42;
     scope FunDG f = () { return &x; };
     return fun(f);
}

void* fun (scope FunDG ptr)  safe
{
     return ptr();
}
```

I'm not quite sure how one would express taking a pointer to a 
scope value though.
Nov 20 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/20/2016 8:41 AM, Mathias Lang wrote:
 First I have a question: What is going to happen with `scope foo = new Object`
?
 Are we keeping it the way it currently is ?
Yes.
 void main ()  safe
 {
     scope o = new Object();
 }
 ```

 Results in the following error:
 scope2.d(3): Error: delete o is not  safe but is used in  safe function main

 Which works with the compiler I'm currently using, 2.071.2.
I'll look into that.
 By the way, I reiterate my point that tying `scope` verification to ` safe`
 function is highly misleading and not necessary.
I understand your point. The trouble is, it will break too much code.
 I found that the following code compiles:

 ```
 alias FunDG = void delegate ()  safe;

 void main ()  safe
 {
     int x = 42;
     scope FunDG f = () { assert(x == 42); };
     return fun(&f);
 }

 FunDG fun (scope FunDG* ptr)  safe
 {
     return *ptr;
 }
 ```
I get: bug9.d(7): Error: cannot return non-void from void function
 As well as the following:
 ```
 alias FunDG = void* delegate ()  safe;

 void main ()  safe
 {
     void* x = fwd();
 }

 void* fwd ()  safe
 {
     int x = 42;
     scope FunDG f = () { return &x; };
     return fun(f);
 }

 void* fun (scope FunDG ptr)  safe
 {
     return ptr();
 }
I'll have to investigate that one.
Nov 22 2016
next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Wednesday, 23 November 2016 at 03:25:20 UTC, Walter Bright 
wrote:
 I understand your point. The trouble is, it will break too much 
 code.
Maybe I am missing something here: I didn't consider it as breaking more code than it's ` safe` counterpart.
 I found that the following code compiles:
 [...]
I get: bug9.d(7): Error: cannot return non-void from void function
My bad, I tried to reduce the snippet a bit more and broke it :) Here's the corrected version: ``` void main () safe { int x; void* escape = bar(&x); } void* bar (scope void* x) safe { return fun(&x); } void* fun (scope void** ptr) safe { return *ptr; } ``` Using the top of your branch, that is: b8df860 allow 'return' attribute on nested functions (68 minutes ago) <Walter Bright>
Nov 23 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2016 3:05 AM, Mathias Lang wrote:
 [...]
It doesn't actually escape on your version, as main() doesn't return a void*. Corrected example: -------------------- void* abc () safe { int x; void* escape = bar(&x); return escape; // <== correction } void* bar (scope void* x) safe { return fun(&x); // where the problem lies } void* fun (scope void** ptr) safe { return *ptr; } --------------------- Probably the best solution is to not allow taking the address of a scope local. Good catch!
Nov 23 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/22/2016 7:25 PM, Walter Bright wrote:
 As well as the following:
 ```
 alias FunDG = void* delegate ()  safe;

 void main ()  safe
 {
     void* x = fwd();
 }

 void* fwd ()  safe
 {
     int x = 42;
     scope FunDG f = () { return &x; };
     return fun(f);
 }

 void* fun (scope FunDG ptr)  safe
 {
     return ptr();
 }
I'll have to investigate that one.
I amended: https://github.com/dlang/dmd/pull/6253 to fix it.
Nov 23 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/20/2016 8:41 AM, Mathias Lang wrote:
 And from a quick test (Using `-dip25 -transition=safe` and 5928249 which is the
 commit of this P.R.), the following code:

 ```
 void main ()  safe
 {
     scope o = new Object();
 }
 ```

 Results in the following error:
 scope2.d(3): Error: delete o is not  safe but is used in  safe function main

 Which works with the compiler I'm currently using, 2.071.2.
This was done here back in June: https://github.com/dlang/dmd/commit/e64ae1d3e5aa078a036242864a68499617c9b278 and was added to fix: https://issues.dlang.org/show_bug.cgi?id=16195 The trouble is, operator delete is not safe. What do you suggest?
Nov 22 2016
next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Wednesday, 23 November 2016 at 03:35:16 UTC, Walter Bright 
wrote:
 On 11/20/2016 8:41 AM, Mathias Lang wrote:
 And from a quick test (Using `-dip25 -transition=safe` and 
 5928249 which is the
 commit of this P.R.), the following code:

 ```
 void main ()  safe
 {
     scope o = new Object();
 }
 ```

 Results in the following error:
 scope2.d(3): Error: delete o is not  safe but is used in  safe 
 function main

 Which works with the compiler I'm currently using, 2.071.2.
This was done here back in June: https://github.com/dlang/dmd/commit/e64ae1d3e5aa078a036242864a68499617c9b278 and was added to fix: https://issues.dlang.org/show_bug.cgi?id=16195 The trouble is, operator delete is not safe. What do you suggest?
If I understand correctly, it's not safe because it might free a reference which is still used elsewhere, making it dangling. If we have the correct `scope` enforcement though, we can allow it *only at the scope which `new`ed it*, because we know it's not possible than any other place has a reference to it, as the reference would have to be `scope` as well. Does that make sense ?
Nov 23 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2016 3:08 AM, Mathias Lang wrote:
 If I understand correctly, it's not  safe because it might free a reference
 which is still used elsewhere, making it dangling.
Yes.
 If we have the correct `scope` enforcement though, we can allow it *only at the
 scope which `new`ed it*, because we know it's not possible than any other place
 has a reference to it, as the reference would have to be `scope` as well.

 Does that make sense ?
Yes. I'll see if I can make that work tomorrow.
Nov 23 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
https://issues.dlang.org/show_bug.cgi?id=16747
Nov 24 2016
prev sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Monday, 24 October 2016 at 19:30:16 UTC, Walter Bright wrote:
 On 10/24/2016 2:13 AM, Mathias Lang wrote:
 So it means that the storage class of 'e' changes during 
 semantic analysis of
 the function. I don't think that's something we want to ever 
 do.
Why? D already infers lots of things.
To answer your question: Because it isn't even inference. The specs explicitly says "Unlike module level declarations, declarations within function scope are processed in order." (https://dlang.org/spec/function.html) Changing the storage class of a previous declaration is then explicitly breaking that line of the spec, and breaking a key assumption the specs / compiler relies on. I pointed that problem using delegate, I could have done so using templates, nested functions, or whatever part of the language that relies on said assumption. On Monday, 24 October 2016 at 19:32:45 UTC, Walter Bright wrote:
 On 10/24/2016 9:49 AM, Mathias Lang wrote:
 Why is this necessary / what would be the problem in allowing 
 it in  system ?
For one thing, it would require all users to learn scope semantics and annotate them correctly. It would be like requiring const correctness.
Users will have to learn `scope` semantic to write ` safe` code. If one doesn't want to learn `scope` semantics, one just doesn't use it, `scope` is opt-in after all. I fail to see any benefit for someone that doesn't want to learn `scope`, and find it extremely confusing for users that want to use it but can't make their code ` safe` (because for example, they're accepting a delegate which can be system so they can't just wrap code in ` trusted`).
Oct 24 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 2:02 PM, Mathias Lang wrote:
 On Monday, 24 October 2016 at 19:30:16 UTC, Walter Bright wrote:
 On 10/24/2016 2:13 AM, Mathias Lang wrote:
 So it means that the storage class of 'e' changes during semantic analysis of
 the function. I don't think that's something we want to ever do.
Why? D already infers lots of things.
To answer your question: Because it isn't even inference. The specs explicitly says "Unlike module level declarations, declarations within function scope are processed in order." (https://dlang.org/spec/function.html) Changing the storage class of a previous declaration is then explicitly breaking that line of the spec, and breaking a key assumption the specs / compiler relies on.
I don't believe it breaks anything the language relies on.
 I pointed that problem using delegate, I could have done so using templates,
 nested functions, or whatever part of the language that relies on said
assumption.
Please wait until I investigate that one.
 On Monday, 24 October 2016 at 19:32:45 UTC, Walter Bright wrote:
 On 10/24/2016 9:49 AM, Mathias Lang wrote:
 Why is this necessary / what would be the problem in allowing it in  system ?
For one thing, it would require all users to learn scope semantics and annotate them correctly. It would be like requiring const correctness.
Users will have to learn `scope` semantic to write ` safe` code. If one doesn't want to learn `scope` semantics, one just doesn't use it, `scope` is opt-in after all. I fail to see any benefit for someone that doesn't want to learn `scope`,
Writing system code can be done without paying attention to that stuff, and there is value in that. safe code comes into play when writing larger scale, longer lived (!) code, and code that is worked on by a team.
 and find it extremely confusing for users that want to use it but can't make
their
 code ` safe` (because for example, they're accepting a delegate which can be
  system so they can't just wrap code in ` trusted`).
That's the reason for inference. (I'd also like to do inference for 'const'. Inference of pure, nothrow, etc., has been very successful, and I don't see any fundamental issue with taking it further.)
Oct 24 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 2:13 AM, Mathias Lang wrote:
 Crafting a test case that exploit this was trivial:
 void* escape3 (scope void* p)  safe
 {
     Escaper e;
     scope dg = () { return e.e; };
     e.e = p;
     return dg();
 }
A complete example: -------- struct Escaper { void* e; } void* escape3 (scope void* p) safe { Escaper e; e.e = p; scope dg = () { return e.e; }; // Error: scope variable e may not be returned return dg(); } ------- But changing the order of things: -------- struct Escaper { void* e; } void* escape3 (scope void* p) safe { Escaper e; scope dg = () { return e.e; }; // no error e.e = p; return dg(); } ------- This definitely is a bug, but is fixable. Thanks for posting it!
Oct 24 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Monday, 24 October 2016 at 23:10:46 UTC, Walter Bright wrote:
 This definitely is a bug, but is fixable. Thanks for posting it!
Let's please talk about how this needs fixing beforehand, see my post below. It seems to me this requires 2 semantic passes through function statememts :o.
Oct 25 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/25/2016 12:09 AM, Martin Nowak wrote:
 Let's please talk about how this needs fixing beforehand, see my post below.
 It seems to me this requires 2 semantic passes through function statememts :o.
One way of fixing it besides doing multiple passes is noting its use inside dg, and setting a flag saying it cannot be made scope, and so the scoped assignment becomes an error. General flow analysis generally requires an arbitrary number of passes to converge on a solution (the global optimizer does this), but we can do it in one pass with the conservative approach above, and just accept that it's conservative.
Oct 25 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Tuesday, 25 October 2016 at 10:05:50 UTC, Walter Bright wrote:
 One way of fixing it besides doing multiple passes is noting 
 its use inside dg, and setting a flag saying it cannot be made 
 scope, and so the scoped assignment becomes an error.
A tristate tainting, unkown/scope/escape, should indeed work here.
Oct 25 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/25/2016 5:59 PM, Martin Nowak wrote:
 On Tuesday, 25 October 2016 at 10:05:50 UTC, Walter Bright wrote:
 One way of fixing it besides doing multiple passes is noting its use inside
 dg, and setting a flag saying it cannot be made scope, and so the scoped
 assignment becomes an error.
A tristate tainting, unkown/scope/escape, should indeed work here.
I implemented it, but now the PR got thoroughly fouled up after I rebased it on master.
Nov 01 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Tuesday, 1 November 2016 at 09:19:55 UTC, Walter Bright wrote:
 I implemented it, but now the PR got thoroughly fouled up after 
 I rebased it on master.
You need help fixing this? In case of failed rebases it's usually best to git rebase --abort. If it's too late for that, you cat look at .git/logs/refs/heads/branch_name, to find the git commit hash before the rebase, then git reset --hard to that, and try again.
Nov 05 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/5/2016 3:58 AM, Martin Nowak wrote:
 On Tuesday, 1 November 2016 at 09:19:55 UTC, Walter Bright wrote:
 I implemented it, but now the PR got thoroughly fouled up after I rebased it
 on master.
You need help fixing this? In case of failed rebases it's usually best to git rebase --abort. If it's too late for that, you cat look at .git/logs/refs/heads/branch_name, to find the git commit hash before the rebase, then git reset --hard to that, and try again.
I'm concerned that if I dink around with it, I might wind up scrambling/deleting the commit history of it. I'd rather learn this stuff on something less important, so I'd prefer to defer it to you.
Nov 06 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Monday, 24 October 2016 at 09:13:58 UTC, Mathias Lang wrote:
 void* escape3 (scope void* p)  safe
 {
     Escaper e;
     scope dg = () { return e.e; };
     e.e = p;
     return dg();
 }
Seems like this would require that the scope tainting is done in a separate pass, before going through the statements in lexical order. The dg needs to know that e will escape in order to change to return scope. Not sure the compiler is ready for 2-pass function semantics. While it's clearly powerful, my intuition also goes a bit against it. - Does the compiler architecture allow for the necessary semantics. - What's a use-case that requires this? - Might the scope inference graph have cycles? - Can humans understand scope inference, and can we print proper error messages? It might be a bit too ambituous, and turn out to be an unforced error creating too many scope holes.
Oct 25 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 22 October 2016 at 15:52:17 UTC, Mathias Lang wrote:
 It's the same as 'return ref', which is already there, except 
 it's 'return scope'. The examples on how it works all boil 
 down to a couple lines of code.
Most people seem to disagree with that opinion. I was looking at the DIP, and checked out the P.R. branch to test it, and I cannot see how it could work if `scope` is not transitive.
Let's clarify the term transitive, it's quite possible different people have different understandings of that term. Only drawing on my conclusion from rewriting the RC/Unique implementations. What's needed is a "container" to wrap return values so that they cannot be aliased or escaped, a bit similar to how you can't take the address of an rvalue. If you could get a named alias (e.g. scope ref/ptr) into the returned scope value, then the lifetimes of the two variables are entangled, and it would be trivial to end up with a dangling scoped ref/ptr. scope S s: scope int* p = s.getInternalRef; // could be a malloced class ref instead of a pointer if (flipCoin) s.destroy; // or s = S(); To make this work, we'd need to disallow any mutation of s, while references to it are reachable. I wonder if that's worth the trouble over simply disallowing any binding to a named (and thus reachable) variable. Would require chaining of return values though, s.get.cannot.put.that.hot.potatoe.anywhere :/.
Oct 25 2016
parent Martin Nowak <code dawg.eu> writes:
On Tuesday, 25 October 2016 at 07:51:21 UTC, Martin Nowak wrote:
 To make this work, we'd need to disallow any mutation of s, 
 while references to it are reachable.
Didn't mention this explicitly. Binding the returned scope value as named scope ref argument to a chained function call must of course be possible, but that still wouldn't allow to bind to a variable that outlives the expression. So it's really somewhat similar to rvalues.
Oct 25 2016
prev sibling parent Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
 <mtzackphhklkrtotdbbd forum.dlang.org>
In-Reply-To: <mtzackphhklkrtotdbbd forum.dlang.org>

--F2hGmWhEbQt1Ddxi5Lf0t64BP2RxI75gX
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

On 10/20/2016 01:12 AM, Martin Nowak wrote:
 But IMHO we should ensure that we have sound specs (backed by tests)
 that solve the __right__ use-cases before discussing implementations. A=
s
 part of the implementation we'll have to transform this into a more
 easily understandable spec anyhow, so we might as well do that in the
 beginning.
This is exactly what I am trying to do in this thread (which is continuation of e-mail exchange with Walter on same topic). Coming up with set of useful practical snippets based on proposed implementation and using them to figure out subtle semantical details. Once I understand what is supposed to work and what is not, I'll give my try at complete rewrite of DIP1000 to make it more coherent with intentio= ns. --F2hGmWhEbQt1Ddxi5Lf0t64BP2RxI75gX--
Oct 20 2016
prev sibling next sibling parent reply Dicebot <public dicebot.lv> writes:
 protected-headers="v1"
From: Dicebot <public dicebot.lv>
Newsgroups: d,i,g,i,t,a,l,m,a,r,s,.,D,.,i,n,t,e,r,n,a,l,s
Subject: Re: DIP1000 discussion and testing
References: <nu00a6$t5i$1 digitalmars.com>
In-Reply-To: <nu00a6$t5i$1 digitalmars.com>

--63kxrBHdAJWHTHoSJlgg4BO9eS1mR6nPu
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable

Ok, so I am revisiting DIP1000 right now based on recent comments from
Walter. One thing remains unclear to me though, consider this example in
DIP1000 document:

```
 safe struct RefCountedSlice(T) {
	private T[] payload;

	// ...



	scope ref T opIndex(size_t i) {
		return payload[i];
	}
}
```

It looks like with current semantics this statement is very wrong.
Assuming that you have meant `scope return` instead (because plain scope
does nothing for `this` reference) Right now it does not matter how
`opIndex` is annotated if variable holding `RefCountedSlice` itself is
not annotated/deduced as scope:

```
 safe struct S
{
    // RefCountedSlice code reduced to bare minimum

    private int* payload;

    this (int) {
        this.payload =3D new int;
    }

     trusted ~this () {
        delete this.payload;
    }

    int* get () scope return {
        return this.payload;
    }
}

int* global;

 safe unittest
{
    scope s =3D S(42);
    global =3D s.get(); // Error: scope variable s assigned to global wit=
h
longer lifetime
}

 safe unittest
{
    auto s =3D S(42);
    global =3D s.get(); // compiles, oops
}
```

And indeed, with your explanations it seems to work as intended - `scope
return` annotation is simply forwarding lifetime of `S.payload` to
return value, which happens to be infinity by lifetime rules (GC
allocation). But that means that statement saying destructor can be
 trusted is wrong.

Walter, would you mind showing reference counted example that actually
works under your implementation?


--63kxrBHdAJWHTHoSJlgg4BO9eS1mR6nPu--
Oct 22 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/22/2016 4:22 PM, Dicebot wrote:
 Walter, would you mind showing reference counted example that actually
 works under your implementation?
I can't, because I'm waiting for Andrei to make available what he considers a correct ref counted design. But you are correct in that the 'return' rules can be used to transfer the lifetime from an argument to the return value, even if the return value is not derived from the argument.
Oct 23 2016
next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 23 October 2016 at 07:36:39 UTC, Walter Bright wrote:
 I can't, because I'm waiting for Andrei to make available what 
 he considers a correct ref counted design.

 But you are correct in that the 'return' rules can be used to 
 transfer the lifetime from an argument to the return value, 
 even if the return value is not derived from the argument.
Glad to hear that. Even after 1 hour thinking in circles we couldn't figure out where scope could be used without this. From my still limited understanding, the lifetime of the return value should be bound to the minimum (shortest) lifetime of all arguments that could potentially be aliased. So if the return value is a struct containing an int* and a S*, it might alias any argument containing/or being an int or S (and maybe untyped memory). This seems to be essential for any wrapper using malloced memory to safely return sth. (e.g. a range) with a pointer into that memory.
Oct 25 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 10/25/2016 5:39 PM, Martin Nowak wrote:
 From my still limited understanding, the lifetime of the return value should be
 bound to the minimum (shortest) lifetime of all arguments that could
potentially
 be aliased.
That's right.
 So if the return value is a struct containing an int* and a S*, it might alias
 any argument containing/or being an int or S (and maybe untyped memory).
If that parameter is marked "return scope", then yes.
Oct 26 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 23 October 2016 at 07:36:39 UTC, Walter Bright wrote:
 On 10/22/2016 4:22 PM, Dicebot wrote:
 Walter, would you mind showing reference counted example that 
 actually
 works under your implementation?
I can't, because I'm waiting for Andrei to make available what he considers a correct ref counted design.
We were working on that about more than a year ago when return ref landed. It didn't solve the escape problem for classes, but we have fairly solid understanding how a baseline implementation for unique, ref-counted, and weak-ref ownership should work. Since those are primitives, we should be able to use them for the design of scope w/o a concrete RC. https://github.com/dlang/phobos/pull/3259 If necessary, I can finish the rewrite to inform further discussion.
Oct 25 2016
prev sibling next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Sunday, 16 October 2016 at 13:45:42 UTC, Dicebot wrote:
 So, there is a pull request 
 (https://github.com/dlang/dmd/pull/5972)
 which is intended to implement DIP1000
 (https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md).
One thing about the DIP that I'd like more info on is: https://github.com/dlang/DIPs/blob/master/DIPs/DIP1000.md#safe
 Errors for scope violations are only reported in  safe code.
Why is this necessary / what would be the problem in allowing it in system ?
Oct 24 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 10/24/2016 9:49 AM, Mathias Lang wrote:
 Why is this necessary / what would be the problem in allowing it in  system ?
For one thing, it would require all users to learn scope semantics and annotate them correctly. It would be like requiring const correctness.
Oct 24 2016
parent Martin Nowak <code dawg.eu> writes:
On Monday, 24 October 2016 at 19:32:45 UTC, Walter Bright wrote:
 For one thing, it would require all users to learn scope 
 semantics and annotate them correctly. It would be like 
 requiring const correctness.
Could we just turn off scope inference in system code but escape check explicit scope usage?
Oct 25 2016
prev sibling next sibling parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 16 October 2016 at 13:45:42 UTC, Dicebot wrote:

To summarize the current state, the Skype call with Walter 
cleared most of the misunderstandings, DIP1000 is about 2 simple 
points.

- make scope work (fix all escaping whatsoever)

It's more complex than return ref because pointers, other than 
references, can be bound to variables, hence the lifetime algebra.

Since it was already possible to create pointers in  safe code 
(ubyte[] a
= staticArray[];), and we finally can prevent escaping of 
pointers,
DIP1000 also includes allowing to take the address of sth. in 
 safe
code, similarly to how &ary[$] is now possible (see 
https://github.com/dlang/dmd/pull/6253).

- add return scope to allow returning scoped arguments (or sth. 
derived
from them)

The current state is to conservatively assume the return value 
has the
minimum lifetime of all return scope parameters (no smartness 
about
possible aliasing of return value to parameters), even for 
 trusted
functions. This should enable RC/Unique like patterns or ranges on
containers.

- Adding scope inference for parameters and variables, we already 
have return inference from DIP25.

Inference will make this actually usable, otherwise we'd end up 
with a huge amount of required extra annotations.

====================

Open questions from my side:

- what about unwanted/incorrect return scope inference

Mixed return branches would conservatively assume a scoped return 
value?
Is this a valid use-case? Should at least work with  system code.

int* func()(int* arg, bool allocate)
{
     return allocate ? new int : arg;
}

- what about return ref without scope

Does return ref on parameters makes sense without scope?
Can you take the address of an unscoped return ref result?
Nov 22 2016
next sibling parent reply Marc =?UTF-8?B?U2Now7x0eg==?= <schuetzm gmx.net> writes:
On Tuesday, 22 November 2016 at 14:09:49 UTC, Martin Nowak wrote:
 similarly to how &ary[$] is now possible
Huh? How could this be safe? Does the resulting pointer somehow become "tainted" to avoid dereferencing it?
Nov 23 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 11/23/2016 7:32 AM, Marc Schütz wrote:
 On Tuesday, 22 November 2016 at 14:09:49 UTC, Martin Nowak wrote:
 similarly to how &ary[$] is now possible
Huh? How could this be safe? Does the resulting pointer somehow become "tainted" to avoid dereferencing it?
It's only allowed if it is the operand of a comparison operation.
Nov 24 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 11/22/2016 6:09 AM, Martin Nowak wrote:
 - what about unwanted/incorrect return scope inference

 Mixed return branches would conservatively assume a scoped return value?
 Is this a valid use-case? Should at least work with  system code.

 int* func()(int* arg, bool allocate)
 {
     return allocate ? new int : arg;
 }
?: returns the most restrictive scope of any of its operands. This applies equally as well to a function with numerous parameters that are return scope or return ref.
 - what about return ref without scope

 Does return ref on parameters makes sense without scope?
Yes, see dip25.
 Can you take the address of an unscoped return ref result?
Taking the address of a ref makes it a scope.
Dec 09 2016
next sibling parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Saturday, 10 December 2016 at 05:00:42 UTC, Walter Bright 
wrote:
 On 11/22/2016 6:09 AM, Martin Nowak wrote:
 - what about unwanted/incorrect return scope inference

 Mixed return branches would conservatively assume a scoped 
 return value?
 Is this a valid use-case? Should at least work with  system 
 code.

 int* func()(int* arg, bool allocate)
 {
     return allocate ? new int : arg;
 }
?: returns the most restrictive scope of any of its operands. This applies equally as well to a function with numerous parameters that are return scope or return ref.
Not quite: https://issues.dlang.org/show_bug.cgi?id=16037 Also, another reason why tainting the STC of an aggregate is not a good idea: struct Foo { int* ptr; } void main () safe { Foo f; escape1(f); assert(*f.ptr == 42); } void escape1 (ref Foo f) safe { int i = 42; f.ptr = &i; }
Dec 11 2016
next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
Moving from 
https://forum.dlang.org/post/o2llgd$2tuc$1 digitalmars.com

On Monday, 12 December 2016 at 08:01:19 UTC, Walter Bright wrote:
 On 12/11/2016 11:15 PM, Mathias Lang wrote:
 On Monday, 12 December 2016 at 01:05:20 UTC, Walter Bright 
 wrote:
 On 12/10/2016 6:14 AM, Mathias Lang via Dlang-internal wrote:
 Since scope class are not testable, I wouldn't say it's 
 finished.
Please be more specific.
- Due to a regression, we cannot have `scope`-allocated classes in ` safe` code (https://github.com/dlang/dmd/pull/6279).
That's both testable, and has been fixed.
? The P.R. and bug report are still open. When was it fixed ?
Dec 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2016 7:21 AM, Mathias Lang wrote:
 The P.R. and bug report are still open. When was it fixed ?
It's ready to go.
Dec 12 2016
parent Mathias Lang <mathias.lang sociomantic.com> writes:
On Monday, 12 December 2016 at 21:54:38 UTC, Walter Bright wrote:
 On 12/12/2016 7:21 AM, Mathias Lang wrote:
 The P.R. and bug report are still open. When was it fixed ?
It's ready to go.
Oh, I haven't noticed that you updated it, great! Martin: Could you make sure the scope branch is updated to include what's in master once everything gets merged back?
Dec 12 2016
prev sibling next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
```
T* foo (T) (T* arg)
{
     return arg;
}

int* escape ()  safe
{
     int b;
     return foo(&b);
}

void main ()  safe
{
     int* ptr = escape();
}
```

Compiles with `./src/dmd -transition=safe -run test.d`   
ac6f655030f814cd352a33b2c9490df16c84459d
Dec 12 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2016 11:36 AM, Mathias Lang wrote:
 ```
 T* foo (T) (T* arg)
 {
     return arg;
 }

 int* escape ()  safe
 {
     int b;
     return foo(&b);
 }

 void main ()  safe
 {
     int* ptr = escape();
 }
 ```

 Compiles with `./src/dmd -transition=safe -run test.d`  
 ac6f655030f814cd352a33b2c9490df16c84459d
Thanks. Will investigate.
Dec 12 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/12/2016 1:53 PM, Walter Bright wrote:
 Thanks. Will investigate.
I have it corrected now. Dunno which branch to push it to at the moment, will check with Martin.
Dec 13 2016
prev sibling parent reply Martin Nowak <code dawg.eu> writes:
On Monday, 12 December 2016 at 07:17:11 UTC, Mathias Lang wrote:
 Not quite: https://issues.dlang.org/show_bug.cgi?id=16037
Thanks for all the test cases, it's really the most useful approach to close all escape holes. How about tracking all scope bugs in Bugzilla using "[scope]" in the title and safe as keyword? If there are still design-concerns/unclarities about (non-)transitivity, we could benefit from a good test case illustrating the problem.
Dec 15 2016
parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Thursday, 15 December 2016 at 20:34:28 UTC, Martin Nowak wrote:
 Thanks for all the test cases, it's really the most useful 
 approach to close all escape holes. How about tracking all 
 scope bugs in Bugzilla using "[scope]" in the title and safe as 
 keyword?
IMO bugzilla is a more public setup than this NG is - people not familiar with the design of scope, or it's goal, might misinterpret it. I also doubt it would scale to more feature branches. But I have no strong feeling about it so if it fits your workflow better, I'm fine with that.
 If there are still design-concerns/unclarities about 
 (non-)transitivity, we could benefit from a good test case 
 illustrating the problem.
Lack of transitivity is one thing, inability to express something more complex than a top level scope is another. There is no way to have a scope pointer to a scope pointer to an non scope pointer. When it comes to scope, you actually want the inverse effect we have for const transitivity: the transitivity should go "up" instead of going "down". If we borrow C's syntax for `const` for a minute, you could have: - `scope * scope * int*` => scope ptr to scope ptr to non scope int ptr - `scope * scope * scope int*` => scope ptr to scope ptr to scope ptr to int - ` * scope * int*` => non scope ptr to a scope ptr to a non scope int ptr The first 2 examples are valid as long as each "level" has a lifetime <= to the previous one. What we absolutely want to disallow is the ability to have a pointer to something with `>` lifetime. With our current grammar, if scope was transitive, you could express the second example by: `scope int***` or `scope(scope(scope(int*)*)*)`, or any variation. However, there would be no ability to express the first, valid case. Obviously we can put a blanket ban on those cases, but to me they sound valid. Now, a more practical issue, once again related to changing the STC of some variable mid-way through a function semantic analysis: ``` struct Foo { int* ptr; } void main () safe { int* a = escape(); } int* escape () safe { int i; Foo f; scope int** x = &f.ptr; f.ptr = &i; return bar(x); } int* bar ( scope int** ptr ) safe { return *ptr; } ``` Compiles with `./src/dmd -transition=safe -dip25 test.d` on commit fbbfce8f6e0afb716ec96ba3dc2e82b3536cbaac (with the latest 2 P.R. merged).
Dec 16 2016
next sibling parent reply Mathias Lang <mathias.lang sociomantic.com> writes:
On Friday, 16 December 2016 at 16:43:30 UTC, Mathias Lang wrote:
 Lack of transitivity is one thing, inability to express 
 something more complex than a top level scope is another.
Another, hopefully even better test case for what I'm trying to express: ``` void main () safe { int* a = escape(); } int* escape () safe { int i; int*[3] a = [ &i, null, null ]; return bar(&a[0]); } int* bar (scope int** x) safe { return foo(*x); } int* foo (int* x) safe { return x; } ``` As long as we have: - No way to express different `scope` levels (the "backward" transitivity) - A way to take the address of `scope` pointers We will still have that hole.
 Compiles with `./src/dmd -transition=safe -dip25 test.d` on 
 commit fbbfce8f6e0afb716ec96ba3dc2e82b3536cbaac (with the 
 latest 2 P.R. merged).
Dec 16 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/16/2016 9:08 AM, Mathias Lang wrote:
 As long as we have:
 - A way to take the address of `scope` pointers
Taking the address of a scope pointer is invalid, I'll fix the method you used to do it.
Dec 16 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Saturday, 17 December 2016 at 07:29:17 UTC, Walter Bright 
wrote:
 On 12/16/2016 9:08 AM, Mathias Lang wrote:
 As long as we have:
 - A way to take the address of `scope` pointers
Taking the address of a scope pointer is invalid, I'll fix the method you used to do it.
What is the reasoning for disallowing that? Is dealing with second order indirections (scope int**) somehow different from from other escape checks?
Dec 17 2016
next sibling parent Martin Nowak <code dawg.eu> writes:
On Sunday, 18 December 2016 at 04:03:55 UTC, Martin Nowak wrote:
 Taking the address of a scope pointer is invalid, I'll fix the 
 method you used to do it.
What is the reasoning for disallowing that? Is dealing with second order indirections (scope int**) somehow different from from other escape checks?
Also see https://github.com/dlang/dmd/pull/6328/files#r92933626 on that question.
Dec 17 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 8:03 PM, Martin Nowak wrote:
 On Saturday, 17 December 2016 at 07:29:17 UTC, Walter Bright wrote:
 On 12/16/2016 9:08 AM, Mathias Lang wrote:
 As long as we have:
 - A way to take the address of `scope` pointers
Taking the address of a scope pointer is invalid, I'll fix the method you used to do it.
What is the reasoning for disallowing that?
scope is not transitive, and there's no way to declare a scope pointer to a scope pointer.
 Is dealing with second order indirections (scope int**) somehow different from
 from other escape checks?
Yes. Only the top level is checked. By disallowing taking the address of a scope variable, only the top level needs to be checked.
Dec 17 2016
parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/18/2016 06:59 AM, Walter Bright wrote:
 On 12/17/2016 8:03 PM, Martin Nowak wrote:
 On Saturday, 17 December 2016 at 07:29:17 UTC, Walter Bright wrote:
 On 12/16/2016 9:08 AM, Mathias Lang wrote:
 As long as we have:
 - A way to take the address of `scope` pointers
Taking the address of a scope pointer is invalid, I'll fix the method you used to do it.
What is the reasoning for disallowing that?
scope is not transitive, and there's no way to declare a scope pointer to a scope pointer.
That's not really a reason, it's a symptom of the implementation and design ;). Let me suggest a reason, transitive scope checking would be more complex to implement, is it that?
 Is dealing with second order indirections (scope int**) somehow
 different from
 from other escape checks?
Yes. Only the top level is checked. By disallowing taking the address of a scope variable, only the top level needs to be checked.
The biggest limitation of return ref (DIP25) was that there was no way to obtain indirections (e.g. a named pointer variable). Hence you'd always need to write (and execute) a.b.c.d.e, with no way to store temporary results (tmp = a.b.c; tmp.d.e). No we get 1, but only 1, level of indirections (`tmp = &a.b.c; tmp.d.e`). Two questions that arise from that. - How certain are we that this is enough? In other words, why is `tmp = &a.b; tmp2 = &tmp.c; tmp2.d.e` not necessary. - Let's imagine for a short moment, what it would mean to make scope transitive at a later point, if we'd ever find enough reasons to do so. Maximizing future options is always good. - Allowing `&scoped` would be simple b/c it doesn't compile atm. - Changing `lifetime(*p) = ∞` to `lifetime(*p) = p` later, could be harder I guess. How restricting would it be to already add it now? - The requirement for `ptr = &var` that `lifetime(var) >= reachability(ptr)` would remain the same. Hope I fully got the point, I'm not that much into the details of DIP1000, so please correct me if that's nonsense.
Dec 18 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 9:28 AM, Martin Nowak wrote:
 Let me suggest a reason, transitive scope checking would be more complex
 to implement, is it that?
It's a heluva lot more complicated for the user. And since it is not necessary, why pay such a large bill?
 - How certain are we that this is enough? In other words, why is `tmp =
 &a.b; tmp2 = &tmp.c; tmp2.d.e` not necessary.
It would be necessary in order to build non-trivial data structures on the stack. How necessary is that? I've never needed to in decades of programming. I don't think there are any instances in Phobos or dmd. If one really wants to do that, there's always system code. Is it worth burdening everything with a feature when the use case for it is rare? I don't think so. For example, consider 'volatile' in C++. The use case for it is rare (very rare). But there's an enormous amount of spec verbiage and thought dedicated to how to weave it into the type system, overloading rules, conversion rules, casting rules, promotion rules, mangling rules, deduction rules, on and on. It's just not worth it.
 - Let's imagine for a short moment, what it would mean to make scope
 transitive at a later point, if we'd ever find enough reasons to do so.
 Maximizing future options is always good.
This kind of annotation system was first proposed 10 years ago. I've been thinking about how to make it simpler ever since, and the original has not improved with age. I also have a lot of experience with transitive const, and how disruptive that was, and how many years it took to work the problems out (Andrei says they're still a problem).
Dec 18 2016
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
BTW, Rust is a system with annotations that can be applied at any level. Yet it 
still needs  system code for many data structures (any with cycles in it).

Rust's system also (so I've read) requires one to re-architect data structures. 
It's not remotely a matter of just adding annotations.

That alone makes it unacceptable for D.

D's 'return scope' system covers routine coding use. More complex things can be 
encapsulated in  trusted code. I believe this is the sweet spot that will make 
it easy to transition to, use and be effective.

---

It's true that I've repeatedly had problems explaining how it works to you, 
Andrei, Dicebot and Mathias. I can only infer that having a much more complex 
system, which scope as a type constructor would be, would be infinitely worse.
I 
find the prospect of debugging a data structure with 'scope' sprinkled around
on 
random edges in it to be terrifying (as well as a combinatorial template 
instantiation explosion problem).
Dec 18 2016
prev sibling parent Mathias Lang <mathias.lang sociomantic.com> writes:
On Sunday, 18 December 2016 at 17:56:26 UTC, Walter Bright wrote:
 On 12/18/2016 9:28 AM, Martin Nowak wrote:
 Let me suggest a reason, transitive scope checking would be 
 more complex
 to implement, is it that?
It's a heluva lot more complicated for the user. And since it is not necessary, why pay such a large bill?
 - How certain are we that this is enough? In other words, why 
 is `tmp =
 &a.b; tmp2 = &tmp.c; tmp2.d.e` not necessary.
It would be necessary in order to build non-trivial data structures on the stack. How necessary is that? I've never needed to in decades of programming. I don't think there are any instances in Phobos or dmd. If one really wants to do that, there's always system code. Is it worth burdening everything with a feature when the use case for it is rare? I don't think so. For example, consider 'volatile' in C++. The use case for it is rare (very rare). But there's an enormous amount of spec verbiage and thought dedicated to how to weave it into the type system, overloading rules, conversion rules, casting rules, promotion rules, mangling rules, deduction rules, on and on. It's just not worth it.
 - Let's imagine for a short moment, what it would mean to make 
 scope
 transitive at a later point, if we'd ever find enough reasons 
 to do so.
 Maximizing future options is always good.
This kind of annotation system was first proposed 10 years ago. I've been thinking about how to make it simpler ever since, and the original has not improved with age. I also have a lot of experience with transitive const, and how disruptive that was, and how many years it took to work the problems out (Andrei says they're still a problem).
Thank you so much for this explanation. It clears up many of the points on which I had questions. Once I run out of simple test cases, I'll try to toy around with a more advanced RefCounted design.
Dec 22 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/16/2016 9:08 AM, Mathias Lang wrote:
 void main ()  safe
 {
     int* a = escape();
 }

 int* escape ()  safe
 {
     int i;
     int*[3] a = [ &i, null, null ];
     return bar(&a[0]);
 }

 int* bar (scope int** x)  safe
 {
     return foo(*x);
 }

 int* foo (int* x)  safe { return x; }
https://github.com/dlang/dmd/pull/6329
Dec 17 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Saturday, 17 December 2016 at 14:09:59 UTC, Walter Bright 
wrote:
 On 12/16/2016 9:08 AM, Mathias Lang wrote:
 void main ()  safe
 {
     int* a = escape();
 }

 int* escape ()  safe
 {
     int i;
     int*[3] a = [ &i, null, null ];
     return bar(&a[0]);
 }

 int* bar (scope int** x)  safe
 {
     return foo(*x);
 }

 int* foo (int* x)  safe { return x; }
https://github.com/dlang/dmd/pull/6329
Merged both of your P.R. in a local branch, and tested again: ``` void main () safe { int[] a = escape(); } int[] escape () safe { Foo f; return f.foo; } struct Foo { int[10] v; int[] foo () return safe { return this.v; } } ```
 Taking the address of a scope pointer is invalid
Thanks for the clarification. I think this will prove pretty limiting in the future. For example, a stack allocator will return data that is already typed as `scope`, and as a consequence, interior pointers will not be expressible. I'll see if I can form an example in terms of code. And since I started toying around with aggregate methods, here's another reason why changing the STC of a field member is a very bad idea: ``` void main () safe { int* a = escape(); } int* escape () safe { int i; Foo f; f.v = &i; return f.foo; } struct Foo { int* v; int* foo () safe { return this.v; } } ``` Note: The definition of `Foo` might be in another module, and just visible via an header (.di) file.
Dec 17 2016
next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 7:38 AM, Mathias Lang wrote:
 Taking the address of a scope pointer is invalid
Thanks for the clarification. I think this will prove pretty limiting in the future. For example, a stack allocator will return data that is already typed as `scope`, and as a consequence, interior pointers will not be expressible. I'll see if I can form an example in terms of code.
It should work fine.
 And since I started toying around with aggregate methods, here's another reason
 why changing the STC of a field member is a very bad idea:
It doesn't change the STC of field members. Only of locals. --- I'll investigate the reports. Thanks!
Dec 17 2016
parent reply Martin Nowak <code dawg.eu> writes:
On Sunday, 18 December 2016 at 02:43:59 UTC, Walter Bright wrote:

Just a general remark, not specific to the content of this 
conversation.

Being a bit more specific in answers easily makes them less 
misunderstandable.

 On 12/17/2016 7:38 AM, Mathias Lang wrote:
 For example, a stack allocator will return data that is 
 already typed as
 `scope`, and as a consequence, interior pointers will not be 
 expressible. I'll
 see if I can form an example in terms of code.
It should work fine.
What does "it" refer to here, doesn't hurt much to write "Internal pointers in scope structs" or so. Equally "work fine" doesn't state how the "it" is supposed to work.
 And since I started toying around with aggregate methods, 
 here's another reason
 why changing the STC of a field member is a very bad idea:
It doesn't change the STC of field members. Only of locals.
What's it here? Your DIP or the dmd implementation?
Dec 17 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 7:56 PM, Martin Nowak wrote:
 On 12/17/2016 7:38 AM, Mathias Lang wrote:
 For example, a stack allocator will return data that is already typed as
 `scope`, and as a consequence, interior pointers will not be expressible. I'll
 see if I can form an example in terms of code.
It should work fine.
What does "it" refer to here, doesn't hurt much to write "Internal pointers in scope structs" or so. Equally "work fine" doesn't state how the "it" is supposed to work.
A stack allocator is conceptually returning a slice of a static array allocated on the stack. This works with scope. Understanding scope always goes back to understanding how it works with pointers and addresses. Adding layers of abstraction over that can be confusing by obfuscating that relationship, but rewrite the abstract as pointers and addresses, and the behavior then becomes clear. The bugs Mathias has posted have (so far) all been the result of an abstraction not conforming to how pointers and addresses work, due to a bug in dmd.
 And since I started toying around with aggregate methods, here's another reason
 why changing the STC of a field member is a very bad idea:
It doesn't change the STC of field members. Only of locals.
What's it here? Your DIP or the dmd implementation?
The dmd implementation.
Dec 17 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Sunday, 18 December 2016 at 05:56:42 UTC, Walter Bright wrote:
 On 12/17/2016 7:56 PM, Martin Nowak wrote:
 On 12/17/2016 7:38 AM, Mathias Lang wrote:
 For example, a stack allocator will return data that is 
 already typed as
 `scope`, and as a consequence, interior pointers will not be 
 expressible. I'll
 see if I can form an example in terms of code.
It should work fine.
What does "it" refer to here, doesn't hurt much to write "Internal pointers in scope structs" or so. Equally "work fine" doesn't state how the "it" is supposed to work.
A stack allocator is conceptually returning a slice of a static array allocated on the stack. This works with scope. Understanding scope always goes back to understanding how it works with pointers and addresses. Adding layers of abstraction over that can be confusing by obfuscating that relationship, but rewrite the abstract as pointers and addresses, and the behavior then becomes clear. The bugs Mathias has posted have (so far) all been the result of an abstraction not conforming to how pointers and addresses work, due to a bug in dmd.
Thanks for expanding on that, as Martin pointed out, it helps reviewers a lot. I don't feel the point about internal pointers to stack-allocated data have been addressed, but maybe it'll be better if I just craft a test case.
 https://github.com/dlang/dmd/pull/6331
The problem with this solution is that it forces `scope` on unrelated method. E.g. ``` int* escape () safe { int i; Foo f; f.v = &i; return f.foo; } struct Foo { int* v; int* foo () safe { return null; } } ``` Doesn't compile anymore. It might be your intention, but it needs to be clearly expressed in the DIP. Moreover, this code is obviously correct to the reader: ``` struct Foo { int* v; int foo () safe { return *v; } } ``` But won't compile. I tried with static arrays just to be sure, same outcome.
Dec 18 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 3:04 AM, Mathias Lang wrote:
 The problem with this solution is that it forces `scope` on unrelated method.
 E.g.

 ```
 int* escape ()  safe
 {
     int i;
     Foo f;
     f.v = &i;
     return f.foo;
 }

 struct Foo
 {
     int* v;
     int* foo ()  safe { return null; }
 }
 ```

 Doesn't compile anymore.
And it shouldn't. f contains a pointer to a local, and so f must be scope. A scoped object can only be used on a scoped method, because otherwise the method can squirrel away an escaping copy.
 It might be your intention, but it needs to be clearly
 expressed in the DIP.
The rules need to be clearly expressed, but I don't know how to express every consequence of the rules.
 Moreover, this code is obviously correct to the reader:

 ```
 struct Foo
 {
     int* v;
     int foo ()  safe { return *v; }
 }
 ```

 But won't compile. I tried with static arrays just to be sure, same outcome.
It compiles when I try it.
Dec 18 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Sunday, 18 December 2016 at 15:24:34 UTC, Walter Bright wrote:
 Moreover, this code is obviously correct to the reader:

 ```
 struct Foo
 {
     int* v;
     int foo ()  safe { return *v; }
 }
 ```

 But won't compile. I tried with static arrays just to be sure, 
 same outcome.
It compiles when I try it.
I didn't meant as a standalone example, but in the context of the previous snippet. Full code: ``` int bar () safe { int i = 42; Foo f; f.v = &i; return f.foo(); // Doesn't compile } struct Foo { int* v; int foo () safe { return *v; } } ```
Dec 18 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 8:23 AM, Mathias Lang wrote:
 I didn't meant as a standalone example, but in the context of the previous
 snippet. Full code:

 ```
 int bar ()  safe
 {
     int i = 42;
     Foo f;
     f.v = &i;
     return f.foo(); // Doesn't compile
 }

 struct Foo
 {
     int* v;
     int foo ()  safe { return *v; }
 }
 ```
That doesn't compile because the Foo could be written as: static int* s; struct Foo { int* v; int foo () safe { s = v; return *v; } } The compiler can't know that from the interface to Foo.foo, and so must assume the worst. The safety features are about guarantees, not having faith that the programmer didn't do something like that. The purpose in adding the 'scope' annotation to Foo.foo is to tell the compiler that there's no hanky-panky going on in the implementation. When the compiler does compile the implementation, it can then check it against the 'scope' semantics.
Dec 18 2016
prev sibling next sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 7:38 AM, Mathias Lang wrote:
 void main ()  safe
 {
     int[] a = escape();
 }

 int[] escape ()  safe
 {
     Foo f;
     return f.foo;
 }

 struct Foo
 {
     int[10] v;
     int[] foo () return  safe { return this.v; }
 }
I've said it's all about pointers and addresses. So let's remove the abstractions, and rewrite it in the equivalent pointers and addresses form: ``` void main () safe { int* a = escape(); } int* escape () safe { int f; return foo(f); // Error: escaping reference to local variable f } int* foo (return ref int i) safe { return &i; } ``` and indeed it does fail to compile. So the bug in the compiler is failing to recognize the abstract version of the pointers and addresses logic.
Dec 17 2016
parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 10:29 PM, Walter Bright wrote:
 So the bug in the compiler is failing to
 recognize the abstract version of the pointers and addresses logic.
https://github.com/dlang/dmd/pull/6330
Dec 17 2016
prev sibling parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 7:38 AM, Mathias Lang wrote:
 int* escape ()  safe
 {
     int i;
     Foo f;
     f.v = &i;
     return f.foo;
 }

 struct Foo
 {
     int* v;
     int* foo ()  safe { return this.v; }
 }
Rewriting to the equivalent: ``` int* escape() safe { int i; Foo f; f.v = &i; return foo(g); // Error: scope variable f assigned to non-scope // parameter g calling bug.foo } struct Foo { int* v; } int* foo(ref Foo g) safe { return g.v; } ``` So the trouble here is dmd not recognizing that f.foo() is the same as foo(f).
Dec 17 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/17/2016 11:47 PM, Walter Bright wrote:
 So the trouble here is dmd not recognizing that f.foo() is the same as foo(f).
https://github.com/dlang/dmd/pull/6331
Dec 18 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Sunday, 18 December 2016 at 10:04:57 UTC, Walter Bright wrote:
 On 12/17/2016 11:47 PM, Walter Bright wrote:
 So the trouble here is dmd not recognizing that f.foo() is the 
 same as foo(f).
https://github.com/dlang/dmd/pull/6331
With the two last P.R. you submitted merged, the following compiles ``` void escape () safe { Foo f; bar(f); } void bar (scope ref Foo f) safe { int[10] i; f.v = i; } struct Foo { int[] v; } ```
Dec 18 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/18/2016 3:09 AM, Mathias Lang wrote:
 With the two last P.R. you submitted merged, the following compiles

 ```
 void escape ()  safe
 {
     Foo f;
     bar(f);
 }

 void bar (scope ref Foo f)  safe
 {
     int[10] i;
     f.v = i;
 }

 struct Foo
 {
     int[] v;
 }
 ```
Thanks, I'll look at it tomorrow.
Dec 18 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Sunday, 18 December 2016 at 15:25:55 UTC, Walter Bright wrote:
 On 12/18/2016 3:09 AM, Mathias Lang wrote:
 With the two last P.R. you submitted merged, the following 
 compiles

 ```
 void escape ()  safe
 {
     Foo f;
     bar(f);
 }

 void bar (scope ref Foo f)  safe
 {
     int[10] i;
     f.v = i;
 }

 struct Foo
 {
     int[] v;
 }
 ```
Thanks, I'll look at it tomorrow.
Just tested again with the HEAD of scope (e46873f66), it compiles with just `ref` instead of `scope ref` for bar's parameter. Which brings me to the question, is `scope ref` supposed to apply to the `ref` or to the type (i.e. `Foo` in this case) ?
Dec 27 2016
parent reply Walter Bright <newshound2 digitalmars.com> writes:
On 12/27/2016 2:18 PM, Mathias Lang wrote:
 Which brings me to the question, is `scope ref` supposed to apply to the `ref`
 or to the type (i.e. `Foo` in this case) ?
scope refers to the parameter's value, ref refers to the parameter's address. The two are independent of each other.
Dec 27 2016
parent reply Mathias Lang <pro.mathias.lang gmail.com> writes:
On Wednesday, 28 December 2016 at 01:25:22 UTC, Walter Bright 
wrote:
 On 12/27/2016 2:18 PM, Mathias Lang wrote:
 Which brings me to the question, is `scope ref` supposed to 
 apply to the `ref`
 or to the type (i.e. `Foo` in this case) ?
scope refers to the parameter's value, ref refers to the parameter's address. The two are independent of each other.
Thanks (Y) I just started testing methods, but didn't get far, because the following doesn't compile: ``` void getObj () safe { scope o = new Bar; auto p = o.foo; // Line 9 } class Bar { Object foo () return scope safe { return this; } } // scope1.d(9): Error: scope variable o assigned to non-scope parameter this calling scope1.Bar.foo ``` However making `foo` a function which accept a class works: ``` Object foo (return scope Object this_) safe { return this_; } ``` Also, it looks like delegates to member functions are not marked scope where they should. The following compiles, but obviously should not since the context pointer of the delegate will be the instance `b`, which is on the stack. ``` void main () safe { fwd(); auto x1 = Generator1(); auto x2 = Generator2(); } int delegate() safe Generator1; int delegate() safe Generator2; struct G1 { int generator () safe { return 42; } } class G2 { int generator () safe { return 42; } } void fwd () safe nogc { G1 g1; scope g2 = new G2; Generator1 = &g1.generator; Generator2 = &g2.generator; } ```
Dec 30 2016
next sibling parent Walter Bright <newshound2 digitalmars.com> writes:
Thanks, I'll look into these.
Dec 30 2016
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/30/2016 01:12 PM, Mathias Lang wrote:

Can we please start to track all DIP1000 implementation bugs on
Bugzilla. It's becoming quite chaotic to keep an overview of this.
Until someone creates a scope or dip1000 keyword (does  braddr maintain
Bugzilla), we can use [scope] in the title, that's search-able.

https://issues.dlang.org/buglist.cgi?list_id=212784&query_format=advanced&resolution=---&short_desc=%5Bscope%5D&short_desc_type=substring
Dec 31 2016
next sibling parent Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 12/31/2016 05:18 PM, Martin Nowak wrote:
 On 12/30/2016 01:12 PM, Mathias Lang wrote:
 
 Can we please start to track all DIP1000 implementation bugs on
 Bugzilla. It's becoming quite chaotic to keep an overview of this.
 Until someone creates a scope or dip1000 keyword (does  braddr maintain
 Bugzilla), we can use [scope] in the title, that's search-able.
 
 https://issues.dlang.org/buglist.cgi?list_id=212784&query_format=advanced&resolution=---&short_desc=%5Bscope%5D&short_desc_type=substring
Just filed the first one, https://issues.dlang.org/show_bug.cgi?id=17049.
Dec 31 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 31 December 2016 at 16:18:11 UTC, Martin Nowak wrote:
 https://issues.dlang.org/buglist.cgi?list_id=212784&query_format=advanced&resolution=---&short_desc=%5Bscope%5D&short_desc_type=substring
Let's use the existing keyword (safe) AND the [scope] tag in the title.
Dec 31 2016
prev sibling parent Walter Bright <newshound2 digitalmars.com> writes:
On 12/16/2016 8:43 AM, Mathias Lang wrote:
 ```
 struct Foo { int* ptr; }

 void main ()  safe
 {
     int* a = escape();
 }

 int* escape ()  safe
 {
     int i;
     Foo f;
     scope int** x = &f.ptr;
     f.ptr = &i;

     return bar(x);
 }

 int* bar ( scope int** ptr )  safe
 {
   return *ptr;
 }
 ```

 Compiles with `./src/dmd -transition=safe -dip25 test.d` on commit
 fbbfce8f6e0afb716ec96ba3dc2e82b3536cbaac (with the latest 2 P.R. merged).
https://github.com/dlang/dmd/pull/6328
Dec 17 2016
prev sibling parent Martin Nowak <code dawg.eu> writes:
On Saturday, 10 December 2016 at 05:00:42 UTC, Walter Bright 
wrote:
 - what about return ref without scope

 Does return ref on parameters makes sense without scope?
Yes, see dip25.
 Can you take the address of an unscoped return ref result?
Taking the address of a ref makes it a scope.
Thanks, that was likely the piece of information I was missing. So even with all the scope changes, a return ref parameter provides the same guarantees, it can't be escaped in a safe function.
Dec 15 2016
prev sibling parent reply Martin Nowak <code+news.digitalmars dawg.eu> writes:
On 10/16/2016 03:45 PM, Dicebot wrote:
 Trying to solve it, I have been experimenting lately with various
 acceptance tests trying to come up with useful design snippets enabled
 by that PR. Rationale is that if actual semantics are clear and examples
 are all validated as included test cases, review of the code itself
 becomes less important.
 
 I'd like to have more open feedback about it because in its current
 shape PR5972 doesn't seem useful enough to me to be merged at all - and
 much less powerful than DIP1000 seemed to promise.
 
 I'll start separate sub-threads with more details.
Just build a scope preview, http://nightlies.dlang.org/dmd-scope/. You can just download it or use curl -fsS https://dlang.org/install.sh | bash -s dmd-scope to try it (also works on Travis-CI anything else using the install.sh script). -Martin
Dec 19 2016
parent reply ZombineDev <petar.p.kirov gmail.com> writes:
On Tuesday, 20 December 2016 at 00:15:23 UTC, Martin Nowak wrote:
 On 10/16/2016 03:45 PM, Dicebot wrote:
 Trying to solve it, I have been experimenting lately with 
 various acceptance tests trying to come up with useful design 
 snippets enabled by that PR. Rationale is that if actual 
 semantics are clear and examples are all validated as included 
 test cases, review of the code itself becomes less important.
 
 I'd like to have more open feedback about it because in its 
 current shape PR5972 doesn't seem useful enough to me to be 
 merged at all - and much less powerful than DIP1000 seemed to 
 promise.
 
 I'll start separate sub-threads with more details.
Just build a scope preview, http://nightlies.dlang.org/dmd-scope/. You can just download it or use curl -fsS https://dlang.org/install.sh | bash -s dmd-scope to try it (also works on Travis-CI anything else using the install.sh script). -Martin
That's great Martin! What's the delay between a PR merged in https://github.com/dlang/dmd/tree/scope and being available for testing via the installer? Is it a nightly build of the scope branch, as the URL suggests?
Dec 21 2016
parent Martin Nowak <code dawg.eu> writes:
On Wednesday, 21 December 2016 at 13:48:11 UTC, ZombineDev wrote:
 That's great Martin! What's the delay between a PR merged in 
 https://github.com/dlang/dmd/tree/scope and being available for 
 testing via the installer? Is it a nightly build of the scope 
 branch, as the URL suggests?
Preview builds of branches are manually build at the moment (whenever there is sth. interesting). Nightlies are only for master atm.
Dec 23 2016