www.digitalmars.com         C & C++   DMDScript  

digitalmars.D - Proposal for design of 'scope' (Was: Re: Opportunities for D)

reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
Moving this to a new thread as suggested by Jacob.

On Wed, Jul 09, 2014 at 04:57:01PM -0700, H. S. Teoh via Digitalmars-d wrote:

 On Wed, Jul 09, 2014 at 03:16:37PM -0700, H. S. Teoh via Digitalmars-d wrote:
 [...]

 	https://issues.dlang.org/show_bug.cgi?id=13085


 [...]
 
 Hmm, apparently, this is a long-standing known issue:
 
 	https://issues.dlang.org/show_bug.cgi?id=5270
 
 Judging from this, a big missing piece of the current implementation is
 the actual enforcement of 'scope'.
 
 So here's a first stab at refining (and extending) what 'scope' should
 be:
 
 - 'scope' can be applied to any variable, and is part of its type. Let's
   call this a "scoped type", and a value of this type a "scoped value".
 
 - Every scoped type has an associated lifetime, which is basically the
   scope in which it is declared.
 
    - The lifetime of a scoped variable is PART OF ITS TYPE.
 
    - An unscoped variable is regarded to have infinite lifetime.
 
    - For function parameters, this lifetime is the scope of the function
      body.
 
    - For local variables, the lifetime is the containing lexical scope
      where it is declared.
 
 - Taking the address of a scoped value returns a scoped pointer, whose
   lifetime is the lexical scope where the address-of operator is used.
 
 - A scoped type can only be assigned to another scoped type of identical
   or narrower lifetime.  Basically, the idea here is that a scoped value
   can only have its scope narrowed, never expanded. In practice, this
   means:
 
    - If a scoped type is a reference type (class or pointer or ref), it
      can only be assigned to another scoped type whose associated
      lifetime is equal or contained within the source value's associated
      lifetime.
 
    - If a scoped type is a value type with indirections, it can only be
      assigned to an lvalue of the same scoped type (with the same
      associated lifetime).
 
    - If a scoped type is a value type with no indirections, it's freely
      assignable to a non-scoped lvalue of compatible type.
 
    - A function's return type can be scoped (not sure what syntax to use
      here, since it may clash with scope delegates).
 
       - The lifetime of the return value is the containing scope of the
         function definition -- if it's a module-level function, then its
         lifetime is infinite. If it's an inner function, then its
         lifetime is the containing lexical scope of its definition.
         Example:
 
             class C {}
             void func() {
                 // return type of helper is scope(C) with lifetime up to
                 // the end of func's body.
                 scope(C) helper() { ... }
             }
 
       - Returning a value from a function is considered to be equivalent
         to assigning the value to a variable of the return type of the
         function. Thus:
 
             class C {}
             void func() {
                 scope(C) c1;
 
                 // helper's return type has lifetime == func's body
                 scope(C) helper() {
                     scope(C) c2;
                     if (cond)
                         return c1; // OK, c1's lifetime == func's body
                     else
                         return c2; // ILLEGAL: c2's lifetime < func's body
                 }
             }
 
       - Since a scoped return type has its lifetime as part of its type,
         the type system ensures that scoped values never escape their
         lifetime. For example, if we are sneaky and return a pointer to
         an inner function, the type system will prevent leakage of the
         scoped value:
 
             class C {}
             auto func() {
                 scope(C) c1;
 
                 // Return type of sneaky is scope(C) with lifetime =
                 // body of func.
                 scope(C) sneaky() {
                     // This is OK, because c1's lifetime == body of
                     // func, which is compatible with its return type.
                     return c1;
                 }
 
                 // Aha! we now we have broken scope... or have we?
                 return &sneaky;
             }
 
             void main() {
                 // Let's see. Get a function pointer to a function that
                 // "leaks" a scoped value...
                 auto funcptr = func();
 
                 // But this doesn't compile, because the return type of
                 // funcptr() is a scoped variable whose lifetime is
                 // inside the body of func(), but since we're outside of
                 // func here, the lifetime of x doesn't match the
                 // lifetime of funcptr()'s return value, so the
                 // following assignment is rejected as having
                 // incompatible types:
                 auto x = funcptr();
 
                 // This will actually work... but it's OK, because we
                 // aren't actually storing the return value of
                 // funcptr(), so the scoped value is actually not leaked
                 // after all.
                 funcptr();
             }
 
 - Aggregates:
 
    - It's turtles all the way down: members of scoped aggregates also
      have scoped type, with lifetime inherited from the parent
      aggregate. In other words, the lifetime of the aggregate is
      transitive to the lifetime of its members. For example:
 
         class C {}
         struct S {
             C c;
             int x;
         }
         int func(scope S s) {
             // N.B. lifetime of s is func's body.
             auto c1 = s.c; // typeof(c1) == scope(C) with lifetime = func's
body
             C d;
             d = c1; // illegal: c1 has shorter lifetime than d.
             return s.x; // OK, even though typeof(s.x) has lifetime =
                         // func's body, it's a value type so we're
                         // actually copying it to func's return value,
                         // not returning the actual scoped int.
         }
 
 - Passing parameters: since unscoped values are regarded to have
   infinite lifetime, it's OK to pass unscoped values into scoped
   function parameters: it's a narrowing of lifetime of the original
   value, which is allowed. (What's not allowed is expanding the lifetime
   of a scoped value.)
 
 I'm sure there are plenty of holes in this proposal, so destroy away.
 ;-)
 
 
 T
 
 -- 
 If I were two-faced, would I be wearing this one? -- Abraham Lincoln



T

-- 
All problems are easy in retrospect.
Jul 10 2014
next sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jul 10, 2014 at 01:56:39PM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 10/07/14 01:57, H. S. Teoh via Digitalmars-d wrote:
 

[...]
I'm sure there are plenty of holes in this proposal, so destroy away.
;-)


 
 You should post this in a new thread.


Done.



 I'm wondering if a lot more data can be statically allocated. Then
 passed by reference to functions taking scope parameters. This should
 be safe since the parameter is guaranteed to outlive the function
 call.


Yep. In whatever shape or form we finally implement 'scope', it would
allow us to address such issues.


On Thu, Jul 10, 2014 at 01:26:54PM +0000, Wyatt via Digitalmars-d wrote:

 On Wednesday, 9 July 2014 at 23:58:39 UTC, H. S. Teoh via Digitalmars-d
 wrote:

So here's a first stab at refining (and extending) what 'scope'
should be:


 In general, I like it, but can scopedness be inferred?  The impression
 I get from this is we're supposed to manually annotate every scoped
 everything, which IMO kind of moots the benefits in a broad sense.


I think in some cases it can be inferred. For example, taking the
address of a local variable should return an appropriately-scoped
pointer type, so that the type system can statically verify that it
doesn't leak past the variable's lifetime:

	int* myFunc(int* q) {
		int x;
		auto ptr = &x;	// typeof(ptr) == scoped(int*) with lifetime == x.lifetime
		if (cond)
			return q;	// OK, q has infinite lifetime
		else
			return ptr;	// compile error: cannot convert scoped(int*) to int*
	}

There is an interesting subtlety here, in that local variables
themselves are not necessarily scoped, for example:

	class C {}
	C createObj() {
		auto obj = new C;	// obj is a local variable
		return obj;		// but its lifetime can be extended outside the function
	}

Yet addresses of local variables are scoped:

	class C {}
	C* createObj() {
		auto obj = new C;
		return &obj;	// INVALID: this would allow the caller
				// to access a local variable that's
				// gone out of scope
	}

This leads to some further complications, for example:

	class C {
		C* getRef() { return &this; }
	}
	C* func(scope C obj) {
		// In here, obj's lifetime is the body of func

		// But this violates obj's lifetime:
		return obj.getRef();
	}

The problem is, how do we statically prevent this sort of abuse? since
the definition of C.getRef may not know anything about func, and so
can't possibly return a type whose lifetime is restricted to func, yet
that's what's needed to prevent the invalid return of obj.getRef's value
outside func.



 If it _cannot_ be inferred (even if imperfectly), then I wonder if it
 doesn't make more sense to invert the proposed default and require
 annotation when scope restrictions need to be eased.


The way I see it, scope is really a way for a function to state that it
won't keep a persistent reference to the scoped parameter, and so it's
OK to pass in, say, a reference to a temporary that will disappear after
the function returns. Defaulting to scope will cause far too much
breakage, I think, esp. given that code like the following is probably
common in the wild:

	class C {}
	C myFunc(C obj) {
		obj.doSomething();
		return obj; // will be rejected if parameters are scoped by default
	}


T

-- 
One disk to rule them all, One disk to find them. One disk to bring them all
and in the darkness grind them. In the Land of Redmond where the shadows lie.
-- The Silicon Valley Tarot
Jul 10 2014
next sibling parent "Meta" <jared771 gmail.com> writes:
On Thursday, 10 July 2014 at 18:16:57 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 There is an interesting subtlety here, in that local variables
 themselves are not necessarily scoped, for example:

 	class C {}
 	C createObj() {
 		auto obj = new C;	// obj is a local variable
 		return obj;		// but its lifetime can be extended outside the 
 function
 	}


They are scoped, but to the lifetime of the GC, which is the 
length of the program. Anything that the GC owns can safely be 
marked scope(static) or however you would write it.
Jul 10 2014
prev sibling parent reply Jacob Carlborg <doob me.com> writes:
On 10/07/14 20:15, H. S. Teoh via Digitalmars-d wrote:


 	class C {}
 	C myFunc(C obj) {
 		obj.doSomething();
 		return obj; // will be rejected if parameters are scoped by default
 	}


Hmm, why wouldn't that work? The scope where you called "myFunc" is 
guaranteed to outlive "myFunc".

-- 
/Jacob Carlborg
Jul 10 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jul 11, 2014 at 08:56:10AM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 10/07/14 20:15, H. S. Teoh via Digitalmars-d wrote:
 

	class C {}
	C myFunc(C obj) {
		obj.doSomething();
		return obj; // will be rejected if parameters are scoped by default
	}


 
 Hmm, why wouldn't that work? The scope where you called "myFunc" is
 guaranteed to outlive "myFunc".


[...]

Because the scope of the parameter 'obj' is defined to be the scope of
myFunc only, according to the current proposal.


T

-- 
What are you when you run out of Monet? Baroque.
Jul 11 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 2014-07-11 16:29, H. S. Teoh via Digitalmars-d wrote:


 Because the scope of the parameter 'obj' is defined to be the scope of
 myFunc only, according to the current proposal.


Wouldn't it be possible to define the scope of a parameter to the 
caller's scope?

-- 
/Jacob Carlborg
Jul 13 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Jul 13, 2014 at 12:07:58PM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 2014-07-11 16:29, H. S. Teoh via Digitalmars-d wrote:
 

Because the scope of the parameter 'obj' is defined to be the scope
of myFunc only, according to the current proposal.


 
 Wouldn't it be possible to define the scope of a parameter to the
 caller's scope?


[...]

We could, but how would that help static analysis within the function's
body, since the caller's scope is unknown?


T

-- 
Truth, Sir, is a cow which will give [skeptics] no more milk, and so they are
gone to milk the bull. -- Sam. Johnson
Jul 13 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 13/07/14 16:37, H. S. Teoh via Digitalmars-d wrote:


 We could, but how would that help static analysis within the function's
 body, since the caller's scope is unknown?


Won't the caller's scope always outlive the callee's?

-- 
/Jacob Carlborg
Jul 14 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Mon, Jul 14, 2014 at 10:41:10AM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 13/07/14 16:37, H. S. Teoh via Digitalmars-d wrote:
 

We could, but how would that help static analysis within the
function's body, since the caller's scope is unknown?


 
 Won't the caller's scope always outlive the callee's?


[...]

Yes, but since the extent of this scope is unknown from inside the
function body, it doesn't easily lend itself nicely to check things like
this:

	int* ptr;
	void func(scope int* arg) {
		ptr = arg; // should this be allowed?
	}

If we only know that 'arg' has a longer lifetime than func, but we don't
know how long it is, then we don't know if it has the same lifetime as
'ptr', or less. So it doesn't really let us do useful checks.


T

-- 
"I suspect the best way to deal with procrastination is to put off the
procrastination itself until later. I've been meaning to try this, but haven't
gotten around to it yet. " -- swr
Jul 14 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 15/07/14 01:48, H. S. Teoh via Digitalmars-d wrote:


 Yes, but since the extent of this scope is unknown from inside the
 function body, it doesn't easily lend itself nicely to check things like
 this:

 	int* ptr;
 	void func(scope int* arg) {
 		ptr = arg; // should this be allowed?
 	}

 If we only know that 'arg' has a longer lifetime than func, but we don't
 know how long it is, then we don't know if it has the same lifetime as
 'ptr', or less. So it doesn't really let us do useful checks.


I was thinking that "arg" would have at least the same lifetime as the 
caller, i.e. the same as "ptr".

-- 
/Jacob Carlborg
Jul 15 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Tue, Jul 15, 2014 at 09:19:34AM +0200, Jacob Carlborg via Digitalmars-d
wrote:

 On 15/07/14 01:48, H. S. Teoh via Digitalmars-d wrote:
 

Yes, but since the extent of this scope is unknown from inside the
function body, it doesn't easily lend itself nicely to check things
like this:

	int* ptr;
	void func(scope int* arg) {
		ptr = arg; // should this be allowed?
	}

If we only know that 'arg' has a longer lifetime than func, but we
don't know how long it is, then we don't know if it has the same
lifetime as 'ptr', or less. So it doesn't really let us do useful
checks.


 
 I was thinking that "arg" would have at least the same lifetime as the
 caller, i.e. the same as "ptr".


[...]

But what if 'ptr' is declared in a private binary-only module, and only
the signature of 'func' is known? Then what should 'scope' mean to the
compiler when 'func' is being called from another module?


T

-- 
ASCII stupid question, getty stupid ANSI.
Jul 15 2014
parent Jacob Carlborg <doob me.com> writes:
On 2014-07-15 16:58, H. S. Teoh via Digitalmars-d wrote:


 But what if 'ptr' is declared in a private binary-only module, and only
 the signature of 'func' is known? Then what should 'scope' mean to the
 compiler when 'func' is being called from another module?


Hmm, I didn't think of that :(

-- 
/Jacob Carlborg
Jul 15 2014
prev sibling next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
I've been working on a proposal for ownership and borrowing since 
some time, and I seem to have come to a very similar result as 
you have. It is not really ready, because I keep discovering 
weaknesses, and can only work on it in my free time, but I'm glad 
this topic is finally addressed. I'll write about what I have now:

First of all, as you've already stated, scope needs to be a type 
modifier (currently it's a storage class, I think). This has 
consequences for the syntax of any parameters it takes, because 
for type modifiers there need to be type constructors. This 
means, the `scope(...)` syntax is out. I suggest to use template 
instantiation syntax instead: `scope!(...)`, which can be freely 
combined with the type constructor syntax: 
`scope!lifetime(MyClass)`.

Explicit lifetimes are indeed necessary, but dedicated 
identifiers for them are not. Instead, it can directly refer to 
symbol of the "owner". Example:

     int[100] buffer;
     scope!buffer(int[]) slice;

Instead of lifetime intersections with `&` (I believe Timon 
proposed that in the original thread), simply specify multiple 
"owners": `scope!(a, b)`. This works, because as far as I can see 
there is no need for lifetime unions, only intersections.

A problem that has been discussed in a few places is safely 
returning a slice or a reference to an input parameter. This can 
be solved nicely:

     scope!haystack(string) findSubstring(
         scope string haystack,
         scope string needle
     );

Inside `findSubstring`, the compiler can make sure that no 
references to `haystack` or `needle` can be escape (an 
unqualified `scope` can be used here, no need to specify an 
"owner"), but it will allow returning a slice from it, because 
the signature says: "The return value will not live longer than 
the parameter `haystack`."

     // fixed-size arrays (new syntax of Kenji's PR)
     string[$] text = "Old McDonald had a farm.";
     auto sub = findSubstring(text, "had");
     // typeof(sub) is scope!text(string),
     // `haystack` gets substituted by `text`
     assert(sub == "had a farm".);

Have multiple parameters? No problem:

     scope!(a,b)(string) selectOneAtRandom(
         scope string a,
         scope string b
     );
     // => a _and_ b will outlive return value

For methods, `scope!this` can be used to. It's really no 
different from other parameters, as `this` is just a special 
implicit parameter.

There is also a nice extension: `scope!(const owner)`. This 
means, that as long as the value designated as such live, `owner` 
will be treated as const.

An interesting application is the old `byLine` problem, where the 
function keeps an internal buffer which is reused for every line 
that is read, but a slice into it is returned. When a user 
naively stores these slices in an array, she will find that all 
of them have the same content, because they point to the same 
buffer. See how this is avoided with `scope!(const ...)`:

struct ByLineImpl(Char, Terminator) {
private:
     Char[] line;
     // ...

public:
     // - return value must not outlive `this` (i.e. the range)
     // - as long as the return value exists, `this` will be const
      property scope!(const this)(Char[]) front() const {
         return line;
     }
     void popFront() { // not `const`, of course
         // ...
     }
     // ...
}

void main() {
     alias Line = const(char)[];
     auto byline = stdin.byLine();
     foreach(line; byline) {
         write(line); // OK, `write` takes its parameters as scope
         // (assuming the widespread usage of scope throughtout 
Phobos)
     }
     Line[] lines;
     foreach(line; byline) {
         lines ~= line;
         // ERROR: `line` has type scope!(const byline)(Line), not 
Line
     }
     // let's try to work around it:
     scope!(const byline)(Line)[] clines;
     foreach(line; byline) {     // ERROR: `byline` is const
         clines ~= line;
     }
     // => nope, won't work
     // another example, to show how it works:
     auto tmp = byline.front;    // OK
     // `byline` is const as long as `tmp` exists
     write(byline.front);        // OK, `front` is const
     byline.popFront();          // ERROR: `byline` is const
}

Describing what happens here: As long as any variable (or 
temporary) with the type `scope!(const byline)` exists, `byline` 
itself will be treated as const. "Exists" in this case only 
referes to lexical scope: A variable is said to "exist" from the 
point it is declared, to the end of the scope it's declared in. 
Loops, gotos, and exceptions don't have an effect. This means 
that it can be easily checked by the compiler, without it having 
to perform complicated control flow analysis.

I also thought about allowing `scope!return` for functions, to 
specify that it a value will not outlive the value returned from 
the function, but I'm not sure whether there is an actual use 
case, and the semantics are not clear.

An open question is whether there needs to be an explicit 
designation of GC'd values (for example by `scope!static` or 
`scope!GC`), to say that a given values lives as long as it's 
needed (or "forever").

Specifying an owner in the type also integrates naturally with 
allocators. Assuming an allocator releases all of it's memory to 
operating system when it is destroyed, there needs to be a 
guarantee that none of its contents is referenced anymore at this 
point. This can be achieved by returning a borrowed reference:

     struct MyAllocator {
         scope!this(T) alloc(T)() if(T == class) {
             // ...
         }
     }

Note that this does not preclude the allocator from doing garbage 
collection while it exists; in this manner, `scope!GC` might just 
be an application of this pattern instead of a special syntax.

Now, for the problems:

Obviously, there is quite a bit of complexity involved. I can 
imagine that inferring the scope for templates (which is 
essential, just as for const and the other type modifiers) can be 
complicated.

On the upside, at least it requires no control or data flow 
analysis. It's also a purely additive change: If implemented 
right, no currently working code will break.

Then I encountered the following problem, and there are several 
different variations of it:

     struct S {
         int* p;
         void releaseBuffer() scope {
             // `scope` in the signature applies to `this`
             free(this.p);
             this.p = null;
         }
     }
     int bar(scope ref S a, scope int* b) {
         a.releaseBuffer();
         return *b; // use after free
     }
     S s;
     bar(s, s.p);

The root cause of the problem here is the call to `free()`. I 
_believe_ the solution is that `free()` (and equivalent functions 
of allocators as well as `delete`) must not accept scope 
parameters. More realistic candidates for such situations are 
destructors in combination with move semantics. Therefore, 
`~this()` needs to be marked as scope, too, for it to be callable 
on a borrowed object. If a scope object has a non-scope 
destructor, but no scope one, and is going to be destroyed, this 
needs to be a compile error. (Rust avoids that problem by making 
any object const while there are borrowed references, but this 
requires its complex borrow checker, which we should avoid for D.)

I also have a few ideas about owned types and move semantics, but 
this is mostly independent from borrowing (although, of course, 
it integrates nicely with it). So, that's it, for now. Sorry for 
the long text. Thoughts?
Jul 10 2014
next sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

 snip


That's curious. I actually replied to H. S. Teoh in his new 
thread, but somehow it ended up here...
Jul 10 2014
parent reply "John Colvin" <john.loughran.colvin gmail.com> writes:
On Thursday, 10 July 2014 at 20:13:18 UTC, Marc Schütz wrote:

 On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

 snip


 That's curious. I actually replied to H. S. Teoh in his new 
 thread, but somehow it ended up here...


You are in the new thread...
Jul 10 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 10 July 2014 at 20:39:45 UTC, John Colvin wrote:

 On Thursday, 10 July 2014 at 20:13:18 UTC, Marc Schütz wrote:

 On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

 snip


 That's curious. I actually replied to H. S. Teoh in his new 
 thread, but somehow it ended up here...


 You are in the new thread...


Not in the forum, unfortunately:

http://forum.dlang.org/group/digitalmars.D

The two threads named "Proposal for design of 'scope'" have no 
replies, and my post appears in the original thread:

http://forum.dlang.org/thread/lphnen$1ml7$1 digitalmars.com?page=16#post-dvtohzrxoyzhhegzkfqf:40forum.dlang.org
Jul 10 2014
prev sibling next sibling parent Justin Whear <justin economicmodeling.com> writes:
On Thu, 10 Jul 2014 20:10:36 +0000, Marc Schütz wrote:


 So, that's it, for now. Sorry for the long text.
 Thoughts?


This is crazy interesting, thanks for writing it up.  Need to digest it a 
bit before offering more feedback.
Jul 10 2014
prev sibling next sibling parent reply Timon Gehr <timon.gehr gmx.ch> writes:
On 07/10/2014 10:10 PM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 ...
 Instead of lifetime intersections with `&` (I believe Timon proposed
 that in the original thread)


I didn't. I suggested that _some_ suitable syntax might need to be found. :)


 The root cause of the problem here is the call to `free()`.


(Proving the safety of code containing manual free requires different 
machinery.)
Jul 10 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 10 July 2014 at 21:01:23 UTC, Timon Gehr wrote:

 On 07/10/2014 10:10 PM, "Marc Schütz" <schuetzm gmx.net>" wrote:

 ...
 Instead of lifetime intersections with `&` (I believe Timon 
 proposed
 that in the original thread)


 I didn't. I suggested that _some_ suitable syntax might need to 
 be found. :)


Apologies, then :-)

But the point still stands - I don't think we need to introduce 
"invented" lifetime names as Rust does, we can refer to existing 
identifiers/symbols instead.


 The root cause of the problem here is the call to `free()`.


 (Proving the safety of code containing manual free requires 
 different machinery.)


I think that once manual memory management is involved, it's 
impossible to make safety guarantees in the general case. But I 
also believe that not all is lost: Much can be won when the 
_implementers_ of libraries take special care (in our case by not 
marking free() as scope), so that library _users_ cannot 
accidentally run into these problems. And because the 
trouble-makers can be clearly identified, there is even a simple 
rule that covers probably 99% of the cases: "Don't make memory 
deallocators take borrowed pointers."

The reason I'm not completely sure about it is that move 
semantics can have the same effect as manual memory management: 
It's possible to move a unique object into a lower scope, which 
is then destroyed at the end of that scope. It's just much less 
obvious...
Jul 10 2014
prev sibling next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

     struct S {
         int* p;
         void releaseBuffer() scope {
             // `scope` in the signature applies to `this`
             free(this.p);
             this.p = null;
         }
     }
     int bar(scope ref S a, scope int* b) {
         a.releaseBuffer();
         return *b; // use after free
     }
     S s;
     bar(s, s.p);

 The root cause of the problem here is the call to `free()`. I 
 _believe_ the solution is that `free()` (and equivalent 
 functions of allocators as well as `delete`) must not accept 
 scope parameters.


Thinking more about it:

     struct S {
          int* p;
          void releaseBuffer() scope {
              free(this.p);
              this.p = null;
          }
      }
      int bar(void delegate() a, scope int* b) {
          a();
          return *b; // use after free
      }
      S s;
      bar({ s.releaseBuffer(); }, s.p);

So, for what I suggested (`free()` mustn't accept scope) to work, 
an additional rule is required: While a borrowed reference exist, 
the original must also be treated as scope.

Now, this is much more complicated to implement :-( Maybe there's 
a better way?
Jul 10 2014
prev sibling next sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

 Instead of lifetime intersections with `&` (I believe Timon 
 proposed that in the original thread), simply specify multiple 
 "owners": `scope!(a, b)`. This works, because as far as I can 
 see there is no need for lifetime unions, only intersections.


There are unions.

class A {
    scope!s1(A) a;
}

scope!s2(A) b;

b.a; // <= this has union lifetime of s1 and s2.
Jul 10 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 11 July 2014 at 06:49:26 UTC, deadalnix wrote:

 On Thursday, 10 July 2014 at 20:10:38 UTC, Marc Schütz wrote:

 Instead of lifetime intersections with `&` (I believe Timon 
 proposed that in the original thread), simply specify multiple 
 "owners": `scope!(a, b)`. This works, because as far as I can 
 see there is no need for lifetime unions, only intersections.


 There are unions.

 class A {
    scope!s1(A) a;
 }

 scope!s2(A) b;

 b.a; // <= this has union lifetime of s1 and s2.


How so? `s2` must not extend after `s1`, because otherwise it 
would be illegal to store a `scope!s1` value in `scope!s2`. From 
the other side, `s1` must not start after `s2`.

This means that the lifetime of `b.a` is `s1`, just as it has 
been annotated, no matter what the lifetime of `b` is. In fact, 
because `s1` can be longer than `s2`, a copy of `a.b` may safely 
be kept around after `b` is deleted (but of course not longer 
than `s1`).
Jul 11 2014
prev sibling parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Thu, Jul 10, 2014 at 08:10:36PM +0000, via Digitalmars-d wrote:

 I've been working on a proposal for ownership and borrowing since some
 time, and I seem to have come to a very similar result as you have. It
 is not really ready, because I keep discovering weaknesses, and can
 only work on it in my free time, but I'm glad this topic is finally
 addressed. I'll write about what I have now:
 
 First of all, as you've already stated, scope needs to be a type
 modifier (currently it's a storage class, I think). This has
 consequences for the syntax of any parameters it takes, because for
 type modifiers there need to be type constructors. This means, the
 `scope(...)` syntax is out. I suggest to use template instantiation
 syntax instead: `scope!(...)`, which can be freely combined with the
 type constructor syntax: `scope!lifetime(MyClass)`.
 
 Explicit lifetimes are indeed necessary, but dedicated identifiers for
 them are not. Instead, it can directly refer to symbol of the "owner".
 Example:
 
     int[100] buffer;
     scope!buffer(int[]) slice;


Hmm. Seems that you're addressing a somewhat wider scope than what I had
in mind. I was thinking mainly of 'scope' as "does not escape the body
of this block", but you're talking about a more general case of being
able to specify explicit lifetimes.

[...]

 A problem that has been discussed in a few places is safely returning
 a slice or a reference to an input parameter. This can be solved
 nicely:
 
     scope!haystack(string) findSubstring(
         scope string haystack,
         scope string needle
     );
 
 Inside `findSubstring`, the compiler can make sure that no references
 to `haystack` or `needle` can be escape (an unqualified `scope` can be
 used here, no need to specify an "owner"), but it will allow returning
 a slice from it, because the signature says: "The return value will
 not live longer than the parameter `haystack`."


This does seem to be quite a compelling argument for explicit scopes. It
does make it more complex to implement, though.


[...]

 An interesting application is the old `byLine` problem, where the
 function keeps an internal buffer which is reused for every line that
 is read, but a slice into it is returned. When a user naively stores
 these slices in an array, she will find that all of them have the same
 content, because they point to the same buffer. See how this is
 avoided with `scope!(const ...)`:


This seems to be something else now. I'll have to think about this a bit
more, but my preliminary thought is that this adds yet another level of
complexity to 'scope', which is not necessarily a bad thing, but we
might want to start out with something simpler first.


[...]

 An open question is whether there needs to be an explicit designation
 of GC'd values (for example by `scope!static` or `scope!GC`), to say
 that a given values lives as long as it's needed (or "forever").


Shouldn't unqualified values already serve this purpose?


[...]

 Now, for the problems:
 
 Obviously, there is quite a bit of complexity involved. I can imagine
 that inferring the scope for templates (which is essential, just as
 for const and the other type modifiers) can be complicated.


I'm thinking of aiming for a design where the compiler can infer all
lifetimes automatically, and the user doesn't have to. I'm not sure if
this is possible, but based on what Walter said, it would be best if we
infer as much as possible, since users are lazy and are unlikely to be
thrilled at the idea of having to write additional annotations on their
types.

My original proposal was aimed at this, that's why I didn't put in
explicit lifetimes. I was hoping to find a way to define things such
that the lifetime is unambiguous from the context in which 'scope' is
used, so that users don't ever have to write anything more than that.
This also makes the compiler's life easier, since we don't have to keep
track of who owns what, and can just compute the lifetime from the
surrounding context. This may require sacrificing some precision in
lifetimes, but if it helps simplify things while still giving adequate
functionality, I think it's a good compromise.


[...]

 I also have a few ideas about owned types and move semantics, but this
 is mostly independent from borrowing (although, of course, it
 integrates nicely with it). So, that's it, for now. Sorry for the long
 text. Thoughts?


It seems that you're the full borrowed reference/pointer problem, which
is something necessary. But I was thinking more in terms of the baseline
functionality -- what is the simplest design for 'scope' that still
gives useful semantics that covers most of the cases? I know there are
some tricky corner cases, but I'm wondering if we can somehow find an
easy solution for the easy parts (presumably the more common parts),
while still allowing for a way to deal with the hard parts.

At least for now, I'm thinking in the direction of finding something
with simple semantics that, at the same time, produces complex
(interesting) effects when composed, that we can use to solve the
borrowed pointer problem.


T

-- 
Computers are like a jungle: they have monitor lizards, rams, mice, c-moss,
binary trees... and bugs.
Jul 11 2014
parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 11 July 2014 at 21:04:05 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 On Thu, Jul 10, 2014 at 08:10:36PM +0000, via Digitalmars-d 
 wrote:
 Hmm. Seems that you're addressing a somewhat wider scope than 
 what I had
 in mind. I was thinking mainly of 'scope' as "does not escape 
 the body
 of this block", but you're talking about a more general case of 
 being
 able to specify explicit lifetimes.


Indeed, but it includes what you're suggesting. For most use 
cases, just `scope` without an explicit lifetime annotation is 
fully sufficient.


 [...]

 A problem that has been discussed in a few places is safely 
 returning
 a slice or a reference to an input parameter. This can be 
 solved
 nicely:
 
     scope!haystack(string) findSubstring(
         scope string haystack,
         scope string needle
     );
 
 Inside `findSubstring`, the compiler can make sure that no 
 references
 to `haystack` or `needle` can be escape (an unqualified 
 `scope` can be
 used here, no need to specify an "owner"), but it will allow 
 returning
 a slice from it, because the signature says: "The return value 
 will
 not live longer than the parameter `haystack`."


 This does seem to be quite a compelling argument for explicit 
 scopes. It
 does make it more complex to implement, though.


 [...]

 An interesting application is the old `byLine` problem, where 
 the
 function keeps an internal buffer which is reused for every 
 line that
 is read, but a slice into it is returned. When a user naively 
 stores
 these slices in an array, she will find that all of them have 
 the same
 content, because they point to the same buffer. See how this is
 avoided with `scope!(const ...)`:


 This seems to be something else now. I'll have to think about 
 this a bit
 more, but my preliminary thought is that this adds yet another 
 level of
 complexity to 'scope', which is not necessarily a bad thing, 
 but we
 might want to start out with something simpler first.


It's definitely an extension and not as urgently necessary, 
although it fits well into the general topic of borrowing: 
`scope` by itself provides mutable borrowing, but `scope!(const 
...)` provides const borrowing, in the sense that another object 
temporarily takes ownership of the value, so that the original 
owner can only read the object until it is "returned" by the 
borrowed value going out of scope. I mentioned it here because it 
seemed to be an easy extension that could solve an interesting 
long-standing problem for which we only have workarounds today 
(`byLineCopy` IIRC).

And I have to add that it's not completely thought out yet. For 
example, might it make sense to have `scope!(immutable ...)`, 
`scope!(shared ...)`, and if yes, what would they mean...


 [...]

 An open question is whether there needs to be an explicit 
 designation
 of GC'd values (for example by `scope!static` or `scope!GC`), 
 to say
 that a given values lives as long as it's needed (or 
 "forever").


 Shouldn't unqualified values already serve this purpose?


Likely yes. It might however be useful to contemplate, especially 
with regards to allocators.


 [...]

 Now, for the problems:
 
 Obviously, there is quite a bit of complexity involved. I can 
 imagine
 that inferring the scope for templates (which is essential, 
 just as
 for const and the other type modifiers) can be complicated.


 I'm thinking of aiming for a design where the compiler can 
 infer all
 lifetimes automatically, and the user doesn't have to. I'm not 
 sure if
 this is possible, but based on what Walter said, it would be 
 best if we
 infer as much as possible, since users are lazy and are 
 unlikely to be
 thrilled at the idea of having to write additional annotations 
 on their
 types.


I agree. It's already getting ugly with `const pure nothrow  safe 
 nogc`, adding another annotation should not be done 
lightheartedly. However, if the compiler could infer all the 
lifetimes (which I'm quite sure isn't possible, see the 
haystack-needle example), I don't see why we'd need `scope` at 
all. It would at most be a way not to break backward 
compatibility, but that would be another case where you could say 
that D has it backwards, like un- safe by default...


 My original proposal was aimed at this, that's why I didn't put 
 in
 explicit lifetimes. I was hoping to find a way to define things 
 such
 that the lifetime is unambiguous from the context in which 
 'scope' is
 used, so that users don't ever have to write anything more than 
 that.
 This also makes the compiler's life easier, since we don't have 
 to keep
 track of who owns what, and can just compute the lifetime from 
 the
 surrounding context. This may require sacrificing some 
 precision in
 lifetimes, but if it helps simplify things while still giving 
 adequate
 functionality, I think it's a good compromise.


I agree it looks a bit intimidating at first glance, but as far 
as I can tell it should be relatively straightforward to 
implement. I'll explain how I think it could be done:

The obvious things: The parser needs to recognize the new syntax, 
and scope needs to be turned into a type modifier and stored in 
the internal data structures accordingly.

It is then possible to define a hierarchy of lifetimes. At the 
top are global and static variables and the GC heap 
(`scope!static` or just unannotated), then the come function 
parameters, then local variables in function bodies, and finally 
local variables in lower scopes like `if` blocks. This is purely 
based on lexical scope and order of declaration (local variables 
are destroyed in inverse order of construction, for example); it 
can be derived from the AST. Furthermore, it is a strict 
hierarchy; lifetimes higher in the hierarchy are strict super 
sets of lower lifetimes.

A variables effective lifetime is then its place in this 
hierarchy, or the lifetime of its owner if one is specified.

Once that's done, the semantic phase needs to be extended to 
check for scope correctness. This seems complicated, but actually 
needs to touch only a few places. Any time a scope value is 
copied, by assignment, returning from a function, passing to a 
function, throwing, and what else I may have missed, the compiler 
needs to check that the destination's effective lifetime is not 
wider than that of the source.

For function calls, an additional step is necessary, but it isn't 
really complicated either. Let's take `findSubstring` as an 
example:

     scope!haystack(string) findSubstring(
         scope string haystack,
         scope string needle
     );

     void foo() {
         string[$] h = "Hello, world!";
         auto found = findSubstring(h, ", ");
         // `typeof(found)` is now `scope!h`
     }

As owners in function signatures may refer to other parameters 
(or `this`), the compiler needs to match up these parameters with 
what is passed in, and substitute them accordingly for type 
deduction (only for `auto` return values).

And that's it, AFAICS. Notice that none of this requires flow 
control analysis or inter-procedural things, it can all be 
decided locally at the place of assignment/calling/etc.


 [...]

 I also have a few ideas about owned types and move semantics, 
 but this
 is mostly independent from borrowing (although, of course, it
 integrates nicely with it). So, that's it, for now. Sorry for 
 the long
 text. Thoughts?


 It seems that you're the full borrowed reference/pointer 
 problem, which
 is something necessary. But I was thinking more in terms of the 
 baseline
 functionality -- what is the simplest design for 'scope' that 
 still
 gives useful semantics that covers most of the cases? I know 
 there are
 some tricky corner cases, but I'm wondering if we can somehow 
 find an
 easy solution for the easy parts (presumably the more common 
 parts),
 while still allowing for a way to deal with the hard parts.

 At least for now, I'm thinking in the direction of finding 
 something
 with simple semantics that, at the same time, produces complex
 (interesting) effects when composed, that we can use to solve 
 the
 borrowed pointer problem.


I already wrote this in a reply to Walter. I believe in some 
cases we can allow automatic borrowing without any annotation at 
all, not even bare `scope`. The most obvious examples are pure 
functions with signatures that guarantee that nothing can be 
escaped from them:

     void foo(int[] p) pure;    // obvious, function has no 
opportunity
                                // to keep a reference to `p`
     int bar(int[] p) pure;     // returns an `int` but that's a 
value
                                // type, and that's ok
     int[] baz(const(int)[] p) pure;
                                // the return type is not `const` 
and thus
                                // cannot come from `p`

Maybe there are some cases with non-pure functions, too. But on 
the other hand, I also think that in the end we won't get around 
introducing explicit annotations, because the above rules can 
never cover enough cases to disregard the remaining ones.

Anyway, I don't believe that explicit annotations will be needed 
often enough to turn the users away. It will be mostly library 
writers who have to use them, and Phobos can set a good example 
there and work out a good style, just as it has done for other 
matters.

It also helps to take a glance at Rust's standard library, to see 
how frequent or infrequent lifetime annotations will be. They 
keep popping up here and there, but they are not littered all 
over the source code. They're frequent enough to confirm my 
suspicion that they cannot be disregarded, but they're also 
infrequent enough not to be an annoyance. (I only looked at a few 
modules, though.)
Jul 12 2014
prev sibling next sibling parent reply "Meta" <jared771 gmail.com> writes:
On Thursday, 10 July 2014 at 17:04:24 UTC, H. S. Teoh via 
Digitalmars-d wrote:

Can you copy and paste the text from your original post? It's 
difficult to read on the web interface.
Jul 10 2014
parent "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jul 11, 2014 at 01:01:20AM +0000, Meta via Digitalmars-d wrote:

 On Thursday, 10 July 2014 at 17:04:24 UTC, H. S. Teoh via Digitalmars-d
 wrote:
 
 Can you copy and paste the text from your original post? It's
 difficult to read on the web interface.


I'm working on a DIP for this, actually. Hopefully it will be more
fleshed out than what I posted. I'll post the link when it's ready.


T

-- 
The problem with the world is that everybody else is stupid.
Jul 10 2014
prev sibling parent reply "deadalnix" <deadalnix gmail.com> writes:
I'm bugging around with a similar proposal for a while, but quite 
fail to put the last pieces in place. It goes similar in

On Thursday, 10 July 2014 at 17:04:24 UTC, H. S. Teoh via 
Digitalmars-d wrote:

    - For function parameters, this lifetime is the scope of the 
 function
      body.


Some kind of inout scope seem less limiting. The caller know the 
scope, the callee know that is is greater than itself. It is 
important as local variable in the outer scope of the function 
have more restricted scope and must not be assignable.

Each parameter have a DIFFERENT lifetime, but it is impossible to 
tell which one is larger from the callee perspective. Thus you 
must have a more complex lifetime definition than grater/smaller 
lifetime. Yup, when you get into the details, quantum effects 
start to arise.


    - An unscoped variable is regarded to have infinite 
 lifetime.
 




So it is not unscoped, but I'm simply nitpicking on that one.


       - Since a scoped return type has its lifetime as part of 
 its type,
         the type system ensures that scoped values never 
 escape their
         lifetime. For example, if we are sneaky and return a 
 pointer to
         an inner function, the type system will prevent 
 leakage of the




This get quite tricky to define when you can have both this and a 
context pointer. Once again, you get into a situation where you 
have 2 non sortable lifetime to handle. And worse, you'll be 
creating values out of that mess :)


 - Aggregates:
 
    - It's turtles all the way down: members of scoped 
 aggregates also
      have scoped type, with lifetime inherited from the parent
      aggregate. In other words, the lifetime of the aggregate 
 is
      transitive to the lifetime of its members.




Yes rule for access is transitivity. But the rule to write is 
"antitransitive". It gets tricky when you consider that a member 
variable may have to be able to "extend" the lifetime of one of 
its member.

IE a member of lifetime B in a value of lifetime A sees it 
lifetime becoming max(A, B). Considering lifetime aren't always 
sortable (as show in 2 examples), this is tricky.

This basically means that you have to define what happen for non 
sortable lifetime, and what happen for union/intersection of 
lifetime. As you see, I've banged my head quite a lot on that 
one. I'm fairly confident that this is solvable, but definitively 
require a lot of effort to iron out all the details.


 - Passing parameters: since unscoped values are regarded to 
 have
   infinite lifetime, it's OK to pass unscoped values into 
 scoped
   function parameters: it's a narrowing of lifetime of the 
 original
   value, which is allowed. (What's not allowed is expanding 
 the lifetime
   of a scoped value.)
 




Get rid of the whole concept of unscopped, and you get rid of a 
whole class of redundant definition that needs to be done.


 I'm sure there are plenty of holes in this proposal, so 
 destroy away.
 ;-)
 




Need some more iron. But I'm happy to see that some people came 
up with proposal that are close to what I had in mind.

The above mentioned detail may seems scary, but I'm confident 
these will only cause problems in a small variety of cases.

An aspect of the proposal that isn't mentioned is postblit and 
destructions. scoping will need to redefine theses.

Ultimately I love the idea and think D should go in that 
direction at some point. But now I'd prefers see things ironed 
out in general in D ( safe is a good example).
Jul 10 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Fri, Jul 11, 2014 at 06:41:47AM +0000, deadalnix via Digitalmars-d wrote:
[...]

 On Thursday, 10 July 2014 at 17:04:24 UTC, H. S. Teoh via Digitalmars-d
 wrote:

   - For function parameters, this lifetime is the scope of the
   function body.


 
 Some kind of inout scope seem less limiting. The caller know the
 scope, the callee know that is is greater than itself. It is important
 as local variable in the outer scope of the function have more
 restricted scope and must not be assignable.
 
 Each parameter have a DIFFERENT lifetime, but it is impossible to tell
 which one is larger from the callee perspective. Thus you must have a
 more complex lifetime definition than grater/smaller lifetime. Yup,
 when you get into the details, quantum effects start to arise.


Looks like we might need to use explicit lifetimes for this. Unless
there's a way to simplify it -- i.e., we don't always need exact
lifetimes, as long as the estimated lifetime is never larger than the
actual lifetime. From the perspective of the callee, for example, if the
lifetimes of both parameters are longer than it can see (i.e., longer
than the lifetimes of its parent lexical scopes) then it doesn't matter
what the exact lifetimes are, it can be treated as an unknown value with
a lower bound, as long as it never tries to assign anything with
lifetime >= that lower bound. The caller already knows what these
lifetimes are from outside, but the function may not need to know.

At least, I'm hoping this kind of simplifications will still allow us to
do what we need, while reducing complexity.



   - An unscoped variable is regarded to have infinite lifetime.




 
 So it is not unscoped, but I'm simply nitpicking on that one.


Well, yes, the reason I wrote that line was to make the definition
uniform across both scoped and unscoped types. :)



      - Since a scoped return type has its lifetime as part of its
        type, the type system ensures that scoped values never
        escape their lifetime. For example, if we are sneaky and
        return a pointer to an inner function, the type system will
        prevent leakage of the




 
 This get quite tricky to define when you can have both this and a
 context pointer. Once again, you get into a situation where you have 2
 non sortable lifetime to handle. And worse, you'll be creating values
 out of that mess :)


Is it possible to simplify this by taking the minimum of the two
lifetimes (i.e. intersection)? Or will that run into unsolvable cases?



- Aggregates:

   - It's turtles all the way down: members of scoped aggregates
     also have scoped type, with lifetime inherited from the parent
     aggregate. In other words, the lifetime of the aggregate is
     transitive to the lifetime of its members.




 
 Yes rule for access is transitivity. But the rule to write is
 "antitransitive". It gets tricky when you consider that a member
 variable may have to be able to "extend" the lifetime of one of its
 member.
 
 IE a member of lifetime B in a value of lifetime A sees it lifetime
 becoming max(A, B). Considering lifetime aren't always sortable (as
 show in 2 examples), this is tricky.
 
 This basically means that you have to define what happen for non
 sortable lifetime, and what happen for union/intersection of lifetime.
 As you see, I've banged my head quite a lot on that one. I'm fairly
 confident that this is solvable, but definitively require a lot of
 effort to iron out all the details.


Along these lines, I'm wondering if "turtles all the way down" is the
wrong way of looking at it. Consider, for example, an n-level deep
nesting of aggregates. If obj.nest1 is const, then obj.nest1.nest2.x
must also be const, because otherwise we break the const system. So
const is transitive downwards. But if obj.nest1 is a scoped reference
type with lifetime L1, that doesn't necessarily mean obj.nest1.y only
has lifetime L1. It may be a pointer that points to an infinite lifetime
object, for example, so it's not a problem that the pointer goes out of
scope before the object pointed to. OTOH, if obj.nest1 has scope L1,
then obj itself cannot have a longer lifetime than L1, otherwise we may
access obj.nest1 after its lifetime is over. So the lifetime of
obj.nest1 must propagate *upwards* (or outwards).

This means that scope is transitive outwards, which is the opposite of
const/immutable, which are transitive inwards! So it's not turtles all
the way down, but "pigeons all the way up". :-P



- Passing parameters: since unscoped values are regarded to have
  infinite lifetime, it's OK to pass unscoped values into scoped
  function parameters: it's a narrowing of lifetime of the original
  value, which is allowed. (What's not allowed is expanding the
  lifetime of a scoped value.)




 
 Get rid of the whole concept of unscopped, and you get rid of a whole
 class of redundant definition that needs to be done.


OK, let's just say unscoped == scope with infinite lifetime. :)



I'm sure there are plenty of holes in this proposal, so destroy
away.  ;-)




 
 Need some more iron. But I'm happy to see that some people came up
 with proposal that are close to what I had in mind.


[...]

Maybe what we should do, is to have everyone post their current
(probably incomplete) drafts of what scope should do, so that we have
everything on the table and we can talk about what should be kept, what
should be discarded, etc.. It may be, that the best design is not what
any one of us has right now, but some combination of multiple current
proposals.


T

-- 
Be in denial for long enough, and one day you'll deny yourself of things you
wish you hadn't.
Jul 11 2014
next sibling parent "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 11 July 2014 at 22:03:37 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 Along these lines, I'm wondering if "turtles all the way down" 
 is the
 wrong way of looking at it. Consider, for example, an n-level 
 deep
 nesting of aggregates. If obj.nest1 is const, then 
 obj.nest1.nest2.x
 must also be const, because otherwise we break the const 
 system. So
 const is transitive downwards. But if obj.nest1 is a scoped 
 reference
 type with lifetime L1, that doesn't necessarily mean 
 obj.nest1.y only
 has lifetime L1. It may be a pointer that points to an infinite 
 lifetime
 object, for example, so it's not a problem that the pointer 
 goes out of
 scope before the object pointed to. OTOH, if obj.nest1 has 
 scope L1,
 then obj itself cannot have a longer lifetime than L1, 
 otherwise we may
 access obj.nest1 after its lifetime is over. So the lifetime of
 obj.nest1 must propagate *upwards* (or outwards).


I'm not so sure about transitivity either, although I started 
with it. One reason that `const`, `immutable` and `shared` need 
to be transitive is that we can then use this fact to infer other 
properties from it, e.g. thread-safety. I don't really see such 
advantages for `scope`, but instead it would make handling 
`scope` in aggregates extremely complicated. And it wouldn't make 
much sense either IMO, because aggregates are usually defined 
somewhere else than where they are used, and often by a different 
author, too. Therefore, they come with their own ownership 
strategies, and it's simply not possible to force a different one 
onto them from the outside. For this reason, I now tend to 
intransitivity.

There also something else that became clear to me: If an 
important use case is found that requires transitivity, nothing 
is really lost. We know all the types that are involved, and we 
can check whether all references contained in them are marked as 
`scope` using introspection, even without additional compiler 
support beyond a simple trait, just as we can today check for 
things like `hasUnsharedAliasing`! Therefore, we wouldn't even 
close the doors for further improvements if we decide for 
intransitivity.
Jul 12 2014
prev sibling parent reply "Marc =?UTF-8?B?U2Now7x0eiI=?= <schuetzm gmx.net> writes:
On Friday, 11 July 2014 at 22:03:37 UTC, H. S. Teoh via 
Digitalmars-d wrote:

 Maybe what we should do, is to have everyone post their current
 (probably incomplete) drafts of what scope should do, so that 
 we have
 everything on the table and we can talk about what should be 
 kept, what
 should be discarded, etc.. It may be, that the best design is 
 not what
 any one of us has right now, but some combination of multiple 
 current
 proposals.


I've just done so for mine:
http://wiki.dlang.org/User:Schuetzm/scope
Jul 13 2014
parent reply "H. S. Teoh via Digitalmars-d" <digitalmars-d puremagic.com> writes:
On Sun, Jul 13, 2014 at 02:58:27PM +0000, via Digitalmars-d wrote:

 On Friday, 11 July 2014 at 22:03:37 UTC, H. S. Teoh via Digitalmars-d wrote:

Maybe what we should do, is to have everyone post their current
(probably incomplete) drafts of what scope should do, so that we have
everything on the table and we can talk about what should be kept,
what should be discarded, etc.. It may be, that the best design is
not what any one of us has right now, but some combination of
multiple current proposals.


 
 I've just done so for mine:
 http://wiki.dlang.org/User:Schuetzm/scope


Mine is here:

	http://wiki.dlang.org/User:Quickfur/DIP_scope


T

-- 
Turning your clock 15 minutes ahead won't cure lateness---you're just making
time go faster!
Jul 14 2014
parent reply Jacob Carlborg <doob me.com> writes:
On 14/07/14 18:16, H. S. Teoh via Digitalmars-d wrote:


 Mine is here:

 	http://wiki.dlang.org/User:Quickfur/DIP_scope


 From the DIP:

"The 'scope' keyword has been around for years, yet it is barely 
implemented and it's unclear just what it's supposed to mean"

I don't know if it's worth clarify but "scope" currently has various 
features.

1. Allocate classes on the stack: "scope bar = new Bar()"
2. Forcing classes to be allocated on the stack: "scope class Bar {}"
3. The scope-statement: "scope(exit) file.close()"
4. Scope parameters. This is the part that is unclear what is means/is 
supposed to mean in the current language

-- 
/Jacob Carlborg
Jul 14 2014
next sibling parent reply simendsjo <simendsjo gmail.com> writes:
On 07/15/2014 08:42 AM, Jacob Carlborg wrote:

 On 14/07/14 18:16, H. S. Teoh via Digitalmars-d wrote:
 

 Mine is here:

     http://wiki.dlang.org/User:Quickfur/DIP_scope


 
 From the DIP:
 
 "The 'scope' keyword has been around for years, yet it is barely
 implemented and it's unclear just what it's supposed to mean"
 
 I don't know if it's worth clarify but "scope" currently has various
 features.
 
 1. Allocate classes on the stack: "scope bar = new Bar()"
 2. Forcing classes to be allocated on the stack: "scope class Bar {}"
 3. The scope-statement: "scope(exit) file.close()"
 4. Scope parameters. This is the part that is unclear what is means/is
 supposed to mean in the current language
 


Isn't both 1 and 2 deprecated?
Jul 14 2014
parent Jacob Carlborg <doob me.com> writes:
On 15/07/14 08:46, simendsjo wrote:


 Isn't both 1 and 2 deprecated?


Depends on what you mean by "deprecated". People are keep saying that 
but it's not. Nothing, except for people saying that, indicates that. No 
deprecation message, no warning, nothing about it in the documentation. 
Even if/when that will be deprecated that it's not unclear what it does.

-- 
/Jacob Carlborg
Jul 15 2014
prev sibling parent reply "Adam D. Ruppe" <destructionator gmail.com> writes:
On Tuesday, 15 July 2014 at 06:42:20 UTC, Jacob Carlborg wrote:

 1. Allocate classes on the stack: "scope bar = new Bar()"
 4. Scope parameters. This is the part that is unclear what is 
 means/is supposed to mean in the current language


These are actually the same thing: if something is stack 
allocated, it must not allow the reference to escape to remain 
memory safe... and if the reference is not allowed to escape, 
stack allocating the object becomes an obvious automatic 
optimization.

People keep calling them deprecated but they really aren't - the 
escape analysis to make it memory safe just isn't implemented.


 2. Forcing classes to be allocated on the stack: "scope class 
 Bar {}"


I think this is the same thing too, just on the class instead of 
the object, but I wouldn't really defend this feature, even if 
implemented correctly, since ALL classes really ought to be scope 
compatible if possible to let the user decide on their lifetime.
Jul 15 2014
parent Jacob Carlborg <doob me.com> writes:
On 15/07/14 14:47, Adam D. Ruppe wrote:


 These are actually the same thing: if something is stack allocated, it
 must not allow the reference to escape to remain memory safe... and if
 the reference is not allowed to escape, stack allocating the object
 becomes an obvious automatic optimization.

 People keep calling them deprecated but they really aren't - the escape
 analysis to make it memory safe just isn't implemented.


Yes, I agree.


 I think this is the same thing too, just on the class instead of the
 object, but I wouldn't really defend this feature, even if implemented
 correctly, since ALL classes really ought to be scope compatible if
 possible to let the user decide on their lifetime.


If a class is allocated on the stack, its destructor will be called (at 
least according to the spec). If you declare a class "scope" you know it 
will always be allocated on the stack and can take advantage of that. 
Even if all classes are "scope" compatible some might _only_ be 
compatible with "scope".

-- 
/Jacob Carlborg
Jul 15 2014